Posts Tagged ‘big data’

Open eGovernment Data

Monday, November 28th, 2011

The Open Source movement is moving into government data.  Governments are finding a new source of untapped economic stimulus with the mountains of data they collect.  The  data is collected for the ultimate good of the public but rarely shared because information access was too people intensive and expensive up until recently.  Things have changed.

GOV opendata1 300x168 Open eGovernment Data

ETALAB (France), data.gov.uk (UK), data.dc.gov (Washington, DC, US), whitehouse.gov/open (US), and countless other local and national governments have open their data coffers.  In the case of DC for instance, the cost of publishing the data was $50K for the city. The DC government expected it to spur the creation of a few new ventures, and a bit of private investments.  Instead, 50 startups were born and $3M invested.  There is a world of open data coming to the private software industry.

Open Government data is also going to be Big Data.  The size of data collected is by definition larger larger than traditional “enterprise data” for instance (especially at the national level).  The tools being developed for big data will solve some of the issues with access and real time analytics that exit with government data.  Exorbyte MatchMaker is one of these tools.  That’s why government agencies have already chosen MatchMaker for their search and data access challenges (2 national European census agencies, German Finance Ministry, and more).

Are you ready for open government data?  Any ideas what would make sense to build with this data?

Big Data Search

Monday, November 28th, 2011

Big Data1 300x225 Big Data SearchEvery economic cycle comes with its host of enterprise software trends.  Big Data hs become a recognized phenomenon in 2011.  In May 2011 McKinsey released the “Big data: The next frontier for innovation, competition, and productivity” report. It started with:  ”The amount of data in our world has been exploding and analyzing large data sets—so-called big data—will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus”.

IBM, Oracle, SAP, Microsoft, SalesForce.com, and others are all aiming their development efforts at Big Data (see vldb.org).  The amount of data produced, collected and stored by online activities to which companies, their customers, their partners, and their sales channels participate has grown enormously.  Tools are being developed that allow affordable long-term storage.  New columnar in-memory database formats have emerged that enable near-real-time analytics.  Fast growing stratups and open source solutions have also converged with their own new NoSQL formats (InfiniDB, LucidDB, InfoBright, Hadoop, NoSQL, etc.)     MatchMaker, Exoryte’s Universal Search platform, is the perfect answer to search within Big Data.

The challenges of search within big data are:

  • Searching Big Data though SQL queries is simply too slow and inflexible – fuzzy or advanced search requires a search indexer layer or  something different than traditional on-disk relational DB formats.
  • Indexing large databases can be long, disruptive to normal database operation and require complex hardware infrastructures.
  • Running complex queries and fuzzy logic requires so much calculation and lookups that new search strategies are required.

Exorbyte MatchMaker is made to address these challenges and our professional services team has proven repeatedly tht they can be addressed:  Allianz (the world’s 12th-largest financial services group),  German Finance Ministry, and more blue chip and government organizations tun o us each year for that very expertise.

What do you think of Big Data?