Thoughts on the Big Data Landscape for 2014

bigdataAs we head into the new year, I thought I would take a moment to reflect on the past year in Big Data and Analytics, while contemplating what the upcoming year may bring.

We’ve seen different vendors continue to build out their big data solutions — either by doubling down on what they already have, partnering with other vendors, or going in new directions.

We’ve seen in-memory computing  take the market by storm, largely driven by the marketing arm of SAP — regardless of the actual results for true in-memory computing.  We’ve also seen Oracle change their messaging on Exadata with the release of X4 (they no longer refer to X3 as an “in-memory” machine, which was stretching the truth in no uncertain terms).

And, we’ve seen IBM launch DB2 with BLU Acceleration, which takes in-memory computing to the next level, while maintaining “appliance simplicity.”

We’ve also seen vendors like Teradata, Microsoft, and Oracle expand their partnerships for Hadoop with HortonWorks for the first two, and Cloudera for Oracle… but more importantly we’ve seen the first workable logical data warehouse architectures emerge that augment traditional data warehousing with Hadoop and Stream computing.

So where do we go from here?  Cloud computing is emerging bringing in a new era of “agile data warehousing” starting with SAP’s HANA One in late 2012 followed almost immediately by Amazon RedShift, and then IBM’s BLU for Cloud technology preview later in 2013.  And appliances as a delivery model for big data continue to prove capable and cost-effective — for use cases that fit their economics and sweet spot.

My bold (and some not so bold) predictions for 2014 and beyond:

  • In-memory and solid-state data warehousing will emerge as a viable solution for the “less than 50 TB” crowd, while traditional media will remain king on the 100+ TB implementations
  • BLU Acceleration from IBM will emerge as the first of the next-generation in-memory systems that solve many of the current problems associated with in-memory (cost, scalability)
  • Cloud analytics and data warehousing will consistently chip away at the low-end of the appliance market relegating it to large-scale, highly targeted implementations by the end of 2015, with private and public cloud implementations emerging as the dominant player for the entry-level, SMB, and development environments
  • The company that solves the big data ecosystem seamlessly, providing compatible solutions from the low-end all the way up to the high-end with tight ecosystem integration will have the greatest successes (IBM, Oracle, and Teradata are all well-positioned to take this role!)

So let’s enjoy the last days and hours of 2013, and look to the future in the highly competitive big data landscape.

And as we look to the future, we must remember the past, and recognize that the problem of big data is not a new one, but one that we have been dealing with for a long long time.  Indeed, I stumbled upon this lovely quote  from Daniel Boorstin dating back to 1983 when the effort was undertaken to computerize the Library of Congress:

Technology is so much fun but we can drown in our technology. The fog of information can drive out knowledge.

Sound familiar?

Happy New Year to all my Big Data colleagues!


Leave a Reply