Big Data meets Open Source meets Service Providers

Earlier this week, I had the privilege of attending 451 Group’s Client Conference in San Francisco. After a couple of days of listening to sessions on “The Data Structure of the Cloud,” I came away with some thoughts on what’s going on in our industry right now.

The trends are clear, and the camps are clear. It’s new and old. It’s not new vs. old. The real question is where, when and how to get to the new.

OK what’s ‘new’: a) IT as a service; b) data-driven businesses; and c) reduced software licensing costs. Theses collectively drive technology companies to new creative community powered business models.

Well, I guess that’s not really new. These trends have been at work for years. But what is new is that it’s finally happening and at an accelerating pace.

Cloud is real. Hadoop is real. VC funding for open source models is real.

The accelerating pace of these trends make it exciting to be in the technology business! I leave the conference reflecting on the last two decades and coming to the time-proven conclusion that the new needs to eat even if we starve the old.

Remember token-ring networks and life without cell phones and Facebook? I honestly can’t, and my children couldn’t fold a newspaper or mail a letter or sit down for an hour to play a board game if you asked them to. My 14-year old son recently asked me what the flags was for on the mailbox, my daughter texted me on the ski lift chair (while I was sitting next to her) and my 9–year old is single-handedly supporting Zynga. The old stuff still exists, but the new is crowding it out.

It’s clear to me that the old relational databases, system management tools, software selling/licensing models, business applications……will run for as long as Unix is around. But it’s equally clear that the web-generation and scientific/government communities are spawning the businesses and technologies that will consume the bulk of new investment budgets.

OK, that’s still nothing new. Yep, until you think about intersections of the new. Rachael Chalmers’ panel with Opscode (creators of open source framework Chef), Cloudera (the Hadoop people) and an unfortunately missing Amazon EC2 GM (I don’t need to say cloud here), was both fascinating and inspiring. Fascinating to think about its ramifications on society—like trust and ownership and responsibility when, after you’ve sequenced your genome, you realize that you also just sequenced your parents and siblings (one example given by Cloudera’s Mike Olsen). Inspiring in its message to go forth with the new (even when we don’t know how the railroad or Internet or cloud will turn out) once we understand why the old is limiting.

Takeaway: Find the applications at the intersection of cloud, big data and open systems models and go for it—mindset not technology is the only thing holding us back.