What does “Big Data Storage” Look Like?

When we went from PC apps to Web apps we lost a bunch of things, but the worst was: the UI and UX went to hell.  Even with all the rich interaction add-ons, Web apps today are just not as (for want of a better word) delightful as native apps were, and won’t be for some time.  That’s probably reason #1 for mobile apps today: they are much more delightful than anything achievable through a web browser.

There’s an analogy with Big Data: the paradigm of distributing computation to the storage (which I talked about a bit last fall) is pretty powerful, but it’s a step backwards in terms of storage.  There were all kinds of good reason to centralize storage (or at least decouple it from computing), and while putting compute next to storage makes all kinds of sense, you now have the move the data into the storage that’s near the compute before you can compute on it.  Not bad, perhaps, if you have to do a whole passle of computing on a relatively static batch of data, but a real non-starter in near-real-time or even regularly-changing batches of data.

Which should be leading some teams to think from the ground up about how to design a storage and compute system that manages to distribute storage with respect to computing but centralize it with respect to locality or reference and management.

We’ve seen a couple of ideas in this area, and would like to see more.

Your thoughts?