Managing "big data" is an interesting challenge. When data becomes too large to fit on a single system, characteristics such as volume, throughput and variety become important. On the other hand, consistency and isolation become more difficult to accomplish, especially as data is often replicated to different physical locations.
Many new "NoSQL" databases abandon normalized tables in favor of denormalized collections. This approach exposes applications to old problems, such as how to maintain consistent indexes without periodically rebuilding them, that truly relational databases have already solved.
Intelligent embedded systems and devices are increasingly being used to store and modify working subsets of a "big data". I will be exploring ways to bridge common data from our embedded database and "big data" technology in my upcoming posts.
