Bigdata is now challenging businesses and IT leaders. There are opportunities and challenges with Big data transforming how data is collected(ingest), analyzed(range queries) and served(random reads). Analytics framework like Hadoop and No-SQL distributed, real-time databases like Cassandra and Membase are laying down new requirements on data center stacks. They affect all datacenter layers:
– how can the compute/storage layer handle SSD’s,
– how can the networking layer do more non-blocking IO’s and
– how can storage stack support more reliability, more availability, how would replication be handled with such large data-sets, what about versioning, snapshoting, what about backup and restore ?
– how can cluster management be done, how can the cluster be self healing ?
Many of the architectural decisions have to be looked from the perspective of big data lenses.