Data Lake and Lakehouse Architectures: Evolution, Design Principles, and Open Challenges
Madhu Sudan Gandotra *
Harrisburg University, Harrisburg, Pennsylvania, USA.
*Author to whom correspondence should be addressed.
Abstract
The rapid proliferation of big data has made the tasks of data storage, processing, management and analytics considerably more complex, thereby necessitating scalable data management frameworks. This research work reviews the emergence of lakehouse and data lake architectures. The discussion covers the design principles, architectural types, technical aspects, advantages and disadvantages of these data architectures. This paper includes several types of data lake architecture, including Lambda architecture, Kappa architecture, mono-zone architecture and data-pond architecture, and discusses the development of lakehouse architecture, which integrates the strengths of both data lakes and data warehouses. This review covers various technical components of data lakes, including storage mechanisms, file formats, table formats, metadata catalogues and computing engines, as well as features such as ACID transactions, schema support, caching, indexing, data versioning and query optimisation. The main conclusion of the current literature review is that lakehouses represent a step forward in contemporary data management because they address several drawbacks associated with conventional data lakes. Nevertheless, these architectures cannot resolve all issues associated with data handling, particularly those involving interoperability, security, privacy, scalability, performance and data quality. Through an analysis of previous literature, this study identifies possible areas for future research in data management.
Keywords: Data Lakehouse, data lake, data warehouse, big data, acid transactions, data governance, scalability, data security