International Journal of AI and Advanced Computing
Review Article
• Open Access
Modern Data Architectures for Intelligent Decision Support: From Data Warehouses and Data Lakes to the Lakehouse Paradigm
View PDFAbstract
The exponential growth of data volumes, variety, and velocity has fundamentally transformed the landscape of decision support systems, necessitating architectural innovations that can accommodate both structured analytical workloads and unstructured data processing. This comprehensive review examines the evolution of data management architectures for decision support, tracing the progression from traditional data warehouses through data lakes to the emergent Data Lakehouse paradigm. The paper analyzes the technological foundations, architectural patterns, and practical implementations of these modern data architectures, with particular emphasis on the integration of open table formats (Delta Lake, Apache Iceberg, Apache Hudi) and the medallion architecture pattern for data quality management. Through systematic synthesis of recent literature and empirical studies, the review investigates the critical role of data ingestion frameworks, including Apache Kafka, Apache Flink, Apache Airflow, and Apache NiFi, in enabling real-time data processing and analytics. The findings reveal that the transition from isolated data warehouses and data lakes to unified Lakehouse architectures addresses fundamental limitations of previous approaches, providing transactional reliability (ACID compliance), schema evolution capabilities, time travel functionality, and optimized analytical performance. Empirical evidence from the insurance sector demonstrates that data warehousing adoption is associated with a 4.8% reduction in loss ratios and a 5.6% reduction in combined ratios, with an additional 3.2 percentage points of improvement from complementary IT investments. The study also examines applications in higher education, where data lake implementations enable the integration of structured, semi-structured, and unstructured data from learning management systems, social media platforms, and IoT devices. Key challenges identified include data governance complexities, metadata management, scalability constraints, and the need for specialized skills. Emerging solutions such as cloud-native architectures, serverless computing, and AI-driven data management offer promising pathways to address these barriers. The findings contribute to the evolving field of data engineering by providing insights into best practices, architectural decision frameworks, and identification of research gaps for future exploration.
Keywords
Decision Support Systems, Data Warehouse, Data Lake, Data Lakehouse, Open Table Formats, Data Ingestion, Medallion Architecture, Business Intelligence, Data GovernanceReferences
Alam, M. A., Sohel, A., Uddin, M. M., & Siddiki, A. (2024). Big Data and Chronic Disease Management Through Patient Monitoring And Treatment With Data Analytics. Academic Journal on Artificial Intelligence, Machine Learning, Data Science and Management Information Systems, 1(01), 77-94.Armbrust, M., Das, T., Torres, J., Yavuz, B., Zhu, S., Xin, R., Ghodsi, A., Stoica, I., & Zaharia, M. (2020). Delta Lake: High-performance ACID table storage over cloud object stores. Proceedings of the VLDB Endowment, 13(12), 3411-3424.
Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2021). Lakehouse: A new generation of open platforms that unifies data warehousing and advanced analytics. Proceedings of the 11th Annual Conference on Innovative Data Systems Research (CIDR '21).
Blazic, G., Poscic, P., & Jaksic, D. (2017). Data warehouse architecture classification. 2017 40th International Convention on Information and Communication Technology, Electronics and Microelectronics, 1491-1495.
Dabbechi, H., Haddar, N. Z., Elghazel, H., & Haddar, K. (2021). Social Media Data Integration: From Data Lake to NoSQL Data Warehouse. In Intelligent Systems Design and Applications (pp. 701-710). Springer.
Databricks. (2023). What is a medallion architecture? https://docs.databricks.com/en/lakehouse/medallion.html
Dehury, C. K., Jakovits, P., Srirama, S. N., Giotis, G., & Garg, G. (2022). TOSCAData: Modeling data pipeline applications in TOSCA. Journal of Systems and Software, 186, 111164.
El Aissi, M. E. M., Benjelloun, S., Loukili, Y., Lakhrissi, Y., Boushaki, A. E., Chougrad, H., & Elhaj Ben Ali, S. (2022). Data Lake Versus Data Warehouse Architecture: A Comparative Study. Lecture Notes in Electrical Engineering, 745, 201-210.
Elkina, H., & Zaki, T. (2023). Data Ingestion Frameworks for Data Lakes: An Overview. Journal of Research Technology and Digital Development, 6(10s2), 1700-1708.
Fang, H. (2015). Managing data lakes in big data era: What's a data lake and why has it became popular in data management ecosystem. 2015 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems, 820-824.
Giebler, C., Gröger, C., Hoos, E., Eichler, R., Schwarz, H., & Mitschang, B. (2021). The Data Lake Architecture Framework. Gesellschaft für Informatik, Bonn.
Harasis, A. (2026). Impact of data warehousing adoption on underwriting and claims performance in Saudi insurance firms. Insurance Markets and Companies, 17(1), 65-77.
Hassan, M. S., Islam, M. A., Abdullah, A. B. M., & Nasir, H. (2024). End-user perspectives on fintech services adoption in the Bangladesh insurance industry. Journal of Financial Services Marketing, 29(4), 1377-1395.
Herden, O. (2020). Architectural Patterns for Integrating Data Lakes into Data Warehouse Architectures. Lecture Notes in Computer Science, 12581, 12-27.
Holub, B. L., & Shevchenko, D. V. (2026). Modern Design Paradigms for Decision Support Systems: From Integrating Data Warehouses and Data Lakes to the Data Lakehouse Architecture. Information Technologies and Electronic Engineering, 1(1), 28-38.
Inmon, W. H., & Linstedt, D. (2019). Data architecture: A primer for the data scientist (2nd ed.). Morgan Kaufmann.
Ionescu, S.-A., Diaconita, V., & Radu, A.-O. (2025). Engineering sustainable data architectures for modern financial institutions. Electronics, 14(8), 1650.
Martín, C., Langendoerfer, P., Zarrin, P. S., Díaz, M., & Rubio, B. (2022). Kafka-ML: Connecting the data stream with ML/AI frameworks. Future Generation Computer Systems, 126, 15-33.
Miloslavskaya, N., & Tolstoy, A. (2016). Big Data, Fast Data and Data Lake Concepts. Procedia Computer Science, 88, 300-305.
Morshed, A. (2025a). Cultural norms and ethical challenges in MENA accounting: The role of leadership and organizational climate. International Journal of Ethics and Systems, 41(3), 630-656.
Morshed, A. (2025d). Sustainable energy revolution: Green finance as the key to the Arab Gulf States' future. International Journal of Energy Sector Management, 20(2), 556-577.
Mukherjee, R., & Kar, P. (2017). A Comparative Review of Data Warehousing ETL Tools with New Trends and Industry Insight. 2017 IEEE 7th International Advance Computing Conference, 943-948.
Poojara, S. R., Dehury, C. K., Jakovits, P., & Srirama, S. N. (2022). Serverless data pipeline approaches for IoT data in fog and cloud computing. Future Generation Computer Systems, 130, 91-105.
Ravat, F., & Zhao, Y. (2019). Data Lakes: Trends and Perspectives. In Database and Expert Systems Applications (pp. 304-313). Springer.
Reis, J., & Housley, M. (2022). Fundamentals of data engineering: Plan and build robust data systems (1st ed.). O'Reilly Media.
Saggar, S., Bitoni, C., Khurana, I., & Alhawat, R. (2022). Data Warehouse with Big Data Technology for Higher Education. SSRN Electronic Journal.
Salaki, R. J., & Ratnam, K. A. (2018). Agile Analytics: Applying in the Development of Data Warehouse for Business Intelligence System in Higher Education. In Trends and Advances in Information Systems and Technologies (pp. 1038-1048). Springer.
Santoso, L. W., & Yulia. (2017). Data Warehouse with Big Data Technology for Higher Education. Procedia Computer Science, 124, 93-99.
Sawadogo, P., & Darmont, J. (2021). On data lake architectures and metadata management. Journal of Intelligent Information Systems, 56(1), 97-120.
Seshagiri, S., & Prema, K. V. (2025). Efficient handling of data imbalance in health insurance fraud detection using meta-reinforcement learning. IEEE Access.
Singh, P. (2019). Airflow. In Learn PySpark (pp. 67-84). Apress.
Smolinski, M. (2018). Impact of Storage Space Configuration on Transaction Processing Performance for Relational Database in PostgreSQL. In Beyond Databases, Architectures and Structures (pp. 157-167). Springer.
Sreemathy, J., Joseph V., I., Nisha, S., Prabha I., C., & Priya R.M., G. (2020). Data Integration in ETL Using TALEND. 2020 6th International Conference on Advanced Computing and Communication Systems, 1444-1448.
Sukare, N., & Al, E. (2021). Smart Classroom Environment using IoT in advanced and lebanese French university Education. Turkish Journal of Computer and Mathematics Education, 12(7).
Vaisman, A., & Zimányi, E. (2014). Data Warehouse Systems. Springer.
Williamson, B. (2018). The hidden architecture of higher education: Building a big data infrastructure for the 'smarter university.' International Journal of Educational Technology in Higher Education, 15(1), 12.
Wyatt, L., Caufield, B., & Pol, D. (2009). Principles for an ETL Benchmark. In Performance Evaluation and Benchmarking (pp. 183-198). Springer.