Real-Time Big Data Analytics: Frameworks, Techniques, and Applications for Intelligent Decision-Making

Prof. A. Mohammad

doi:10.5281/zenodo.21212313

International Journal of AI and Advanced Computing

Volume 1 , Issue 2

Research Article • Open Access

Real-Time Big Data Analytics: Frameworks, Techniques, and Applications for Intelligent Decision-Making

Prof. A. Mohammad

Pages 85-94

DOI: 10.5281/zenodo.21212313

View PDF

Abstract

The exponential growth of data generated by social media platforms, IoT devices, financial systems, and healthcare applications has necessitated the development of sophisticated frameworks for real-time data processing and analysis. Traditional batch processing methods, while effective for historical analysis, are insufficient for addressing the velocity and volume requirements of modern data streams. This comprehensive review examines the landscape of real-time big data analytics, focusing on stream processing frameworks, integration with machine learning and artificial intelligence, and applications across critical domains including cybersecurity, healthcare, financial services, e-commerce, and intelligent transportation systems. Through systematic synthesis of recent literature and case studies, this paper investigates the architectural patterns, performance characteristics, and implementation challenges of leading stream processing frameworks including Apache Kafka, Apache Flink, and Spark Streaming. The review reveals that Apache Flink's true stream processing architecture achieves sub-second latency for complex event processing, while Kafka excels in high-throughput data ingestion with its distributed log-based architecture. The integration of machine learning models with stream processing frameworks has emerged as a critical enabler for real-time predictive analytics, with online learning algorithms demonstrating adaptability to evolving data patterns. Applications across industries demonstrate significant improvements in operational efficiency, fraud detection accuracy, and personalized user experiences. Key challenges identified include scalability limitations under extreme workloads, latency optimization for time-sensitive applications, data security and privacy concerns, and the complexity of integrating diverse data sources. Emerging solutions such as edge computing, federated learning, and explainable AI offer promising pathways to address these barriers while enabling more robust, efficient, and transparent real-time analytics. The findings contribute to the evolving field of streaming big data analytics by providing insights into best practices, comparative analysis of frameworks, and identification of research gaps for future exploration.

Keywords

Real-Time Analytics, Big Data, Stream Processing, Apache Kafka, Apache Flink, Spark Streaming, Machine Learning, Cybersecurity, Healthcare Analytics, Fraud Detection, Edge Computing,

References

Ajagbe, S. A., Amuda, K. A., Oladipupo, M. A., Afe, O. F., & Okesola, K. I. (2021). Multi-classification of alzheimer disease on magnetic resonance images (MRI) using deep convolutional neural network (DCNN) approaches. International Journal of Advanced Computer Research, 11(53), 51-60.
Alam, M. A., Sohel, A., Uddin, M. M., & Siddiki, A. (2024). Big Data and Chronic Disease Management Through Patient Monitoring And Treatment With Data Analytics. Academic Journal on Artificial Intelligence, Machine Learning, Data Science and Management Information Systems, 1(01), 77-94.
Asberg, M., Nolte, T., Kato, S., & Rajkumar, R. (2012). RTCSA - ExSched: An External CPU Scheduler Framework for Real-Time Systems. 2012 IEEE International Conference on Embedded and Real-Time Computing Systems and Applications, 240-249.
Babcock, B., Babu, S., Datar, M., Motwani, R., & Thomas, D. (2004). Operator scheduling in data stream systems. The VLDB Journal, 13(4), 333-353.
Banús, J. M., Arenas, A., & Labarta, J. (2002). PDPTA - An Efficient Scheme to Allocate Soft-Aperiodic Tasks in Multiprocessor Hard Real-Time Systems.
Block, A., Brandenburg, B. B., Anderson, J. H., & Quint, S. (2008). ECRTS - An Adaptive Framework for Multiprocessor Real-Time System. 2008 Euromicro Conference on Real-Time Systems, 23-33.
Gao, C., Yan, J., Zhou, S., Varshney, P. K., & Liu, H. (2019). Long short-term memory-based deep recurrent neural networks for target tracking. Information Sciences, 502, 279-296.
Holmes, J. H., Sun, J., & Peek, N. (2014). Technical Challenges for Big Data in Biomedicine and Health: Data Sources, Infrastructure, and Analytics. Yearbook of Medical Informatics, 23(1), 42-47.
Hussen, N., Elghamrawy, S. M., Salem, M., & El-Desouky, A. I. (2023). A Fully Streaming Big Data Framework for Cyber Security Based on Optimized Deep Learning Algorithm. IEEE Access, 11, 65675-65688.
Jia, Y., Gu, Z., Jiang, Z., Gao, C., & Yang, J. (2023). Persistent graph stream summarization for real-time graph analytics. World Wide Web, 26(5), 2647-2667.
Kastner, K.-H., Keber, R., Pau, P., & Samal, M. (2014). Real-Time Traffic Conditions with SUMO for ITS Austria West. Springer Berlin Heidelberg, 146-159.
Khan, M. S., et al. (2025). Big Data Analytics for Cybersecurity: A Comprehensive Review. Journal of Cybersecurity Research.
Khoshkhah, K., Pourmoradnasseri, M., Hadachi, A., Tera, H., Mass, J., Kesh, E., & Wu, S. (2022). Real-Time System for Daily Modal Split Estimation and OD Matrices Generation Using IoT Data: A Case Study of Tartu City. Sensors, 22(8), 3030.
Kim, A. C., Park, M., & Lee, D. H. (2020). AI-IDS: Application of Deep Learning to Real-Time Web Intrusion Detection. IEEE Access, 8, 70245-70261.
Kleiminger, W., Kalyvianaki, E., & Pietzuch, P. (2011). ICDE Workshops - Balancing load in stream processing with the cloud. 2011 IEEE 27th International Conference on Data Engineering Workshops, 16-21.
Ko, J.-H., Kook, Y., & Shin, K. (2020). KDD - Incremental Lossless Graph Summarization. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 317-327.
Kwon, J., Cho, H., & Ravindran, B. (2012). JTRES - A framework accommodating categorized multiprocessor real-time scheduling in the RTSJ. Proceedings of the 10th International Workshop on Java Technologies for Real-time and Embedded Systems, 18-25.
Lai, T. L. (2004). Likelihood Ratio Identities and Their Applications to Sequential Analysis. Sequential Analysis, 23(4), 467-497.
Leang, B., Ean, S., Ryu, G.-A., & Yoo, K.-H. (2019). Improvement of Kafka Streaming Using Partition and Multi-Threading in Big Data Environment. Sensors, 19(1), 134.
Liqing, C., Li, J., Lu, Y., & Zhang, Y. (2020). Adaptively secure certificate-based broadcast encryption and its application to cloud storage service. Information Sciences, 538, 273-289.
Luo, L., Zhou, L., & Song, P. X. K. (2022). Real-Time Regression Analysis of Streaming Clustered Data With Possible Abnormal Data Batches. Journal of the American Statistical Association, 118(543), 2029-2044.
Ma, L., Li, X., Wang, Y., & Wang, H. (2009). SAC - Real-time scheduling for continuous queries with deadlines. Proceedings of the 2009 ACM symposium on Applied Computing, 1516-1517.
Mishra, S., Sachan, R., & Rajpal, D. (2020). Deep Convolutional Neural Network based Detection System for Real-time Corn Plant Disease Recognition. Procedia Computer Science, 167, 2003-2010.
Nair, L. R., Shetty, S. D., & Shetty, S. D. (2017). Streaming Big Data Analysis for Real-Time Sentiment based Targeted Advertising. International Journal of Electrical and Computer Engineering, 7(1), 402-407.
Peddireddy, K. (2023). Streamlining Enterprise Data Processing, Reporting and Realtime Alerting using Apache Kafka. 2023 11th International Symposium on Digital Forensics and Security (ISDFS), 1-4.
Peddireddy, K., & Banga, D. (2023). Enhancing Customer Experience through Kafka Data Steams for Driven Machine Learning for Complaint Management. International Journal of Computer Trends and Technology, 71(3), 7-13.
Peek, N., Holmes, J. H., & Sun, J. (2014). Technical Challenges for Big Data in Biomedicine and Health: Data Sources, Infrastructure, and Analytics. Yearbook of Medical Informatics, 9(1), 42-47.
Puthal, D., Nepal, S., Ranjan, R., & Chen, J. (2017). A dynamic prime number based efficient security mechanism for big sensing data streams. Journal of Computer and System Sciences, 83(1), 22-42.
Ramachandra, M. N., Srinivasa Rao, M., Lai, W. C., Parameshachari, B. D., Ananda Babu, J., & Hemalatha, K. L. (2022). An Efficient and Secure Big Data Storage in Cloud Environment by Using Triple Data Encryption Standard. Big Data and Cognitive Computing, 6(4), 101.
Sahoo, P. K., Mohapatra, S. K., & Wu, S.-L. (2016). Analyzing Healthcare Big Data With Prediction for Future Health Condition. IEEE Access, 4, 9786-9799.
Shaikh, A., & Gupta, P. (2022). Real-time intrusion detection based on residual learning through ResNet algorithm. International Journal of System Assurance Engineering and Management.
Sharma, S., Gaherwal, S., Aman, & Sharma, S. (2024). Real-Time Big Data Analytics in Social Media: Enhancing User Behavior Prediction. Proceedings of the 3rd International Conference on Artificial Intelligence, Machine Learning and Cybersecurity.
Stankovic, J. A., Son, S. H., & Hansson, J. (1999). Misconceptions about real-time databases. Computer, 32(6), 29-36.
Sun, Y., Liu, Q., Chen, X., & Du, X. (2020). An Adaptive Authenticated Data Structure With Privacy-Preserving for Big Data Stream in Cloud. IEEE Transactions on Information Forensics and Security, 15, 3295-3310.
Toulis, P., & Airoldi, E. M. (2017). Asymptotic and finite-sample properties of estimators based on stochastic gradients. The Annals of Statistics, 45(4), 1694-1727.
Valls, M. G., Lopez, I. R., & Villar, L. F. (2013). iLAND: An Enhanced Middleware for Real-Time Reconfiguration of Service Oriented Distributed Real-Time Systems. IEEE Transactions on Industrial Informatics, 9(1), 228-236.
Wei, Y., Prasad, V., & Son, S. H. (2007). ISORC - QoS Management of Real-Time Data Stream Queries in Distributed Environments. 10th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing, 241-248.
Wu, H., Shang, Z., Peng, G., & Wolter, K. (2020). ISSRE - A Reactive Batching Strategy of Apache Kafka for Reliable Stream Processing in Real-time. 2020 IEEE 31st International Symposium on Software Reliability Engineering, 207-217.
Xu, J., Meng, Q., Wu, J., Zheng, J. X., Zhang, X., & Sharma, S. (2021). Efficient and Lightweight Data Streaming Authentication in Industrial Control and Automation Systems. IEEE Transactions on Industrial Informatics, 17(6), 4279-4287.