Optimizing Data Ingestion Processes using a Serverless Framework on Amazon Web Services
  IJCTT-book-cover
 		          
 
© 2024 by IJCTT Journal
Volume-72 Issue-6
Year of Publication : 2024
Authors : Rajesh Remala, Krishnamurty Raju Mudunuru, Sevinthi Kali Sankar Nagarajan
DOI :  10.14445/22312803/IJCTT-V72I6P116

How to Cite?

Rajesh Remala, Krishnamurty Raju Mudunuru, Sevinthi Kali Sankar Nagarajan, "Optimizing Data Ingestion Processes using a Serverless Framework on Amazon Web Services," International Journal of Computer Trends and Technology, vol. 72, no. 6, pp. 118-125, 2024. Crossref, https://doi.org/10.14445/22312803/IJCTT-V72I6P116

Abstract
This paper presents a novel approach to data ingestion leveraging serverless architecture on Amazon Web Services (AWS). Traditional data ingestion methods often face challenges such as scalability limitations and high operational overhead. In contrast, serverless computing offers a promising solution by abstracting infrastructure management and scaling resources dynamically based on demand. We demonstrate the effectiveness of our approach through experimentation and performance evaluation. Results show significant improvements in scalability, resource utilization, and cost efficiency compared to traditional approaches. Additionally, we discussed the design considerations, implementation details, and best practices for deploying and managing the serverless data ingestion framework on AWS. Overall, our framework provides a robust solution for efficiently ingesting data into cloud environments, offering benefits in terms of scalability, flexibility, and cost-effectiveness. By utilizing serverless architecture, the framework enables automatic scaling and resource provisioning, reducing operational overhead and optimizing costs.

Keywords
Amazon Web services, Cost-Effectiveness, Data ingestion, Framework, Serverless, Scalability.

Reference

[1] Omar Al-Debagy, and Peter Martinek, “A Comparative Review of Microservices and Monolithic Architectures,” 2018 IEEE 18th International Symposium on Computational Intelligence and Informatics (CINTI), Budapest, Hungary, pp. 000149-000154, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[2] Werner Vogels, AWS re: Invent 2018-Keynote, 2018. [Online]. Available: https://amzn.to/2FKc7zk
[3] Josef Spillner, “Quantitative Analysis of Cloud Function Evolution in the AWS Serverless Application Repository,” ArXiv preprint, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[4] Peter Sbarski, and Sam Kroonenburg, “Serverless Architectures On AWS: With Examples Using AWS Lambda,” Simon and Schuster, 2017.
[Google Scholar] [Publisher Link]
[5] Garrett McGrath, and Paul R. Brenner, “Serverless Computing: Design, Implementation, and Performance,” 2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW), Atlanta, GA, USA, pp. 405-410, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[6] V. Giménez-Alventosa, Germán Moltó, and Miguel Caballer, “A Framework and A Performance Assessment for Serverless Mapreduce on AWS Lambda,” Future Generation Computer Systems, vol. 97, pp. 259-274, 2019.
[CrossRef] [Google Scholar] [Publisher Link]
[7] Scott Patterson, “Learn AWS Serverless Computing: A Beginner’s Guide to Using AWS Lambda, Amazon API Gateway, And Services from Amazon Web Services,” Packt, 2019.
[Google Scholar] [Publisher Link]
[8] Danilo Poccia, “AWS Lambda in Action: Event-Driven Serverless Applications,” Simon and Schuster, 2016.
[Google Scholar] [Publisher Link]
[9] D. Marupaka and S. Rangineni, “Machine Learning-Driven Predictive Data Quality Assessment in ETL Frameworks," International Journal of Computer Trends and Technology, vol. 72, no. 3, pp. 53-60, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[10] R. Arokia Paul Rajan, “Serverless Architecture - A Revolution in Cloud Computing,” 2018 Tenth International Conference on Advanced Computing (ICoAC), Chennai, India, pp. 88-93, 2018.
[CrossRef] [Google Scholar] [Publisher Link]
[11] John Chapin, and Mike Roberts, “Programming AWS Lambda: Build and Deploy Serverless Applications with Java,” O'Reilly Online Learning, 2020.
[Google Scholar] [Publisher Link]
[12] Sandeep Rangineni, and Arvind Kumar Bhardwaj, “Analysis of DevOps Infrastructure Methodology and Functionality of Build Pipelines,” EAI Endorsed Transactions on Scalable Information Systems, vol. 11, no. 4, pp. 1-8, 2024.
[CrossRef] [Google Scholar] [Publisher Link]
[13] Naga Simhadri Apparao Polireddi et al., “A Novel Study on Data Science for Data Security and Data Integrity with Enhanced Heuristic Scheduling in Cloud,” 2023 2nd International Conference on Automation, Computing and Renewable Systems (ICACRS), Pudukkottai, India, pp. 1862-1868, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[14] Changyuan Lin, and Hamzeh Khazaei, “Modeling and Optimization of Performance and Cost of Serverless Applications,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 3, pp. 615-632, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[15] Sudhakar Kalyan, “Amazon Web Services (AWS) Glue,” International Journal of Management, IT and Engineering, vol. 8, no. 9, pp. 108-122, 2018.
[Google Scholar] [Publisher Link]
[16] Daniel Bardsley, Larry Ryan, and John Howard, “Serverless Performance and Optimization Strategies,” 2018 IEEE International Conference on Smart Cloud (SmartCloud), New York, USA, pp. 19-26, 2018.
[CrossRef] [Google Scholar] [Publisher Link]