A DW is needed to efficiently and continuously prepare data for practical real-time data analysis and forecasting with machine learning (ML) algorithms. This paper discusses extraction, transformation, and loading process (ETL) automation and the critical subsystems integration for algorithmic trading ML modelling and stock market forecasting. Additionally, we discuss a new API designed for direct access to the DW. It opens vast opportunities for performance and data processing time to become critical nonfunctional requirements.
This research intends to implement a data warehouse (DW) and ETL (Extract, Transform, Load) process to handle time-series data used by an XGBoost model to predict stock market prices. The ETL pipeline automates data ingestion from Financial Modelling Prep (FMP) API [2], performs necessary transformation, and stores it for future use. Furthermore, it leverages its design to optimize resource usage. The paper also explores the DW performance and estimations for growing workloads.