Stocks Tracker ETL
End to end stock price pipeline orchestrated with Apache Airflow, deployed on AWS with a scheduled daily run. Pulls from the Alpha Vantage API into PostgreSQL, with Metabase for visualization.
Technologies Used
PythonAirflowPostgreSQLDockerAWS
Features
- • Extract, transform and load live data from API into PostgreSQL using Python scripts
- • The ETL pipeline is orchestrated by Airflow DAG
- • Scheduled daily run using cronjob on AWS EC2
- • Visualization using SQL queries demonstrated through Metabase
Code Snippet
# run_pipeline.py
# Standalone test script — chains extract -> transform -> load manually.
# Use this to verify the whole pipeline works BEFORE wiring it into Airflow.
# Run from project root: python run_pipeline.py
from etl.extract import extract
from etl.transform import transform
from etl.load import load
import time
symbols = ["AAPL", "NVDA"] # Insert stock ticker(s)
for symbol in symbols:
print(f"
--- Running pipeline for {symbol} ---")
extracted = extract(symbol)
if extracted is None:
print(f"[run_pipeline] Skipping {symbol} — extract failed (check rate limit or API key)")
continue # Move to next symbol instead of crashing
records = transform(extracted)
load(records)
time.sleep(15)
print("
--- Pipeline run complete ---")