ETL Pipeline - Real Open Data Processing

📡

EXTRACT

Pull data from USGS Earthquake API, Open-Meteo weather API, and other open sources in real-time.

🔧

TRANSFORM

Clean, normalize coordinates, calculate derived fields, filter by region.

📊

ANALYZE

Aggregate statistics, compute magnitude distribution, regional summaries.

💾

LOAD

Output to JSON/CSV for dashboard consumption and archival storage.

📡 Data Sources

🌍

USGS Earthquake

Real-time seismic data worldwide

LIVE

🌤️

Open-Meteo

Weather API - no API key needed

LIVE

🗺️

OpenStreetMap

Overpass API for geospatial

READY

⚙️ Transform Logic

# Python ETL Pipeline Example import requests, json # 1. Extract - Fetch USGS data def extract(): url = "https://earthquake.usgs.gov/...geojson" data = requests.get(url).json() return data["features"] # 2. Transform - Filter Indonesia def transform(features): filtered = [] for f in features: lat = f["geometry"]["coordinates"][1] lng = f["geometry"]["coordinates"][0] if -15 <= lat <= 10 and 90 <= lng <= 145: filtered.append({ "lat": lat, "lng": lng, "magnitude": f["properties"]["mag"], "place": f["properties"]["place"] }) return filtered # 3. Load - Output JSON def load(data): with open("output.json", "w") as f: json.dump(data, f, indent=2)

🔄 ETL Pipeline: Real Open Data Processing