📊 Data Pipeline Flow (ETL)
📥 RAW APIs
→
🔍 Validate
→
🧹 Cleanse
→
🔄 Transform
→
📊 Aggregate
→
💾 Warehouse
📈 Processing Distribution by Region
🔑 Data Sources Configuration
PYTHON from pyspark import SparkSession
# Apache Spark-style pipeline
# Data Sources Configuration
DATA_SOURCES = {
"indonesia_population": {
"url": "bps.go.id/api/v1",
"format": "JSON",
"records": 2_400_000
},
"seasia_weather": {
"url": "open-meteo.com/v1",
"format": "GeoJSON",
"records": 1_200_000
},
"osm_indonesia": {
"url": "overpass-api.de",
"format": "XML",
"records": 4_500_000
}
}
# Spark Pipeline Execution
spark = SparkSession.builder \
.appName("IndonesiaBigData") \
.config("spark.sql.shuffle.partitions", 200) \
.getOrCreate()
📍 Live Data Points - Indonesia
📋 Recent Processing Jobs
| Job ID | Source | Region | Records | Duration | Status |
|---|---|---|---|---|---|
| Loading... | |||||