Multi-source Ingestion Framework
A configuration-driven ingestion framework, so adding a new data source is a config change rather than a new codebase.
- Sector
- Cross-industry
- Discipline
- Data Engineering
- Technology
- Python, Apache Airflow, PySpark, PostgreSQL, AWS
The problem
Each new data source had arrived as its own script, with its own retry logic, its own idea of scheduling and its own failure modes. Nothing was shared, so every source multiplied the maintenance burden and no two failed in the same way.
What we built
We built a configuration-driven ingestion framework where a source is declared rather than coded. Connection, schedule, schema, incremental strategy and destination are configuration; retries, pagination, checkpointing, logging and alerting are handled once by the framework.
Architecture
How the system fits together.
- 01
Declare
Each source defined in configuration: connection, schedule, expected schema and incremental key.
- 02
Extract
Shared connectors handling pagination, rate limits, retries with backoff and checkpointed incremental reads.
- 03
Land
Raw payloads written immutably and partitioned, so any downstream change can be replayed from source.
- 04
Transform
Typed transformations into the warehouse model, with schema-drift detection on every run.
- 05
Orchestrate
Dependency-aware scheduling with per-source alerting and a single view of pipeline health.
Outcome
What the system delivers.
Capability delivered, stated plainly. We do not attach business metrics to this work that we are not in a position to verify.
- New sources added by configuration rather than by writing another pipeline
- Retry, checkpoint and alerting behaviour identical across every source
- Immutable raw landing zone enabling replay after any downstream change
- Schema drift detected on ingest instead of surfacing later as broken reports
- Python
- Apache Airflow
- PySpark
- PostgreSQL
- AWS