**From Simple Scripts to Production‑Ready Pipelines: Key Lessons from My Second ETL Project**
In my earlier article, I shared a 12‑month roadmap to transition from data analyst to data engineer and described my first ETL project—fetching GitHub repository data and automating it with GitHub Actions. That project taught me that writing the transformation logic is the easy part; the hard part is building a system that runs reliably over time.
My second project was an RSS ingestion pipeline, designed specifically to explore the engineering challenges that arise once a script works. Instead of focusing on the business logic, I used this simple, well‑defined ETL to learn about packaging, orchestration, retries, idempotency, scheduling, and observability.
Why I Built Another ETL Pipeline
By the time I finished the first project, I realized the difficult questions weren’t about new datasets or transformations. They were about system behavior:
– How should the pipeline be executed and scheduled?
– How should it recover from failures?
– How do you make it run consistently across environments?
– Where should orchestration responsibility end and application responsibility begin?
An RSS feed was intentionally simple, so I could focus on these engineering concerns rather than business logic. I also wanted to move beyond GitHub Actions to a tool built for data workflows—Kestra.
First Architectural Decision: Docker Before Kestra
I resisted starting with Kestra. Instead, I built the pipeline in layers:
1. A small Python ETL that fetches, parses, and saves articles.
2. Idempotent PostgreSQL inserts using ON CONFLICT DO NOTHING.
3. A Docker container encapsulating the application.
4. Kestra to orchestrate the container.
This incremental approach let me validate each layer. When I finally introduced Kestra, I no longer tried to make it run Python directly. I treated Kestra as an orchestrator for a self‑contained Docker image responsible for its own execution. This clarified responsibilities and simplified debugging.
The Assumption That Turned Out to Be Wrong
Initially, I assumed Kestra should execute my Python environment. Quickly, I learned that Kestra is an orchestrator, not a runtime for application dependencies. Once I shifted to treating the workflow as “orchestrate a ready‑to‑run Docker image,” the architecture became clean and resilient. Each component kept a single responsibility:
– Python: fetch, parse, persist.
– Docker: consistent execution artifact.
– Kestra: scheduling, retries, and workflow management.
This separation also made failures easier to isolate: if a run failed, I could immediately rule out the ETL and focus on the orchestration layer.
Running Automatically Changes Everything
Scheduling introduced the reality of repeated runs. I configured an hourly trigger in Kestra and intentionally broke the pipeline to see what happens. Thanks to retries configured in Kestra and idempotent writes in PostgreSQL, failures were handled gracefully—without changes to the application itself.
Observability matured alongside reliability. I replaced noisy debug prints with concise logs that explained each execution, turning raw output into a clear narrative of what happened during the run.
The Final Architecture
By the end, the stack looked like this:
– A simple Python ETL packaged as a Docker image.
– PostgreSQL for idempotent storage.
– Docker as the deployment unit.
– Kestra for scheduling, retries, and monitoring.
Each layer was validated before the next was added, and each owned a single responsibility.
What This Project Changed About How I Think
I used to believe the hard part was writing ETL code. Now I know the real engineering begins after the script works—when the pipeline must run unattended, handle failures, avoid duplicates, and provide useful logs.
Key takeaways:
– Keep responsibilities separate: execution belongs to the application; orchestration belongs to the platform.
– Docker is not just packaging—it’s the contract between your app and the orchestrator.
– Incremental validation reduces cognitive load and risk.
– Good observability means the right logs, not more logs.
Looking Ahead
This project didn’t introduce complex transformations, but it fundamentally changed my approach to data engineering. I now design systems with clear boundaries, reliable failure modes, and incremental validation in mind. I’m excited to apply these principles to the next challenge.
