What is an ETL pipeline, and when do you need one?
An ETL pipeline extracts data from a source, transforms it into the shape your product needs, and loads it somewhere queryable. You need one when the data lives somewhere you do not control and reading it live is too slow, too rate-limited or too unreliable. Onpolar builds these on Go and PostgreSQL.
- Largest we operate
- 19 government datasets, refreshed daily
- Core stack
- Go · PostgreSQL · River
- Patterns
- Batch ETL · streaming · webhooks
- Built this way
- Deal.ee · ERPFlow.ai
The problem
Integrations fail quietly, which is the whole problem
A broken page gets reported within minutes. A sync that silently stopped three weeks ago gets discovered when somebody notices the numbers have been wrong for a month.
The causes are boring and repeatable: a retry that hammers a rate limit until the provider blocks you, a webhook processed twice because nobody made the handler idempotent, a schema change upstream that surfaces as a null somewhere far away, a timezone.
The fix is not cleverness. It is idempotent handlers, a real job queue with visible failures, replayable runs, and alerting on the absence of expected work rather than only on thrown errors. Done properly the data layer disappears and the product rides on top of it.
What you get
What a pipeline actually consists of
Third-party API integrations
CRMs, ERPs, payment, shipping and accounting systems wired in with retries, backoff and a rate-limit budget rather than optimism.
Batch ETL pipelines
Scheduled extract, transform and load runs that are replayable, so a bad day upstream gets re-run instead of reconciled by hand.
Webhook and event ingestion
Idempotent handlers with signature verification and a dead-letter path, because every webhook gets delivered twice eventually.
Query-optimised stores
Data modelled for how the product reads it, indexed for the queries that actually run, with materialised aggregates where they earn their keep.
Job queues and scheduling
Durable background work on River, with retries, backoff, and a queue you can inspect rather than infer from log lines.
Data quality gates
Row counts, schema checks and freshness assertions that fail the run rather than quietly publishing nonsense downstream.
Monitoring and alerting
Alerts on work that did not happen, not only on errors that did. Silence is the failure mode that costs the most.
One API over the result
The merged data exposed behind a single OpenAPI contract, so every consumer reads it the same way instead of each inventing its own.
How it works
How a pipeline gets built
- 01Week 1
Map the sources honestly
What each source really returns, how often it changes, its rate limits, and how it fails. Documentation and reality routinely disagree; we go by reality.
- 02Weeks 1–2
Model the destination
A target schema designed around the queries the product runs — not a mirror of whatever shape the source happened to send.
- 03Weeks 2–4
Build it replayable
Idempotent handlers, a durable queue, quality gates and backfill from day one, so the first bad upstream day is a re-run rather than an incident.
- 04Week 5
Instrument and hand over
Dashboards, freshness alerts, a runbook per failure mode, and a backfill somebody other than us can run.
Proof of work
How this looks in production
Both are ours, which means the pipelines are ones we operate rather than ones we delivered and left. Deal.ee is the data-scale end of this work; ERPFlow is the regulated-format end.
Deal.ee — AI-Indexed Business Intelligence and Marketplace for Estonia
Deal.ee turns Estonia's public business data into a fast, AI-citable intelligence layer — ~368,000 companies across 19 government datasets, refreshed daily in Estonian, Russian and English — and adds a verified-company marketplace where businesses claim their profile by national eID, with an offers layer — products, services, real estate, jobs and investments — rolling out on top. Built in Go on PostgreSQL, engineered to be cited by ChatGPT, Claude, Perplexity and Gemini.
ERPFlow — API-First Multi-Tenant SaaS ERP with EU E-Invoicing
ERPFlow is a multi-tenant SaaS ERP — invoicing, CRM, inventory, staff and finance in one platform — built API-first on a Go + PostgreSQL backend behind a single OpenAPI contract, with native Peppol e-invoicing and per-seat Stripe billing.
Have data stuck somewhere useless?
Bring the sources and what you wish the product could do with them. Thirty minutes is usually enough to say which pattern it needs.
Pattern fit
Which integration pattern fits which problem
Most integration pain comes from picking the wrong pattern for the data, then adding retries until it looks fine.
| Pattern | Use it when | What it costs you |
|---|---|---|
| Live API call | The data must be current to the second and the source is fast and reliable | Your uptime is now their uptime, and their rate limit is your ceiling |
| Webhook ingestion | The source can push changes and you need them within seconds | Handlers must be idempotent and signature-checked; missed deliveries need a reconciliation path |
| Streaming ingestion | High-volume events feeding dashboards or real-time product features | Ordering and exactly-once semantics become your problem to solve |
| Scheduled batch ETL | Large datasets that change on a known cadence — registries, exports, reports | Data is only as fresh as the last run, which makes the schedule a product decision |
| Change data capture | You need a replica of a database whose application you do not own | Tight coupling to the source schema; their migrations become your outages |
Deal.ee leans on scheduled batch: nineteen government datasets refreshed daily, serving roughly 2.2 million URLs at native speed.
FAQ
Questions we get asked
How do you stop a daily pipeline from silently breaking?
Freshness assertions and alerting on missing work, not only on errors. If a run that should produce 40,000 rows produces 12, the run fails and somebody gets told. Deal.ee refreshes nineteen government datasets every day on exactly that basis.
What if the third-party API is badly documented or unreliable?
That is the normal case, not the exception. We probe it, record what it really returns including its failure shapes, and build against that. Rate limits get a budget, flaky endpoints get backoff, and undocumented behaviour gets written down so the next person is not surprised by it.
Can you connect systems that have no API at all?
Usually. Scheduled file exports, SFTP drops, a database replica, or a scraper where the terms permit one. A worse interface is not a blocker — it just changes which failure modes we have to design for.
Does this need a data warehouse?
Usually not. If the destination is a product surface, PostgreSQL modelled for the queries you actually run is faster, cheaper and simpler than a warehouse you then have to serve from. A warehouse earns its place when analysts rather than applications are the consumer.
How long does an integration take?
A single well-documented API is days. A pipeline across several sources with quality gates and backfill is three to five weeks. The variable is almost never our end — it is how strange the source data turns out to be once you look at all of it.
Who owns and runs it afterwards?
You do, in your infrastructure, with a runbook per failure mode and a backfill your team can run without calling us. We would much rather write the runbook than be the runbook.
Can the pipeline feed an AI feature?
Increasingly that is the reason to build one. Grounding an assistant in your real data means the data has to be current, permission-aware and queryable — which is precisely what this work produces. Deal.ee was built so answer engines could cite it as a primary source.
What about e-invoicing and other regulated formats?
Handled where it is in scope. ERPFlow exports every invoice as Peppol BIS Billing 3.0 in UBL 2.1 XML, the standard for cross-border B2B invoicing in the EU.
Start an integration project
Tell us what needs connecting and how it currently breaks. We reply within one business day.