Integrations and data pipelines that never fail quietly
In short
What is a data pipeline?
A data pipeline, often called ETL, extracts data from a source, transforms it into the shape your product needs and loads it somewhere fast to query. You need one when the data lives in a system you do not control. Onpolar builds them on Go and PostgreSQL, with checks and alerts so a failure never goes unnoticed.
- Largest running
- 19 government datasets, refreshed daily
- Core stack
- Go · PostgreSQL · River
- Patterns
- Batch · streaming · webhooks
Updated
Example
One nightly run
An illustration of how a daily pipeline runs.
- 01:00FetchEach source is pulled within its rate limits.
- 01:20CheckRow counts, schema and freshness are tested first.
- 01:30TransformData is reshaped for the queries your product runs.
- 01:45LoadThe new data goes live in one switch, or not at all.
- 07:00Fresh by morningOr an alert has already told someone why not.
What you get
API integrations
CRMs, ERPs, payments, shipping and accounting connected, with rate limits respected.
Data pipelines
Scheduled runs that can be replayed, so a bad day upstream is re-run, not fixed by hand.
Webhooks and events
Handled once even when delivered twice, with signatures checked.
Monitoring and alerts
Alerts when expected work does not happen, not only when something errors.
How it works
- 01
Map the sources
What each system really returns, how often it changes and how it fails.
- 02
Build it replayable
Safe retries, a durable job queue, quality checks and backfill from day one.
- 03
Hand over
Dashboards, alerts and a runbook your team can use without us.
Compared
A monitored pipeline vs a quick script
| Compared on | Monitored pipeline | Quick script |
|---|---|---|
| When it fails | An alert when a run is late or short | Noticed weeks later |
| A bad day upstream | Re-run the job | Fix the data by hand |
| Duplicate events | Handled once | Counted twice |
| Who can run it | Your team, with a runbook | Only the person who wrote it |
Questions
How do you stop a daily data pipeline from silently breaking?
With freshness checks and alerts on missing work, not only on errors. If a run that should produce 40,000 rows produces 12, the run fails and somebody is told. Deal.ee refreshes nineteen government datasets every day on exactly that basis.
What if the third-party API is badly documented or unreliable?
That is the normal case. Onpolar probes the API, records what it really returns, including how it fails, and builds against that. Rate limits get a budget, flaky endpoints get backoff, and undocumented behaviour gets written down for the next person.
Can you connect systems that have no API?
Usually. Scheduled file exports, SFTP drops, a database replica, or a scraper where the terms allow one. A worse interface is not a blocker; it only changes which failures the pipeline has to be designed for.
Do we need a data warehouse?
Usually not. If the data feeds a product, PostgreSQL modelled for the queries you actually run is faster, cheaper and simpler than a warehouse. A warehouse earns its place when analysts, rather than an application, are the main users of the data.
How long does an API integration take?
A single well-documented API takes days. A pipeline across several sources, with quality checks and backfill, takes three to five weeks. The variable is almost always how unusual the source data turns out to be once all of it is examined.