How I structure API integrations with Python, SQL and GitHub Actions

Start with the boundary

An integration should have one clear place for credentials, one place for provider calls, and one place for downstream modeling. Mixing these concerns makes retries and audits harder.

Keep raw payloads

The raw layer preserves provider responses before business rules change them. SQL views and modeled tables can evolve without losing the original shape of the API response.

Use idempotent IDs

Repeated collection windows are normal. External IDs, upserts, and controlled logs let a job replay without duplicating records or hiding partial failures.

Why GitHub Actions fits

For lightweight scheduled ingestion, GitHub Actions gives history, manual dispatch, secret storage, and predictable execution without maintaining a permanent worker.

Provider APIs are unstable contracts

Even well documented APIs change behavior in small ways: optional fields disappear, rate limits move, pagination behaves differently, and tokens expire at awkward times. A good integration expects that drift. The code should isolate provider clients, normalize errors, record response metadata, and make it clear which part of the pipeline failed. That is the difference between a script that works today and an integration that can be maintained next month.

Scheduling needs operational memory

A scheduled job needs more than a cron expression. It needs to remember which window it collected, which records were inserted, which records were skipped, and whether a retry is safe. GitHub Actions helps by giving run history and manual dispatch, but the pipeline itself still needs logs and idempotent storage. Without those pieces, a retry can create duplicates or hide the reason a report is incomplete.

SQL keeps business rules visible

When the raw API response is saved first, business rules can move into SQL models where they are easier to inspect. Joins, filters, date windows, and status definitions become explicit instead of being buried inside Python transformations. This also makes stakeholder changes easier. If a report definition changes, the integration does not need to recollect every payload; the modeled layer can evolve from the same auditable source.