A pattern for ingesting a continuous external stream in Go: a dumb, fast Kafka consumer that lands raw messages in Postgres, with per-partition fan-out, idempotent batched writes, and a landing table that never expires.

A Go service for consuming real-time UK railway data from Kafka, with table partitioning and integration testing using Docker

Build Kafka producers and consumers in Go using the franz-go library. Covers Kafka architecture, topic management, message serialisation, and practical examples for real-time data streaming.

Explore our journey in revolutionising device data processing using Golang and Kafka, where we tackled high-frequency data challenges, improved throughput, and laid the groundwork for future scalability and efficiency.