Aran Wilkinson

Resume

Unknown — Thu, 11 Jun 2026 00:00:00 +0000

Introducing Headcode: A Unified API for UK Rail Data

Unknown — Wed, 27 May 2026 00:00:00 +0000

UK rail data is brilliantly open and freely available. It is also completely fragmented across dozens of discrete, historically siloed sources. If you want to build an application that tracks trains, you have to piece together obscure feeds from the Rail Data Marketplace, Network Rail Open Data, NaPTAN, and the Office of Rail and Road.

Much of this fragmentation goes back to the legacy of British Rail and the complexities of privatisation. You will encounter Darwin, the real-time passenger information engine operated by the Rail Delivery Group, which feeds the physical departure screens on the platforms. You will stumble into TRUST and TOPS, the legacy Network Rail systems that monitor real-time train progression against schedules and track individual pieces of rolling stock. You will see raw signals from Train Describer (TD) berths tracking train movements piece-by-piece via trackside axle counters.

Integrating directly with official rail data means wrangling huge CSV datasets, legacy XML endpoints, and complex Kafka streams while constantly translating complex data structures. There are open-source proxies like Huxley that wrap Darwin, and excellent mature tools built around this data like Realtime Trains for power users or OpenTrainTimes for live signalling maps. But if you are a typical developer who just wants a clean JSON payload of what trains are leaving a station right now, the barrier to entry is absurdly high.

I am building Headcode to solve this. The premise is straightforward: one integration instead of twelve, with intelligence baked in. Headcode consumes these scattered raw feeds, performs the complex reconciliation, normalisation, and enrichment, and exposes the result through clean REST endpoints. Downstream developers get the data they need without maintaining stream ingestion pipelines.

The name comes from the terminology used by the UK rail industry itself. A "headcode" is the official Train Reporting Number assigned to every single service to uniquely identify it to signallers and routing systems. For example, 1A65</code>. The 1</code> indicates the class of train, such as an express passenger service. The A</code> specifies the destination area. The 65</code> is the individual identifying number for that specific service. It is a precise identifier, which is exactly what a good API should provide.

The API experience</a></h2> I wanted the developer experience to feel like a modern platform, not a raw scrape of a government database. Every API request should return insights or connected data that you physically cannot get from a single raw feed alone. The service needs to add value in the middle layer. One of the biggest points of friction in UK rail data is the identifier systems. The industry uses several overlapping codes for the exact same physical location: CRS codes: the three-letter customer-facing codes you see on passenger tickets, like PAD</code> for London Paddington.</li> TIPLOCs: seven-character codes used for routing and scheduling, like PADTON</code>.</li> STANOX codes: a five-digit internal numbering system used by legacy signalling systems.</li> </ul> If you query a raw Darwin live feed, it will identify a station using a TIPLOC. If you query a retail ticketing feed, you will get a CRS. Headcode treats these identifiers natively and forgivingly. You can query the API with any of these codes. The service understands the cross-references and translates them on the fly, returning all known identifiers in the response payload. You do not have to maintain your own mapping tables synced with CORPUS or constantly query reference databases to translate what the API gives you. You ask for Paddington using PAD</code>, and the API knows exactly what you mean. The data returned is strictly unified. Headcode seamlessly blends real-time live data with static and historic datasets. When you pull a list of delays or the train order for a specific platform, the response is enriched with static accessibility information, station operator details, and physical coordinates sourced from NaPTAN. You get the whole picture in a single structured response, rather than having to stitch together four separate HTTP calls. What you can build with it</a></h2> Right now, Headcode handles the heavy lifting of continuous ingestion. It takes in massive static reference dumps alongside live Darwin Kafka streams, translating thousands of real-time operational messages a second into structured relational data. You can hit the API for live station departure and arrival boards directly derived from Darwin. You can pull comprehensive station metadata with fuzzy searching. If a user types "Paddingon" instead of "Paddington", the API will still resolve the correct station and return the full metadata tree without throwing a 404. There are specific endpoints for detailed service descriptions and network disruptions sourced from operational train alerts or station messages. You can query platform train-order endpoints to see exactly which trains are queued to arrive at a specific platform and in what sequence. Calculating platform order is surprisingly hard when you are looking at baseline timetables because trains run out of order all the time. Headcode does the active calculation based on real-time track progression and provides the corrected sequence. The entire service is secured via API-key Bearer authentication sitting on a custom auth layer. To provide a concrete example of what you can build on top of this platform, I am currently developing a Bluesky bot. It will act as an active webhook consumer of the Headcode API (webhooks are a planned feature coming soon to the platform). Instead of aggressively polling endpoints to check if a train is late, it will listen for events and automatically post live train information, severe network disruptions, and active delays for specific high-traffic routes. The bot serves as a demo of the platform in the wild, proving that you can build reactive social tools without needing to understand Kafka messaging protocols or trackside axle counter logic. Under the hood</a></h2> The backend is written entirely in Go. The service is deployed as a monolith. I run multiple instances of the service, each responsible for a subset of the system. The system is dealing with constant firehoses of Kafka messages and concurrent HTTP requests. Go's native concurrency model handles stream ingestion effortlessly without complex worker pool orchestration. The compiled binary gives me predictable memory usage and low-latency API responses. The core product is served as REST JSON. I am using OpenAPI code generation to maintain strict contracts between the server responses and the generated documentation. I use ConnectRPC over HTTP/2 specifically for health checks and internal service state. ConnectRPC provides a lightweight, strictly typed protocol that is perfect for internal monitoring without pulling in the bloat of standard gRPC dependencies. PostgreSQL serves as the backbone of the entire system, handling millions of messages per day. I use pgx</code> for the driver and sqlc</code> for database interaction. What makes sqlc</code> brilliant is that it compiles plain SQL queries directly into type-safe Go code. It removes the need for unpredictable ORMs and keeps exact database access patterns explicit during code reviews. I rely heavily on Postgres extensions to push the heavy lifting down to the database layer. PostGIS powers all the spatial queries. When an endpoint needs to find all stations within a specific physical radius of a GPS coordinate, PostGIS handles the geospatial maths natively and incredibly fast. For text searching, pg_trgm</code> provides trigram matching. This is what enables the fuzzy searching on station names. Standing up and managing a separate Elasticsearch cluster just to handle user typos is unnecessary architectural overhead when Postgres handles it directly and efficiently. The infrastructure is intentionally simple and vertically scaled. The stack runs on a Rocky Linux VM hosted on Proxmox. I chose Rocky Linux for its strict bug-for-bug compatibility with RHEL and sheer stability as a hypervisor guest. Everything is explicitly provisioned with Terraform. Caddy acts as the web server, and traffic is routed securely back to the internet via Cloudflare Tunnels. It is a robust, low-maintenance setup ideal for the current scale of Headcode. I will likely migrate the workloads to something else to scale out the system in the future. But spinning up a monolithic VM gets the product built, shipped, and tested with real user traffic without fighting abstraction layers. Taking it for a spin</a></h2> Because Headcode abstracts away the domain complexities of the UK rail network, developers can spend their time actually building transit applications instead of debugging XML parsers and cross-referencing obscure alphanumeric identifiers. While Headcode is currently in closed beta, the documentation is completely open. You can read the comprehensive API documentation, explore the response schemas, and see how the endpoints are structured right now at docs.headcode.dev</a>. I will be posting further updates on the Bluesky bot, API access availability, and new Headcode features over the coming weeks as the system expands.

Introducing jjw: a workspace manager for jj Unknown — Sat, 16 May 2026 00:00:00 +0000 I've just released jjw</a>, a small command line tool for managing jj</a> workspaces. The short version is that jjw</code> gives me a nicer workflow for creating, switching between, and deleting isolated jj workspaces. The slightly longer version is that I wanted a reliable way to run multiple coding agents in parallel without them trampling over each other's working copies, local state, or setup. It is inspired by wt</a>, a Git worktree manager, but I decided to adapt it for working with jj and jj's workspace and bookmark model. Why I built it</a></h2> I've been using jj</code> more recently and really like the model. It makes a lot of day-to-day version control operations feel calmer, especially once you get used to the working copy being a commit and the operation log being there as a safety net. At the same time, my development workflow has changed. I often want to try a few ideas at once, or hand off different tasks to different LLM coding agents. That works best when each task has its own clean, isolated environment. You can already do this with jj workspace add</code>, but I found myself wanting a bit more structure around it: consistent workspace names and locations</li> matching bookmarks for pull requests</li> quick shell integration so creating or switching workspaces actually moves my shell there</li> lifecycle hooks for setting up or tearing down per-workspace state</li> safe deletion and cleanup once work has been merged</li> </ul> That became jjw</code>. What jjw does</a></h2> jjw</code> manages workspaces inside a configured directory in your repository. A basic setup looks like this: version: 1 workspace_dir: workspaces default_branch: main</code></pre> Once configured, creating a workspace is as simple as: jjw create feature-x</code></pre> Behind the scenes, jjw</code> will: create a jj workspace at workspaces/feature-x</code></li> create a bookmark for the workspace</li> allocate a workspace index, if configured</li> run any configured post_create</code> hooks</li> ask the shell integration to move you into the new workspace</li> </ol> From there you can work normally, push the bookmark with jj git push</code>, open a pull request, and eventually clean things up when the work is merged. The commands</a></h2> The core commands are intentionally small: jjw create feature-x # create workspace + bookmark, then cd into it jjw list # list workspaces jjw cd feature-x # switch to a workspace jjw exit # return to the main repo jjw delete feature-x # delete workspace, bookmark, and files jjw cleanup # remove clean workspaces with merged bookmarks</code></pre> There are also configuration and utility commands: jjw config init jjw config get workspace_dir jjw root jjw version</code></pre> The goal is not to wrap every part of jj. It is only there to smooth over the workspace lifecycle. Hooks for real project setup</a></h2> The feature that makes this particularly useful for me is lifecycle hooks. A lot of projects need a bit of per-workspace setup. Maybe you need a unique dev server port, a generated .env</code> file, a temporary database name, or some other local resource. When running multiple agents in parallel, those details matter a lot more. jjw</code> supports hooks such as: pre_create</code></li> post_create</code></li> pre_delete</code></li> post_delete</code></li> </ul> Hooks receive useful environment variables, including: JJW_NAME JJW_PATH JJW_BOOKMARK JJW_REPO_ROOT JJW_JJ_ROOT JJW_WORKSPACE_DIR JJW_INDEX</code></pre> For example, you can use the allocated index to assign each workspace a different port: hooks: post_create: - script: ./scripts/setup-ports.sh</code></pre>#!/bin/bash PORT=$((3000 + JJW_INDEX)) echo "Dev server port: $PORT"</code></pre> That is the kind of small automation that makes parallel workspaces much easier to live with. Getting started</a></h2> The easiest way to install jjw</code> is with go install</code>: go install github.com/aranw/jjw@latest</code></pre> Alternatively, you can build it from source. This requires Just</a> to be installed. git clone https://github.com/aranw/jjw.git cd jjw just install</code></pre> Then add the shell integration for your shell: # zsh eval "$(jjw init zsh)" # bash eval "$(jjw init bash)" # fish jjw init fish | source</code></pre> jjw</code> currently expects jj</code> 0.25+ and a colocated jj/git repository. In a repository where you want to use it, initialise the config: jjw config init</code></pre> Then create your first workspace: jjw create my-change</code></pre>What's next</a></h2> This is an initial release, so I expect the edges to become clearer as I use it more. The current focus is keeping the tool boring and predictable: create workspaces, connect them to bookmarks, run hooks, and clean them up safely. If you use jj</code>, run multiple local workstreams, or are experimenting with coding agents, I'd be interested to hear whether this fits your workflow. The project is on GitHub here: github.com/aranw/jjw</a>. Why I built a reference Go project (and taught an AI agent to clone it) Unknown — Wed, 13 May 2026 00:00:00 +0000 I get asked about how to structure a Go project often. Not the big architectural questions about microservices versus monoliths, but the mundane stuff: where does the config go, why isn't there a pkg/</code> directory, what's the difference between internal/books/</code> and internal/database/books/</code>. I used to answer with a wall of text. Then I started linking to specific repos I'd worked on, which helped but came with baggage — real projects accumulate weird one-off decisions that don't generalise. So I built a reference instead: example-project-structure</a>, a minimal Go HTTP/RPC service backed by PostgreSQL. A fake bookstore with just enough domain (books, authors, genres) to show joins and type conversions without drowning in features. That was reason one. Reason two turned out to be more interesting. Structure as a document an agent can read</a></h2> I've been spending time with AI coding agents lately, and I kept running into the same friction: every time I start a new project, I want it in a structure I already know and like. But AI output is non-deterministic. Ask an agent to scaffold a project twice and you'll get two different structures with the odd similarity. One run gives you models/</code>, the next gives you entities/</code>, neither matches what you actually want. So I thought: why not give it a concrete example to bootstrap from instead of hoping it guesses right? So I wrote the reference repo with a second reader in mind. The docs/</code></a> directory isn't just for humans — layout.md</code></a> maps every directory and explains what lives where, design-decisions.md</code></a> captures the reasoning behind choices like "Service not Repository" and "package-by-feature not package-by-layer", and adding-an-entity.md</code></a> is a step-by-step walkthrough an agent can follow mechanically. Then I took it a step further. The repo includes a BOOTSTRAP.md</code></a> file that's written specifically for an AI agent. It assumes the agent has already cloned the repo and copied its contents into a new project directory. From there it walks through every rename, every file to strip, every conditional based on what the user actually wants (OpenAPI only? ConnectRPC only? No database?). Each step lists what to change and every place it appears so the agent doesn't need to rediscover the structure on its own. I've paired this with a Claude Code skill (not open source yet, still refining it) that asks the user a handful of questions (project name, module path, API style, database choice) and then follows BOOTSTRAP.md</code> to scaffold a new project. When paired with some details about what you're building, the AI can produce a fairly decent starting point for a project. What the structure actually looks like</a></h2> The project is small by design — just enough to show the patterns without drowning in features: . ├── api/ # protobuf & OpenAPI schemas ├── apigen/ # generated API stubs ├── cmd/bookstore/ # binary entry point └── internal/ ├── books/ # domain package (one per feature) ├── database/ # data access, separated from transport └── server/ # HTTP/RPC wiring</code></pre> There's no pkg/</code>, no models/</code>, no handlers/</code> directory. Each of those absences is a deliberate choice, and I've written up the reasoning in the repo's docs. Rather than cramming everything into one post, I'm breaking the details into separate pieces: The structure itself — the directory tree, what each package owns, the layering rules, and what's deliberately missing</li> Design decisions — why Service over Repository, why package-by-feature wins over time, why manual DI in runner.go</code></li> The BOOTSTRAP.md approach — writing documentation that serves both humans and AI agents, how the Claude Code skill works, and what I've learnt about making repos agent-friendly</li> </ul> For now, the repo is public and the docs are thorough. Have a look at example-project-structure</a> — I'll dig into the design decisions and the BOOTSTRAP.md approach in follow-up posts. Now Unknown — Tue, 12 May 2026 00:00:00 +0000 living in Huddersfield, UK</li> working remotely as a Golang Engineer</li> our little one is coming up on their first birthday, which has been the best and most tiring year of my life</li> exploring UK rail data and building Go services to process real-time train information with Kafka and PostgreSQL (wrote about it here</a>)</li> planning to rework parts of my home smart lighting project, which has been on hold for a bit</li> leaning into AI tools to bring ideas to life quicker, iterating on more projects than I'd have time for otherwise. I did lose a database</a> along the way, which taught me to be more structured about it</li> squeezing in photography and DIY on the house when I get the odd free weekend</li> still writing about projects and Go on the blog</a>, trying to keep a steady pace</li> </ul> About /now</h2> Inspired by Derek Sivers</a>, nownownow.com</a>, and aboutideasnow.com</a>, this is a "now" page. It's a simple way to tell you what I'm up to right now. This page is periodically updated manually to reflect what I'm currently working on, learning, and doing. It's a snapshot of my life at this moment in time. How I lost a database and learned to actually use AI Unknown — Tue, 12 May 2026 00:00:00 +0000 A while back I was working on a personal project, a small homelab setup that collected boiler and heating measurements into a PostgreSQL database. I was using AI to help me with some database migration work, restructuring tables and migrate them to a being partitioned. The AI gave me a sequence of SQL commands, confident and clear. I copied and pasted them into my database editor and ran them. No backups. The data was gone. It was my fault. Not in a hand-wavy "we're all responsible" kind of way. I literally ran the commands without properly understanding them. The AI suggested dropping the original table before I'd verified the temp table was intact, and I just did it. But here's the thing: I didn't feel reckless at the time. The session had been going well. The AI suggested a command, I ran it, it worked. Then another, worked again. Each successful step built a kind of momentum, and by the time we got to the destructive operation my guard was completely down. The prior successes had done their job. I was confident we were on the right track. That's the pattern worth watching out for. It's not that AI gives you bad commands (often it doesn't). It's that a long run of good ones quietly erodes your scepticism, so when it eventually gives you something that doesn't quite do what you expected, you're not in the right headspace to catch it. You've been lulled into executing rather than reviewing. There's a story that did the rounds recently where a developer claimed a Cursor/Claude agent deleted his company's production database. Ibrahim Diallo makes a blunt point about it</a>: the AI didn't do anything. You gave it the access, you ran the process, and somewhere in your architecture there was an endpoint capable of wiping your entire production database. That's not an AI problem. He's right. But I think it's worth unpacking why people end up there, because it's not simply carelessness. AI assistants are unnervingly fluent. They produce SQL, shell commands, and migration scripts with the same confident tone regardless of whether the operation is reversible or catastrophic. There's no hesitation, no flag on the dangerous bit. You have to bring your own scepticism, and after twenty minutes of a productive session where everything has worked exactly as described, that scepticism is the first thing to go. After losing that data I spent some time thinking about what had actually gone wrong in my process, beyond the obvious "take backups" lesson. The answer wasn't complicated: I had no structure around what the AI was doing. I was having a conversation, following its lead, and executing whatever it produced. There were no checkpoints, no moments where I stepped back and asked whether what was happening matched what I actually wanted. Working out a better approach took months, not sessions. It meant experimenting across different projects, paying attention to where AI added real value and where it quietly created more problems than it solved. I don't think that kind of learning can be shortcut by reading someone else's process, but I can at least describe what I landed on. I start by writing a PRD, a short document that captures what needs to happen and why, at a level of abstraction I can actually reason about. I put this together with AI help, but I review it and own it before anything gets built. It's not a chat transcript, it's a document I'd be comfortable handing to a colleague. The point is to have something I've thought through before I start generating code, so that I have a reference point to check against when things start moving fast. From the PRD I generate tasks. Small, atomic ones. Each task includes the specific code changes involved, so when the AI picks one up it knows exactly what it's introducing and modifying. This keeps it focused on a narrow scope rather than trying to hold an entire codebase in context, and it means if something goes wrong I know roughly what were working on, what went wrong and why. Handing it a whole PRD to implement in one go is a reliable way to watch it confidently go sideways halfway through. Smaller tasks mean shorter context, which means fewer hallucinations and less drift. At every stage (PRD, tasks, code) I'm the one verifying that what's happening matches what I intended. The AI is generating, I'm reviewing. For code changes, I commit at logical points as I go using Jujutsu</a>, a version control system that makes frequent checkpointing feel natural rather than ceremonial. Sensible intervals mean that if something goes wrong I'm not scrambling to unpick a large, tangled diff. For commands (SQL, terminal operations, anything running directly against a live system) the approach is different because you can't just roll back to a previous commit. Here I slow down and actually read what the AI has given me before running it. If it's touching a database, I take a backup first. The momentum problem from my opening story is exactly what I'm trying to resist: the temptation to keep executing because the last five commands worked fine. A destructive SQL statement looks just as routine as a SELECT</code> in the flow of a session, and it's easy to treat it the same way. There are a few areas I still want to explore. One is sandboxing tooling like Agent Safehouse</a>, which aims to constrain what an AI agent can actually reach. Another is VM-based AI agents, where the entire environment the agent operates in is disposable and isolated. Both feel like they're attacking the right problem: rather than relying entirely on the human to catch the dangerous command, you limit the blast radius of what can go wrong in the first place. The other thing I want to look at is database branching. The idea is borrowed from version control: instead of running migrations against the database you care about, you create an isolated copy, run your changes there, and only merge when you're confident it's right. Neon</a> offers this as a managed service, using copy-on-write storage so branches are created in seconds without duplicating data. Each branch gets its own connection string, completely isolated from the parent. If my original project had been running on something like that, I could have branched the database before letting the AI anywhere near it and thrown the branch away when it went wrong. What I'm less clear on is how to get something similar locally. There are a handful of tools in this space. pgsh</a> uses Postgres template databases to let you branch and switch between database states from the command line. Database Lab Engine</a> takes a different approach, using ZFS copy-on-write snapshots to create thin clones of full databases. And at the simpler end, Postgres schemas can act as a kind of lightweight namespacing, letting you create isolated environments within a single database by manipulating the search path, which is useful for testing even if it's not true branching. I haven't tried any of these yet, but the general pattern is appealing: make it cheap to create a disposable copy of your database state so that destructive operations have a clear undo path. That would have saved me a lot of pain. All of this is really just structure around one idea: the AI doesn't know which of its suggestions will ruin your afternoon, and it won't tell you. The fluency is real. The judgement isn't. Those are different things, and the longer a session goes on, the easier it is to forget that. The shift in how I think about AI is probably the most useful thing to come out of losing that data. I've stopped treating it as a collaborator who has my back, and started treating it as a very fast, capable tool that will do exactly what you ask without judgement. Services Unknown — Tue, 12 May 2026 00:00:00 +0000 Alongside my day job, I take on a small number of freelance projects — usually teams who need hands-on engineering help with Go, distributed systems, or cloud infrastructure. I'm selective about what I take on. I'd rather do a few things well than spread myself thin, so if your project isn't a good fit for my skills, I'll say so. What I can help with</h2> Backend engineering and system design — I build services in Go: APIs, workers, data pipelines, event-driven architectures. I'm comfortable taking a blank repo to production, or picking up an existing codebase and making it better. Cloud infrastructure — Most of my experience is with Google Cloud — Cloud Run, Compute Engine, Cloud Functions, BigQuery, Pub/Sub, Cloud Storage. I design, build, and operate cloud-native systems that scale without becoming a maintenance headache. Data and integration — Real-time pipelines, Kafka producers and consumers, database design and migration (mostly PostgreSQL), and connecting systems that were never meant to talk to each other. Architecture and technical strategy — Sometimes you just need someone to look at what you've got, work out what's working, and help plan what comes next: architecture reviews, technology choices, migration planning. My primary stack is Go, PostgreSQL, Kafka, and Google Cloud, with Docker, Terraform, GitHub Actions, and gRPC alongside. I've spent time with Rust too, though Go is where I'm most productive. How I work</h2> I like to stay close to your team and codebase rather than working in isolation — I just do it on my own schedule. That might mean evening commits, async code reviews, or catching up on Slack outside your usual hours. I'm based in the UK (GMT), and engagements typically run from a few weeks to a few months. Get in touch</h2> If you think I could help with something you're working on, the easiest way to start is to book a short call — no commitment, just a conversation about what you need and whether I'm the right person for it. Book a call</a> You can also reach me by email at aran@aran.dev</a>. Using CUE to unify IoT sensor data Unknown — Fri, 17 Oct 2025 00:00:00 +0000 I've been building a home automation system that processes motion sensor data from various IKEA Zigbee devices. Each sensor type sends a different data structure, and I needed to extract consistent information from all of them without sacrificing type safety. I spent a long time trying to make Home Assistant work for my setup. The visual automation builder felt limiting once I needed conditional logic beyond simple triggers, and writing automations in YAML became tedious. Adding complex logic like correlating motion across multiple rooms or applying time-weighted logic was difficult. I ended up fighting the system rather than solving the problem. My current home automation setup looks like this: ┌─────────────┐ ┌─────────────┐ │ TRÅDFRI │ │ VALLHORN │ │ Sensor │ │ Sensor │ └──────┬──────┘ └──────┬──────┘ │ │ └────────┬───────┘ Zigbee │ ▼ ┌──────────────────┐ │ zigbee2mqtt │ │ (Zigbee bridge) │ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ MQTT Broker │ │ │ │ Topics: │ │ bedroom/motion │ │ hallway/motion │ │ kitchen/motion │ └────┬─────────┬───┘ │ │ │ │ JSON messages ┌───────┘ └───────┐ ▼ ▼ ┌──────────────┐ ┌──────────────┐ │ Node-RED │ │ Home │ │ │ │ Assistant │ │ - Visual │ │ │ │ flows │◄────►│ - Visual │ │ - Custom │ │ automata │ │ logic │ │ - YAML │ └──────────────┘ └──────┬───────┘ │ │ HTTP API ▼ ┌──────────────────┐ │ Philips Hue Hub │ │ │ │ ┌────┐ ┌────┐ │ │ │Bulb│ │Bulb│ │ │ └────┘ └────┘ │ │ │ │ ┌──────┐ │ │ │Sensor│ │ │ └──────┘ │ └──────────────────┘</code></pre> With Home Assistant</a> and Node-RED</a> handling some of my automation logic, and controlling devices through various hubs like Philips Hue. This works for many people, but I wanted more control and flexibility that I knew I could get with a custom Go service. I could build exactly what I wanted with the language I already knew. I'm building a Go service to replace this entire automation layer—Home Assistant, Node-RED, and the device-specific hubs. The service subscribes directly to MQTT</a> topics where zigbee2mqtt publishes sensor data, processes that data, and sends commands back to control lights and other devices. The system architecture</a></h2> The sensors connect via Zigbee, a low-power wireless protocol common in home automation. I'm running zigbee2mqtt</a> which bridges between Zigbee devices and an MQTT broker. My Go service subscribes to MQTT topics and processes the JSON messages that sensors publish whenever their state changes. Currently, I'm using two different sensor types: IKEA TRÅDFRI motion sensor</a> – the older model that reports light levels as a boolean</li> IKEA VALLHORN motion sensor</a> – the newer model with precise lux readings and motion timing</li> </ul> Eventually, I'll add others including Philips Hue sensors. Each type has its own unique data structure with different fields. I'd seen this pattern before—small differences that compound into maintenance headaches as the codebase grows. The zigbee2mqtt documentation shows what each sensor exposes. The TRÅDFRI reports light levels as a boolean illuminance_above_threshold</code>, whilst the VALLHORN gives you actual lux readings in an illuminance</code> field. The VALLHORN tells you precisely how many seconds have passed since motion stopped via no_occupancy_since</code>; the TRÅDFRI doesn't provide this at all. Both detect motion with an occupancy</code> boolean, but handling these differences in pure Go means type switches, assertions, and duplicated logic across every method that touches sensor data. CUE</a> is an open-source data validation language and inference engine that lets you define schemas, validate data against those schemas, and template data. Unlike JSON Schema which only validates, or Protocol Buffers which require compilation, CUE combines validation and data templating in a single language with a syntax that feels familiar to JSON and Go. From the beginning I knew CUE</a> would let me validate sensor data at the boundary and transform it into consistent structures, meaning the rest of my code doesn't need to know which sensor type it's processing. CUE handles the differences between sensor types in both directions, but this article focuses on the inbound transformation (I'll cover outbound MQTT messages in a follow-up article). This article shows my CUE implementation alongside what a pure Go approach might have looked like, demonstrating where CUE removes the boilerplate. The pure Go approach and its limitations</a></h2> Here's what processing sensor data might have looked like if I'd built this system in pure Go without CUE: // Separate structs for each sensor's raw data type TradfriRawData struct { Battery int `json:"battery"` Occupancy bool `json:"occupancy"` IlluminanceAboveThreshold bool `json:"illuminance_above_threshold"` Linkquality int `json:"linkquality"` } type VallhornRawData struct { Battery int `json:"battery"` Occupancy bool `json:"occupancy"` Illuminance int `json:"illuminance"` NoOccupancySince *int `json:"no_occupancy_since,omitempty"` // Optional, pointer to handle null Linkquality int `json:"linkquality"` Voltage int `json:"voltage"` } type MotionSensorMessage struct { SensorID string `json:"sensor_id"` Timestamp string `json:"timestamp"` Floor string `json:"floor"` SensorType string `json:"sensor_type"` RawData any `json:"raw_data"` } // Processing requires routing based on sensor type func ProcessMQTTMessage(topic string, payload []byte, metadata SensorMetadata) (*MotionSensorMessage, error) { var rawData any switch metadata.SensorType { case "ikea_tradfri_motion": var tradfri TradfriRawData if err := json.Unmarshal(payload, &tradfri); err != nil { return nil, fmt.Errorf("failed to unmarshal tradfri data: %w", err) } rawData = tradfri case "ikea_vallhorn_motion": var vallhorn VallhornRawData if err := json.Unmarshal(payload, &vallhorn); err != nil { return nil, fmt.Errorf("failed to unmarshal vallhorn data: %w", err) } rawData = vallhorn default: return nil, fmt.Errorf("unknown sensor type: %s", metadata.SensorType) } return &MotionSensorMessage{ SensorID: metadata.SensorID, Timestamp: time.Now().Format(time.RFC3339), Floor: metadata.Floor, SensorType: metadata.SensorType, RawData: rawData, }, nil } // Every method needs type switches func (m *MotionSensorMessage) IsMotionDetected() (bool, error) { switch data := m.RawData.(type) { case TradfriRawData: return data.Occupancy, nil case VallhornRawData: return data.Occupancy, nil default: return false, fmt.Errorf("unknown sensor data type") } } func (m *MotionSensorMessage) IsLightSufficient() (bool, bool, error) { switch data := m.RawData.(type) { case TradfriRawData: // Field is inverted - true means dark return true, !data.IlluminanceAboveThreshold, nil case VallhornRawData: // Consider light sufficient if illuminance is above 50 lux return true, data.Illuminance > 50, nil default: return false, false, fmt.Errorf("unknown sensor data type") } } func (m *MotionSensorMessage) GetSecondsSinceMotion() (int, bool, error) { vallhorn, ok := m.RawData.(VallhornRawData) if !ok { return 0, false, nil // Not supported on this sensor } if vallhorn.Occupancy { return 0, false, nil // Motion detected, field not present } if vallhorn.NoOccupancySince == nil { return 0, false, nil // Field is null } return *vallhorn.NoOccupancySince, true, nil }</code></pre> Typed structs don't solve the fundamental problem. Every method that extracts data requires type switches to handle the union. Optional fields like NoOccupancySince</code> need pointers to distinguish between "not present", "null", and "has a value". The VALLHORN sensor provides precise timing about how long a room has been empty, but extracting this value requires checking the sensor type, verifying occupancy is false, checking the pointer isn't nil, and dereferencing it. Add a third sensor type and you extend every type switch with another case. When IKEA releases a firmware update that changes a field, you hunt through multiple functions to fix it. I knew I'd be adding more sensor types and dealing with firmware variations, so I chose CUE from the start rather than fighting this complexity. You could build similar behaviour in pure Go with interfaces and careful abstraction. CUE handles this declaratively. You define schemas with constraints and transformations, and CUE validates the incoming data whilst extracting normalised structures. What CUE provides</a></h2> CUE is a configuration language that handles everything from defining schemas and generating configuration files to validating data and computing transformations. It's used for API definitions, build configurations, policy validation, and more. In this system, I use it to validate sensor data at the boundary and transform it into normalised structures that my Go code can work with consistently. Instead of validating battery levels with scattered if statements through your Go code, the schema catches invalid data at the boundary: battery: int & >=0 & <=100</code></pre> CUE definitions (prefixed with #</code>) work like type definitions in Go. You can reference them and compose them with other schemas, but they don't produce output by themselves. This lets you build a union type for all supported sensors: #MotionSensor: #IkeaTradfriMotionSensorStatus | #IkeaVallhornMotionSensorStatus</code></pre> The language handles both validation and transformation. This means you can turn the TRÅDFRI's boolean light sensor and the VALLHORN's lux reading into a single light_sufficient</code> field without writing Go type switches. Building the schemas</a></h2> I started with a base schema that all sensors share: #MotionSensorBase: { sensor_id!: string timestamp?: string floor!: "basement" | "ground" | "upstairs" | "attic" sensor_type!: string raw_data!: _ }</code></pre> The !</code> operator marks required fields, whilst ?</code> marks optional ones. The raw_data</code> field uses CUE's top type _</code>, which accepts any value. This allows each sensor type to define its own structure for raw_data</code>. Each sensor gets its own schema that composes the base with sensor-specific constraints: #IkeaTradfriMotionSensorStatus: #MotionSensorBase & { sensor_type: "ikea_tradfri_motion" raw_data: { battery: int & >=0 & <=100 occupancy: bool illuminance_above_threshold: bool linkquality: int & >=0 & <=255 } }</code></pre> The &</code> operator composes schemas. Each sensor gets its own constraints that match IKEA's hardware specifications. The VALLHORN schema includes additional fields that the older TRÅDFRI doesn't support: #IkeaVallhornMotionSensorStatus: #MotionSensorBase & { sensor_type: "ikea_vallhorn_motion" raw_data: { battery: int & >=0 & <=100 occupancy: bool illuminance: int & >=0 & <=65535 linkquality: int & >=0 & <=255 // Only present when occupancy is false no_occupancy_since?: null | int & >=0 voltage: int } }</code></pre> The no_occupancy_since</code> field is optional (marked with ?</code>) and can be either null or a positive integer. When motion stops, the sensor starts counting seconds and reports them. Extracting normalised data</a></h2> The extraction schemas contain the conditional logic that would otherwise live in Go type switches. Here's motion detection: #ExtractMotionFromSensor: { sensor: #MotionSensor motion_detected: bool if sensor.sensor_type == "ikea_tradfri_motion" || sensor.sensor_type == "ikea_vallhorn_motion" { motion_detected: sensor.raw_data.occupancy } // Optional field for precise timing on supported sensors no_occupancy_seconds?: int if sensor.sensor_type == "ikea_vallhorn_motion" { // Only present when occupancy is false and sensor supports it if sensor.raw_data.occupancy == false && sensor.raw_data.no_occupancy_since != _|_ && sensor.raw_data.no_occupancy_since != null { no_occupancy_seconds: sensor.raw_data.no_occupancy_since } } }</code></pre> In Go, you'd write a method on each sensor struct to extract this data. Add a new sensor and you update every extraction method. With CUE, you add one conditional block to the schema and the extraction works everywhere you use it. The no_occupancy_seconds</code> field shows how CUE handles sensor-specific features. The VALLHORN provides this timing information, the TRÅDFRI doesn't. Making the field optional means sensors without this capability simply don't include it. Automation logic can check if it exists and use it for precise timing ("turn off lights if no motion for 10 minutes"), or fall back to basic occupancy detection when it's not available. Light detection demonstrates how CUE normalises different measurement approaches: #ExtractAmbientLightFromSensor: { sensor: #MotionSensor light_sufficient?: bool has_ambient_sensor: bool if sensor.sensor_type == "ikea_tradfri_motion" { if sensor.raw_data.illuminance_above_threshold != _|_ { has_ambient_sensor: true // This field is inverted - true means dark light_sufficient: !sensor.raw_data.illuminance_above_threshold } } if sensor.sensor_type == "ikea_vallhorn_motion" { if sensor.raw_data.illuminance != _|_ { has_ambient_sensor: true // 50 lux is sufficient for movement without lights light_sufficient: sensor.raw_data.illuminance > 50 } } }</code></pre> The TRÅDFRI's boolean gets inverted (it indicates darkness rather than light). The VALLHORN's lux reading gets compared to a threshold. Application code doesn't need to know about these differences. It just checks light_sufficient</code>. How the Go code works with CUE</a></h2> With CUE handling validation and transformation, the Go code becomes generic. In my Go code, I've decided to wrap optional fields in an Option type that provides convenience helpers like IsPresent()</code> and MustGet()</code> methods for safely handling values that may or may not exist. Here's the actual implementation: // Generic processor that works for ALL sensor types type MotionSensorProcessor struct { cueCtx *cue.Context extractMotionSchema cue.Value // #ExtractMotionFromSensor extractAmbientLightSchema cue.Value // #ExtractAmbientLightFromSensor } // This function works for Vallhorn, Tradfri, or any future sensor func (p *MotionSensorProcessor) ProcessSensorData(sensorData MotionSensor, statusSchema cue.Value) error { // Encode to CUE and validate against the sensor-specific schema sensorDataAsCUE := p.cueCtx.Encode(sensorData) if err := statusSchema.Unify(sensorDataAsCUE).Validate(); err != nil { return fmt.Errorf("schema validation failed: %w", err) } // Extract motion event using CUE's extraction schema extractMotion := map[string]any{"sensor": sensorData} extractMotionAsCUE := p.cueCtx.Encode(extractMotion) unifiedMotion := p.extractMotionSchema.Unify(extractMotionAsCUE) // Decode back to a clean Go struct var motionEvent MotionEvent if err := unifiedMotion.Decode(&motionEvent); err != nil { return fmt.Errorf("failed to decode motion event: %w", err) } // Identical code for all sensor types if motionEvent.MotionDetected { handleMotionDetected(sensorData.Floor) } // Works for Vallhorn (which has the field) and gracefully handles // Tradfri (which doesn't) via the Option type if motionEvent.NoOccupancySeconds.IsPresent() { seconds := motionEvent.NoOccupancySeconds.MustGet() if seconds > 600 { handleNoMotionTimeout(sensorData.Floor, seconds) } } // Extract ambient light reading using CUE extractAmbientLight := map[string]any{"sensor": sensorData} extractAmbientLightAsCUE := p.cueCtx.Encode(extractAmbientLight) unifiedAmbientLight := p.extractAmbientLightSchema.Unify(extractAmbientLightAsCUE) var lightReading AmbientLightReading if err := unifiedAmbientLight.Decode(&lightReading); err != nil { return fmt.Errorf("failed to decode ambient light: %w", err) } // Handles both Tradfri's boolean and Vallhorn's lux reading // CUE normalised them both into a single light_sufficient field if lightReading.HasAmbientSensor && lightReading.LightSufficient.IsPresent() { if !lightReading.LightSufficient.MustGet() { handleDarkness(sensorData.Floor) } } return nil } // Vallhorn-specific handler - ONLY handles unmarshalling type VallhornHandler struct { processor *MotionSensorProcessor statusSchema cue.Value // #IkeaVallhornMotionSensorStatus } func (h *VallhornHandler) HandleMQTT(topic string, payload []byte) error { sensorID, floor, err := parseSensorTopic(topic) if err != nil { return fmt.Errorf("failed to parse topic: %w", err) } // THIS IS THE ONLY VALLHORN-SPECIFIC CODE var rawData VallhornRawData if err := json.Unmarshal(payload, &rawData); err != nil { return fmt.Errorf("failed to unmarshal: %w", err) } // Build the generic sensor data structure sensorData := MotionSensor{ SensorID: sensorID, Timestamp: time.Now().Format(time.RFC3339), Floor: floor, SensorType: "ikea_vallhorn_motion", RawData: rawData, } // Everything from here is generic and reusable return h.processor.ProcessSensorData(sensorData, h.statusSchema) } // Tradfri-specific handler - ONLY handles unmarshalling type TradfriHandler struct { processor *MotionSensorProcessor statusSchema cue.Value // #IkeaTradfriMotionSensorStatus } func (h *TradfriHandler) HandleMQTT(topic string, payload []byte) error { sensorID, floor, err := parseSensorTopic(topic) if err != nil { return fmt.Errorf("failed to parse topic: %w", err) } // THIS IS THE ONLY TRADFRI-SPECIFIC CODE var rawData TradfriRawData if err := json.Unmarshal(payload, &rawData); err != nil { return fmt.Errorf("failed to unmarshal: %w", err) } // Build the generic sensor data structure sensorData := MotionSensor{ SensorID: sensorID, Timestamp: time.Now().Format(time.RFC3339), Floor: floor, SensorType: "ikea_tradfri_motion", RawData: rawData, } // Same generic processor works for Tradfri too return h.processor.ProcessSensorData(sensorData, h.statusSchema) }</code></pre> Each handler's sensor-specific code is about five lines: just unmarshalling JSON into the correct struct type. Everything else is generic processing. The same ProcessSensorData</code> function validates, extracts motion events, extracts ambient light readings, and executes business logic without knowing which specific sensor it's processing. The extraction schemas in CUE contain all the conditional logic. The Go code just encodes, unifies, and decodes. No type switches. No conditional chains. What you gain</a></h2> When an MQTT message arrives, validation against the CUE schema happens at the boundary. Invalid data gets rejected with specific error messages about what's wrong rather than causing runtime panics deep in the application. The schema files document the exact structure and valid ranges for each sensor. Adding a new sensor type means defining one schema and making the necessary changes to the extraction schemas for motion and ambient light. The existing generic processing code continues to work. That said, CUE validates structure but tracking application state (like whether a room has been unoccupied for 10 minutes) still requires Go code. CUE tells you what the current state is, not what to do with it. Time-series analysis, rate limiting, and state machines live in your application layer. There's a learning curve if you're coming from Protobuf, JSON Schema or Go structs. CUE's syntax looks familiar but behaves differently, especially around definitions versus concrete values. The documentation helps, but expect some initial friction. Why CUE instead of alternatives</a></h2> JSON Schema can validate incoming data but can't transform it. You'd still need Go type switches to extract normalised structures from different sensor types. Protocol Buffers require a compilation step and don't handle optional fields as elegantly. The no_occupancy_since</code> field that's sometimes present, sometimes null, and sometimes absent would need Protobuf's wrapper types (like google.protobuf.Int32Value</code>) or custom handling logic. CUE does both validation and transformation in one tool. The same schemas that validate incoming MQTT messages also define the extraction logic that normalises the data. You maintain one set of schemas that handles both concerns. What's next</a></h2> I built this system with CUE from the start, and the flexibility it provides compared to a pure Go solution has worked well for this use case. The schemas sit at the boundary where MQTT messages enter the system. Every sensor type gets validated, and the rest of the codebase works with clean, normalised structures. The system currently handles four motion sensors across four floors, controlling lights on two floors. Adding new sensor types means defining a schema and updating the extraction logic. When I add Philips Hue sensors, I'll extend the #MotionSensor</code> union type and make the necessary changes to the extraction schemas for motion and ambient light detection. The generic processing code won't need to change. This article covered inbound sensor data transformation. But in the coming weeks, I'll be writing more about this system, starting with how CUE handles the outbound side: generating device-specific MQTT commands for different bulb types. There's a lot more to explore around building automation rules, managing state, and handling edge cases in a distributed sensor network. Thanks</a></h2> Thanks to Paul, Roger and Daniel from the CUE</a> team for their help and support with CUE and this article. Building an MQTT client in Go Unknown — Tue, 30 Sep 2025 00:00:00 +0000 MQTT (Message Queuing Telemetry Transport) connects millions of IoT devices worldwide, from smart home sensors to industrial monitoring systems. Whilst many Go developers reach for existing libraries, building your own client wrapper teaches you the protocol's nuances and gives you precise control over connection handling, message routing, and error recovery. This article walks through building a production-ready MQTT client wrapper using the Eclipse Paho Go library</a>. You'll see how to handle concurrent message processing, implement topic wildcard matching, and manage connection lifecycle properly. The anatomy of MQTT communication</a></h2> MQTT operates as a publish-subscribe protocol where clients connect to a central broker. Publishers send messages to named topics, subscribers register interest in topic patterns, and the broker routes messages accordingly. The protocol's power lies in its topic hierarchy and wildcard system. Topics use forward slashes as separators (sensors/temperature/living-room</code>), whilst wildcards enable flexible subscriptions: +</code> matches exactly one level (sensors/+/living-room</code> catches all sensor types in the living room), and #</code> matches zero or more levels (sensors/#</code> catches everything under sensors). Quality of Service (QoS) levels control delivery guarantees: QoS 0 sends messages once with no confirmation, QoS 1 ensures at-least-once delivery through acknowledgements, and QoS 2 provides exactly-once delivery via a four-step handshake. Design decisions and architecture</a></h2> The client wrapper needs to solve several problems that basic MQTT libraries don't address well: concurrent message handling, automatic reconnection with subscription restoration, topic-based message routing, and graceful shutdown coordination. The core design uses a connection manager from the autopaho library for automatic reconnection, a handler registry that maps topic patterns to functions, and a wait group to coordinate graceful shutdown of message processors. type Client struct { config ClientConfig conn *autopaho.ConnectionManager handlers map[string]MessageHandler wg *conc.WaitGroup mu sync.Mutex handlersMu sync.RWMutex isConnected bool }</code></pre> Two separate mutexes prevent deadlocks: mu</code> protects connection state changes, whilst handlersMu</code> guards the handler registry during concurrent read operations when messages arrive. The MessageHandler</code> type accepts a context and payload, enabling handlers to access topic information and implement timeouts: type MessageHandler func(ctx context.Context, payload []byte) error</code></pre>Implementing connection management</a></h2> Connection handling requires balancing automatic reconnection with application control. The autopaho library provides connection management, but you need to coordinate this with your handler system. Local brokers like Mosquitto running on your Home Assistant instance offer significant advantages: sub-millisecond latency, no internet dependency, and predictable connection behaviour. func (c *Client) Connect(ctx context.Context) error { c.mu.Lock() defer c.mu.Unlock() if c.isConnected { return nil } if err := c.conn.AwaitConnection(ctx); err != nil { return fmt.Errorf("connecting to MQTT broker: %w", err) } c.isConnected = true return nil }</code></pre> The AwaitConnection</code> method blocks until the connection establishes or the context cancels. This approach lets callers implement their own timeout strategies whilst ensuring the client tracks connection state accurately. With a local Mosquitto broker, connections typically establish within milliseconds rather than the seconds required for internet-based brokers. Authentication configuration happens during client creation. For local setups, you might skip authentication entirely or use simple username/password pairs: // For local broker without authentication brokerURL, _ := url.Parse("tcp://homeassistant.local:1883") // For local broker with basic authentication brokerURL, _ := url.Parse("tcp://username:password@homeassistant.local:1883")</code></pre>Topic wildcard matching implementation</a></h2> MQTT's wildcard system requires careful implementation to match the specification exactly. The topicMatches</code> function handles three cases: exact matches, single-level wildcards (+</code>), and multi-level wildcards (#</code>). func topicMatches(pattern, topic string) bool { if pattern == topic { return true } if strings.HasSuffix(pattern, "/#") { prefix := pattern[:len(pattern)-2] return topic == prefix || strings.HasPrefix(topic, prefix+"/") } patternParts := strings.Split(pattern, "/") topicParts := strings.Split(topic, "/") // Implementation continues... }</code></pre> The function splits topics into segments and compares them level by level. Special handling for /#</code> patterns ensures they match both the exact prefix and longer paths. Testing wildcard matching requires comprehensive examples: sensors/+/temperature</code> matches sensors/bedroom/temperature</code> but not sensors/bedroom/living-room/temperature</code></li> sensors/#</code> matches sensors</code>, sensors/bedroom</code>, and sensors/bedroom/temperature</code></li> +/temperature</code> matches bedroom/temperature</code> but not sensors/bedroom/temperature</code></li> </ul> Concurrent message processing</a></h2> Message handlers run concurrently to prevent slow handlers from blocking others. Each incoming message spawns a goroutine that processes handlers matching the topic pattern. func (c *Client) handleMessage(publish *paho.Publish) { topic := publish.Topic handlers := c.findMatchingHandlers(topic) for _, handler := range handlers { currentHandler := handler c.wg.Go(func() { ctx := WithTopic(context.Background(), topic) err := currentHandler(ctx, publish.Payload) if err != nil { // Handle error appropriately } }) } }</code></pre> The currentHandler</code> variable prevents closure capture issues where all goroutines might execute the same handler. The wait group tracks active handlers for graceful shutdown. Context enhancement provides handlers with topic information without requiring additional parameters: func WithTopic(ctx context.Context, topic string) context.Context { return context.WithValue(ctx, topicKey, topic) } func GetTopic(ctx context.Context) string { v, ok := ctx.Value(topicKey).(string) if !ok { return "" } return v }</code></pre>Graceful shutdown and cleanup</a></h2> Proper shutdown coordination ensures no messages are lost during application termination. The disconnect process waits for active handlers to complete before closing the connection. func (c *Client) Disconnect(ctx context.Context) error { ctx, cancel := context.WithTimeout(ctx, 20*time.Second) defer cancel() waitDone := make(chan struct{}) go func() { c.wg.Wait() close(waitDone) }() select { case <-waitDone: // Handlers completed case <-ctx.Done(): return errors.New("timeout waiting for handlers") } return c.conn.Disconnect(ctx) }</code></pre> The timeout prevents indefinite blocking if handlers don't complete promptly. Applications can choose appropriate timeout values based on their handler behaviour. Practical usage patterns</a></h2> Real applications typically use multiple handlers for different message types. A home automation system might separate temperature, humidity, and motion sensor handlers: client.AddHandler("sensors/+/temperature", handleTemperature) client.AddHandler("sensors/+/humidity", handleHumidity) client.AddHandler("motion/+", handleMotion) client.AddHandler("alerts/#", handleAlerts)</code></pre> Handler registration automatically subscribes to topics when the client is connected. This eliminates the common pattern of manual subscription after connection establishment. Error handling in handlers should be defensive. Network issues, malformed payloads, or database connection problems shouldn't crash the entire client: func handleTemperature(ctx context.Context, payload []byte) error { topic := GetTopic(ctx) var reading TemperatureReading if err := json.Unmarshal(payload, &reading); err != nil { log.Printf("Invalid temperature data from %s: %v", topic, err) return nil // Don't retry malformed data } if err := database.Store(reading); err != nil { log.Printf("Failed to store temperature reading: %v", err) return err // Retry database errors } return nil }</code></pre>Testing strategies</a></h2> Testing MQTT clients requires either a test broker or careful mocking. The Eclipse Mosquitto project provides a public test broker at test.mosquitto.org:1883</code> for development and testing. Integration tests should verify handler registration, message routing, and connection recovery: func TestHandlerRouting(t *testing.T) { client, err := NewClient(ctx, testConfig) require.NoError(t, err) var received []string client.AddHandler("test/+", func(ctx context.Context, payload []byte) error { received = append(received, string(payload)) return nil }) client.Publish(ctx, "test/sensor1", []byte("data1")) client.Publish(ctx, "test/sensor2", []byte("data2")) // Verify received contains expected data }</code></pre> Unit tests for topic matching don't require a broker connection and can run quickly: func TestTopicMatching(t *testing.T) { cases := []struct { pattern, topic string expected bool }{ {"sensors/+/temperature", "sensors/bedroom/temperature", true}, {"sensors/+/temperature", "sensors/bedroom/humidity", false}, {"sensors/#", "sensors/bedroom/temperature", true}, } for _, c := range cases { result := topicMatches(c.pattern, c.topic) assert.Equal(t, c.expected, result) } }</code></pre>Performance considerations and optimisations</a></h2> Message throughput depends on handler performance and goroutine overhead. For high-volume scenarios, consider message batching or worker pools instead of spawning unlimited goroutines. The handler registry uses a simple map with read-write mutex protection. Applications with thousands of topic patterns might benefit from more sophisticated data structures like prefix trees or regular expression compilation. Connection configuration affects performance and reliability. Shorter keepalive intervals detect network issues faster but increase bandwidth usage. Longer retry delays reduce broker load during outages but slow recovery. Security and authentication</a></h2> Production deployments should use TLS encryption and authentication. MQTT supports username/password authentication, client certificates, and OAuth tokens depending on broker configuration. URL-based authentication configuration keeps credentials in one place: // For username/password authentication brokerURL, _ := url.Parse("tcp://username:password@broker.example.com:1883") // For TLS without authentication brokerURL, _ := url.Parse("tls://broker.example.com:8883")</code></pre> Client certificates require additional configuration in the autopaho client setup. OAuth token handling needs custom authentication callbacks. Extending the client</a></h2> The basic client provides a foundation for more sophisticated features. Message persistence can store and replay messages during connection outages. Message transformation can validate, filter, or modify payloads before handler execution. Rate limiting prevents handlers from overwhelming downstream systems during message bursts. Circuit breakers can disable handlers that repeatedly fail, preventing cascading failures. Metrics collection helps monitor client health. Track connection uptime, message rates, handler execution times, and error frequencies to understand system behaviour. Common pitfalls and solutions</a></h2> Handler goroutines without proper error handling can cause memory leaks through panic recovery. Always handle errors gracefully and avoid panics in handler code. I learned this the hard way when a malformed JSON payload from a failing Xiaomi sensor crashed my entire monitoring service. Connection state races occur when multiple goroutines check IsConnected()</code> simultaneously. The client's internal locking prevents these issues, but application code should handle connection errors appropriately. During Home Assistant restarts, expect temporary connection failures and design handlers to gracefully handle them. Topic subscription ordering matters for wildcard patterns. More specific patterns should be registered before general ones to ensure proper message routing precedence. In my setup, device-specific handlers like energy/evcharger/power</code> register before the catch-all energy/#</code> pattern. Reconnection loops can overwhelm brokers if retry delays are too short. The autopaho library includes exponential backoff, but applications should monitor connection attempt frequency. Home Assistant's built-in Mosquitto addon handles reconnections well, but external brokers might have stricter connection limits. Battery-powered Aqara devices introduce their own challenges. These sensors often send irregular updates or enter sleep modes that affect message timing. Design your handlers to accommodate missing data and use reasonable timeouts when waiting for sensor responses. Integrating with Home Assistant architecture</a></h2> My MQTT client sits alongside Home Assistant rather than replacing it, handling custom analytics and backup monitoring that complement the main automation system. Home Assistant manages device discovery, user interfaces, and real-time automation, whilst my Go services handle historical analysis, custom alerting, and data export. This separation works well because each system excels at different tasks. Home Assistant's visual interface makes it perfect for configuring automation rules and monitoring device status. Go services excel at data processing, custom calculations, and integration with external systems like energy supplier APIs. The MQTT broker becomes the coordination layer between systems. Home Assistant publishes state changes and device readings, my Go services process this data and publish derived metrics, and both systems can subscribe to each other's topics for coordination. Within my home lab I run Home Assistant in a VM on a thin pc and my Go services run in separate VMs. If a system needs maintenance, the others can continue operating independently, with MQTT message retention ensuring no data loss during brief outages. Conclusion</a></h2> Building an MQTT client wrapper teaches you the protocol's subtleties whilst providing exactly the functionality your application needs. The combination of automatic reconnection, concurrent message processing, and graceful shutdown handling creates a robust foundation for IoT and messaging applications. The complete implementation demonstrates Go's strengths for network programming: excellent concurrency primitives, clear error handling, and strong typing that prevents common messaging bugs. Start with the basic client and extend it based on your specific requirements. Monitor its behaviour in production and adjust timeouts, retry logic, and error handling based on real usage patterns. Go 1.25's new JSON encoding package Unknown — Thu, 11 Sep 2025 00:00:00 +0000 ⚠️ Important: The JSON v2 packages are experimental and their APIs may change in future releases. Use GOEXPERIMENT=jsonv2</code> to enable these features for testing. Go 1.25 arrived on 12th August 2025. This release included no language changes compared to recent versions. The most notable addition however was the inroduction of an experimental JSON implementation that improves encoding and decoding performance whilst offering more flexible JSON data handling. Why JSON v2 matters</a></h2> JSON processing creates significant overhead in many Go applications. The new experimental package addresses this directly. You can enable it with GOEXPERIMENT=jsonv2</code> at build time. The experimental JSON v2 package addresses several long-standing issues with the current encoding/json</code> implementation. These include performance bottlenecks in both marshalling and unmarshalling operations, inconsistent behaviour around edge cases like duplicate keys and case sensitivity, limited flexibility in controlling JSON processing behaviour, and memory allocation patterns that impact garbage collection performance. Enabling the experimental JSON v2</a></h2> Go 1.25 includes the new experimental JSON implementation, which you enable by setting the environment variable GOEXPERIMENT=jsonv2</code> at build time. # Enable the experimental JSON v2 implementation GOEXPERIMENT=jsonv2 go build your-app.go # Or set it for your entire build process export GOEXPERIMENT=jsonv2 go build your-app.go</code></pre> Alternatively, you can use the build tag: # Using build tag instead of environment variable go build -tags=goexperiment.jsonv2 your-app.go</code></pre> When enabled, this experiment has two major effects. The existing encoding/json</code> package uses the new JSON implementation under the hood whilst maintaining full API compatibility. New packages become available: encoding/json/v2</code> and encoding/json/jsontext</code> for more advanced use cases. Architecture: separation of concerns</a></h2> encoding/json/jsontext</code> provides low-level, high-performance JSON tokenizers (Decoder) and encoders (Encoder). This package focuses solely on the syntactic aspects of JSON. encoding/json/v2</code> handles the semantic conversion between Go types and JSON values, built upon jsontext. This separation of syntactic and semantic analysis improves code clarity, enables better performance through specialised low-level operations, and provides greater flexibility for custom JSON processing needs. Internal modernisation of JSON v1</a></h2> A crucial aspect of the JSON v2 experiment is that the existing encoding/json</code> package is internally modernised to use the new v2 infrastructure under the hood. This means when you enable GOEXPERIMENT=jsonv2</code>, all your existing JSON code benefits from the new technology whilst maintaining full API compatibility. This internal modernisation approach provides several benefits: You can incrementally adopt v2 features whilst keeping existing code unchanged. Options allow you to configure behaviour anywhere from entirely v1-compatible to entirely v2-style, enabling smooth transitions. This means you're not forced into an all-or-nothing migration approach. As new features are added to v2, they automatically become available in v1. For example, v2's new struct tag options like inline</code> and format</code>, plus the more performant MarshalJSONTo</code> and UnmarshalJSONFrom</code> interface methods, become available to existing v1 code without any changes required on your part. Having a single underlying implementation also reduces the maintenance burden significantly. Bug fixes, performance improvements, and new functionality benefit both API versions simultaneously, so you get the benefits regardless of which API you're using. This approach ensures you can test the new implementation with your existing codebase simply by enabling the experiment, without changing any code. Performance improvements: the numbers</a></h2> JSONv2 delivers significant speed improvements across different operations. The official blog reports unmarshal performance improvements of up to 10x, with marshal performance roughly at parity. For detailed benchmarks, the go-json-experiment/jsonbench</a> repository provides comprehensive analysis showing performance gains that vary by operation type: Unmarshalling Performance: Concrete types</a>: 2.7x to 10.2x faster than JSONv1</li> Interface types</a> (any</code>, map[string]any</code>, []any</code>): 2.3x to 5.7x faster than JSONv1</li> Raw JSON values</a>: 10.2x to 21.1x faster than JSONv1</li> </ul> Marshalling Performance: Interface types</a>: 1.6x to 3.6x faster than JSONv1</li> Raw JSON values</a>: 5.6x to 12.0x faster than JSONv1</li> Concrete types</a>: 1.4x faster to 1.2x slower than JSONv1</li> </ul> Note: These specific benchmark figures are from the external jsonbench repository. Individual results may vary based on your specific use cases and data structures. The performance improvements stem from the architectural separation between syntax parsing and semantic marshalling. The low-level jsontext</code> package handles tokenisation efficiently, whilst the higher-level package focuses purely on type conversion. This eliminates redundant work that occurred in the monolithic v1 implementation. Memory allocation patterns also improve dramatically. The new implementation reduces heap allocations during unmarshalling, which decreases garbage collection frequency. This leads to more predictable performance characteristics across different workload types. Enhanced streaming performance</a></h2> You can achieve additional performance benefits by switching from regular MarshalJSON and UnmarshalJSON to their streaming alternatives (MarshalJSONTo and UnmarshalJSONFrom). The Go team reports this converts certain O(n²) runtime scenarios into O(n). Switching from UnmarshalJSON to UnmarshalJSONFrom in the k8s OpenAPI spec improved performance by orders of magnitude. The streaming API works by processing JSON data directly through encoders and decoders rather than creating intermediate byte slices. This eliminates copying overhead and reduces memory pressure for large documents. Custom types can implement streaming methods to take advantage of this approach. Behavioural changes and new options</a></h2> JSON v2 introduces several important behavioural changes that improve consistency and security. Case sensitivity</a></h3> JSONv1 matches struct fields case-insensitively when unmarshalling. A JSON field called "firstname" would successfully map to a Go struct field named "FirstName". JSONv2 changes this behaviour to case-sensitive matching by default, requiring exact case matches between JSON keys and struct field names or json tags. This change prevents potential security issues where different case variations might be processed unexpectedly. You can restore the original case-insensitive behaviour using the MatchCaseInsensitiveNames(true)</code> option if needed for backward compatibility. Duplicate key handling</a></h3> JSONv1 silently allows duplicate names in JSON objects, keeping the value from the last occurrence. This behaviour can create security vulnerabilities where malicious JSON might include duplicate keys to override expected values. JSONv2 rejects duplicate object names with an error by default. This stricter parsing helps prevent security issues but may break code that previously relied on duplicate key handling. The AllowDuplicateNames(true)</code> option restores the original permissive behaviour when necessary. Nil slice and map handling</a></h3> The marshalling behaviour for nil slices and maps changes between versions. JSONv1 marshals nil slices and maps as JSON null</code> values. JSONv2 marshals nil slices as empty arrays ([]</code>) and nil maps as empty objects ({}</code>). This change creates more consistent JSON output where the structure remains predictable regardless of whether collections are nil or empty. You can restore the original null-marshalling behaviour using FormatNilSliceAsNull(true)</code> and FormatNilMapAsNull(true)</code> options. Flexible options system</a></h2> JSON v2 provides a comprehensive options system for controlling marshalling and unmarshalling behaviour. The options system allows fine-grained control over parsing strictness, formatting preferences, and compatibility modes. You can combine multiple options using JoinOptions</code> to create custom configurations for different use cases. Available options include multiline formatting for pretty-printed output, legacy semantics compatibility for smooth transitions from v1, HTML escaping control, and various parsing strictness settings. This flexibility ensures you can configure JSONv2 to match your specific requirements whilst taking advantage of the performance improvements. Caller-specified customisation</a></h2> Beyond the options system, JSON v2 introduces a powerful feature allowing callers to specify custom JSON representations for any type, where caller-specified functions take precedence over type-defined methods or default behaviour. package main import ( "encoding/json/v2" "encoding/json/jsontext" "fmt" "time" ) func main() { // Custom marshaler for time.Time that formats as Unix timestamp marshalers := json.MarshalFunc(func(t time.Time) ([]byte, error) { return []byte(fmt.Sprintf("%d", t.Unix())), nil }) data := struct { Name string `json:"name"` Time time.Time `json:"timestamp"` }{ Name: "example", Time: time.Now(), } // Use custom marshaler result, err := json.Marshal(data, json.WithMarshalers(marshalers)) if err != nil { panic(err) } fmt.Println(string(result)) // Output: {"name":"example","timestamp":1725897600} }</code></pre> This feature is particularly powerful when you need to define JSON behaviour for types you don't control, such as protocol buffer messages or external library types. It's also useful when different parts of your application need different JSON representations for the same types based on context. You can even override the default JSON representation of any type, including built-in types, without modifying the type definition itself. The streaming variants (MarshalToFunc</code> and UnmarshalFromFunc</code>) provide even better performance by working directly with encoders and decoders. In practice, this becomes especially valuable when integrating with third-party APIs that expect specific JSON formats for common types like timestamps or UUIDs. Real-world impact</a></h2> For applications processing significant amounts of JSON data, these improvements deliver measurable benefits. Reduced latency in API responses becomes particularly noticeable under load. Lower memory pressure from reduced allocations means better resource utilisation in constrained environments. Improved security through stricter parsing defaults helps prevent issues with malformed or malicious JSON inputs. Looking forward</a></h2> JSON v2 is expected to potentially become the default in Go 1.26, though this depends on feedback from the current experiment. As the official blog states, the outcome may result in "anything from abandonment of the effort, to adoption as stable packages of Go 1.26." This timeline provides opportunity to test and provide feedback whilst the API continues to evolve. The design of encoding/json/v2</code> will continue evolving based on community feedback. Developers should try the new API and provide input on the proposal issue. Conclusion</a></h2> Go 1.25's experimental JSON v2 package delivers substantial performance improvements alongside enhanced security through stricter defaults. The architectural separation between syntactic parsing and semantic marshalling provides a solid foundation for future enhancements. For developers working with APIs and data-intensive applications, testing with GOEXPERIMENT=jsonv2</code> provides a straightforward way to evaluate these improvements. As the experiment progresses towards potential adoption in Go 1.26, now is the time to explore these capabilities and provide feedback that will shape the future of JSON processing in Go. Go 1.24's new tool directive in go.mod Unknown — Fri, 31 Jan 2025 00:00:00 +0000 Go 1.24 introduces a fantastic new feature with the tool directive, simplifying how you manage project dependencies by integrating tools directly into your go.mod</code> file. In distributed systems with numerous microservices, teams often prioritise consistency over progression, leading to tool version stagnation. The challenge of coordinating tool updates across a large estate of services frequently results in teams sticking to older, familiar versions rather than adopting beneficial new features. The tool directive addresses this by making tool version management explicit and trackable within each project's go.mod</code>. This visibility helps teams make informed decisions about updates and reduces the friction of rolling out new tool versions across services. Understanding the Tool Directive</a></h2> Prior to Go 1.24, developers faced challenges in ensuring all team members used consistent tool versions. The new tool directive simplifies this process by encapsulating tool requirements within your module's configuration. Example with a typical go.mod</code> (Go 1.23): module github.com/aranw/go-124-new-tool-directive go 1.23</code></pre> To incorporate the github.com/dmarkham/enumer</a> tool, you'd previously run: go install github.com/dmarkham/enumer@latest</code></pre> With Go 1.24, add the tool to your go.mod</code> using: go get -tool github.com/dmarkham/enumer@latest</code></pre> Updated go.mod</code>: module github.com/aranw/go-124-new-tool-directive go 1.24 tool github.com/dmarkham/enumer require ( github.com/dmarkham/enumer v1.5.10 // indirect ... )</code></pre> Now we have the enumer</code> tool specified in our go.mod</code> file. It is available for us to use in our project via the Go toolchain without needing separate installation on our system. This is a great improvement for teams working on projects that require specific tools to be installed. It ensures that everyone is using the same version of the tools and reduces the risk of version drift. Version drift is a common issue with distributed systems. So much so that teams will often stick with older versions of tools to avoid the hassle of updating across multiple services. The tool directive in Go 1.24 simplifies this process by making tool version management explicit and trackable within each project's go.mod</code>. To call the enumer</code> tool in your project, you can use the following command: go tool enumer -type=Direction -json</code></pre>Utilizing the Tool Directive in Code Generation</a></h2> The addition of the tool directive enhances code generation workflows by eliminating manual tool installations. Traditionally, you'd use: //go:generate enumer -type=Direction -json</code></pre> Relying on the tool being installed on your system. Now, we can leverage Go's built-in tool execution with: //go:generate go tool enumer -type=Direction -json</code></pre> This approach executes the specified tool using the one defined in your go.mod</code>, ensuring consistency across your development environment. Under-the-Hood Mechanics</a></h2> When you include a tool via the directive, Go's module system handles its installation. The tool is stored in your project's go.mod</code> file, making it available for execution without separate installations. This mechanism prevents version drift and dependency conflicts, ensuring reliability across team members. Best Practices and Considerations</a></h2> The tool directive helps maintain consistent development environments through centralised tool management in go.mod</code>. This approach prevents version drift whilst eliminating the need for manual tool installation across team members. By leveraging Go's caching system for efficient tool retrieval and execution, your team can focus on development rather than tool management. Conclusion</a></h2> The introduction of the 'tool' directive in Go 1.24 represents a substantial advancement in managing dependencies within Go modules. By incorporating this feature, developers can ensure that dependencies remain current and consistent across different environments, thereby enhancing development efficiency and minimising version discrepancies between team members. In summary, integrating the 'tool' directive is a recommended best practice for modern Go projects aiming to maintain control over their dependencies. Following the guidelines outlined above will enable developers to fully utilise this powerful new feature and ensure their projects remain robust and scalable. Info </div> </div> Checkout my follow up article on how to separate your tool dependencies with a separate go.mod</code> file here</a>. </div> </div> Building an HTTP server in Go with zero dependencies Unknown — Sun, 19 Jan 2025 00:00:00 +0000 Go 1.22 brought enhancements to the net/http</code> package's router. Enabling method matching and wildcard patterns for HTTP routes. The standard library now provides a robust router that can handle most use cases, reducing the need for third-party routing libraries. Zero dependencies or minimal dependencies are a common requirement for many projects. In this post we will explore how we can remove one more dependency from our Go projects by implementing a zero dependency HTTP server. Prior to Go 1.22</a></h2> Before Go 1.22, the standard library's net/http</code> package did not provide a router. Developers often reached for third-party libraries such as gorilla/mux</code> or chi</code> to handle routing in their applications. These libraries provide additional features such as middleware, sub-routers, and more. For example, with chi you can define routes with method verbs mux.Get("/users", handleUsers)</code>. import ( "net/http" "github.com/go-chi/chi/v5" ) func handleUser(w http.ResponseWriter, r *http.Request) { // Handler implementation } func createUser(w http.ResponseWriter, r *http.Request) { // Handler implementation } func main() { r := chi.NewRouter() r.Get("/users/{id}", handleUser) r.Post("/users", createUser) http.ListenAndServe(":8080", r) }</code></pre> Go 1.22 introduced HTTP method verbs in routes for the standard library HTTP package e.g. http.HandleFunc("GET /users", handleUsers)</code>. A key difference between the standard library and third-party libraries is the lack of middleware support. Middleware is a common pattern in web applications to add additional functionality to HTTP handlers. For example, logging, authentication, and error handling. Third party libraries such as gorilla/mux</code> and chi</code> provide middleware support out of the box. Building a Zero Dependency HTTP Server</a></h2> To build our zero dependency HTTP server we will use the standard library's net/http</code> package. We will create a simple HTTP server that listens on port 8080</code> and responds with a Hello, World!</code> message. package main import ( "context" "encoding/json" "log" "net/http" "os" "os/signal" "time" ) func main() { // Create new mux for routing mux := http.NewServeMux() // Add routes mux.HandleFunc("GET /", func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusOK) json.NewEncoder(w).Encode("Hello, World!") }) // Create the HTTP server srv := &http.Server{ Addr: ":8080", Handler: mux, } // Start server in a goroutine to allow for graceful shutdown go func() { log.Printf("server starting on :8080") if err := srv.ListenAndServe(); err != http.ErrServerClosed { log.Fatalf("server error: %v", err) } }() // Wait for interrupt signal to gracefully shut down the server quit := make(chan os.Signal, 1) signal.Notify(quit, os.Interrupt) <-quit // Create a deadline for shutdown ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) defer cancel() if err := srv.Shutdown(ctx); err != nil { log.Fatal("server forced to shutdown:", err) } }</code></pre> In the above code snippet we have defined a simple HTTP server that listens on port 8080</code>. So far it looks very similar to our previous third party solution using Chi. The key difference is that we are now using the standard library's http.ServeMux</code> to handle routing. The server implementation includes graceful shutdown handling, which is important for production services. When the server receives an interrupt signal (Ctrl+C), it: Stops accepting new requests</li> Allows in-flight requests to complete</li> Times out after 10 seconds if requests haven't completed</li> </ol> This ensures that requests are handled properly even when the server is shutting down, rather than abruptly terminating connections. Adding Routes</a></h3> To add routes to our server we can use the http.ServeMux</code>'s HandleFunc</code> method. For example, to add a route that responds with a Hello, World!</code> message: mux.HandleFunc("GET /", func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusOK) json.NewEncoder(w).Encode("Hello, World!") })</code></pre> In our example we created a route that responds to GET /</code> requests with a Hello, World!</code> message. The HandleFunc</code> method takes a string that defines the HTTP method and path. The string is split by a space to separate the method and path. The second argument is a function that takes an http.ResponseWriter</code> and http.Request</code>. The router also supports path parameters. For example, we can capture dynamic values from the URL: type User struct { ID string `json:"id"` } mux.HandleFunc("GET /users/{id}", func(w http.ResponseWriter, r *http.Request) { // Extract the id parameter from the URL userID := r.PathValue("id") // Create a user object with the ID from the URL user := User{ ID: userID, } // Set response headers and encode JSON response w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusOK) json.NewEncoder(w).Encode(user) })</code></pre> In the above example, we're using Go 1.22's new path parameter syntax {id}</code> to capture a variable part of the URL. When a request is made to /users/123</code>, the PathValue()</code> method makes it easy to extract the id</code> value without any string manipulation or regular expressions. This is a significant improvement over the old router where you'd need to parse URL segments manually or rely on third-party libraries. The PathValue()</code> method returns an empty string if the parameter isn't found, making it safe to use without additional error checking for parameter existence. This gives us a clean way to handle dynamic routes while keeping our handler code simple and readable. For more complex routing patterns, you can use wildcard patterns to match multiple segments in a URL. For example, to match all requests to /static/...</code>: // Example wildcard pattern mux.HandleFunc("GET /static/...", func(w http.ResponseWriter, r *http.Request) { // Handle static files })</code></pre> The ...</code> wildcard at the end of a pattern matches the remainder of the URL path, allowing you to create flexible routing patterns. For example, /static/...</code> will match any path under /static/</code>, making it perfect for serving static files or handling nested routes. Wildcard patterns can only appear at the end of a pattern. Middleware</a></h3> Middleware is a common pattern in web applications to add functionality to HTTP handlers. Common use cases for middleware include logging, authentication, and error handling among others. In the absence of a third-party router, we will need to implement our own set of helper functions to provide middleware functionality. // Middleware is a function that wraps an http.HandlerFunc type Middleware func(http.HandlerFunc) http.HandlerFunc // WithLogging logs the request method and path along with the time taken func WithLogging(next http.HandlerFunc) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { start := time.Now() next(w, r) log.Printf("%s %s took %v", r.Method, r.URL.Path, time.Since(start)) } } // WithRecovery catches panics in request handling func WithRecovery(next http.HandlerFunc) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { defer func() { if err := recover(); err != nil { log.Printf("panic: %v", err) http.Error(w, "Internal Server Error", http.StatusInternalServerError) } }() next(w, r) } }</code></pre> In this snippet we define a Middleware</code> type that wraps an http.HandlerFunc</code>. We have also defined two common middlewares WithLogging</code> and WithRecovery</code>. WithLogging</code> logs the request method and path along with the time taken to process the request. WithRecovery</code> catches panics in the request handling and returns an Internal Server Error</code> response. Using the middleware is simple, we just need to wrap our handler functions with the middleware we want to apply: mux.HandleFunc("GET /", WithLogging(WithRecovery(func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusOK) json.NewEncoder(w).Encode("Hello, World!") })))</code></pre> This demonstrates how you can achieve the common middleware pattern without having to depend on external third-party router libraries. The middleware functions are simple to test and compose, allowing us to add cross-cutting concerns to our handlers while keeping the core request handling logic clean. A common helper function for chaining middleware is: func Chain(h http.HandlerFunc, m ...Middleware) http.HandlerFunc { for i := len(m) - 1; i >= 0; i-- { h = m[i](h) } return h }</code></pre> This function takes a handler function and a list of middleware functions and chains them together. This allows you to apply multiple middleware functions to a single handler function. // Usage example mux.HandleFunc("GET /users/{id}", Chain( handleUser, WithLogging, WithRecovery, ))</code></pre> When defining a chain of middleware functions, the order in which they are applied matters. Middleware functions are applied in the order they are passed to the Chain</code> function. For example, in our usage example above, logging will be performed before recovery. Conclusion</a></h2> Go 1.22's enhanced HTTP router brings many features we previously relied on third-party libraries for directly into the standard library. With method-based routing, path parameters, and wildcard support, we can now build robust HTTP servers with zero external dependencies. While libraries like Chi and Gorilla Mux offer additional features, the standard library's router is now capable of handling many common use cases. This means: Fewer dependencies to manage</li> No version compatibility concerns</li> Smaller binary sizes</li> Code that's more likely to remain stable over time</li> </ul> For simple to medium complexity HTTP servers, the standard library solution demonstrated in this post might be all you need. However, if your application requires more advanced routing features like subrouters, regular expression matching, or extensive middleware chains, then a third-party router might still be the better choice. Consider starting with the standard library approach and only reaching for external dependencies when you have specific requirements that aren't met by net/http</code>. Processing UK rail data in real-time Unknown — Fri, 10 Jan 2025 00:00:00 +0000 Railway systems generate an endless stream of real-time data that needs to be captured, processed, and stored efficiently. When I was approached for a freelance project to build a Kafka consumer in Go using franz-go and integrate it with PostgreSQL, I knew this would be an interesting challenge. The result was a service designed to reliably consume real-time train information and persist it into a database. What started as a straightforward Kafka consumer evolved into a robust system, capable of handling millions of daily messages about train movements, schedules, and disruptions across the rail network. This project demonstrates modern Go service architecture, robust data processing patterns, and integration with industry-standard messaging and storage systems. Working with real-time railway data presents unique challenges. Trains don't always run according to schedule, and tracking these deviations requires a system that's both resilient and highly available. The solution leverages Go's concurrency model to efficiently process messages whilst maintaining data consistency. Through careful consideration of failure scenarios, I've built a system that consumers of this data can rely on to receive accurate, timely information about train movements and delays for integration into their own applications. In this post, I'll walk through some of the key architectural decisions, challenges faced, and lessons learned while building this system. Whether you're working with Kafka, building data processing pipelines, or interested in Go-based microservices, you'll find practical insights from my experience developing this project. Project Overview</a></h2> The service serves as a critical component in a rail information system, bridging the gap between real-time data streams and persistent storage. Its primary responsibilities are: Consuming train data from Darwin JSON Kafka Topic</li> Processing and validating incoming messages</li> Storing processed data in a PostgreSQL database using table partitioning</li> Providing health checks and monitoring endpoints</li> </ol> System Architecture</a></h2> Message Processing Pipeline</a></h3> The system processes railway data through a series of well-defined stages, starting with Kafka consumption and ending with persistent storage. At its core, it uses the franz-go</a> library for reliable Kafka interaction. The consumer supports SASL authentication and configurable consumer groups for secure, scalable message processing. Message flow is managed through Go channels, providing natural backpressure and buffering. Messages accumulate in the channel before being passed to the database layer. While the system currently processes messages individually, it's designed to support batch processing if needed. This approach keeps the implementation simple while leaving room for optimization. func (c *Consumer) processMessages(ctx context.Context, msg kafka.Message) error { select { case c.messageChan <- msg: return nil case <-ctx.Done(): return ctx.Err() } }</code></pre> Error handling and shutdown procedures ensure system reliability. When errors occur, detailed logs capture message context and error metadata. During shutdown, the system: Stops consuming new messages</li> Processes remaining buffered messages</li> Commits Kafka offsets</li> Closes database connections gracefully</li> </ul> Database Management</a></h3> PostgreSQL serves as the persistent storage layer, using table partitioning for efficient data management. The base table structure uses RANGE partitioning on the created_at</code> column: CREATE TABLE IF NOT EXISTS messages ( id UUID, message JSONB NOT NULL, created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (id, created_at) ) PARTITION BY RANGE (created_at);</code></pre> For a detailed explanation of PostgreSQL table partitioning patterns and benefits, see my article on PostgreSQL Table Partitioning</a>. The service manages partitions through a dedicated partitioner that runs both on startup and periodically: type Partitioner struct { db *sql.DB } func (p *Partitioner) Ensure(ctx context.Context, date time.Time) error { tableName := fmt.Sprintf("messages_%04d_%02d_%02d", date.Year(), date.Month(), date.Day()) startDate := time.Date(date.Year(), date.Month(), date.Day(), 0, 0, 0, 0, time.UTC) endDate := startDate.AddDate(0, 0, 1) query := fmt.Sprintf(` CREATE TABLE IF NOT EXISTS %s PARTITION OF messages FOR VALUES FROM ('%s') TO ('%s')`, tableName, startDate.Format("2006-01-02 15:04:05"), endDate.Format("2006-01-02 15:04:05")) _, err := p.db.ExecContext(ctx, query) return err } func (p *Partitioner) Start(ctx context.Context) { // Create partitions for today and tomorrow on startup now := time.Now() p.Ensure(ctx, now) p.Ensure(ctx, now.AddDate(0, 0, 1)) // Run partition creation every 6 hours ticker := time.NewTicker(6 * time.Hour) go func() { for { select { case <-ticker.C: if err := p.Ensure(ctx, time.Now()); err != nil { log.Printf("failed to ensure partition: %v", err) } case <-ctx.Done(): ticker.Stop() return } } }() }</code></pre> The 6-hour interval provides redundancy in partition creation, preventing gaps if the service experiences downtime. Creating tomorrow's partition on startup ensures message processing continues seamlessly across midnight UTC. Testing Strategy</a></h3> Integration tests form the backbone of the testing approach. Using Docker containers, tests run against real Kafka and PostgreSQL instances: func TestMessageProcessing(t *testing.T) { kafka := test.NewKafkaContainer(t) postgres := test.NewPostgresContainer(t) // Test setup and assertions }</code></pre> Utility packages handle container lifecycle and test data setup, making it straightforward to write comprehensive tests. While unit tests with mocks supplement coverage, integration tests have proven more valuable for catching real-world issues, particularly around: Transaction handling</li> Error scenarios</li> Message ordering</li> Database interactions</li> </ul> Deployment</a></h2> The service runs on a Virtual Private Server (VPS), deployed alongside its PostgreSQL instance. This co-location minimises latency for database operations. Kafka runs as an external service that the system connects to - I have no visibility into its configuration or infrastructure. The VPS runs Fedora Linux, which uses systemd as its init system and service manager. With systemd being a core part of Fedora, there's no need to install additional process managers like Supervisor or PM2. This reduces complexity and potential points of failure. systemd provides process supervision, automatic restarts on failure, and centralised logging through journald. Here's the systemd service configuration: [Unit] Description=Railway Service Systemd ConditionPathExists=/opt/railway/railway-linux-amd64 After=network.target [Service] ExecStart=/opt/railway/railway-linux-amd64 -config=/etc/railway/railway.yaml Restart=always</code></pre> Before moving to production, I needed to validate that the service worked correctly with the provided VPS architecture and infrastructure. To achieve this, the service underwent a thorough 7-day staging validation period. This validation phase was crucial for verifying several critical aspects of the system. The service successfully handled server restarts, maintaining data consistency and resuming message processing automatically. Database writes remained reliable throughout the test period. No manual intervention was needed during this validation phase, demonstrating the robustness of the systemd configuration and overall system design. This gave confidence that the service would operate reliably in the production environment using the provided infrastructure. Conclusion</a></h2> Building reliable data pipelines requires careful attention to detail. This project showcases that through its clean architecture and robust Kafka consumer implementation. The production deployment on Fedora, with each component designed with reliability in mind, keeps the service running smoothly. The testing approach proved particularly valuable. Integration tests with containerised dependencies caught real-world issues early. Error handling and monitoring ensure problems are visible and quickly resolved. systemd's built-in features provide robust process management. What started as a simple Kafka consumer evolved into a production-ready service. It reliably processes railway data day after day, demonstrating how Go's simplicity and powerful concurrency model make it an excellent choice for data processing systems. For more on the technologies used in this project, see my posts on getting started with Go and Kafka using franz-go</a> and PostgreSQL table partitioning with Go</a>. Engineering arrogance Unknown — Mon, 06 Jan 2025 00:00:00 +0000 Innovation and rapid execution drive success in technology companies. When engineering teams collaborate effectively, they iterate quickly and maintain competitive advantages. However, arrogance within engineering teams can silently erode this foundation. Arrogance, ego and passive-aggressive attitudes within teams can stifle creativity, foster hostility, and hinder innovation. What is Engineering Arrogance?</a></h2> Engineering arrogance manifests when technical expertise becomes a barrier for others. It transforms code reviews into battlegrounds, technical decisions into power plays, and learning opportunities into ego threats. Instead of driving innovation, this behavior weaponises expertise for personal validation, prioritising ego over teamwork. The impacts surface in daily engineering work through ego battles during code reviews, inflexible "my way or the highway" mentalities, gatekeeping of knowledge, and undermining of new ideas. While not every team experiences all these behaviors, even a few can poison team dynamics. When arrogance becomes ingrained in engineering culture, it leads to something far worse: technology stagnation. This mindset fundamentally corrupts the purpose of technical expertise. Rather than serving as a foundation for collaboration and growth, engineering arrogance builds barriers where bridges for collaboration should exist, ultimately undermining the very innovation it claims to protect. Ego in Engineering Arrogance</a></h2> Engineering expertise naturally builds confidence. Yet unchecked ego transforms that confidence into an obstacle. While healthy confidence drives engineers to excel and push boundaries, ego left unchecked is dangerous, converting constructive technical discussions into battles for dominance. This transformation often begins subtly. Engineers start equating their technical expertise with their professional worth. Every code review and design discussion feels like a referendum on their competence. When an engineer's identity becomes intrinsically tied to their technical decisions, admitting mistakes transforms from a learning opportunity into a perceived threat, and this defensive posture erodes the collaborative environment needed for innovation. The impact compounds over time in unpredictable ways. Senior engineers might cling to outdated patterns. They dismiss fresh ideas to maintain their perceived authority, turning technical discussions into exercises in defending territory rather than exploring solutions. Engineers may double down on problematic approaches instead of acknowledging mistakes. These choices turn temporary technical compromises into permanent architectural decisions that live with projects for years. These behaviours eventually reshape the systems themselves into rigid structures resistant to change and innovation. Teams spend more time maintaining legacy approaches and accumulating technical debt than evolving their systems to meet future needs. What begins as personal ego turns into institutional stagnation. Resulting in platforms becoming difficult to change as the mindsets that maintain them. Innovation dies quietly. Teams stop brainstorming freely, fearing ridicule or dismissal. Technical discussions lose their vibrancy. What should be dynamic exploration becomes a predictable exercise in confirming existing beliefs. This quiet conformity doesn't just stifle individual creativity—it systematically undermines the collaborative innovation that drives technology forward, leaving teams trapped in cycles of maintaining increasingly outdated systems. Toxic Code Reviews: The Battleground for Ego</a></h2> Code reviews should make code better and help teams grow together. But when arrogance creeps in, they turn into painful ego battles where showing technical superiority matters more than actual improvement. Nitpicking is probably the most common toxic review pattern. A reviewer gets hung up on tiny details like variable names or where the brackets go, completely missing the bigger picture. Sure, coding standards matter - but when someone's correcting every little thing just to show they're in charge, it's not about code quality anymore. It wastes everyone's time and kills motivation to improve things. Then there's the reviewers who can't be bothered to explain themselves. They drop comments like "this is bad" or "needs work" without saying why. It's like they're too important to explain their thinking. When engineers get this kind of useless feedback, they either have to guess what the reviewer wants or go back and forth endlessly. Pretty soon people stop trying to contribute meaningfully. The worst offenders take it public. They'll mock code in team channels with comments like "you clearly don't understand how things work here" or "this is what happens when people don't learn our architecture first.". Instead of helping someone learn, they're just showing off their supposed superiority. Each time this happens, it teaches everyone a lesson: keep your head down, don't try anything new, and definitely don't ask questions that might make you look inexperienced. When code reviews become about ego rather than improvement, the whole system breaks down. Good ideas get buried because people are more worried about avoiding criticism than solving problems. Teams that should be collaborating end up divided into camps of harsh reviewers and defensive contributors. The very process that should make engineering better ends up holding everyone back. Technological Stagnation</a></h2> Engineering arrogance kills innovation in predictable ways. It murders risk-taking and every mistake gets ridiculed in code reviews or design meetings, engineers learn to play it safe. Instead of trying bold solutions, they stick to what won't attract attention. The codebase fills up with minor tweaks and safe changes while big architectural problems go unsolved. This safety-first mentality spreads. Engineers stop speaking up in meetings. They keep their ideas to themselves rather than risk getting shot down. Pretty soon, the loudest voices - usually the most arrogant ones - are the only ones heard. The team ends up doing things "the way we've always done them" not because it's right, but because challenging the status quo isn't worth the hassle. The damage gets worse when senior engineers start hoarding knowledge. They treat their expertise like a treasure to guard rather than something to share. Junior engineers struggle with the same problems over and over because the "experts" are too busy proving how smart they are to actually teach anyone. Each team ends up solving the same problems differently because knowledge is stuck in silos. All of this results in a codebase that looks exactly like the team culture that built it - rigid, fragmented, and resistant to change. New features take forever because everyone's afraid to touch the "critical" parts that only one person understands. Technical debt piles up because fixing it would mean admitting previous approaches were wrong. The platform becomes a museum of past decisions that everyone's afraid to update. This stagnation costs money and time. Teams move slowly not because they're being careful, but because they're paralysed. Innovation dies - buried under layers of "that's not how we do things here" and "you don't understand the context.". What started as engineering arrogance ends as engineering paralysis. Passive-Aggressive Arrogance</a></h2> Not all engineering arrogance is obvious. Sometimes it's hidden in subtle behaviors - the eye rolls during standups, or that teammate who always says "interesting..." in a tone that clearly means "that's stupid." This quieter version of arrogance might seem less harmful than outright criticism, but it's just as toxic to team culture. You'll spot it in a hundred little ways. There's the engineer who sits silently through design discussions, then sends passive-aggressive messages picking apart decisions after everyone's left the room. Or the one who responds to new ideas with "Well, you can try that if you want..." in a tone that suggests you're about to crash and burn. They'll agree to changes in meetings but then never quite find time to review the PR, letting it rot in the queue until everyone forgets about it. The damage runs deep because it's harder to call out. When someone says "I'm just asking questions..." while systematically undermining a technical decision, or responding to every suggestion with "maybe you know something I don't..." they're creating doubt while maintaining plausible deniability. These behaviors shut down innovation just as effectively as direct criticism, but they do it so quietly that many teams don't realise what's happening until their culture is thoroughly poisoned. These passive-aggressive engineers often see themselves as the "voice of reason" or the "technical conscience" of the team. But instead of openly discussing concerns, they express disapproval through sighs in meetings, frowning, pointed questions that aren't really questions, and minimal engagement with anything they didn't propose. When their ideas aren't chosen, they'll technically comply while doing everything possible to prove they were right all along - even if it means quietly undermining team decisions. This behavior spreads. Other engineers start adopting the same tactics because direct communication feels too risky. Soon you've got a team full of people speaking in subtexts and hidden meanings instead of actually solving problems. Technical discussions become exercises in reading between the lines, and the energy that should go into innovation gets spent on navigating personality politics instead. Strategies for Combating Arrogance</a></h2> Fighting engineering arrogance isn't about grand gestures - it's about changing daily habits and interactions. The fix starts with both individual engineers and their leaders making different choices in everyday situations. For engineers, it begins with letting go of code ownership. Your code isn't your baby - it's just one possible solution to a problem. When someone suggests changes, they're not attacking you personally. They're trying to make the solution better. Try responding to feedback with curiosity: "That's an interesting point - could you help me understand your concerns?". Instead of defending your approach, explore alternatives. Remember that the best engineers aren't the ones who write perfect code - they're the ones who help their teams deliver better solutions. If you're leading engineering teams, you've got the power to reshape the culture through your actions. When was the last time you admitted a mistake in front of your team? Or openly changed your mind after hearing a junior engineer's perspective? These moments matter. Show your team it's okay to be wrong sometimes. Celebrate the engineer who caught a serious bug in their own code before it hit production. Highlight the team that scrapped two weeks of work because they found a better approach. Make it clear that learning beats looking smart. The real change happens in small moments. It's the senior engineer taking time to explain a complex system to a new hire. It's the team lead who asks for input instead of dictating solutions. It's the developer who starts their code review with what they learned from reading their colleague's approach. These daily choices might feel small, but they will slowly transform the culture from one of competition to one of collective growth. Building a better engineering culture is not a one-time fix - it is an ongoing practice. The goal is not to eliminate all ego (we're human after all), but to create an environment where learning and improvement matter more than looking smart. When teams get this right, they don't just write better code - they build a place where great engineers want to stay and grow. Conclusion</a></h2> Engineering arrogance isn't born from having the smartest person in the room - it comes from creating environments where everyone can contribute their best work. Collaborative innovation flourishes when arrogance is set aside in favor of humility and curiosity. Teams don't just tolerate diverse opinions, they actively seek them out. The real impact happens in daily choices: how we review code, how we share knowledge, and how we respond to new ideas. These small moments shape whether our teams build bridges or walls between each other. The next time you're about to dismiss an idea because "that's not how we do things here," pause. Ask yourself what you might learn instead. What looks like a mistake might be the seed of your next breakthrough. Building an environment of trust and respect isn't just good for team morale - it drives real technological progress. When engineers feel safe to experiment and contribute, they create solutions that last. Building with the Google Sheets API in Go Unknown — Sun, 05 Jan 2025 00:00:00 +0000 Getting started with the Google Sheets First things first. You'll need Here's a basic configuration structure }< }< }< }< }< A1 notation is the backbone of working Let's make this simpler for us }< }< }< }< }< }</sp }< }< }< }< }< }< }< }< }< }< Consistency is key when creating new }< }</sp }< }</sp }</sp }< }< }</sp }< }< }< }< Scale changes everything. When your }< }< }< }< }</sp }< }< }< APIs can be unpredictable. Network }< } }</sp }< }< }< }</sp }< }< Now that we have a understanding of }< }</sp }< }< }< }< }< When working with the Sheets API at Always batch your operations. The API Put in place proper error handling Be careful with range calculations. Consider caching frequently accessed The Google Sheets API opens up interesting The patterns we've covered here Building Stria to streamline payment Unknown — Fri, 03 Jan 2025 00:00:00 +0000 API can be challenging for Go developers. From understanding A1 notation to managing batch operations. There are several concepts and patterns to understand. This guide walks you through building a robust integration with the Google Sheets API in Go. class="zola-anchor" href="#setting-up-your-environment" aria-label="Anchor link for: setting-up-your-environment">Setting Up Your Environment</a></h2> to set up your development environment by installing the required dependencies: class="giallo-l">go get google.golang.org/api/sheets/v4 go get golang.org/x/oauth2/google</code></pre> we'll use throughout our examples. It's straightforward but flexible enough for most use cases. class="giallo-l">type SheetsConfig struct { CredentialsFile string TokenFile string SpreadsheetID string Range string F;span> func NewSheetsClient(config SheetsConfig) (*sheets.Service, error) { ctx := context.Background() b, err := os.ReadFile(config.CredentialsFile) if err != nil { return nil, fmt.Errorf("reading credentials file: %v", err) /span> config, err := google.ConfigFromJSON(b, sheets.SpreadsheetsScope) if err != nil { return nil, fmt.Errorf("parsing credentials: %v", err) /span> client := getClient(config, config.TokenFile) srv, err := sheets.New(client) if err != nil { return nil, fmt.Errorf("creating sheets client: %v", err) /span> return srv, nil F;span></code></pre>Understanding Sheet Ranges</a></h2> with the Sheets API. A1 notation is the standard way to reference cells in spreadsheets. Letters represent columns (A, B, C...) while numbers represent rows (1, 2, 3...). When combined, they pinpoint specific cells - A1 is the first cell in the top-left corner, B3 is the cell in the second column and third row. To work with multiple cells at once, use a colon to show a range, like A1:B10. The system continues past 26 columns by doubling up letters - after Z comes AA, then AB, and so on. in Go with a helper that turns complex range calculations into intuitive operations. class="giallo-l">// Range represents a spreadsheet range in A1 notation type Range struct { Sheet string StartCol string EndCol string StartRow int EndRow int F;span> z-definition z-comment">// String returns the A1 notation representation of the range. z-definition z-comment">// For example: "Sheet1!A1:B2" or "Sheet1!A1:A" for a single column. func (r Range) String() string { if r.EndCol == "" { r.EndCol = r.StartCol /span> if r.EndRow == 0 { return fmt.Sprintf("%s!%s%d:%s", r.Sheet, r.StartCol, r.StartRow, r.EndCol) /span> return fmt.Sprintf("%s!%s%d:%s%d", r.Sheet, r.StartCol, r.StartRow, r.EndCol, r.EndRow) F;span> z-definition z-comment">// isValidColumn checks if a column name consists of uppercase letters A-Z func isValidColumn(col string) bool { if col == "" { return false /span> for _, c := range col { if c < 'A' || c > 'Z' { return false an> /span> return true F;span> z-definition z-comment">// NewRange creates a Range from a sheet name and column range. z-definition z-comment">// It returns an error if the input parameters are invalid. func NewRange(sheet, startCol, endCol string, startRow, endRow int) (*Range, error) { if sheet == "" { return nil, fmt.Errorf("sheet name cannot be empty") /span> if !isValidColumn(startCol) { return nil, fmt.Errorf("invalid start column: %s", startCol) /span> if endCol != "" && !isValidColumn(endCol) { return nil, fmt.Errorf("invalid end column: %s", endCol) /span> if startRow <= 0 { return nil, fmt.Errorf("start row must be positive, got: %d", startRow) /span> if endRow < 0 { return nil, fmt.Errorf("end row cannot be negative, got: %d", endRow) /span> if endRow != 0 && endRow < startRow { return nil, fmt.Errorf("end row (%d) cannot be less than start row (%d)", endRow, startRow) /span> return &Range{ Sheet: sheet, StartCol: startCol, EndCol: endCol, StartRow: startRow, EndRow: endRow, span>, nil F;span></code></pre>Creating and Configuring Spreadsheets</a></h2> spreadsheets. A well-structured sheet makes future operations smoother. Here's a pattern that works well in practice. class="giallo-l">type SheetConfig struct { Name string Headers []string Frozen bool F;span> func CreateConfiguredSheet(srv *sheets.Service, title string, configs []SheetConfig) (*sheets.Spreadsheet, error) { spreadsheet := &sheets.Spreadsheet{ Properties: &sheets.SpreadsheetProperties{ Title: title, an>, Sheets: make([]*sheets.Sheet, len(configs)), /span> for i, config := range configs { sheet := &sheets.Sheet{ Properties: &sheets.SheetProperties{ Title: config.Name, }, an> if config.Frozen { sheet.Properties.GridProperties = &sheets.GridProperties{ FrozenRowCount: 1, } an> spreadsheet.Sheets[i] = sheet /span> created, err := srv.Spreadsheets.Create(spreadsheet).Do() if err != nil { return nil, fmt.Errorf("failed to create spreadsheet: %v", err) /span> z-definition z-comment"> // Configure headers for each sheet for i, config := range configs { if len(config.Headers) > 0 { range := NewRange(config.Name, "A", columnToLetter(len(config.Headers)), 1) values := [][]interface{}{ make([]interface{}, len(config.Headers)), } for j, header := range config.Headers { values[0][j] = header } valueRange := &sheets.ValueRange{ Range: range.String(), Values: values, } _, err := srv.Spreadsheets.Values.Update(created.SpreadsheetId, range.String(), valueRange). ValueInputOption("RAW").Do() if err != nil { return nil, fmt.Errorf("failed to set headers for sheet %s: %v", config.Name, err) } an> /span> return created, nil F;span> z-definition z-comment">// columnToLetter converts a column number to A1 notation letter func columnToLetter(col int) string { var result string for col > 0 { col-- result = string(rune('A'+col%26)) + result col /= 26 /span> return result F;span></code></pre>Efficient Data Operations</a></h2> application needs to handle larger datasets, batch operations become crucial. They're faster and more reliable. Here's a pattern that's proven effective in production environments. class="giallo-l">type BatchUpdate struct { updates map[string][][]interface{} ranges []string F;span> func NewBatchUpdate() *BatchUpdate { return &BatchUpdate{ updates: make(map[string][][]interface{}), ranges: make([]string), /span> F;span> func (b *BatchUpdate) AddUpdate(range string, values [][]interface{}) { b.updates[range] = values b.ranges = append(b.ranges, range) F;span> func (b *BatchUpdate) Execute(srv *sheets.Service, spreadsheetId string) error { var data []*sheets.ValueRange for _, range := range b.ranges { data = append(data, &sheets.ValueRange{ Range: range, Values: b.updates[range], an>) /span> req := &sheets.BatchUpdateValuesRequest{ ValueInputOption: "RAW", Data: data, /span> _, err := srv.Spreadsheets.Values.BatchUpdate(spreadsheetId, req).Do() return err F;span></code></pre>Error Handling and Retry Logic</a></h2> calls might fail. Rate limits could kick in. A robust retry mechanism makes all the difference. class="giallo-l">type RetryConfig struct { MaxAttempts int InitialDelay time.Duration MaxDelay time.Duration F;span> func WithRetry(config RetryConfig, operation func() error) error { var lastErr error delay := config.InitialDelay for attempt := 0; attempt < config.MaxAttempts; attempt++ { if err := operation(); err == nil { return nil class="z-keyword"> else { lastErr = err if !isRetryable(err) { return err } time.Sleep(delay) delay = min(delay*2, config.MaxDelay) an> /span> return fmt.Errorf("operation failed after %d attempts: %v", config.MaxAttempts, lastErr) F;span> func isRetryable(err error) bool { if err == nil { return false /span> z-definition z-comment"> // Check for specific Google Sheets API errors that are retryable if gerr, ok := err.(*googleapi.Error); ok { switch gerr.Code { case 429, // Too Many Requests 500, // Internal Server Error 503: // Service Unavailable return true an> /span> return false F;span></code></pre>Practical Usage Example</a></h2> ranges, batch operations, and error handling, let's put everything together in a real-world scenario. In this example we will look at managing product inventory. class="giallo-l">type Product struct { ID string Name string Stock int LastUpdated time.Time F;span> func UpdateInventory(srv *sheets.Service, spreadsheetId string, products []Product) error { batch := NewBatchUpdate() for _, product := range products { values := [][]interface{}{ { product.ID, product.Name, product.Stock, product.LastUpdated.Format(time.RFC3339), }, an> range := NewRange("Inventory", "A", "D", findProductRow(srv, spreadsheetId, product.ID)) batch.AddUpdate(range.String(), values) /span> return WithRetry(RetryConfig{ MaxAttempts: 3, InitialDelay: time.Second, MaxDelay: time.Second * 5, span>, func() error { return batch.Execute(srv, spreadsheetId) span>) F;span> func findProductRow(srv *sheets.Service, spreadsheetId, productId string) int { z-definition z-comment"> // Implementation to find the row number for a product z-definition z-comment"> // This would typically involve searching the ID column z-definition z-comment"> // and returning the appropriate row number z-definition z-comment"> // Returns a new row number if product not found F;span></code></pre>Best Practices and Common Pitfalls</a></h2> scale, following these best practices will help you build more reliable applications. has rate limits. Batching helps you stay well within them while improving performance. Our BatchUpdate pattern shows how to do this effectively. and retry logic. Network hiccups happen. The Sheets API occasionally returns 429 or 503 errors. Your application should handle these gracefully. A1 notation can be confusing. That's why we built the Range struct. Always validate your ranges before performing operations. data locally. If you're reading the same ranges repeatedly, a local cache can work wonders. Just remember to implement proper invalidation. href="#conclusion" aria-label="Anchor link for: conclusion">Conclusion</a></h2> possibilities for Go applications. It's powerful but requires careful handling. Start simple. Get the basics right. Then build up to more complex operations. will help you avoid common pitfalls. They're battle-tested and ready for production use. But don't stop here. Explore the API's advanced features. Try out formatting capabilities. Experiment with conditional formatting and chart creation. data integrations I've been working on a new side project. Stria</a>, a service that will make payment data integration straightforward and reliable. As I get closer to launching I wanted to share a bit more about what I've been building and why. The Problem I'm Solving</a></h2> Every time I've needed payment data in the past in analytics tools or reporting systems. I've ended up building a custom solution which has turned into the same headache. What should be a "quick integration" with Stripe's API becomes weeks of: Wrestling with edge cases</li> Building retry logic for API rate limits</li> Managing webhook reliability</li> Maintaining yet another piece of infrastructure</li> </ul> I figured there had to be a better way. What is Stria?</a></h2> Stria will sync your payment data to a desired output. I'm starting with a Stripe integration. But in the future I will expand to other payment providers. Instead of building and maintaining complex custom integrations yourself. Stria will manage the complexity, syncing to databases, Google Sheets or other destinations. Technical Implementation</a></h2> Here's a look under the hood at what I'm building: The backend is being built in Go, chosen for its reliability, performance, and excellent standard library support. This has proven invaluable when working with HTTP services and API integrations. For infrastructure, I'm deploying on Fly.io</a> to ensure scalability as the service grows. In the future I may need to expand on this and utilise GCP's managed services. At the core of Stria is a custom sync engine. This ensures that your data stays consistent even when dealing with network issues or API downtime. From day one, I've built comprehensive observability into the system. This means I can identify and resolve any issues that might arise, ensuring reliable service for users. Initial Feature Set</a></h2> For the initial release, I'm focusing on: Stripe integration</li> Google Sheets integration</li> Scheduled syncing</li> Real-time syncing</li> Reliable data consistency</li> Simple pricing starting at $5/month</li> </ul> What's Coming Next</a></h2> While I'm focused on getting the core functionality rock-solid, I'm already planning: Additional payment provider integrations</li> More destination options</li> Advanced filtering capabilities</li> Custom webhook notifications</li> </ul> Early Access</a></h2> I'm getting close to launch and will be looking for early users soon. If you're interested in: Testing the integration</li> Providing feedback on features</li> Being among the first to use Stria</li> </ul> Feel free to reach out to me on X or Bluesky. Building in Public</a></h2> I believe in building in public and sharing the journey. I'll be sharing here on my blog, on Stria's website and social media. I'll be sharing more technical deep dives about: The challenges of building reliable sync systems</li> Handling webhook idempotency</li> Managing API rate limits effectively</li> Lessons learned along the way</li> </ul> Stay tuned for updates as I get closer to launch and check out Stria at stria.io</a>! PostgreSQL table partitioning with Go Unknown — Wed, 18 Dec 2024 00:00:00 +0000 PostgreSQL offers several ways to partition tables, including by range, list, or hash. There are various partitioning strategies available. This post will focus on a practical example using date-based partitioning from a recent project of mine. Demonstrating the concepts and implementation of table partitioning in practice. The Problem</a></h2> Let's start with a simple table that stores messages: CREATE TABLE messages ( id UUID PRIMARY KEY, message JSONB NOT NULL, created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP );</code></pre> As this table grows, queries filtering on specific columns might become slower, and managing the data becomes more challenging. Partitioning can help address these concerns. Implementing Partitioning</a></h2> For our example, we'll partition the table by date using the PARTITION BY RANGE</code> clause: CREATE TABLE messages ( id UUID NOT NULL, message JSONB NOT NULL, created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (created_at, id) ) PARTITION BY RANGE (created_at);</code></pre> Notice two important changes: The PRIMARY KEY now includes both created_at</code> and id</code></li> We've added the PARTITION BY RANGE clause</li> </ol> Creating Partitions</a></h2> In our example, each partition will represent a single day of data: CREATE TABLE messages_y2024m12d13 PARTITION OF messages FOR VALUES FROM ('2024-12-13 00:00:00') TO ('2024-12-14 00:00:00'); CREATE TABLE messages_y2024m12d14 PARTITION OF messages FOR VALUES FROM ('2024-12-14 00:00:00') TO ('2024-12-15 00:00:00');</code></pre>Working with Partitioned Tables</a></h2> When it comes to inserting data you don't need to worry about inserting into a specific partition. All you need to do is insert into the parent table. Postgres handles the routing of data to the correct partition automatically based on your partition key. -- Postgres automatically routes this insert to the correct partition INSERT INTO messages (id, message, created_at) VALUES ( 'some-uuid', '{"key": "value"}'::jsonb, '2024-12-13 14:30:00'::timestamptz );</code></pre>Automating Partition Management</a></h2> In a production environment, you'll likely want to automate the creation of new partitions. Using Go you can create a function that helps create partitions: func createPartition(db *sql.DB, date time.Time) error { tableName := fmt.Sprintf("messages_y%04dm%02dd%02d", date.Year(), date.Month(), date.Day()) startDate := time.Date(date.Year(), date.Month(), date.Day(), 0, 0, 0, 0, time.UTC) endDate := startDate.AddDate(0, 0, 1) query := fmt.Sprintf(` CREATE TABLE IF NOT EXISTS %s PARTITION OF messages FOR VALUES FROM ('%s') TO ('%s')`, tableName, startDate.Format("2006-01-02 15:04:05"), endDate.Format("2006-01-02 15:04:05")) _, err := db.Exec(query) return err }</code></pre> You could then automate the creating of partitions for future dates: func ensurePartitions(db *sql.DB, daysAhead int) error { now := time.Now() for i := 0; i <= daysAhead; i++ { partitionDate := now.AddDate(0, 0, i) if err := createPartition(db, partitionDate); err != nil { return fmt.Errorf("failed to create partition for %s: %w", partitionDate.Format("2006-01-02"), err) } } return nil }</code></pre>Benefits of Partitioning</a></h2> Queries can quickly find relevant data by skipping partitions that don't match the search criteria, leading to better performance across large datasets</li> Drop a partition. That's all it takes to remove old data</li> Database maintenance operations like index rebuilds and VACUUM run more efficiently on smaller, individual partitions, reducing the overall maintenance window for your database</li> </ol> Conclusion</a></h2> Table partitioning is a powerful feature in Postgres that can help manage large datasets efficiently. By using declarative partitioning and automating partition creation. You can maintain a clean and efficient database design while letting Postgres handle the complexity of routing data to the correct partitions. Success with table partitioning depends on key practices. Always insert data through the parent table. Automate the creation of new partitions to prevent maintenance overhead becoming unmanageable. Partition keys should be included in the primary key for optimal performance. Plan your partition strategy to align with your specific query patterns. Following these fundamentals, you'll be well-positioned to leverage the full benefits of Postgres partitioning. For a real-world example of table partitioning in action, see my post on processing UK rail data in real-time</a> where I use date-based partitioning to handle millions of daily messages from Kafka. YAML configuration: an unexpected bug Unknown — Mon, 09 Dec 2024 00:00:00 +0000 Few weeks ago I wrote about YAML configuration</a> and why I believe they are the best of a bad bunch of formats. Within the post I talked abouit adding the ability for users to do string replacement with $LOG_LEVEL</code> bash style variables. This worked great in situations where you had a variable on it's own e.g. log: level: "$LOG_LEVEL"</code></pre> But I did not take into account secrets that may contain dollars in them. This ended up causing an unexpected bug in production adding a secret containing a dollar symbol into a yaml configuration file. In the end I have decided to go back on my original design and plan to support $LOG_LEVEL</code> in favour of only supporting ${LOG_LEVEL}</code> style variables. Bugs and unexpected behaviour happens and it is important to be able to admit when a solution is flawed and you have to go back on a design. Now my configuration library only supports ${LOG_LEVEL}</code> which I believe is enough. Adding many configuration options for environment variables added complexity. YAML configuration: the best of a bad bunch Unknown — Sat, 23 Nov 2024 00:00:00 +0000 Over the years, I’ve worked on a variety of systems, each with its own approach to configuration. From system.properties</code> in Java apps to wp-config.php</code> in WordPress, .ini</code> and .toml</code> files, JSON, and, of course, environment variables. None of these solutions are perfect. When dealing with application configuration you try to aim for a balance between: readability, maintainability, and flexibility. These qualities can often be at odds with each other. After years of using various methods for configuration I have eventually settled on what I believe to be the most pragmatic approach. Combining YAML for structured configuration with environment variables. I've found this hybrid method aligns with modern development practices, especially in containerised environments, while remaining developer-friendly. Golang's built-in os.ExpandEnv</code> function can handle basic environment variable expansion, but it lacks support for default values when variables are undefined. To address this limitation, I extended the functionality in my YAML configuration library to provide a fallback behaviour, ensuring graceful handling of missing environment variables. My Approach: YAML + Enhanced os.ExpandEnv</code></a></h2> Using YAML as the base configuration format allows me to define structured defaults that are easy to read, maintain, and version-control. Environment variables then handle dynamic values for environment-specific customisations, with custom logic to support default values for additional flexibility. Why YAML is Superior to Environment Variables</a></h3> While environment variables are popular thanks to methologies like Twelve-Factor applications and containerised runtimes, they often create hidden dependencies when developers use os.Getenv</code> directly within code. This practice can obscure configuration requirements and make application behavior unpredictably dependent on environment state. I believe YAML offers a more robust approach through. Maintaining all configuration in a single, structured file loaded at startup, a YAML configuration file ensures settings are visible, traceable, and maintainable. A YAML configuration file establishes a clear boundary between configuration and code, enabling consistent behavior, simplified validation, and more effective testing. Of course environment variables remain valuable for specific use cases like managing secrets and environment-specific overrides. I believe however, YAML proves more effective as the primary configuration method. YAML with Placeholders</a></h3> Here’s an example of a YAML configuration file: log: level: "info" database: postgres: url: "postgres://postgres:postgres@127.0.0.1:5432/default?sslmode=disable" web: host: 127.0.0.1 port: 8974</code></pre> This file might be named config.local.yaml</code> to clearly indicate it’s for local development. It provides sensible defaults that work out of the box for other developers. In production, we can easily override these values using environment variables, as per the Twelve-Factor App principles. For instance: log: level: "$LOG_LEVEL" database: postgres: url: "$DATABASE_URL" web: host: "$WEB_HOST" port: "$WEB_PORT"</code></pre>LOG_LEVEL=debug DATABASE_URL=postgres://prod_user:prod_pass@db.example.com:5432/prod WEB_PORT=80</code></pre> This approach works well for dynamic settings like credentials or deployment-specific URLs. However, some configuration variables—such as the web port—rarely change once defined. Keeping these "static defaults" in the YAML file avoids overloading the environment with unnecessary variables. To merge YAML configuration with environment variables, I use os.ExpandEnv</code>. However, this function has a key limitation. Why os.ExpandEnv</code> Falls Short</a></h3> While os.ExpandEnv</code> is excellent for basic environment variable substitution, it doesn’t support fallback values. If an environment variable isn’t defined, the placeholder is simply replaced with an empty string. This limitation can lead to brittle configurations where missing environment variables cause unexpected behaviour. For example, if DATABASE_URL</code> isn’t set in the example above, it will result in an invalid database configuration. To address this, I extended the behaviour of os.ExpandEnv</code> to support default values using a ${ENV_NAME:default}</code> syntax. This is where some custom logic and using os.LookupEnv</code> can be combined together to provide even more flexibility. Enhanced Logic for Default Values</a></h3> Here’s the function I use to handle both $ENV_NAME</code> and ${ENV_NAME:default}</code> placeholders: // parseEnv replaces $ENV_NAME and ${ENV_NAME:default} placeholders. // Default values are only supported with ${ENV_NAME:default}. func parseEnv(input string) string { // Regex to match both $ENV_NAME and ${ENV_NAME:default} re := regexp.MustCompile(`\$(\w+)|\$\{(\w+)(?::([^}]*))?\}`) return re.ReplaceAllStringFunc(input, func(match string) string { parts := re.FindStringSubmatch(match) if len(parts) == 0 { return match // No match, return as-is } // Check if it's $ENV_NAME or ${ENV_NAME:default} if parts[1] != "" { // $ENV_NAME style varName := parts[1] if value, found := os.LookupEnv(varName); found { return value } return "" // If not found, replace with an empty string } // ${ENV_NAME:default} style varName := parts[2] defaultValue := parts[3] if value, found := os.LookupEnv(varName); found { return value } return defaultValue // Fallback to default value if not found }) }</code></pre>How It Works</a></h4> $ENV_NAME</code>: This is replaced by the corresponding environment variable's value, or an empty string if not set.</li> ${ENV_NAME}</code>: This is replaced by the corresponding environment variable's value, or an empty string if not set.</li> ${ENV_NAME:default}</code>: This adds a fallback mechanism. If the variable isn’t set, the specified default value is used instead.</li> </ul> Example</a></h4> Given the following environment variables: LOG_LEVEL=debug</code></pre> The configuration file: log: level: "${LOG_LEVEL:info}" database: postgres: url: "${DATABASE_URL:postgres://localhost:5432/default}" web: host: "${WEB_HOST:127.0.0.1}" port: "${WEB_PORT:8080}"</code></pre> Would result in: log: level: "debug" database: postgres: url: "postgres://localhost:5432/default" web: host: "127.0.0.1" port: "8080"</code></pre>Benefits of This Approach</a></h2> This proposed solution combines structured YAML files with environment variables to create a robust configuration system. YAML provides a version-controlled foundation with clear defaults, while environment variables enable flexible overrides for different deployment environments. Extending Golang's environment variable handling with custom default logic, we ensure graceful fallbacks while maintaining clarity. This hybrid approach delivers both the stability of structured configuration with flexibility needed for modern deployment scenarios. Trade-Offs</a></h2> While this configuration approach offers a balance of flexibility and structure, it does comes with notable trade-offs. The custom parsing logic will increases the maintenance overhead. The system may also require additional validation steps to ensure configuration completeness and correctness. Conclusion: A Balanced Approach</a></h2> For me, combining YAML’s readability and structure with the flexibility of environment variables—and augmenting Go’s os.LookupEnv</code> to support default values—creates a configuration system that balances developer productivity and operational needs. It is not a perfect solution, and there are limitations to this approach, but it strikes a balance that has proven reliable and effective across a variety of projects. For a more complete solution for parsing YAML configuration, I’ve created my own library for loading, parsing, and validating configurations. If you're interested in checking it out, you can find it on my GitHub: aranw/yamlcfg</a>. Update: an unexpected bug</a></h2> After using my updated YAML Configuration library in production I came across an unexpected bug where configuration values that contained a dollar $</code> symbol would trigger the environment variable parsing logic unexpectedly. I wrote a follow up post regarding this and the fix ultimatelly deciding to remove support for $ENV_VAR</code> style variables. You can checkout the post here</a>. Deploying NixOS on Proxmox as an LXC container Unknown — Sat, 26 Oct 2024 13:00:00 +0000 I've been using NixOS</a> for a while now, primarily to provision my personal and work Mac systems. While Nix can be complex and sometimes challenging to debug, I’ve enjoyed its unique approach to system configuration. However, I haven't explored much beyond that scope. For some time, I’ve become particularly interested in using it for containers, as NixOS offers compelling features—reproducibility, declarative configuration, and reliable upgrades—which, in theory, make it an excellent choice. How well this theory holds up is something I’m eager to explore further. In this post, I’m documenting my steps for installing NixOS on Proxmox</a>, my chosen virtual environment solution for my home server, as deploying NixOS onto my home server should help me further explore using it for configuring my containers Loading the NixOS Container Image</a></h2> The first step is to load the NixOS container image onto my Proxmox host as a CT template, so that new containers can be created with this base image. At the time of writing, the latest NixOS LXD container image I found was version 276346783</a>, but you’ll likely want to download the most recent image available here</a>. You’ll need to copy the URL under 'Link' for the system tarball; for my build, this was https://hydra.nixos.org/build/276346783/download/1/nixos-system-x86_64-linux.tar.xz</code>. Next, in Proxmox, I loaded the container using the 'Download from URL' button on the "CT Templates" page for my hosts local storage. Clicking on the button a modal will open up, enter the URL, and gave it a descriptive name, for example I used 'nixos-2024-10-25-system-x86_64-linux.tar.xz'. Create a NixOS Container</a></h2> At first, I attempted to configure a new container through the Proxmox UI. However, based on various resources I found online, there were some issues with this approach, so I ultimately needed to create a new container via the command line. After referencing the NixOS Wiki and other sources, I came up with the following command to create a new container: pct create "$(pvesh get /cluster/nextid)" \ --arch amd64 \ "local-lvm:vztmpl/nixos-2024-10-25-system-x86_64-linux.tar.xz" \ --ostype unmanaged \ --description nixos \ --hostname "thermic" \ --net0 name=eth0,bridge=vmbr0,ip=dhcp,firewall=1 \ --storage "local-lvm" \ --memory "2048" \ --rootfs local-lvm:10 \ --unprivileged 1 \ --features nesting=1 \ --cmode console \ --onboot 1 \ --start 1</code></pre> Some of these settings are specific to my setup and may differ for yours, but hopefully, this provides a helpful starting point. Logging into NixOS</a></h2> NixOS should now be running as an LXC container on Proxmox. If you navigate to the console, you may initially see a black screen, which might seem odd. However, simply press 'Enter,' and you'll be prompted with a login screen. Enter the username 'root' and press 'Enter' again to log into your fresh NixOS container. Next Steps</a></h2> Now that I have the container up and running, my next steps are to create a configuration to install PostgreSQL, setup any monitoring I need and to set up and run a Go app I’ve been developing. Looking ahead, I’m also interested in building a preconfigured NixOS image with all necessary configuration and setup, ready to run any application I choose. Resources</a></h2> https://nixos.wiki/wiki/Proxmox_Linux_Container</a></li> </ul> Why Go uses composition over inheritance Unknown — Mon, 24 Jun 2024 15:00:00 +0000 Recently, Andrew Cairns</a> shared a post on Twitter/X</a> featuring a shortened written version of his fantastic video, Composition over Inheritance Explained by Games</a>. As I read through the article and watched the video again, I reflected on how, in Go, we don’t face the same concerns. Go's design philosophy emphasises simplicity, clarity, and efficiency, strongly favouring composition over inheritance. By leveraging composition, Go encourages the use of small, self-contained components that can be easily combined to build complex systems. This approach not only enhances code readability and maintainability but also provides greater flexibility and control over implementation details. Additionally, Go's interface system, which supports implicit implementation, further promotes the modular and reusable nature of compositional design, helping developers avoid common pitfalls associated with inheritance, such as the fragile base class problem. Go's approach to inheritance is fundamentally different from the traditional class-based inheritance found in languages like PHP, TypeScript, and Java. Instead of relying on class hierarchies, Go uses structs and interfaces to achieve behaviour composition. Behaviour is added to structs through methods, and interfaces are implemented implicitly, allowing for flexible and decoupled design. While traditional object-oriented programming languages like PHP, TypeScript, and Java rely on explicit interfaces, class hierarchies, abstract classes, and mixins or traits to achieve polymorphism and reuse, Go uses implicit interfaces and struct embedding to compose behaviours and decouple the definition of an interface from its implementation. This approach avoids the complexities and rigidity of deep inheritance hierarchies, promoting better modularity, readability, and maintainability. Go's straightforward and flexible interface system makes its design simpler and more efficient, fostering robust and adaptable codebases. Composition vs Inheritance</a></h2> Composition and Inheritance are two fundamental concepts in object-oriented programming (OOP). Composition involves building complex types by combining objects of other types. It promotes the creation of small, reusable components that can be assembled to provide more complex functionalities. This approach encourages flexibility and modularity, as components can be easily replaced or modified without affecting the entire system. Inheritance, on the other hand, allows a class to inherit properties and methods from another class. This establishes a hierarchical relationship between the parent (superclass) and child (subclass), enabling code reuse and the extension of existing functionality. However, inheritance can lead to rigid structures and tight coupling, making systems harder to modify and maintain. How Go Implements Composition and Inheritance</a></h2> Now that we have an understanding of composition and inheritance, let's take a look at how Go implements composition through the use of structs and interfaces. Go allows developers to create flexible and modular code without relying on traditional inheritance hierarchies. Structs and Embedding</a></h3> In Go, structs are the primary way to define complex data types. A struct is a sequence of named elements, called fields, each field has a name and a type. Field names can be specified explicitly or implicitly using embedding. Composition is achieved by embedding one struct within another either named explicitly or implicitly. This allows the embedded struct to become a part of the containing struct, effectively allowing the latter to inherit the fields and methods of the former. type Base struct { Num int } type Container struct { Base // Embedded struct Str string }</code></pre> In this example, we have two structs: Base</code> and Container</code>. The Base</code> struct is embedded into the Container</code> struct, allowing users of the Container</code> struct to access the Num</code> field found in the Base</code> struct. A working example can be found at Go by Example: Struct Embedding</a>. Interfaces and Implicit Implementation</a></h3> Go uses an implicit interface implementation, which means that Go types must implement a set of method signatures defined in a Go interface in order to be considered to conform to that interface. Unlike other languages where you must explicitly declare that a type implements an interface, if a type has all the methods defined in an interface, it is considered to implement that interface automatically. type Geometry interface { Area() float64 Perimeter() float64 } type Rectangle struct { width, height float64 } type Circle struct { radius float64 } func (r Rectangle) Area() float64 { return r.width * r.height } func (r Rectangle) Perimeter() float64 { return 2*r.width + 2*r.height } func (c Circle) Area() float64 { return math.Pi * c.radius * c.radius } func (c Circle) Perimeter() float64 { return 2 * math.Pi * c.radius } func Measure(g Geometry) { fmt.Println(g) fmt.Println(g.Area()) fmt.Println(g.Perimeter()) }</code></pre> Here we have two structs: Circle</code> and Rectangle</code>. These both have Area</code> and Perimeter</code> methods, meaning both structs implement the Geometry</code> interface and can be passed into the Measure</code> method. A working example can be found at Go By Example: Interfaces</a>. Conclusion</a></h2> Go's design philosophy emphasises simplicity, clarity, and efficiency. This is clearly reflected in its preference for composition over inheritance. By encouraging the use of small, self-contained components, Go enables developers to build flexible and modular systems. This compositional approach not only enhances code readability and maintainability but also provides greater control over implementation details, resulting in more adaptable and robust codebases. The simplicity that Go brings to this design is evident in the code produced with the language. Without the need to worry about class hierarchies and architectural issues like the fragile base class problem, developers can focus more on implementation. The use of structs and embedding, combined with Go's interface system that supports implicit implementation, avoids the pitfalls of deep inheritance hierarchies found in traditional object-oriented programming languages like PHP, TypeScript, and Java. This makes Go's approach to software design both innovative and practical. For those looking to delve deeper into these concepts, exploring the resources mentioned below can provide valuable insights. By embracing Go’s compositional model, developers can create more maintainable, scalable, and efficient software systems, aligning with the modern demands of software development. Benefits of Go's Composition and Inheritance Approach</a></h3> Flexibility: By composing types using struct embedding and implicit interfaces, Go provides greater flexibility in how objects are constructed, interact, and are used.</li> Modularity: Components can be developed, tested, and maintained independently, promoting cleaner and more modular code.</li> Simplicity: Avoiding deep inheritance hierarchies keeps the codebase simpler and easier to understand.</li> Reusability: Methods and functionalities can be reused across different types through embedding and interfaces without the constraints of traditional inheritance.</li> </ol> Further Reading and Resources</a></h2> For further research and resources, consider these sources: Patterns of Enterprise Application Architecture</a> by Martin Fowler, which covers subjects such as Inheritance, Class Table Inheritance and more.</li> Andrew Cairns</a> fantastic Composition over Inheritance Explained by Games</a> video.</li> Composition over inheritance</a> on Wikipedia</li> The Go Programming Language</a> by Alan A. A. Donovan and Brian W. Kernighan, which provides comprehensive coverage of Go’s design and idioms.</li> Effective Go</a>, an official document that offers guidelines and best practices for writing clear and idiomatic Go code.</li> Go by Example: Struct Embedding</a></li> Go By Example: Interfaces</a></li> codecademy: Go Composition</a></li> </ul> Devlog 2 - Eventbus and SaaS model Unknown — Sat, 01 Jun 2024 12:00:00 +0000 It's been a while since I wrote a devlog. I aim to do these more regularly, but I got busy, sidetracked, and simply forgot. I kept reminding myself to get back to it but never did. In today’s devlog, I want to discuss the custom event bus I've written in Go to handle internal event delivery within my codebase. I have numerous events that need to trigger various actions, so this was a crucial component. Additionally, I want to touch on pivoting and changing my mind regarding the product's direction, whether it pertains to internal system components, the development approach, or other aspects. Initially, I aimed to build this product as a self-hosted solution with a perpetual licence, and that remains a goal. However, I've also decided to support, or potentially support, a SaaS model in the future. For this reason, I've shifted from using SQLite to Postgres. SQLite is a fantastic product and database, and I will undoubtedly use it for future projects. However, the current product is becoming more complex, and supporting a SaaS model introduces the challenge of managing tenanted SQLite databases. While this is likely feasible, especially with tools like libSQL</a>, I prefer to avoid this complexity for now. For a simpler product, I might explore libSQL as a solution. Event Bus</a></h2> Integrating an external system like Stripe into an application can be complex. However, if you only need to respond to a handful of events and perform specific tasks based on those events, it can be straightforward. For example, you receive a webhook for a charge, update a user's subscription within your app, and you are done. In my situation, I wanted to consume as many Stripe events as possible and build custom workflows based on multiple events and scenarios. This required a more robust solution. My goal was to provide users with the ability to trigger custom workflows, which necessitated access to as many events as Stripe emits. As of writing this, Stripe can emit 229 different event types, and I aim to support as many custom workflows as possible. Writing handling logic for all of these events would be complex, so I am utilising code generation and an internal event bus library that I've built to simplify this process. The event bus utilises the publish/subscribe design pattern to ensure loose coupling and separation of concerns between components. It allows any service within my codebase to easily publish or subscribe to events using a simple function call with the appropriate Golang struct. This enables any service to subscribe to a particular event effortlessly. For example the subscription for an event containing the stripe.Change</code> entity it would look something like this. eventbus.Subscribe[*stripeevents.Event[stripe.Charge]]()(eventbus.HandlerFunc[*stripeevents.Event[stripe.Charge]](s.HandleChargeEvent))</code></pre> This event can be triggered by 6 of the 229 stripe.EventType</code> we use a switch statement to catch these events publishing the event in the correct stripe struct. In this example we're publishing a stripe.Charge</code> entity. case stripe.EventTypeChargeExpired, stripe.EventTypeChargeFailed, stripe.EventTypeChargePending, stripe.EventTypeChargeRefundUpdated, stripe.EventTypeChargeRefunded, stripe.EventTypeChargeSucceeded, stripe.EventTypeChargeUpdated: if err := publish(ctx, data, stripe.Charge{}, inEventType); err != nil { slog.Error("publishing event", "event_type", inEventType, "error", err) if errors.Is(err, ErrEventDataParsing) { return http.StatusBadRequest } return http.StatusInternalServerError }</code></pre> The publisher is far simpler, requiring just a simple call to the generic Publish function: eventbus.Publish[Event[T]]()(ctx, e)</code></pre> Currently, the library relies on the singleton pattern, which means there is a single instance of the event bus shared across the entire application. This approach simplifies access and management of events, but it can also introduce limitations in flexibility and scalability. In the future, I may refactor this to support multiple instances to better handle different use cases and improve modularity. However, in its current form, it suits my needs. Eventually, I plan to open-source the event bus library, but I feel like right now it lacks some features that I want, such as middleware support and improved concurrent handling of event handlers. SaaS Model</a></h2> I've decided to potentially support a SaaS model of the product earlier in the development process. While I could have probably built this using the original database choice, SQLite, it presented complexities that I preferred to avoid at this stage. Although libSQL</a> might have simplified some aspects, it introduced its own challenges. Despite being a fork of SQLite, libSQL lacks the extensive support and documentation that SQLite offers, making it harder to ensure everything is implemented correctly. The differences between libSQL and SQLite added another layer of difficulty, complicating the alignment between the two. In the future, I might explore and utilise the multi-tenant features of libSQL for another project. However, given the current complexities I'm already managing, I decided it was best to reduce potential issues by not incorporating libSQL at this time. Next devlog</a></h2> In my next devlog, I hope to demo some of the onboarding process for my application. When a user signs up, we need to request the Stripe API Key and Stripe Webhook Secret from the user. These values must be stored securely in the database to prevent accidental leaks and ensure that bad actors cannot read them, even if they are accidentally returned to the user. Access to the secrets will be managed through specific functions that handle decryption securely. wip: wifi-qr Zig Rewrite Unknown — Mon, 27 May 2024 21:00:00 +0000 A couple of years ago, I frequently needed to join WiFi networks on external devices for testing. Entering long, randomly generated passwords multiple times per day became quite tedious. To address this, I created a simple bash script that queries the macOS Keychain to retrieve the current WiFi network's password and generates a QR code to simplify the process. The project can be found on my GitHub: aranw/wifi-qr</a>. The functionality of the project has always been limited due to using Bash. I've wanted to rewrite the project and have always thought it would be a great candidate for rewriting in various languages due to its simplicity and number of features. Zig is a language I've been interested in but hadn't found a project to use it with until recently. Last year, I explored the excellent ziglings/exercises</a> project, which offers a delightful learning experience. You solve a series of broken programmes, using hints to fix them and gradually learn the language. Over the May bank holiday, I decided to revisit the Ziglings exercises to see what had changed in Zig, as it is an actively developed language. Ziglings is a great way to learn about these updates. After completing just over half of the exercises, I wanted to try Zig with a real project and thought my wifi-qr</a> project would be a perfect candidate for a rewrite. Rewriting the initial requirements for the project in Zig was fairly straightforward. Using articles, Ziglings, and code examples from projects like Ghostty</a>, I managed to implement the logic for retrieving the password from the macOS Keychain. The QR code generation part proved to be the most challenging. Initially, I tried to integrate a C QR encoding library from GitHub (nayuki/QR-Code-generator</a>), but I struggled with the C to Zig calls. After numerous failed attempts, I eventually discovered a Zig implementation (bensengupta/qr</a>), which was straightforward to integrate and use. Overall, the experience was really enjoyable, and the language was a joy to work with. Currently, my rewrite does not have all the features of the bash script, and I will need to add some missing features to achieve feature parity between the two versions. There are also a few additional features I always wanted to add but found difficult to implement in Bash, which should be much easier in a more feature-rich programming language like Zig. Now that I have the basic functionality working, the next step is to implement additional features. These include: Linux support: Ensuring the script can retrieve WiFi passwords and generate QR codes on Linux systems.</li> Automatic WiFi network detection: Automatically retrieving the current WiFi network if none is provided via the CLI.</li> Windows support: Extending the script to work on Windows platforms.</li> Multiple QR code formats: Enabling the generation of QR codes in various formats.</li> </ul> My progress on the rewrite can currently be found on a branch in my aran/wifi-qr</a> repository. Devlog 1 - Foundations Unknown — Wed, 15 May 2024 18:00:00 +0000 Over the past few months, I've played around with multiple side projects. While working on these and the Stripe integrations, I realised I'd been through this process before—integrating Stripe, writing custom logic to handle Stripe Webhooks and building out features driven from the integration and wondered whether I had to repeat this each time. I decided to create a standalone application that provides all the necessary functionality for my side projects. This application would sit alongside any project, offering hooks, notifications, and other essential features. I've named the project "stria" for now, though this might change. I'm considering selling it under a perpetual licence with free minor/patch updates. Major releases would be available as paid upgrades for existing users. These devlogs aim to keep me on track. In the first entry, I will cover the technology choices for the project. Technology Choices</a></h2> I'm trying to minimise external dependencies and rely primarily on the Go standard library. Config</a></h3> Configuration is managed through YAML config files and environment variables. Initially, I used Pkl</a>, but it required the Pkl CLI tool for parsing. While Pkl shows promise, it currently lacks a pure Go implementation and requires the Java CLI tool, which is a significant dependency. Therefore, I opted not to include it in my applications for now. The current YAML-based solution uses gopkg.in/yaml.v3</a> and the standard library's os.ExpandEnv</a> function. This transforms any ${SOME_ENV_VAR}</code> or $SOME_ENV_VAR</code> in the YAML file before unmarshalling it into the application's config structure. Although this is a simple library I'm proud of the solution that this has evolved into it's easy to understand and not overly complicated. The library can be found on Github aranw/yamlcfg</a>. Database: SQLite</a></h3> For data storage I've picked SQLite or more specifically Turso</a>'s libSQL fork. The libSQL fork of SQLite means I require CGO. Right now I'm still unsure if this is the right choice and may ultimately change this part I could switch back to SQLite or even PostgreSQL. I use the sqlc</a> library to code generate a bunch of the service layer code so if needed I think it should be fairly easy to swap out. Migrations</a></h3> go-migrate/migrate</a> is my current solution for migrations. While it supports "up" and "down" migrations, I don't agree with the concept of "down" migrations. They often create more issues than they resolve, especially since not all database operations can be safely undone and deleted data is often irrecoverable. Ideally, I prefer up-only migrations. If changes need to be undone, a new up migration should be applied to fix the issue. In the future, I hope to develop a custom library to address this need. Frontend: Templ</a></h3> One of the core goals of the project is to ensure easy distribution. To achieve this, I aimed to integrate both the frontend and backend into a single binary. Go's standard library for templating has its challenges, so I opted for the fantastic templ</a> library for the frontend. Coupled with HTMX</a> and alpinejs</a>, this setup provides excellent interactivity and a modern user interface. HTTP Router</a></h3> Currently, I use chi</a> as the solution for the HTTP router. Chi is a lightweight and idiomatic HTTP router that easily integrates with other Go HTTP handlers, facilitating the composition of functionality. In the long term, I'd like to replace this with Go 1.22's new and improved HTTP multiplexer. However, I still need to devise an effective approach for transitioning from Chi to this new multiplexer. Sessions</a></h3> Having encrypted cookies to store session state is vital. To address this, I chose the alexedwards/scs</a> library. I selected this library due to its comprehensive feature set and support for a wide range of storage backends for sessions. Additional Features</a></h2> In the future I have other features that I want to add and integrate and those will require other technology choices to be hard. I have some of them decided but some are still up in the air as to what I want to use. What is next?</a></h2> I've built a strong foundation and am progressing well with the project. Over the next few weeks, I aim to make further advancements and document these developments. However, my progress is currently slow since I have to fit in development whenever possible—on weekends, before and after work, and during holidays. Despite these challenges, I'm excited about tackling the upcoming tasks, which include implementing webhook receivers, setting up password authentication, automating various processes, and syncing workflows. I'm looking forward to sharing more updates as I continue to refine and expand the project. yamlcfg: a Go library for YAML and environment variable config Unknown — Tue, 14 May 2024 16:00:00 +0000 Introducing yamlcfg</a>, a new library designed to simplify YAML and Environment Variable configuration in Golang applications. I'm excited to introduce yamlcfg, a new library designed to simplify YAML and Environment Variable configuration in Golang applications This library utilises gopkg.in/yaml.v3</a> and the os.ExpandEnv</a> function from Go’s standard library to replace any ${SOME_ENV_VAR}</code> or $SOME_ENV_VAR</code> placeholders in a YAML file before unmarshalling the content into the application's configuration structure Currently, I'm focusing on keeping this library lightweight and straightforward. However, there are some enhancements in the pipeline. The next planned feature is to enable loading configuration files from embed.FS</code>. I anticipate additional features will be incorporated as the project evolves Encrypting and decrypting with Go's crypto/aes and crypto/cipher Unknown — Wed, 08 May 2024 20:00:00 +0000 In an upcoming side project, I needed to encrypt and decrypt values stored in a SQLite database. This article offers an introductory overview of encryption and decryption practices but is not intended as a comprehensive guide. Encryption is a complex field that requires careful implementation to ensure data security. While searching for resources on this topic, I encountered inconsistencies and issues with the examples provided, prompting me to document what I learned for future reference. For this example we will use the crypto/aes</code></a>, crypto/cipher</code></a>, and encoding/base64</code></a> Golang packages to handle encrypting/decrypting and encoding/decoding our secrets. Encrypting</a></h2> In cryptography, encryption is the process of concealing information by converting it into a coded format that can only be decoded and read by someone possessing the correct key. This practice, dating back to ancient civilizations such as Greece and Rome, employed various methods to secure communications. Today, encryption remains a critical component of modern software development, where implementing it correctly is essential to prevent the accidental leakage of sensitive information. To encrypt in Go, we will use multiple packages from the crypto</code> package, specifically the crypto/aes</code></a> and crypto/cipher</code></a> packages. To begin with, a secret key is essential for encryption. In this post, we will focus on symmetric key encryption, though an alternative method involves the use of public-key encryption which we might explore in a future post. For this example, I used a random string generator to generate a 32-byte-long string from random.org/strings</a>. In a production environment, this would likely be configured outside of our application and passed in, but for our simple example, we will hard code this value. package main import ( "crypto/aes" "crypto/cipher" "crypto/rand" "encoding/base64" "io" "fmt" ) func Encrypt(secret, value string) (string, error) { block, err := aes.NewCipher([]byte(secret)) if err != nil { return "", err } plainText := []byte(value) // The IV needs to be unique, but not secure. Therefore it's common to // include it at the beginning of the ciphertext. ciphertext := make([]byte, aes.BlockSize+len(plainText)) iv := ciphertext[:aes.BlockSize] if _, err := io.ReadFull(rand.Reader, iv); err != nil { return "", err } stream := cipher.NewCFBEncrypter(block, iv) stream.XORKeyStream(ciphertext[aes.BlockSize:], plainText) return base64.RawStdEncoding.EncodeToString(ciphertext), nil } func main() { in := "Hello World" secret := "uMnkGpBn9q4nnwQws8NSRhpXpFdQDBXg" encrypted, err := Encrypt(secret, in) if err != nil { panic(err) } fmt.Printf("Encrypted: %s\n", encrypted) }</code></pre> If you execute the command go run main.go</code> from the example above, the output will be something like Encrypted: 2PK2Q/Lj8OTU2j+Ak3qjaHiPJX8KMamxtIO2</code>, with the encrypted value varying with each run. Let's delve deeper into what happens inside the Encrypt function. First, we create a cipher.Block</code> by calling aes.NewCipher</code> and providing it with our secret key. This cipher.Block</code> enables us to encrypt and decrypt input. The length of our secret key determines the AES encryption standard used: 16, 24, or 32 bytes for AES-128, AES-192, or AES-256, respectively. Next, we need to determine where to store our encrypted input, also known as ciphertext. Relying solely on our secret key for encryption is insufficient; we also require an element of randomness. This is where the initialization vector (IV)</a> comes into play. Typically, the IV must be random or pseudorandom, although in some cases, it only needs to be unpredictable or unique. At this point we have everything we need to encrypt our input. This is accomplished using a cipher.Stream</code>, which is created by calling cipher.NewCFBEncrypter</code> with our cipher.Block</code> and the initialization vector (IV). The cipher.Stream</code> includes the XORKeyStream</code> function, which takes a sequence of bytes (our input data) and mixes it with a sequence from the cipher's key stream. This key stream is generated using the secret key. The XORKeyStream</code> function effectively scrambles the input data by applying the XOR operation between the data bytes and the key stream bytes. Once we have our encrypted string, the final step is to base64 encode it. This encoding allows us to safely store the values or transmit them via HTTP, ensuring compatibility and preventing data corruption during transfer. Decrypting</a></h2> Now that we have encryption set up, the next step is decrypting, which mirrors the encryption process but in reverse. First, we decode the base64-encoded ciphertext to retrieve the binary data. Next, we initialise our cipher.Block</code> and extract the initialization vector (IV) from the start of the ciphertext. The remaining part is the actual encrypted data. The IV and cipher.Block</code> are then used to create a cipher.NewCFBDecrypter</code>, which provides a cipher.Stream</code>. This stream performs the reverse XOR operation that was used during encryption, effectively decrypting the data. package main import ( "crypto/aes" "crypto/cipher" "encoding/base64" "errors" "fmt" ) func Decrypt(secret, value string) (string, error) { ciphertext, err := base64.RawStdEncoding.DecodeString(value) if err != nil { return "", fmt.Errorf("decoding base64: %w", err) } block, err := aes.NewCipher([]byte(secret)) if err != nil { return "", err } // The IV needs to be unique, but not secure. Therefore it's common to // include it at the beginning of the ciphertext. if len(ciphertext) < aes.BlockSize { return "", errors.New("ciphertext too short") } iv := ciphertext[:aes.BlockSize] ciphertext = ciphertext[aes.BlockSize:] stream := cipher.NewCFBDecrypter(block, iv) // XORKeyStream can work in-place if the two arguments are the same. stream.XORKeyStream(ciphertext, ciphertext) return string(ciphertext), nil } func main() { in := "p6e1TZTF7i50N4Q/9CjvdKDG36ZemI6nZC70" secret := "uMnkGpBn9q4nnwQws8NSRhpXpFdQDBXg" decrypted, err := Decrypt(secret, in) if err != nil { panic(err) } fmt.Printf("Decrypted: %s\n", decrypted) }</code></pre> An interesting difference in the decryption process compared to encryption is that when calling the stream.XORKeyStream</code> function, we pass the ciphertext</code> variable as both arguments. This technique is used because XORKeyStream</code> can modify the input data in-place, turning encrypted data back into plaintext. Conclusion</a></h2> The encryption and decryption processes are crucial for secure data handling in modern applications. By encrypting data, we transform sensitive information into a format that can be safely transmitted or stored. The subsequent decryption ensures that this data can be accurately and safely recovered by authorised parties. Utilising encoding schemes like base64 can further enhance the compatibility and manageability of encrypted data across different systems and environments. Employing these methods will be vital for the side project I am working on, and I am confident that this functionality can also be utilised in my day-to-day work in the future. In follow-up articles, I plan to explore public-key encryption and its potential uses. If you're looking for a simpler, command-line approach to file encryption, check out my post on age: a practical guide to modern file encryption</a> which covers the age tool as a modern GPG alternative. You probably don't need to mock Unknown — Tue, 16 Apr 2024 09:00:00 +0000 With the clickbait title out of the way, we can discuss why mocking may not be as crucial for Go testing as commonly perceived. Testing is vital to ensuring code reliability and functionality. However, the frequent use of mocks to replicate external dependencies can create a misleading sense of safety and test quality. While useful, mocks often fail to fully reflect the true complexity of system functionalities. Go is praised for its simplicity and robust testing features, integrated into its language and toolchain, making testing straightforward. By integrating Docker with Go’s testing tools, developers can create tests that more closely replicate production environments, enhancing both maintainability and efficiency. This post explores the view that extensive mocking is unnecessary, proposing that combining Go with Docker can lead to more reliable and simpler tests. I aim to provide Go developers with insights and strategies to enhance their testing approaches, moving beyond the traditional reliance on mocks prevalent in other languages. Limitations of Mocks in Testing</a></h2> Consider the scenario of testing an application’s interaction with a database where developers use mocks to simulate the interactions between the database and your application. Initially effective, but as the application and its database interactions evolve—through added fields, complex queries, and schema changes—the team often spends substantial time updating mocks to match these developments, inadvertently introducing errors and fostering false confidence. For instance, integrating technologies like Kafka for asynchronous data processing presents challenges. Mocks, while simplifying tests for such systems and the complexities around message delivery, ordering, and failure handling, may not accurately reflect real-world issues such as network latencies or partitioning challenges, leading to unanticipated production failures. These incidents highlight the limitations of mocks: a critical bug slips into production due to subtle discrepancies in handling conditions like null values. Moreover, when production systems update, existing mocks may no longer accurately represent the interactions, leading to further bugs. An additional personal frustration with mocks is their tendency to cause test suites to expand, sometimes becoming more extensive than the actual code being tested. These examples underscore the shortcomings of mocks in capturing the full complexity of real systems. Initially beneficial for their speed and isolation, mocks can lead to significant oversights, emphasising the need for testing methods that more accurately replicate the intricacies of production environments. Essentially, mocks test our assumptions about how a system operates, highlighting the critical need for these assumptions to closely align with the actual complexities of the systems under test. Go's Approach to Testing</a></h2> Go promotes simplicity and efficiency in its testing approach, contrasting with other languages that rely on external frameworks, libraries, and tooling. Go's integrated testing</a> package and tooling supports tests, benchmarks, and examples directly, simplifying setup and encouraging consistent test writing as part of development workflows. With Go, developers do not need to choose which test library or runner they want to use resulting in a consistent test environment across projects. A key feature of Go is its use of interfaces for loose coupling, which simplifies testability. Interfaces in Go are implicitly implemented, allowing developers to inject specific implementations for testing, enhancing both code quality and maintainability. Go also promotes testing practices that closely mirror real-world conditions. For interactions with databases or external systems like message queues, Go, combined with tools like Docker, enables testing against real instances, improving test reliability without the reliance on mocks. While this approach adds complexity, it simplifies certain aspects of testing by eliminating the need for extensive mock setups. Leveraging Go's interfaces and integrating practical tools for real-world testing, developers can achieve more dependable, maintainable tests and improve test quality. This method not only makes the testing process more efficient but also ensures greater confidence in the software’s resilience and functionality. A Go Example</a></h2> We've explored the limitations of mocks and Go's testing philosophy. Now, let's put these concepts into practice with some examples. We'll examine how reliance on mocks can complicate test code, lead to lower quality tests and mask a bug that may not be caught before going to production. Consider a simple example of a function that retrieves a leaderboard using Postgres. Typically, database logic is isolated in a separate database or repository package. This function could be a key dependency in our service layer, responsible for fetching the leaderboard for further actions, such as outputting it via an API or sending an email to a tournament winner. Typically, we would start with a function signature like the following: type Score struct { UserID int Score int Rank int } type Database struct { Conn *pgx.Conn } func (d *Database) GetLeaderboard(ctx context.Context) ([]Score, error) type Service struct { DB *Database } func (s *Service) GetLeaderboard() []string</code></pre> However, the Database</code> struct presents an immediate issue, as it would require us to develop an elaborate mock to replicate the functionality of pgx.Conn</code> so that we could test it. What you would commonly find in situations like this is in our Service</code> we replace the Database</code> with an interface for handling the retrieval of scores from our database. To accommodate our mocked example, we would modify our struct to use an interface as follows: type LeaderboardRetriever interface { Get(ctx context.Context) ([]Score, error) } type Service struct { LBR LeaderboardRetriever } func (s *Service) GetLeaderboard() []string</code></pre> We've introduced an additional layer of abstraction in our code, enabling us to substitute the pgx.Conn</code> implementation detail with our mock. At first glance, this new abstraction might seem innocuous, and it appears we can now easily test our Service and function. However, since we are retrieving a leaderboard from the database, our function depends heavily on a crucial part of the SQL query: SELECT t.user_id, t.high_score, RANK() OVER (ORDER BY t.high_score DESC) AS rank FROM ( SELECT user_id, MAX(score) AS high_score FROM scores GROUP BY user_id ) AS t ORDER BY rank</code></pre> This query is essential as it fetches the highest score for each user and assigns a rank based on those scores in descending order. Consider the scenario where we use a mock for testing, making the assumption that our database logic and SQL query is correct and returns the expected scores: m.EXPECT().Get(context.Background()).Times(1).Return([]withmocks.Score{ {UserID: 1, HighScore: 5, Rank: 1}, {UserID: 2, HighScore: 3, Rank: 2}, }, nil)</code></pre> However, imagine if a subtle change were made to the query during development, like altering the ranking order from descending to ascending: RANK() OVER (ORDER BY t.high_score DESC) AS rank</code> becomes RANK() OVER (ORDER BY t.high_score ASC) AS rank</code>. This minor change might be easily overlooked during a code review with hundreds of lines of code. The tests would still pass because the mock assumes the query returns results in the expected order. Such a change could go into production, resulting in a malfunctioning leaderboard. This subtle bug is demonstrated in our repository within the withmocks</code> package. Using a real Postgres instance for our tests, such as one running in Docker, could help catch this mistake. We could verify that the results returned from the database are correctly ordered through our test assertions. For a complete code example, visit the repository here</a>. This example, while basic, does not cover database migrations, seeding, test isolation, and other crucial considerations when using real databases in tests. Here, our custom Go test libraries utilizing Docker come into play. Practical Alternatives to Mocks in Go</a></h2> In Go, the standard library provides a robust foundation for testing via the testing</a> package, yet it does not include built-in support for mock testing. However, the language’s design, particularly its interface system, naturally facilitates mocking. For more sophisticated mocking requirements, third-party libraries and tools are often sought, especially for generating complex mock objects which attempt to simulate the behaviour of real, intricate objects. Nevertheless, Go does have some built-in packages and features that enable alternative methods for testing, meaning developers do not necessarily need to rely on mocks. An integral part of Go’s design, interfaces naturally lend themselves to creating testable code. By designing components with interfaces, you can easily substitute dependencies with stub implementations for testing purposes. For instance, consider a service that fetches data from an external API. By defining an interface for the API client, you can inject a stub that returns predetermined data during tests, thereby avoiding the need for an actual API call. Go’s httptest</a> package is another example of the standard library’s support for testing without mocks. It allows developers to create in-memory HTTP servers and clients, facilitating the testing of HTTP interactions in isolation. For example, testing an endpoint by spinning up a temporary HTTP server that returns expected responses can ensure that HTTP clients behave as intended when interacting with a real server. The alternatives to mocks in Go testing underscore the language’s commitment to simplicity and effectiveness. By utilising interfaces and leveraging the Go standard library, developers can create robust and maintainable tests that faithfully represent production environments. Go encourages developers to experiment with these strategies, enhancing their testing practices while staying true to Go’s principles. Test Libraries with Go and Docker</a></h2> Let's explore how we can leverage Go's testing capabilities with Docker to emulate real-world systems, such as queues and other external dependencies, enhancing test reliability and realism. Docker is ideal for creating isolated test environments. Using a Docker Compose file to set up and link containers ensures consistent settings that mimic production environments closely. I however prefer a different approach to integrating Docker with Go. I often integrate Docker with Go by building test libraries that run Docker containers for the system under test. I've applied this technique, in multiple previous roles, building test libraries for systems like Postgres, Kafka, RabbitMQ, Google Cloud Spanner, and Google Cloud Pub/Sub. For instance, consider a Go service interacting with a PostgreSQL database. With Docker, we can launch a PostgreSQL container via our Go test library, and run migrations and data seeding, all within the current test. We can extend this method to ensure each test has a separate Postgres database, providing true test isolation between test cases. Similarly, for services using message queues like Kafka or RabbitMQ, Docker can simulate these systems, allowing us to test the full message processing lifecycle in a controlled environment. An example of this approach is detailed at aranw/kafkatest</a>. While this strategy offers significant benefits, it is crucial to follow best practices in managing Docker resources, ensuring data isolation between tests, and effectively integrating these tests into CI/CD pipelines. Proper cleanup and resource management are essential for maintaining the efficiency and reliability of your test suite. Conclusion</a></h2> We've explored Go's testing capabilities and Docker's role in running applications. We've underscored the pivotal message: robust testing doesn't need to rely on traditional mocks. Go’s philosophy, prioritising simplicity and practicality, combined with Docker’s capability to mimic production environments, offers developers a superior approach to crafting meaningful, real-world tests. We've seen how interface-based testing, the httptest</a> package and Docker's containerisation can lead to tests that are not just aligned with real-life scenarios but are also easier to maintain. These methods enhance the development lifecycle, ensuring applications perform as anticipated in their true operating conditions. This approach emphasises thorough testing, focusing on how applications perform in their intended environments. Using Go with Docker meets modern software development needs for building reliable systems. Go developers can use these tools to not only improve testing but also to better understand how applications behave in their environments. Embracing Go's testing capabilities and Docker’s ability to emulate our systems under test offers a pathway to not only better testing but also to more insightful software development. In future posts I plan to write more about how I write tests, the libraries I use and what I believe to be best practices. Continuous deployment with Go and GitHub Actions Unknown — Mon, 11 Mar 2024 09:00:00 +0000 The ability to deploy new features and fixes rapidly and reliably is a significant advantage. Continuous Deployment (CD) plays a crucial role in this context by automating the deployment process, ensuring that code changes are pushed to production or staging environments after successfully passing quality checks. This level of automation not only accelerates the development cycle but it also diminishes the likelihood of errors making it to production. Combining Continuous Integration and Continuous Deployment together makes for a powerful combination for companies and teams of all sizes. For those working with Go, a language celebrated for its simplicity and performance, incorporating Continuous Deployment into our workflows can substantially boost both productivity and the reliability of our applications. GitHub Actions, an integral automation tool within GitHub, provides a straightforward pathway to implement CD pipelines directly within our repositories, eliminating the need for external services. Following the themes of my previous articles Continuous Integration with Go and GitHub Actions</a> and GitHub Actions and Go Private Modules</a>, this article aims to guide software engineers, GitHub Actions users, and Go developers through the process of setting up a Continuous Deployment pipeline. I've written this guide to help you with the knowledge and tools necessary to automate your deployment process, making your development cycle more efficient, secure, and swift. Through my journey in software engineering, I've had the opportunity to implement Continuous Deployment using Go and GitHub Actions across multiple projects at various companies. This experience has proven invaluable, not only in streamlining development processes but also in enhancing team collaboration and developer productivity. Sharing these insights, I hope to help others in leveraguing the full potential of CD in their Go projects. Let's dive into Continuous Deployment with Go and GitHub Actions, exploring the benefits, strategies, and best practices. Understanding Continuous Deployment</a></h2> Continuous Deployment (CD) is a software release process that automates the delivery of code changes to production environments after passing predefined tests and quality checks. This automation ensures that software can be released to users quickly and reliably, making CD a cornerstone of modern DevOps practices. What is Continuous Deployment?</a></h3> At its core, Continuous Deployment automates the steps a code change must go through from being deployed in a production environment. Unlike Continuous Integration (CI), which focuses on the integration and unit testing phases, CD takes automation a step further. It ensures that code changes passing all automated tests are deployed directly to the production environment without manual intervention. However, it's important to note that this automation does not mean a lack of control. Many companies and teams implement protection rules within their CD pipelines, especially for critical environments like production. These rules can require manual approvals before a deployment is allowed to proceed. This hybrid approach allows developers to focus more on development while still providing a safety net, significantly reducing the time and effort required for manual deployments, yet ensuring that critical checks and balances are in place. Benefits of Continuous Deployment with Go</a></h3> Implementing Continuous Deployment for Go projects offers several advantages: Speed and Efficiency: Automating the deployment process reduces the time from code commit to production deployment, enabling faster feedback and iteration cycles.</li> Reliability: Automating deployments reduces the chances of human error. Automated tests ensure that the system deploys only high-quality code to production.</li> Improved Productivity: Developers can concentrate on writing code and improving the product without being bogged down by the operational aspects of deployment.</li> Enhanced Collaboration: Teams can work more efficiently by relying on a consistent and transparent deployment process, fostering a culture of trust and collaboration.</li> </ul> Continuous Deployment vs. Continuous Integration</a></h3> Continuous Integration (CI) and Continuous Deployment (CD) serve distinct roles in software development. CI focuses on integrating code into a shared repository frequently, where it automatically builds and tests. This process aims to detect issues early. On the other hand, CD automates the deployment of code to production after passing all tests, ensuring users always access the latest features and fixes. While CI ensures code is always deployable, CD automates the deployment, making the process seamless Prerequisites</a></h2> In my previous article Continuous Integration with Go and GitHub Actions</a>, we setup a Continuous Integration (CI) pipeline for a Go project using GitHub Actions. We'll build upon that knowledge and extend our CI pipeline to include Continuous Deployment (CD). If you haven't read the previous article, I recommend doing so to familiarise yourself with the basics of GitHub Actions and CI for Go projects. To recap we setup a new Go project, added tests, and configured a GitHub Actions workflow to run our tests on every push to the repository. We'll use the same project and create a new workflow file to extend on our CI pipeline to include CD. To see the full setup, you can find the repository here</a>. Now we'll first start by changing our example project to include a simple web server. We'll then create a new workflow file to deploy our project to a staging environment. In the repositories first pull request we added a simple web server to our project. This demonstrates the current continuous integration pipeline. You can view the pull request here</a>. Fly.io</a></h3> For this article we'll be using Fly</a> to deploy our project. Fly is a platform as a service that allows us to deploy our project close to our users. As a prerequisite we'll need to create a new Fly account and install the Fly CLI. You can find the installation instructions here</a>. Once you have installed the Fly CLI and authenticated with your account you can start creating your applications. For our example we will need two applications, one for staging and one for production. This can be done by following the instructions here</a> and creating a new application for each environment. Ensure you create a new configuration file for each environment. For my example I have created two applications go-github-actions-example-staging</code> and go-github-actions-example-production</code>. I have also created two configuration files fly-staging.toml</code> and fly-production.toml</code>. GitHub Environments</a></h3> We'll also need to create two new environments in our GitHub repository. We can do this by navigating to the Environments</code> tab in our repository settings. We'll create two new environments staging</code> and production</code>. We'll use these environments when deploying our project. For the production</code> environment we can add protection rules ensuring that only certain users can approve deployments to this environment. For this simple example I have put myself as a required reviewer for the production</code> environment. Setting Up Your Continuous Deployment Pipeline</a></h2> We will now explore creating a new GitHub Actions workflow file for our Continuous Deployment of our application. This workflow will be triggered by a push to the main branch and will deploy our project to a staging environment. We'll also copy the testing phases from our CI workflow into our CD workflow. In a new file .github/workflows/deploy-staging.yml</code> we'll add the following: name: Deploy Staging on: push: branches: - main jobs: build: runs-on: ubuntu-latest environment: staging steps: - uses: actions/checkout@v4 - name: Set up Go 1.21 uses: actions/setup-go@v5 with: go-version: 1.21 - name: golangci-lint uses: golangci/golangci-lint-action@v3 with: version: v1.54 - name: Run Tests run: go test -v ./... - name: Build run: go build -v ./... deploy-staging: name: Deploy app to Staging runs-on: ubuntu-latest needs: build steps: - uses: actions/checkout@v4 - uses: superfly/flyctl-actions/setup-flyctl@master - run: flyctl deploy --local-only --config fly-staging.toml env: FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}</code></pre> In this workflow, we have two jobs. The first job build</code> runs our tests and builds our project. It's important to note that we've copied the testing phases from our CI workflow into our CD workflow. This ensures that our project is tested before it's deployed to a our environments. Given more time we could refactor this into a reusable action to avoid duplication. The second job deploy-staging</code> deploys our project to a staging environment using Fly</a>. We use the superfly/flyctl-actions/setup-flyctl</code> action to setup Fly and then use the flyctl deploy</code> command to deploy our project. We pass the FLY_API_TOKEN</code> secret to authenticate with Fly. Something to note with the flyctl</code> command is that we are giving it the --local-only</code> flag. This flag tells the flyctl</code> command to perform builds locally using the local docker daemon. The default is builds are built on remote hosts. This is useful for our GitHub Actions workflow as we don't need to worry about setting up a docker daemon on our runners. Now when we push to the main branch, our project will run the Continuous Integration and Continuous Deployment pipelines. Running both at this point is perhaps not ideal, but it's a good starting point. Now we have staging deployments up and running, we can look at deploying to production. But before we do that, we need to first create releases for our project. GitHub Releases</a></h3> GitHub Releases are a great way to package software releases. They provide a way to package software, release notes, and links to the software's assets. We can use GitHub Releases to create a release for our project and then deploy that release to production. In this example we'll create releases using the GitHub user interface. We'll then use the release tag to deploy our project to production. A future improvement could be to automate this process using GitHub Actions. To create a release, navigate to the releases page by clicking on the Releases</code> header in the right hand side of your repository home. For the first release, click on Create a new release</code>. From this page we need to create our release tag, title, and description. We could also add assets to our release but for this usecase we won't. When it comes to the title I usually keep this the same as the release tag. The description is a good place to list the changes in this release, GitHub has the ability to autogenerate this for you based on the commits included in the release. For my first release I created a v0.1.0</code> release. Once you've created your release, you'll be able to see it on the releases page. However we won't be able to use it to deploy to production just yet. We first need to add a new workflow to our project to deploy to production. In a new file .github/workflows/deploy-production.yml</code> we'll add the following: name: Deploy Production on: push: tags: - v* jobs: build: runs-on: ubuntu-latest environment: production steps: - uses: actions/checkout@v4 - name: Set up Go 1.21 uses: actions/setup-go@v5 with: go-version: 1.21 - name: golangci-lint uses: golangci/golangci-lint-action@v3 with: version: v1.54 - name: Run Tests run: go test -v ./... - name: Build run: go build -v ./... deploy-production: name: Deploy app to Production runs-on: ubuntu-latest needs: build steps: - uses: actions/checkout@v4 - uses: superfly/flyctl-actions/setup-flyctl@master - run: flyctl deploy --local-only --config fly-production.toml env: FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}</code></pre> What you will notice for this new workflow is that it's very similar to the staging deployment workflow. The main difference is that it's triggered by a tag push and it deploys to a production environment. In this example we're using the v*</code> pattern to match any tag that starts with v</code>. This is a common pattern for version tags. Now that we have our production deployment workflow, we can use GitHub Releases to deploy our project to production. To do this we need to create a new release and tag it. Now after we've tagged our next release, we can see in our list of Deployments that we have a new production deployment waiting to be approved. Heading to the Actions</code> tab we can select our Deploy Production</code> workflow and see that it's waiting for approval before it can proceed. Clicking on the Review deployments</code> will open a Review Deployment modal where we can approve the deployment and add an optional comment. Once our deployment has been approved and our workflow has completed we can see that our project has been deployed to production. Conclusion</a></h2> In this article we've explored the benefits of Continuous Deployment and how we can implement it for our Go projects using GitHub Actions. We've also looked at how we can use GitHub Releases to deploy our project to production. Implementing Continuous Deployment alongside Continuous Integration can significantly improve the speed, reliability, and efficiency of your development process and the quality of your software. By automating the deployment process, you can improve the iteration cycle and reduce the likelihood of errors making it to production. Along with ensuring deployments are regular and reliable, Continuous Deployment fosters a culture of trust and collaboration within your team. There are still various improvements and changes we could make to our deployment process. For example, we could automate the creation of releases using a tool like, we could also test and build our application in one job to avoid duplication. Exploring the benefits of Continuous Deployment and its implementation for Go projects using GitHub Actions has illuminated a pathway to enhancing software delivery. The integration with GitHub Releases for deploying projects to production further amplifies the efficacy of this approach. By embracing Continuous Deployment in tandem with Continuous Integration, the development process can achieve remarkable gains in speed, reliability, and efficiency, concurrently elevating the quality of software. Automation of the deployment process not only streamlines the iteration cycle but also significantly diminishes the chances of errors reaching production. Moreover, the adoption of Continuous Deployment cultivates a culture imbued with trust and collaboration within teams. Looking ahead, there are several avenues for augmenting our deployment processes: Automating release creation: Leveraging tools like release-please</a> can streamline the release process, ensuring that new versions are systematically prepared and documented.</li> Optimising job configurations: Consolidating testing and building phases into a single job could reduce redundancy, enhancing the efficiency of our Continuous Integration/Continuous Deployment (CI/CD) pipeline.</li> </ul> These enhancements not only promise to refine the deployment process but also position our projects for greater adaptability and success in the evolving landscape of software development. Deploying Docker images to Fly.io Unknown — Tue, 05 Mar 2024 13:00:00 +0000 Fly.io</a> is a Platform as a Service (PaaS) enabling developers to deploy and run their applications globally, closer to their users. The Fly.io platform, notable for its simplicity and user-friendly design, has recently gained popularity. Fly.io transforms Docker containers into Firecracker VMs and strategically deploys them worldwide. There are a couple of different ways to build and deploy applications on Fly.io, the most straightforward method involves the flyctl</code> CLI tool. With flyctl</code>, you can effortlessly build and deploy your application using the single command flyctl deploy</code>. However, this article will explore other deployment strategies. With Continuous Integration (CI) and Continuous Deployment (CD) workflows, you'll build and push your Docker images to a container registry. This practice facilitates a consistent build process for images and deploying the same image across various environments. This article will show you how to push Docker images to Fly.io's container registry for deployment. Tagging Images</a></h2> When building and pushing Docker images to external registries, it's essential to tag the image with the registry's URL. For Fly.io, the container registry URL is registry.fly.io</code>. Like with other registries, you must include both an image name and a tag. For instance, when using Docker Hub, you tag an image as follows: registry.hub.docker.com/library/<dockerimagename>:<tag></code></pre> For Fly.io, the tagging format is: registry.fly.io/<your-app>:<tag></code></pre> The key distinction is that, unlike Docker Hub, where your Docker image name can be arbitrary, with Fly.io, it must match your application's name. You can tag your Docker image using the following Docker command: ❯ docker tag myapp:latest registry.fly.io/myapp:latest</code></pre> You can also tag your Docker image on build using the docker build</code> command: ❯ docker build -t registry.fly.io/myapp:latest .</code></pre>Pushing Images</a></h2> Pushing Docker images to external registries, like Docker Hub or Fly.io, involves tagging your image with the registry's URL, image name, and tag. External registries allow for secure storage and easy distribution of your containerised applications. Before pushing Docker images, you must authenticate with the external registry. To authenticate with Fly's container registry, you can use the flyctl</code> CLI tool. The command is as follows: ❯ flyctl auth docker Authentication successful. You can now tag and push images to registry.fly.io/{your-app}</code></pre> This command authenticates your Docker client with Fly.io's container registry, allowing you to push images to the registry. This process works by creating an entry in your ~/.docker/config.json</code> file, and for macOS users, it also adds the username and password to your keychain. Now that you're authenticated, push it to Fly.io's container registry. ❯ docker push registry.fly.io/myapp:latest</code></pre>Deploying An Image</a></h2> Once a Docker image is pushed to Fly.io's container registry, you can deploy it to one of your Fly applications. The flyctl</code> CLI tool provides a simple command to deploy your application using the Docker image from the registry. ❯ flyctl deploy --app myapp \ --image registry.fly.io/myapp:latest</code></pre> In future articles and projects, I plan to delve deeper into using manually tagged docker images within CI/CD pipelines. One use case involves pushing and deploying images to ensure consistent builds across multiple environments. For automating deployments to Fly.io with GitHub Actions, see my post on continuous deployment with Go and GitHub Actions</a> which covers setting up staging and production pipelines. I suck at technical interviews Unknown — Tue, 27 Feb 2024 08:00:00 +0000 It's difficult for me to admit, but I struggle with technical interviews. I don't believe it's due to a lack of abilities or insufficient practice on platforms like LeetCode; rather, I think nerves and my dyslexia play a significant role. I tend not to highlight my dyslexia, as I've managed to overcome it in several areas over the years, and I'm concerned that disclosing it might negatively impact my chances if interviewers perceive it as a drawback. Reflecting on a recent interview, I realised I was completely unprepared. A recruiter had sent the interview details to my email, but they ended up in my spam folder. Unaware, I joined the technical interview without any preparation, which, in my mind, led to my downfall. During the interview, I was tasked with reviewing the code of a caching library and was informed that it contained seven or eight bugs. Despite my efforts, I couldn't identify any. It wasn't until the interviewer provided some hints that the bugs began to emerge. Feeling embarrassed, I continued to struggle, repeatedly uttering "erm" and "I'm not sure." This situation is far from ideal for any candidate, as no employer seeks to hire an engineer who struggles with code reviews. I don't always believe the fault lies entirely with me, however. There are times when I have been mentally prepared and entered the interview with confidence, only for the interviewer's approach to unsettle me, making me nervous and causing me to revert to a state where I struggle to read, review code, or solve technical problems. Some past experiences with technical interviews have left me feeling mentally scarred, and shaking off those negative memories has proven challenging. Every time I enter a technical interview, I find myself reflecting on those past experiences. I dwell on them, worrying that history will repeat itself. I fear encountering the same situations as before—interviewers laughing or becoming disengaged, focusing on something else while I'm on camera. Struggling to navigate through a technical challenge under these circumstances could happen again, further unsettling me and impairing my performance. Moving forward, I realise the importance of being more vocal about my struggles with live coding interviews. The negative experiences have significantly impacted my performance, highlighting the need for me to request alternative ways to be assessed. I must advocate for accommodations that align with my strengths, such as using an editor I'm familiar with, participating in take-home tests that allow me to work through challenges in my own time, or facing challenges that reflect either my current daily work or tasks I would be performing in the role. This approach would not only bolster my confidence but also provide a fairer measure of my capabilities. Often, the tasks set during interviews do not mirror the company's actual work or the day-to-day responsibilities of the role, frequently venturing into areas beyond my expertise. It's important to highlight that not all of my experiences with technical interviews have been negative. When matched with the right interviewer and given a challenge within my realm of expertise, I have thrived, delivering creative solutions that have led to job offers. This contrast underscores the inconsistency and often the unfairness of the technical interviewing process. There are plenty of articles, blog posts, and anecdotes from past colleagues that critique the methods of interviewing in the software engineering industry. My experiences, together with those of others, underscore the need for change in the technical interview process. Building your Go toolkit Unknown — Sun, 11 Feb 2024 10:00:00 +0000 After nearly 10 years of working with Go, including over 6 of those years as a full-time Go developer, I've come to rely on a set of tools and libraries that have significantly boosted my productivity. In this guide, I'll share the tools and libraries I use and discuss the importance of curating your own toolkit. Other developers have shared insights on building services and the libraries they depend on, which I highly recommend exploring to aid in assembling your toolkit. Noteworthy examples include Mat Ryer's "How I write services in Go"</a> and his follow up, "How I write Go HTTP services after 13 years"</a> where he shares the libraries he utilises. The primary goal of building your toolkit is to streamline your development process, enabling you to focus more on creating rather than configuring. This guide aims not only to highlight the importance of a personalised toolkit but also to serve as a foundation for developers looking to enhance their efficiency. By curating a set of tools and libraries tailored to your needs, you can significantly reduce cognitive load and boost productivity. The Philosophy Behind a Personal Toolkit</a></h2> Personal toolkits extend beyond Go and are pivotal in software development, offering tailored sets of tools and libraries. Go's ethos of simplicity and efficiency underscores the value of such a personalised approach, especially when large organisations similarly cultivate their own frameworks to boost productivity. Cognitive bandwidth is finite. Decisions, from selecting libraries to choosing debugging tools, consume mental resources, leading to potential fatigue. A curated toolkit can mitigate this by reducing decision-making and providing reliable, familiar tools, thereby conserving energy for creativity and complex problem-solving. The process of curating a toolkit enhances efficiency by minimising the time dedicated to tool-related decisions and familiarising oneself with chosen tools, which expedites development and enables tackling complex problems more confidently. Deep engagement with a limited set of tools promotes mastery and the discovery of advanced features, often overlooked when frequently switching between tools. Furthermore, standardising toolkits within teams simplifies collaboration, easing code sharing and maintaining coding standards. Ultimately, the philosophy behind a personal toolkit aims to enhance efficiency and reduce cognitive load without stifling creativity. It's about building a foundation for development that prioritises solving problems and crafting solutions effectively. Core Components of My Golang Toolkit</a></h2> For my toolkit, I've assembled a set of tools and libraries that have proven invaluable in my development process. While these components are essential for me, they may not be for you. It's crucial to tailor your toolkit to your unique needs. HTTP</a></h3> I typically rely on the net/http</code> package for building services, given its robust foundation for creating HTTP servers and clients. However, its limitations prior to Go 1.21 led me to often use the github.com/go-chi/chi/v5</code> router for added functionality. This preference may evolve with the enhancements in the net/http</code> package introduced in Go 1.22. Concurrency</a></h3> Go's concurrency model is a standout feature. For basic concurrency patterns, the sync</code> package suffices. Yet, for more complex scenarios, I find the github.com/sourcegraph/conc</code> package invaluable for its user-friendly API and handling of intricate cases. Database</a></h3> Favouring Postgres, I predominantly use the github.com/jackc/pgx/v5</code> package for database interactions and connection pooling. This, combined with github.com/golang-migrate/migrate/v4</code> for database migrations and github.com/georgysavva/scany/v2</code> for scanning rows into structs, significantly enhances productivity and efficiency with Postgres. Logging</a></h3> Although I've previously used github.com/rs/zerolog</code> and github.com/sirupsen/logrus</code>, I've recently transitioned to the standard library's log/slog</code> package for its simplicity and one less external dependency. Testing</a></h3> The standard library's testing</code> package meets most of my testing needs. However, for more intuitive assertions, I use github.com/go-quicktest/qt</code>. For integration tests, particularly with systems like Postgres, Google Cloud Spanner, Kafka, etc., I rely on github.com/ory/dockertest/v3</code> to create custom test environments. Tooling</a></h3> So far I've covered libraries, but tooling is also an important part of my toolkit. I use github.com/golangci/golangci-lint</code> for linting, github.com/golang/tools/tree/master/gopls</code> for my language server, air</code> for hot reloading, github.com/a-h/templ</code> for templating and more. These tools are chosen based on project requirements and personal preference. Future Additions</a></h3> Each component in my toolkit is chosen for its fit with my needs and project requirements, reflecting the dynamic nature of software development. As Go evolves, so too does my selection of tools and libraries, ensuring I'm equipped for the challenges of current and future projects. Adaptability is key; my toolkit flexibly incorporates tools like Kafka, gRPC, or Redis as needed, always supported by the latest and most efficient libraries. Currently, I'm evaluating ConnectRPC</a> for its potential in upcoming projects, illustrating my commitment to leveraging emerging technologies to maintain a cutting-edge development environment. Building Your Toolkit</a></h2> Building a Golang toolkit is akin to embarking on a unique adventure, one that is deeply influenced by individual development style, project requirements, and team dynamics. This journey doesn’t follow a linear path; instead, it encourages you to explore, experiment, and curate a set of tools and libraries that align with your specific needs. Begin by reviewing past projects. What challenges did you face, and which libraries or tools helped you overcome them? Identifying patterns in your past choices can provide valuable insights into your preferences and requirements. Perhaps you’ve consistently leaned towards certain routers for web services or found specific ORMs (Object Relational Mapping) or database packages that fit your workflow perfectly. Recognising these tendencies is the first step towards building a toolkit that feels like a natural extension of your development process. Consider the scope of your current and upcoming projects. Are you focusing on web applications, microservices, or CLI tools? Each domain might benefit from different sets of libraries. For example, web services might thrive with routers like chi</a> or frameworks like Gin</a>, while CLI development could benefit from packages like cobra</a>. Personal preferences play a significant role in shaping your toolkit. Do you prioritize minimalism and the Unix philosophy of "doing one thing well," or do you prefer the comprehensive features offered by more extensive frameworks? Your toolkit should reflect this preference, balancing simplicity with functionality as you see fit. Team dynamics cannot be overlooked if you’re working in a collaborative environment. A shared toolkit, built on familiar libraries and tools, can significantly enhance team productivity and project coherence. Engage in discussions to understand your team's preferences and experiences, aiming for a common ground that supports efficient collaboration. For example it might be beneficial to develop your own internal libraries or tools that are more tailored to your team's specific needs, workflows and environments than existing open source libraries. Experimentation is crucial. With a preliminary list of tools and libraries in hand, test them in small-scale projects or components of larger applications. This hands-on approach allows you to evaluate their fit for your needs, ease of use, and performance implications. In essence, building your Golang toolkit is a dynamic process of discovery, evaluation, and refinement. By drawing on your experiences, staying open to new ideas, and fostering collaboration, you can curate a personalised toolkit that not only enhances your productivity but also enriches your development journey. Staying Updated and Flexible</a></h2> It is important that you stay informed about new developments within the Go ecosystem and be ready to reassess and adjust your toolkit. The tech landscape evolves, and so will your preferences and project requirements. This flexibility ensures that your development environment remains efficient, enjoyable, and aligned with your goals. The last thing you want to do is develop a toolkit that you never update or change. This will lead to stagnation and inefficiencies. Instead, be proactive in your approach to your toolkit, always looking for new tools and libraries that could streamline your workflow or help you overcome new challenges. Conclusion</a></h2> Curating a personalised Golang toolkit is a continuous journey that enhances your efficiency and job satisfaction. This guide has introduced the philosophy behind a custom toolkit, essential components, and a strategy for building one that suits your unique needs. The focus has been on tools and libraries, underscoring their importance in your development workflow. However, project structuring, another critical aspect of your toolkit, has not been covered here. Proper structuring is key to scalability and maintainability but will be the subject of future articles. Keep an eye out as we dive deeper into optimizing your Go projects' architecture. Remember, the right toolkit evolves with you, accommodating changes in technology and project demands. Stay adaptable, engage with the Go community, and let your toolkit reflect your development ethos. Here's to the journey of building a toolkit that not only meets your needs but also makes coding a pleasure. Continuous integration with Go and GitHub Actions Unknown — Tue, 06 Feb 2024 10:00:00 +0000 Info </div> </div> Checkout my previous article on how to use private Go modules in local development and GitHub Actions workflows here</a>. </div> </div> Info </div> </div> Checkout my latest post on how to Continuous Deployment with Go and GitHub Actions here</a>. </div> </div> Introduction</a></h2> In a previous article</a>, I explored how to use private Go modules in local development and GitHub Actions workflows. I did not cover how to use GitHub Actions to create a Continuous Integration (CI) workflow for a Go project. In this article, I'll expand on that and explore how to use GitHub Actions to create a Continuous Integration (CI) workflow for a Go project. Getting Started with Go</a></h2> To get started we first need a Go project. In this section we'll delve into setting up a Go project and creating a simple Go application. I'll create a new Go project and initialise a new Go module. ❯ go mod init github.com/aranw/go-github-actions-example</code></pre> This will create a new Go module in the current directory. We can then create a new Go file and add some code. Here we have the main.go</code> file: package main import "fmt" func main() { msg := sayHello("Aran") fmt.Println(msg) } func sayHello(name string) string { return fmt.Sprintf("Hello %s", name) }</code></pre> This is a simple Go application that prints "Hello Aran" to the console. Giving us a simple project to work with. We can also add a test file for our project. Here we have the main_test.go</code> file: package main import "testing" func TestSayHello(t *testing.T) { msg := sayHello("Aran") want := "Hello Aran" if msg != want { t.Errorf("got %s; want %s", msg, want) } }</code></pre> Info </div> </div> As an example for later I'm going to omit the contents of the main_test.go</code> for testing to demonstrate what a failed GitHub Workflow looks like! </div> </div> This is a very simple example but it gives us a good starting point for our Continous Integration workflow with GitHub Actions. Creating a GitHub Actions Workflow</a></h2> GitHub Actions is a powerful automation tool integrated within GitHub's ecosystem, offering a seamless way to automating workflows directly within your repository. In this segment we will look at using GitHub Actions to create a Continuous Integration (CI) workflow specifically tailored for our Go project. GitHub Actions facilitates a variety of automation tasks within the software development lifecycle, from simple code linting to complex deployments. For Go developers, this means leveraging GitHub's infrastructure to automate testing, building and deploying applications. For this article, we will focus on creating a simple CI workflow for our Go project and in future articles we will explore complex workflows like deploying to a cloud provider. Creating a Workflow File</a></h3> Typically the workflow file is a written in YAML and is stored in the .github/workflows</code> directory in the root of your repository. The workflow file contains the definition of the workflow, including the triggers, jobs and steps. A repository can have multiple workflow files, each defining a different workflow. Advanced options can be to manage these workflow files in separate repositories or using an intermediate language like Cue or Jsonnet to generate the workflow files. In future articles we will explore these advanced options. For our Go project we will create a new workflow file called ci.yml</code> in the .github/workflows</code> directory. This file will contain the definition of our CI workflow. name: CI on: [push, pull_request] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Go 1.21 uses: actions/setup-go@v5 with: go-version: 1.21 - name: Run Tests run: go test -v ./...</code></pre> Right now in the workflow file we have defined a simple workflow that runs on every push and pull request to the repository. The workflow contains a single job called build</code> that runs on the latest version of Ubuntu. The job contains three steps: First we checkout the repository codebase into the runner.</li> Then we setup the runner with Go 1.21 using the actions/setup-go</code> action.</li> Finally we run the tests using the go test</code> command.</li> </ul> Now we are only testing our code in the above example and for a more complete Continuous Integration workflow we should also include static analysis, linting and building. We can add these steps to our workflow file to create a more complete CI workflow. Using the golangci-lint</code> action we can easily add linting and other static analysis tools to our workflow. Ideally this step should be done before running the tests. This is because we want to catch any issues with our code before running the tests. - name: golangci-lint uses: golangci/golangci-lint-action@v3 with: version: v1.54</code></pre> By default golangci-lint</code> comes with some sane defaults but you can also configure it to your liking by adding a .golangci.yml</code> file to the root of your repository. For more details on how to configure golangci-lint</code> checkout the official documentation here</a>. We can also add a step to build our Go application. - name: Build run: go build -v ./...</code></pre> Now that we have a complete CI workflow we can commit the workflow file to our repository and push it to GitHub. {{< figure src="./images/github-actions.png" alt="GitHub Actions Failed Workflow" >}} This is a good example of a failing Workflow. In this example the Workflow failed to due to the contents of the test file being missing. Now I'll add the contents to the test file and push the changes to GitHub. {{< figure src="./images/github-actions-1.png" alt="GitHub Actions Workflow Overview" >}} From the overview page we can see both our failed and passed workflows. We can click on the workflow to see the details of the workflow. {{< figure src="./images/github-actions-2.png" alt="GitHub Actions Workflow Details" >}} Info </div> </div> In this screenshot you'll see some warnings about Node.js 16 actions being deprecated. This comes from the Golang CI Linter action and has been fixed in the project's master branch but v3 is still the latest tagged version! </div> </div> Now we have a complete CI workflow for our Go project. We can use this workflow to test our code on every push and pull request to the repository. We can also use this workflow to run other tasks like building and deploying our Go application which we will explore in future articles. To checkout the complete workflow file and example Go project you can visit the GitHub repository here</a>. Conclusion</a></h2> In this article, we explored how we can use GitHub Actions to set up a Continuous Integration (CI) workflow for a Go project. The journey towards automating and optimising our development workflows is both exciting and immensely rewarding. When introduced into a project, CI/CD practices can significantly enhance the quality, reliability, and efficiency of the software development lifecycle. Implementing continuous integration with GitHub Actions brings significant advantages to Go projects and their development teams. It nurtures a culture of transparency and real-time collaboration, significantly reducing integration issues and fostering a proactive approach to bug fixes and code enhancement. Automated workflows provide instant feedback on changes, streamlining the development process by minimizing manual errors and saving valuable time. This level of automation accelerates the development cycle, enabling teams to focus more on innovation and rapidly deliver value to their users. Ultimately, GitHub Actions and Continuous Integration empower development teams to uphold high quality standards, enhance productivity, and facilitate smoother collaboration. While we have touched upon the basics of integrating GitHub Actions with Go, there are several advanced patterns and practices that we have not covered in this article. Here are a few topics that I plan to cover in future articles: Continuous Deployment: Beyond continuous integration, automating the deployment processes ensures that applications are seamlessly and reliably deployed. Future articles will delve into setting up continuous deployment workflows, enabling Go applications to be deployed automatically to various environments, including staging and production.</li> Guarding Pull Requests: Ensuring that only high-quality, tested code makes its way into the project's main branches is crucial. We'll explore strategies for guarding pull requests, such as requiring status checks to pass before merging and implementing automated code reviews, to maintain code quality and stability.</li> Versioning: Managing application versions in a systematic and automated manner is key to tracking releases and ensuring consistency across environments. Future articles will look at how we can automate this process with GitHub Actions workflows, facilitating better release management and compatibility tracking.</li> Local Development: We'll explore how to test GitHub Actions workflows locally, enabling developers to validate their workflows before pushing changes to the repository. Using the nektos/act</a> tool, we can simulate GitHub Actions locally, ensuring that our workflows are functioning as expected.</li> </ul> For a more in-depth understanding of GitHub Actions and how to leverage it for Go projects, I recommend exploring the official docoumentation for GitHub Actions</a>. The documentation provides comprehensive guidance on setting up workflows, creating custom actions, and integrating with various tools and services. Getting started with Go and Kafka using franz-go Unknown — Sat, 03 Feb 2024 22:00:00 +0000 Introduction</a></h2> In this Getting Started guide, we will explore integrating Golang with Apache Kafka. Kafka, is found in many distributed systems, as it excels in handling large-scale and real-time data streams. Its architecture, designed for high throughput and scalability, makes it indispensable in modern data processing pipelines. Golang, with its concise syntax and efficient concurrency management, stands as an ideal ally for Kafka. This blend of Kafka's data handling prowess and Golang's execution efficiency creates a formidable combination, perfect for building high-performance applications. In this guide, we will delve into the mechanics of Kafka, uncovering its core components. We will also explore how Golang's features complement Kafka's capabilities. Our journey will be lined with detailed examples and practical insights, serving both as a reference and a learning experience. Before we proceed any further, it is important to note that this guide focuses primarily on the integration of Kafka with Golang for publishing and consuming messages. Running and operating a Kafka cluster involves intricate details and complexities that are beyond the scope of this article. Our aim is to provide a clear and practical approach to using Kafka within Golang applications, rather than delving into the specifics of Kafka cluster management. For those interested in Kafka cluster operations, numerous comprehensive resources and official documentation are available for a more in-depth exploration. With this understanding, let’s move forward into the heart of integrating Kafka with Golang. Kafka and Golang - A Perfect Match?</a></h2> Apache Kafka is an open-source streaming platform. Kafka is used to build high performance data pipelines, streaming analytics, and event-driven applications. Kafka is a distributed system consisting of servers and clients. The servers are known as brokers and the clients can be either producers or consumers. Kafka is designed to be highly scalable, fault-tolerant, and fast. Kafka is used by many companies including: Cloudflare, Uber, Netflix, LinkedIn and many more. Understanding Kafka</a></h3> Kafka, at its core, is a distributed streaming platform. It is designed to handle large volumes of data in real-time. Initially conceived at LinkedIn, it has grown into a critical component for managing streaming data in coutless industries. Kafka operates on a publish-subscribe model, with key components including producers, consumers, and brokers. Below the diagram shows the high-level architecture of Kafka. {{< figure src="images/kafka-architecture-dark.svg" alt="Kafka Architecture" >}} Some of the key components of Kafka's architecture include: Producers - Producers are clients that publish data to Kafka topics.</li> Consumers - Consumers are clients that subscribe to Kafka topics and consume data.</li> Brokers - Brokers are servers that manage the persistence and replication of data.</li> Topics - Topics are categories that data is published to and consumed from.</li> </ul> Why Golang?</a></h3> Golang is a statically typed, compiled programming language designed at Google. Golang is known for it's simple syntax and powerful concurrency model, making it particularly suited for distributed systems. When it comes to Kafka integration, Golang's features offer several benefits: Efficiency and Performance - Golang's efficient execution model maximises Kafka's throughput and scalability.</li> Simplicity and Readability - The straightforward syntax makes the implementation of Kafka clients more maintainable.</li> Robust Concurrency Model - Golang's goroutines and channels provide a robust environment for handling concurrent Kafka streams.</li> </ul> When it comes to interacting with Kafka in Golang, there are several options available. The most popular are: sarama</a> - A Golang library for Apache Kafka.</li> confluent-kafka-go</a> - Confluent's Golang client</li> kafka-go</a> - Kafka library by Segment</li> franz-go</a> - Feature complete Kafka library in pure Go</li> </ul> Each of these libraries has its own strengths and weaknesses. For this guide, we will focus on the franz-go</code> library. It is a pure Go implementation of Kafka, with no dependencies on C bindings. Crafting the Kafka Producer in Golang</a></h2> Now that we have a better understanding of Kafka and Golang, let's explore how we can integrate the two together. We will start by building a Kafka producer in Golang. A Kafka producer is an application that publishes data to Kafka topics. This process is critical for feeding data in the Kafka ecosystem. Ensure that you have Kafka installed and running. If you don't have Kafka installed, you can follow the official quickstart guide</a> to get up and running. Another alternative is to use Redpanda, a Kafka-compatible event streaming platform built for modern architectures. Redpanda is a drop-in replacement for Kafka, and it is built for performance and scale. You can find the installation instructions for Redpanda here</a>. Setting up a Golang Kafka Producer</a></h3> Getting set up with the twmb/franz-go</code> library is straightforward. We can install the library using the go get</code> command: $ go get github.com/twmb/franz-go@latest</code></pre> The first step for creating a Kafka producer is initialising a Kafka client. We can do this by creating a new client with the kgo.NewClient</code> function. This function takes a variadic list of options that can be used to configure the client. package main import ( "context" "github.com/twmb/franz-go/pkg/kgo" ) func main() { opts := []kgo.Opt{ kgo.SeedBrokers("localhost:9092"), kgo.DefaultProduceTopic("my-topic"), kgo.ClientID("my-client-id"), } client, err := kgo.NewClient(opts...) if err != nil { // TODO: handle/log this error return } defer client.Close() }</code></pre> Now that we have a Kafka client, we can start producing messages to Kafka. This is really simple to do with the franz-go</code> library. But first we need to create our message. record := &kgo.Record{ Value: []byte("Hello World"), Topic: "my-first-topic", }</code></pre> Now that we have our kgo.Record</code> which contains our message and the topic we want to send it to, we can produce the message to Kafka. if err := client.ProduceSync(context.Background(), record).FirstErr(); err != nil { // TODO: handle/log this error return }</code></pre> Lets now put all of this together into a complete example. package main import ( "context" "github.com/twmb/franz-go/pkg/kgo" ) func main() { opts := []kgo.Opt{ kgo.SeedBrokers("localhost:9092"), kgo.DefaultProduceTopic("my-topic"), kgo.ClientID("my-client-id"), } client, err := kgo.NewClient(opts...) if err != nil { // TODO: handle/log this error return } defer client.Close() record := &kgo.Record{ Value: []byte("Hello World"), Topic: "my-topic", } if err := client.ProduceSync(context.Background(), record).FirstErr(); err != nil { // TODO: handle/log this error return } }</code></pre> It should be noted the above example does not handle any errors. In a production system, you would want to handle errors and log them accordingly. You would also want to look at other best practices such as message serialisation and graceful shutdowns. By following the steps above, we have successfully created a Kafka producer in Golang using the franz-go</code> library. This producer is now capable of publishing messages to a Kafka topic, forming the basis of a data pipeline into Kafka. In the next section we will turn our attention to other side of this equation: consuming message from Kafka. Building the Kafka Consumer</a></h2> In the previous section, we explored how to create a Kafka producer in Golang. Now we will turn our attention to building a Kafka consumer. Building a robust Kafka consumer is essential for processing data from Kafka topics, especially high-throughput data streams. The franz-go</code> library is known for being a feature complete Kafka library in pure Go. It provides a robust environment for handling Kafka streams, making it an ideal choice for building Kafka consumers in Golang. Setting up a Golang Kafka Consumer</a></h3> Similar to the Kafka producer, we start with creating a Kafka client. We can do this by creating a new client with the kgo.NewClient</code> function. This function takes a variadic list of options that can be used to configure the client. For a consumer however we need to specify a few other options such as the group id and the topics we want to consume from. package main import ( "context" "fmt" "github.com/twmb/franz-go/pkg/kgo" ) func main() { opts := []kgo.Opt{ kgo.SeedBrokers("localhost:9092"), kgo.ClientID("consumer-client-id"), kgo.ConsumerGroup("my-group-identifier"), kgo.ConsumeTopics("my-topic"), } client, err := kgo.NewClient(opts...) if err != nil { // TODO: handle/log this error return } defer client.Close() }</code></pre> Now that we have a Kafka client, we can start consuming messages from Kafka. This is really simple to do with the franz-go</code> library. for { fetches := client.PollFetches(context.Background()) if errs := fetches.Errors(); len(errs) > 0 { // All errors are retried internally when fetching, but non-retriable errors are // returned from polls so that users can notice and take action. panic(fmt.Sprint(errs)) } // We can iterate through a record iterator... iter := fetches.RecordIter() for !iter.Done() { record := iter.Next() fmt.Println(string(record.Value), "from an iterator!") } }</code></pre> Lets now put all of this together into a complete example. package main import ( "context" "fmt" "github.com/twmb/franz-go/pkg/kgo" ) func main() { opts := []kgo.Opt{ kgo.SeedBrokers("localhost:9092"), kgo.ClientID("consumer-client-id"), kgo.ConsumerGroup("my-group-identifier"), kgo.ConsumeTopics("my-topic"), } client, err := kgo.NewClient(opts...) if err != nil { // TODO: handle/log this error return } defer client.Close() for { fetches := client.PollFetches(context.Background()) if errs := fetches.Errors(); len(errs) > 0 { // All errors are retried internally when fetching, but non-retriable errors are // returned from polls so that users can notice and take action. panic(fmt.Sprint(errs)) } // We can iterate through a record iterator... iter := fetches.RecordIter() for !iter.Done() { record := iter.Next() fmt.Println(string(record.Value), "from an iterator!") } } }</code></pre> This example only shows consuming messages from a single topic and using a very simplistic solution. In a production system, you would want to handle errors, log them accordingly, and look at other best practices. The franz-go</code> library provides a lot of flexibility and features for consuming messages from Kafka. Some best practices that you would want to consider as well in a production system include: Offset Management - Ensure you are tracking which messages you have consumed, avoiding data loss or duplication.</li> Concurrency - Leverage Golang's concurrency model to consume messages in parallel, enhancing throughput and efficiency.</li> Error Handling - Implement robust error handling and re-balances to ensure your consumer remains stable under various conditions.</li> </ul> By following the steps above, you have created a simple Kafka consumer in Golang capable of reading and processing messages from Kafka topics. This consumer forms the foundation for any data processing pipline that requires data from Kafka. In this getting started guide we have explored how to leverage Golang to produce and consume messages from Kafka. Conclusion</a></h2> Through this guide, we have explored the versatile and feature-rich franz-go</code> library for integrating Kafka with Golang. We've uncovered the synergy between Kafka's scalable messaging capabilities and Golang's efficiency and simplicity. This forms a powerful combination for building high-performance data pipelines and event-driven applications. Although producing messages was fairly straightforward, consuming messages requires a bit more consideration and the simplification of the example above is not suitable for production. Kafka consumers require careful management of offsets and error handling in order to ensure data is processed correctly. I have collected the above examples together along with a Docker Compose file that runs Redpanda and a console for managing Redpanda so that you can run the examples yourself. You can find the code and instructions for running the examples here</a>. If you're looking for more on Kafka with Go, I've also written about building a future-proof data pipeline with Go and Kafka</a> covering a real-world high-throughput system, and processing UK rail data in real-time</a> which uses franz-go with PostgreSQL table partitioning. Building a future-proof data pipeline with Go and Kafka Unknown — Thu, 11 Jan 2024 11:00:00 +0000 Introduction</a></h2> The device data processing system is an essential component for business operations. It was, however, becoming unfit for purpose as new systems and teams needed access to the data. The previous system had various limitations, was limited to one cloud platform and was restrictive in how consumers could access the data. We decided to incorporate a new architecture by developing a new system, which presented a significant challenge for stakeholders. They needed quick and efficient access to crucial data but didn't want to introduce what were considered legacy components into new systems. We decided to rebuild the data processing pipeline. Device data arrives in high frequencies every second from many devices. The significant challenges of handling and processing data swiftly led us to choose Golang and Kafka. Golang and Kafka's ability to manage high data throughput efficiently made it ideal for processing and publishing data. Each message contains device readings. These readings can include frequency, state of charge, voltage, and many more. The readings are either tracked on a sub-second or second-level interval. The reading interval depends on the device configuration. The configuration can vary due to various factors. These include device types, sites, and other options. The aim was to build a new system that is robust and flexible. The system had to be able to handle the inconsistent data structures and high volume of data. We chose Kafka as the new system's core messaging platform, using it to streamline data flow between various cloud providers and platforms. This greatly improved our data access and utilisation within the organisation. We've discussed our project's challenges and goals. We will now delve into the technical solution. Technical Deep-Dive</a></h2> At the heart of our system is a single Golang service. This service pulls messages from RabbitMQ. Then, it splits them into smaller messages. In our first approach, we tried to define all the fields into a Golang struct. We then used Golang's standard JSON encoding library to handle the decoding. However, we encountered several challenges due to the nature of the messages. Messages often varied with device configurations, leading to issues such as missing fields and data type inconsistencies, which ultimately caused decoding problems. These hurdles underscored the need for a new approach. Acknowledging these challenges, we decided to shift our message decoding strategy and refactor our message handling. In the refactor, we adopted the tidwall/gjson</a> package for message handling, enabling us to generically handle retrieval of fields from incoming messages. This approach ensured our system remained versatile, capable of adapting to varied data structures with ease. Utilising the gjson package for message decoding allowed us to eliminate the reliance on Golang structs and hard-coded fields, thereby significantly enhancing both performance and efficiency in decoding messages. A standout feature of our revised system is its exceptional flexibility in field handling, eliminating the need for predefined fields. This design allows our service to seamlessly accommodate new fields as they emerge, without requiring updates or deployments to implement these changes. With the technical framework in place, our attention turned to the crucial aspect of performance. Performance Improvements</a></h2> Through custom observability metrics, we meticulously tracked the performance of our message handling logic, ensuring the system operated as anticipated. These metrics offered real-time data and insights, enabling us to pinpoint areas for improvement. Furthermore, they played a crucial role in measuring the impact of our ongoing changes and guiding our optimisation efforts. Initially, our system relied on Golang structs and custom unmarshalling logic, leading to performance constraints. The variability in data from different devices and their settings introduced complexities that necessitated expensive type-checking logic within the code's critical paths, slowing down message processing times. As a result, our throughput was capped at approximately 2 Megabits per second (Mbps). Although the performance was acceptable, each new variation in data required alterations or additional custom unmarshalling logic, rendering this approach unsustainable in the long run. Recognising the need for improving performance and maintainability, we revisited our strategy. Using the metrics from our monitoring platform we were able to do the following: Refactor RabbitMQ message retrieval</li> Identify and fixed bottlenecks with message parsing and handling</li> Develop an improved strategy for monitoring and benchmarking performance going forward</li> </ul> These improvements significantly boosted the service's overall performance. Thanks to our continuous monitoring, we we're able to make informed adjustments, leading to a notable increase in throughput to approximately 4 Mbps. Effectively, these changes doubled our previously recorded throughput and message handling capacity. Addressing the next major challenge required us to investigate alternatives to using hard-coded Golang structs and custom unmarshalling logic. We continued to encounter new message types and discovered missing data fields, the need for a more adaptable approach became evident. Constant modifications to our message handling logic proved to be unsustainable. Thus, we shifted towards a generic solution for message processing, reducing our reliance on intensive type-checking. This transition not only streamlined message processing and simplified maintenance but also significantly decreased the time required to handle each message. The impact of this change was profound, nearly doubling our throughput once more to approximately 8/9 Mbps and enabling our service to process an average of 120,000 messages per second. With these improvements in performance, we are now ready to explore future enhancements. The next phase of our project focuses on expanding our capabilities and exploring new features. We will also examine how we expose those to stakeholders in the future. Future Scope</a></h2> Looking ahead, our journey with this data processing system is far from over. Successfully integrating data into Kafka is merely the initial step of an ongoing saga. The forthcoming crucial phase will involve the transformation of high-frequency data into aggregated streams. These streams will be meticulously segmented into specific time windows, meticulously designed to cater to the unique requirements of diverse teams across the organisation. By doing so, we aim to provide tailored data streams that enable departments to consume and leverage data more effectively, enhancing decision-making processes and operational efficiency. This strategic approach will not only streamline data accessibility but also ensure that each team has the insights needed to drive their initiatives forward. A key focus will be on aggregating the data into specific time windows. These include 50 milliseconds, 1 second, and 30-minute increments. These varied time windows will help stakeholders have a comprehensive view of the data. They cater to both real-time monitoring and longer-term analytical needs. This level of detail will allow for more nuanced insights. It will also improve decision-making processes across departments. Furthermore, we will focus on enhancing our benchmarking and test coverage. Our goal is to establish a robust framework. It can detect any regressions in performance proactively. We aim to maintain the high standards of efficiency and reliability that we've set. Lastly, identifying our current system's limitations is imperative. We intend to use the improved monitoring to analysis and pinpoint potential bottlenecks and scalability challenges. With a clear understanding of these limitations, we'll devise mitigation strategies. These strategies may include adopting new technologies, revising our system architecture, or refining our data processing. Conclusion</a></h2> Our project has enhanced data processing capabilities by harnessing the power of Golang and Kafka, effectively addressing the complexities associated with high-frequency data. Through incremental improvements, we've managed to increase throughput from 2/3 Mbps to 8/9 Mbps. Moving forward, our focus remains on further optimising data streams and scrutinising the scalability of our system. This effort has not only significantly improved our processing speed but also set a strong base for better handling of high-frequency data. If you're interested in getting started with Kafka in Go, check out my Getting started with Go and Kafka using franz-go</a> guide which covers the fundamentals of building producers and consumers. I also wrote about processing UK rail data in real-time</a> using a similar Kafka-based architecture. Age: a practical guide to modern file encryption Unknown — Sun, 25 Jun 2023 09:00:00 +0000 age</code> is a modern alternative to tools like to older file encryption tools like GPG. It is designed to be simple, modern and secure. age</code> is built using modern encryption primitives and is designed to be easy to use. age</code> works by creating or using public/private key pairs to encrypt and decrypt files. The public key can be shared with others to allow them to encrypt files that only you can decrypt. age</code> is designed to be simple and easy to use. age</code> can be found on GitHub here</a>. In this article, I will explore age</code> and demonstrate how to use it to encrypt and decrypt files. Installation</a></h2> age</code> is available for a variety of platforms including Linux, macOS, Windows, and more! The easiest way to install age</code> is to use a package manger for your platform. A list of available places to install age</code> can be found here</a>. Usage</a></h2> age</code> can encrypt and decrypt files with either using a passphrase or a public/private key pair. The following examples demonstrate how to use age</code> to encrypt and decrypt files. Using age with a passphrase</a></h3> To encrypt a file using a passphrase, use the following command: $ age --passphrase --output output.txt.age input.txt Enter passphrase (leave empty to autogenerate a secure one): Confirm passphrase:</code></pre> To decrypt a file using a passphrase, use the age</code> command with the --decrypt</code> option: ❯ age --decrypt --output input.txt output.txt.age Enter passphrase:</code></pre>Using age with a public/private key pair</a></h3> To encrypt a file using a public/private key pair, use the following command: ❯ age-keygen >> ~/.config/age/keys.txt age-keygen: warning: writing secret key to a world-readable file Public key: age1hpct7gdhldvx3q0lj94t5nfp2zhqwq6ksemk65l2xrrvzlykavzsa9ngh0</code></pre> To encrypt a file using a public/private key pair, use the following command: tar cvz ~/data | age -r age1hpct7gdhldvx3q0lj94t5nfp2zhqwq6ksemk65l2xrrvzlykavzsa9ngh0 > data.tar.gz.age</code></pre> To decrypt a file using a public/private key pair, use the following command: ❯ age --decrypt -i ~/.config/age/keys.txt -o data.tar.gz data.tar.gz.age</code></pre>Using age with SSH Keys</a></h3> age</code> can also be used with SSH keys. For example you can use public keys retrieved from GitHub to encrypt files that only the owner of the private key can decrypt. To do this, you can use the following command: ❯ curl https://github.com/aranw.keys | age -R - input.txt > output.txt.age</code></pre> To decrypt the file this will only be possible with my private key. This is a great way to share files with others without the need for a shared secret or passphrase. Using a YubiKey with age</a></h3> Another fantastic feature of age</code> is the ability to use plugins to extend its functionality. For example, you can use a YubiKey</a> to encrypt and decrypt files. To install the age YubiKey plugin, you can visit the GitHub repository</a>. Once installed you can use the following command to setup age to work with your YubiKey: ❯ age-plugin-yubikey</code></pre> This will start an interactive prompt that will help configure age</code> and your YubiKey to work together. Once completed you'll be able to use your YubiKey to encrypt and decrypt files given the correct recipient id and configuration. Next</a></h2> age</code> is a great tool for encrypting files and is a great alternative to older tools like GPG. I think the next steps for me to explore and combining age</code> with sops</code> or Secrets OPerationS</a> from Mozilla. If you're interested in encryption in Go more broadly, I've also written about using Go's crypto/aes and crypto/cipher packages</a> for AES encryption and decryption in your own applications. Exploring Go 1.21: generics in the standard library and more Unknown — Fri, 23 Jun 2023 09:00:00 +0000 Exciting developments are on the horizon for the Go programming language as it enters the release candidate phase of its latest version Go 1.21. Among the most highly anticipated addition is the introduction of generics support to the standard library. This new release not only brings generics to the standard library but it also includes changes to the Golang's tooling, a new structured logging package, an experimental wasi port and more. In this article, we delve into the exciting features of this release candidate, highlighting some of my favourite new features coming in Go 1.21. Go Toolchain</a></h2> The Go toolchain is a collection of tools that are used to compile, test, and run Go code. With the upcoming release of Go 1.21, developers will gain the ability to select their preferred version. This functionality can be leveraged by adding go</code> or toolchain</code> lines either to the projects go.mod</code> or go.work</code> files. Consequently, the go toolchain will seamlessly download, cache and utilise the specified version. If desired, automatic toolchain switching can be disabled. However, in such cases, the go command will refuse to execute within a directory containing a go.mod</code> or go.work</code> file that specifies a newer version the currently running one. In the future, I plan to write more articles that delve into the this new feature in more depth and the possibilities it enables. For more information on the new go toolchain, check out the documentation</a>. min, max, and clear built-in functions</a></h2> One notable addition in this release is the introduction of three new built-in functions: min, max, and clear. These functions aim to provide developers with enhanced convenience and flexibility when working with data. The min and max functions are designed to simplify the process of finding the minimum and maximum values for the given values. With the introduction of these new functions, finding the smallest or largest values becomes a straightforward task, reducing both coding effort and potential errors. Additionally, the clear function offers a convenient way to delete or zero out all elements. This can be particularly useful in scenarios where the reusability of data structures is crucial, allowing developers to easily initalise or reset them without the need for manual iteration or complex workarounds. Check out the documentation for the new min/max</a> and clear</a> functions. Structured Logging with log/slog</a></h2> The log/slog library is a new structured logging package that is being introduced in Go 1.21. This new logger brings immense convenience and eliminates the need for external libraries like zap or zerolog. With the introduction of this new package, developers will be able to easily log structured data without the need for additional dependencies. Developers will no longer need to spend time evulating and integrating external logging libraries, instead they can simply use the log/slog library to log structured data. Open source libraries and projects will also benefit from this new package as it will allow them to provide structured logging without the need for external dependencies or interfaces to other logging libraries. More information about the new log/slog</code> package can be found here</a>. More!</a></h2> In this article, I have provided an overview of some of the features included in Go 1.21. However, it is important to note that there are numerous other exciting additions in this release. For a comprehensive understanding of the upcoming release, I recommend visiting the official Go blog</a> and release notes</a> for detailed information about the new functionalities and enhancements coming in Go 1.21. Using Go private modules with GitHub Actions Unknown — Sat, 10 Jun 2023 15:00:00 +0000 Info </div> </div> I've followed up this post with articles on Continuous Integration with Go and GitHub Actions</a> and Continuous Deployment with Go and GitHub Actions</a>. </div> </div> As the number of repositories within a project increases so does the complexity of the dependencies. As time goes on you begin to want to share code between the repositories. This is where Golangs Module system comes in handy. Creating new private modules is easy but challenges arise when you want to use those modules in your projects. We'll explore how to configure local development and GitHub Actions to use private Go modules. Creating a Private Go Module</a></h2> Creating a private Go module is pretty simple. Initialising a new module by running go mod init github.com/{organisation/{name}</code> will create a new module. You can then add your code to the module and commit it to your repository. To make the module private you can push it to a private repository on GitHub. Now that we have our module we'll want to use it with other Go projects. Local Development</a></h2> For this we'll have to alter our local development configuration to allow the Go tooling to access the private module. The first thing we need to configure on our local development environment is the GOPRIVATE</code> environment variable. This variable tells the Go tooling which modules should be considered private and should not be access via proxies or checksum database. We can set this variable to our GitHub organisation GOPRIVATE=github.com/{organisation}</code>. This variable can include multiple addresses as a comma separated list. This will allow the Go tooling to access any private modules within the organisation. The next thing we need to do is configure the Go tooling to use our GitHub credentials. There are two ways to configure this. The first is to use the .netrc</code> file. This file is used to store credentials for remote hosts. The file is usually located at ~/.netrc</code>. The file should contain the following: machine <host> login <LOGIN> password <PASSWORD></code></pre> Although the file suggests using a password GitHub does not actually support this. Instead we can use a personal access token. You can create a new personal access token by going to Settings > Developer settings > Personal access tokens</code>. Once you have created a new token you can add it to the .netrc</code> file. The other authentication method is to use SSH keys. This is the method I prefer as it allows me to use my existing SSH keys. To use this method we need to change our git configuration to use SSH instead of HTTPS. We can do this by adding the following to our git configuration: [url "ssh://git@github.com/"] insteadOf = "https://github.com/"</code></pre> Now with the GOPRIVATE</code> environment variable and our authentication method configured we can use our private modules in our projects locally. GitHub Actions</a></h2> Now that we have our local development environment configured we need to configure GitHub Actions to use our private modules. Like local development we'll need to configure the GOPRIVATE</code> environment variable and our authentication method. This can be like local either using Personal Access Tokens or SSH keys. Recently I used the SSH key method to configure GitHub Actions. In this article we will explore how to configure GitHub Actions to use SSH keys. The first thing we will configure is the GOPRIVATE</code> environment variable. We can do this by adding the following to our workflow: env: GOPRIVATE: github.com/{organisation}</code></pre> The next step is configuring the SSH Key. To use this we'll need to add a Deploy SSH key to our private module. We can do this by going to Settings > Deploy keys</code> and adding a new key. We'll need to give the key a name and add the public key. The public key can be generated by running ssh-keygen -t ed25519 -C "your_email@example.com" -f github-actions</code>. This will generate a new SSH key pair. We can then add the public key as our Deploy Key. The private key will be used in our GitHub Actions workflow. Adding the private key to our GitHub Actions should be done using GitHub Secrets. We can add the private key to our GitHub Secrets by going to our organisation settings Settings > Organizations</code> then selecting our organisation then to Secrets and variables > Actions</code>. We can then add a new secret with the name GO_MODULE_PRIVATE_KEY</code> and the contents of the generated private key. Now that we have our private key configured we can add it to our GitHub Actions workflow. We can do this by adding the following step to our workflows jobs: - name: Add SSH Go Module Private Key env: SSH_AUTH_SOCK: /tmp/ssh_agent.sock run: | mkdir -p ~/.ssh ssh-keyscan github.com >> ~/.ssh/known_hosts ssh-agent -a $SSH_AUTH_SOCK > /dev/null ssh-add - <<< "${{ secrets.GO_MODULE_PRIVATE_KEY }}" echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> $GITHUB_ENV</code></pre> In this snippet you can see the use of ssh-add</code> to add the private key to the SSH agent and the use of echo</code> to add the SSH agent to the environment. This will allow the Go tooling to use the private key to access the private module. The next thing we will have to add now is the Git configuration making Git and the Go tooling use SSH instead of HTTPS. We can do this by adding the following step to our workflow: - name: Setup access for private go modules run: | git config --global url."ssh://git@github.com/".insteadOf https://github.com/</code></pre> Now that we have all of the necessary configuration we can use our private module in our GitHub Actions workflows. Here's a full workflow example: name: Go Vet & Test on: pull_request: jobs: test: name: "Test" runs-on: ubuntu-20.04 steps: - uses: actions/checkout@v3 - name: Set up Go uses: actions/setup-go@v4 - name: Add SSH Go Module Private Key env: SSH_AUTH_SOCK: /tmp/ssh_agent.sock run: | mkdir -p ~/.ssh ssh-keyscan github.com >> ~/.ssh/known_hosts ssh-agent -a $SSH_AUTH_SOCK > /dev/null ssh-add - <<< "${{ secrets.GO_MODULE_PRIVATE_KEY }}" echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> $GITHUB_ENV - name: Setup access for private go modules run: | git config --global url."ssh://git@github.com/".insteadOf https://github.com/ - name: Run tests run: go test</code></pre> Of course you can make your workflow do more than just run tests. You can use this workflow to build your Go application, deploy to a server or even create a Docker image. Conclusion</a></h2> We explored how to configure local development and GitHub Actions to use private Go modules. We looked at how to configure the GOPRIVATE</code> environment variable and how to configure the Go tooling to use our GitHub credentials. We also looked at how to configure GitHub Actions to use the same configuration as our local development environment. Understanding Go's sync.Cond Unknown — Fri, 31 Mar 2023 22:30:00 +0000 Exploring sync.Cond from the Golang sync Package</a></h2> After a colleague shared a video</a> with me about the Go programming language reviewing the use of Go Generics and the Cond</code> type from the go standard library sync</code> package. The code being reviewed can be found on Github here</a>. I decided to take a closer look at the usefulness of sync.Cond</code> in managing synchronization in concurrent programs. Before watching the video, I had never used sync.Cond</a> but had experience using the other types of the sync</code> package. Intrigued, I started to research more about it and discovered how beneficial it is for preventing race conditions and deadlocks. For Golang developers, the sync</code> package is a popular and helpful tool for synchronizing concurrent operations. The sync.Cond</code> type is specifically designed to help developers synchronize shared resources and is particularly useful when dealing with heavily loaded concurrent systems. It does so by allowing goroutines to pause their execution until a particular condition is met, enabling other goroutines to carry on uninterrupted until the condition is satisfied. Using sync.Cond</code> is surprisingly easy. It works in a very similar manner to sync.Mutex</code> which is more common in Go projects. I think for me a particularly intriguing use case is concurrently accessing and appending items into a slice. For example, say you need a particular number of items from a slice to perform a particular operation or within a particular window say the last 5 seconds then sync.Cond</code> is very useful for this kind of work. Firstly to create a new sync.Cond</code> we can use the sync.NewCond</code> method which takes a parameter that implements the Locker</a> interface. If we are to use a type from the sync</code> package to satisfy the Locker</code> interface then we'll either want to use sync.Mutux</code> or sync.RWMutux</code>. Let's now look at how we can implement this for our example. // List stores T in a slice and provides a way to get the last n items from the list type List[T any] struct { items []T cond *sync.Cond } // NewList creates a new list of T func NewList[T any]() *List[T] { return &List[T]{ items: []T{}, cond: sync.NewCond(&sync.Mutex{}), } }</code></pre> In this generic example, we create a List</code> that can store any type. Next, we need to look at how we can add new items to the list. When using sync.Mutux</code> we need to use the underlying Locker that was passed into the NewCond()</code> call. // Add adds a new item reading to the list func (l *List[T]) Add(item T) { l.cond.L.Lock() defer l.cond.L.Unlock() l.items = append(l.items, item) l.cond.Broadcast() }</code></pre> A call to Lock()</code> is called immediately when we call the Add()</code> function followed by deferring a call to Unlock()</code>. This is still essential as we want to modify the items slice safely. What is different in this Add()</code> function is the call to Broadcast()</code>. The call to broadcast signals all waiting goroutines that they can now wake. Preventing race conditions and deadlocks is essential for developing reliable concurrent programs. A race condition occurs when two or more goroutines access and modify the same shared resource without proper synchronization, leading to unpredictable behaviour and data corruption. The way sync.Cond</code> can broadcast to waiting for goroutines. On the other hand, a deadlock happens when two or more goroutines wait for each other to release a resource, resulting in a situation where none of them can proceed. Both issues can cause programs to crash or behave incorrectly, making them difficult to debug and maintain. Therefore, preventing race conditions and deadlocks is critical for building robust concurrent programs. We can look at how we might implement a Get</code> call that can retrieve a given number of items from the list. // Get returns the last n items readings func (l *List[T]) Get(ctx context.Context, count int) ([]T, error) { l.cond.L.Lock() defer l.cond.L.Unlock() var cancel func() ctx, cancel = context.WithCancel(ctx) defer cancel() go func() { <-ctx.Done() l.cond.Broadcast() }() for len(l.items) < count { l.cond.Wait() if err := ctx.Err(); err != nil { return nil, err } } temps := l.items[:count:count] l.items = l.items[count:] return temps, nil }</code></pre> The most interesting part of this function is the for loop. This loop blocks the function from going any further if there are not enough items in the list to satisfy the caller. If there are not enough items then the function is blocked by the call to Wait()</code>. This call will block until another goroutine signals it by calling either Signal()</code> or Broadcast()</code>. Some interesting additions in the Get</code> code example above are the use of a cancellable context to provide cancellation and a way to prevent goroutines deadlocking. In conclusion, the sync.Cond</code> is an incredibly useful tool for synchronisation with shared resources in highly loaded concurrent systems. A full working copy of the example above can be found here</a>. The never-ending cycle of home DIY projects Unknown — Wed, 29 Mar 2023 22:00:00 +0000 There's nothing quite like starting a new DIY project, but it's not always as straightforward as you may think. There is always a constant worry that a new project may go wrong which could turn it into an even bigger project and uncover more underlying issues. It all starts straightforwardly enough. You have an idea for a new project, you dive headfirst into it, full steam ahead. But as you get into the project, and you start uncovering issues and problems you did not anticipate. Maybe the previous property owner opted to take shortcuts during their DIY projects. Before you know it you have identified more work that was not originally planned. The original project you set out to complete has turned into a monster of endless home improvement projects. To avoid falling into this vicious cycle, it's important to approach any new project with a realistic approach and be prepared that it might not be as simple as you first thought. Don't bite off more than you can chew, take the time to evaluate the project and plan appropriately. One of the biggest mistakes DIYers make is underestimating the time required to complete a project. Don't be fooled by the seemingly simple nature of a task - take the time to realistically estimate how long it might take to complete it and be prepared for the reality that it may take longer. If a project begins to spiral out of control or you find it too daunting to handle on your own, don't hesitate to ask for help. Whether it's from a friend or a professional, getting an extra set of hands can make all the difference. It can save you time, money, and the headache of a never-ending cycle of home DIY projects. While DIY projects can be exciting, they can also lead to a never-ending list of projects if not approached with care. To avoid getting caught in this cycle, take a realistic approach and plan carefully. Try to estimate the time as accurately as possible, and when needed, seek help. With planning and perseverance, you can complete your DIY projects with confidence, hopefully avoiding the never-ending cycle of home improvement projects. Hopefully, with good planning, you avoid the never-ending cycle of home improvement projects. Learning Rust Unknown — Sun, 19 Mar 2023 16:30:00 +0000 I previously wrote a blog post about rust</a> back in June 2022 and I've still not really managed to spend a lot of time learning it. Busy personal and work life makes it difficult sometimes to focus on new projects and hobbies. But after a recently set of power outages I figured I needed a new dead man switch</a> type alert system to notify me when my personal home server hasn't been in contact after a specificed amount of time. Whether I have lost power or internet it would be useful for me to be notified when that might happen. I figured I needed a project to help me focus my efforts. The next thing I needed was something to follow. What better use of AI than asking for an course outline for learning Rust. ChatGPT Response</a></h2> The response I got from ChatGPT is as follows: Step 1: Getting Started</a></h3> Install Rust and set up your development environment</li> Learn the basics of the Rust programming language, including its syntax, data types, control flow, and functions</li> </ul> Step 2: Intermediate Concepts</a></h3> Dive deeper into Rust's ownership model, including borrowing and lifetimes</li> Learn about Rust's built-in concurrency primitives, such as threads and channels</li> Explore Rust's standard library and learn how to use crates from the Rust community</li> </ul> Step 3: Advanced Topics</a></h3> Learn about Rust macros and how to use them to write more concise and expressive code</li> Explore Rust's unsafe features and learn how to use them safely and correctly</li> Learn about Rust's advanced features, such as trait objects and higher kinded types</li> </ul> Step 4: Projects</a></h3> Build small Rust projects to practice and reinforce your knowledge of the language</li> Join the Rust community and contribute to open source Rust projects Build larger Rust projects and gain experience working with Rust's advanced features and tools</li> </ul> Step 5: Continued Learning</a></h3> Read Rust documentation and books to stay up-to-date with new developments in the language</li> Attend Rust conferences and meetups to network with other Rust developers and learn from their experiences</li> Practice coding challenges and participate in coding competitions to improve your Rust skills.</li> </ul> Getting Started with the course</a></h2> Not a bad course it even included a statement at the end telling me to take regular breaks and to avoid burnout. Learning a new language will be a challenge but with persistence and practice it'll hopefully not take me too long to learn. Just write Unknown — Sun, 19 Mar 2023 16:00:00 +0000 Robert Birming</a> recently wrote an article named Just Write</a> and it got me realising that to improve my own writing I need to "Just Write". I need to not care about who I am writing for, what I am writing for and why I am writing. I shouldn't care about my grammar, spelling and other mistakes that I make I think this is often a reason I use to not write and release blog posts. I get scared of what people might think or the mistakes I might write. The Rust programming language Unknown — Sun, 26 Jun 2022 00:00:00 +0000 While working with Golang</a> often read about Rust</a> as an alternative. Looking Rust is a low-level multi-paradigm, general-purpose statically-typed programming language designed for performance and safety. Rust makes it possible to write software that is safe and performant. One of Rust's biggest differences compared to other languages comes from the way it deals with objects and memory. Rust can check at compile-time whether the programmer has performed any incorrect operations on objects which could then to segmentation faults. Ownership is a set of rules that governs how a Rust program manages memory. If the programmer violates any rules, the program will not compile. Ownership rules are checked at compile time resulting in no slow down while your program is running this helps make Rust one of if not the fastest programming language. Along with other cool features and tools Rust has a lot of things that intrigue me and I wanted to try it out. Next steps for learning Rust for me is to learn more about the the language features work and the Rust ecosystem and tools. Couple of topics that I will need to do deep dives into are: Ownership, Cargo, Variables, Control Flows, Smart Pointers and plenty more. Cleanup Unknown — Wed, 17 Nov 2021 20:00:00 +0000 Often end up finding I start to run out of space on my filesystem after a few months of use so I needed to find a way to search for directories that I know I can safely delete. For example for Node.js</a> projects often end up with very large node_modules</code> directory. The following command will list all node_modules</code> from the current directory and there size. # List all directories named node_modules and their sizes $ find . -name 'node_modules' -type d -prune -print | xargs du -chs</code></pre> It always wroth running the command above first to confirm that all the directories it finds are correct. Once validated the directories are safe to delete the command can then be altered to delete these directories and all the files and directories under them. # Remove all node_modules folders (from current directory down) $ find . -name 'node_modules' -type d -prune -print -exec rm -rf '{}' \;</code></pre> Doing this can for node_modules</code> and other similar folders like vendor</code> for PHP projects can help save lots of space. Dot dev Unknown — Sun, 24 Mar 2019 00:00:00 +0000 Like many others, I recently rushed to get hold of one of the brand new dot dev</a> domains waiting for them to become available via Google Domains</a>. I didn't really have a plan for what I'd do with whatever domains I picked up but just knew I wanted aran.dev</a> and aranw.dev</a>. I've been using aranw as my go-to username for a few years now along with some others but I wanted to be the first to get aran.dev and use that as my main domain. After a bit of drama with Google Domains</a> going down and also charging me twice, I ended up getting the domains I wanted. Great! Now, what do I do with them? I've tried a few times in the past to do a blog and always had issues, whether it's related to tooling, hosting and/or commitment to writing on it, I wanted to try again. So here is my latest website, and blog. I've decided to use aran.dev</a> as the primary domain and other domains I own will ultimately get configured to redirect to this domain. For technology, I've gone super simple and decided to use Hugo</a> to statically generate the content with hosting provided by Netlify</a>. This has to be probably the simplest solution I've done and the easiest. At the moment I've made this blog closed source but will ultimately make it open source on my GitHub</a>. About Unknown — Sat, 23 Mar 2019 17:38:20 +0000 Hey, I'm Aran. I'm a software engineer based in Huddersfield, UK</a>. I mostly work in Go, building backend services, data pipelines, and event-driven systems. I've spent a lot of time with Google Cloud (Cloud Run, BigQuery, Pub/Sub, Cloud Functions), PostgreSQL, and Kafka. Most of my work involves helping teams ship reliable systems that handle real traffic, whether that's greenfield builds or sorting out existing architectures that have grown complicated. I've worked across a few different industries, which has taught me that the interesting problems are usually more about how teams work together and how data flows through systems than about picking the right framework. Greenfield projects are where I'm happiest. Starting from scratch with event-driven designs, getting the data model right, and building something that can grow without falling over — that's the work I find most satisfying. I've written about some of this on the blog</a>, covering everything from Go patterns to PostgreSQL partitioning to real-time data processing. I also take on freelance work from time to time — if you've got a project that sounds interesting, I'm happy to chat. Have a look at my services</a> page, or just book a call</a>. Outside of work, I make pizza (proper dough, proper oven), cycle when the weather cooperates, and do an unreasonable amount of DIY on a late Victorian house that always needs something.

Aran Wilkinson

Resume

Introducing Headcode: A Unified API for UK Rail Data

Introducing jjw: a workspace manager for jj

Why I built a reference Go project (and taught an AI agent to clone it)

Now

How I lost a database and learned to actually use AI

Services

Get in touch</h2> If you think I could help with something you're working on, the easiest way to start is to book a short call — no commitment, just a conversation about what you need and whether I'm the right person for it.</p>

Using CUE to unify IoT sensor data

Building an MQTT client in Go

Go 1.25's new JSON encoding package

Go 1.24's new tool directive in go.mod

Building an HTTP server in Go with zero dependencies

Processing UK rail data in real-time

Engineering arrogance

Building with the Google Sheets API in Go

The Google Sheets API opens up interesting The patterns we've covered here

Initial Feature Set</a></h2> For the initial release, I'm focusing on:</p> Stripe integration</li> Google Sheets integration</li> Scheduled syncing</li> Real-time syncing</li> Reliable data consistency</li>

What's Coming Next</a></h2> While I'm focused on getting the core functionality rock-solid, I'm already planning:</p> Additional payment provider integrations</li> More destination options</li> Advanced filtering capabilities</li>

Early Access</a></h2> I'm getting close to launch and will be looking for early users soon. If you're interested in:</p> Testing the integration</li> Providing feedback on features</li>

PostgreSQL table partitioning with Go

Benefits of Partitioning</a></h2> Queries can quickly find relevant data by skipping partitions that don't match the search criteria, leading to better performance across large datasets</li> Drop a partition. That's all it takes to remove old data</li>

YAML configuration: an unexpected bug

YAML configuration: the best of a bad bunch

How It Works</a></h4> $ENV_NAME</code>: This is replaced by the corresponding environment variable's value, or an empty string if not set.</li> ${ENV_NAME}</code>: This is replaced by the corresponding environment variable's value, or an empty string if not set.</li>

Deploying NixOS on Proxmox as an LXC container

Why Go uses composition over inheritance

Devlog 2 - Eventbus and SaaS model

The Google Sheets API opens up interesting
The patterns we've covered here