Building Systems, Breaking Things, and Learning in Public

For the past few years, I’ve spent most of my working life somewhere between building systems and figuring out why they broke.

Sometimes that means writing Go services.

Sometimes it means staring at Kubernetes metrics during a production incident.

Sometimes it means tracing a request through a load balancer, application, database, cache, and three dashboards only to discover that the actual problem was something completely unexpected.

And sometimes it means asking the most dangerous question in engineering:

“What happens if we change this?”

That combination of building, operating, debugging, and occasionally breaking things has taught me far more than I expected when I started my career.

So I finally decided to start writing some of it down.

A little about me

I’m Rahul.

I work primarily around cloud infrastructure, platform engineering, backend systems, and production reliability.

Over the years, I’ve worked with things like Go, Kubernetes, AWS, GCP, Terraform, distributed systems, observability, CI/CD, APIs, and event-driven architectures.

A large part of my experience has also been in video and live-streaming infrastructure, where failures become particularly interesting because thousands or sometimes far more people can be watching while you’re debugging the system.

Production has a wonderful way of exposing assumptions that looked perfectly reasonable in development.

A service that behaves beautifully at normal traffic suddenly develops terrible p99 latency.

A Kubernetes deployment looks healthy while users are timing out.

A database appears fine until a particular query pattern arrives.

A configuration change that should be harmless becomes the beginning of a very long evening.

Those are the situations I find the most interesting.

Why start a blog now?

There is an enormous amount of excellent technical content on the internet.

But a lot of engineering knowledge still lives in places that are difficult to search:

Slack threads.

Incident calls.

Pull-request discussions.

Debugging sessions.

Half-forgotten commands in shell history.

Or simply inside someone's head.

I’ve accumulated plenty of those lessons myself.

Writing forces you to turn:

“I kind of understand how this works.”

into:

“I can explain exactly how this works.”

That distinction matters.

So this blog is partly a place to share what I learn, but it is also a way of forcing myself to understand things more deeply.

What I'll write about

Expect a mixture of practical engineering and deeper technical exploration.

There will probably be plenty of:

Kubernetes and cloud infrastructure

Not just how to deploy something, but what happens when things stop behaving the way you expect.

Networking, scaling, resource limits, containers, EKS, infrastructure-as-code, and the strange problems that only seem to appear in production.

Go and backend engineering

Concurrency, APIs, event-driven systems, worker patterns, reliability, service design, and lessons from building backend tooling.

Go is a language I particularly enjoy because it makes it relatively easy to build simple systems and surprisingly easy to build complicated ones if you're not careful.

Production debugging and incidents

This might become my favorite category.

Things like:

Why is p50 latency fine while p99 is terrible?
Why are requests timing out even though the pods look healthy?
Is the problem the application, ingress, load balancer, database, network, or something else entirely?
What should you actually check first during an incident?

I want these posts to focus on how to think through the problem, rather than simply presenting the final fix.

Distributed systems

Retries.

Queues.

Idempotency.

Caching.

Partial failures.

Event ordering.

Backpressure.

Consistency.

All the things that seem straightforward until several machines have to agree about something.

Observability

Metrics tell you something is wrong.

Logs tell you what happened.

Traces tell you where it happened.

And occasionally all three confidently point you in the wrong direction.

I’ll write about Prometheus, Grafana, Datadog, OpenTelemetry and, more importantly, how to use observability to actually investigate systems.

AI-assisted engineering

AI is increasingly becoming part of how software gets built and operated.

I’m interested less in “AI will replace engineers” and more in the practical question:

How can engineers use these systems to become dramatically more effective?

That includes coding agents, troubleshooting systems, automation, developer tooling, and experiments that may or may not work.

The failed experiments are often worth writing about too.

What this blog won't be

I don't want this to become a collection of perfectly polished tutorials pretending that engineering always follows a clean sequence of steps.

Real engineering looks more like this:

Form a hypothesis.
Check the evidence.
Discover the hypothesis was wrong.
Form a better hypothesis.
Accidentally find another problem.
Fix the original problem.
Spend twenty minutes wondering why the fix worked.
Understand it properly.
Write the runbook you wish existed two hours earlier.

That's the part I want to capture.

The reasoning.

The mistakes.

The trade-offs.

And the little pieces of knowledge that make the next incident easier than the previous one.

Learning in public

There is another reason I'm starting this.

Technology moves ridiculously fast.

You never really finish learning Kubernetes, distributed systems, cloud architecture, Go, AI, or backend engineering.

There is always another layer.

And I think documenting that learning process publicly is more interesting than pretending to have everything figured out.

Some posts will be deep technical dives.

Some will be short notes about something that took me three hours to debug but three minutes to explain afterward.

Some might simply answer a question I wish someone had answered clearly when I first encountered it.

If one of those posts saves another engineer a few hours of staring at logs at 2 AM, then this blog has already done its job.

So, let's build

I have no idea what this blog will look like a few years from now.

That's part of the fun.

For now, I'll use it to document what I'm building, what I'm learning, what breaks, why it broke, and what I understand afterward that I didn't understand before.

There will be code.

There will be architecture.

There will definitely be Kubernetes.

And statistically speaking, there will probably be at least one DNS problem.

Welcome to the blog.