A Dropped Log Is Worse Than an Ugly One

Summary: The previous post ended with a request ID threaded through six services. That's only useful if you can read all six services' logs in one place, which is what this post is about.

The design turns on a single rule that I'd argue for in any logging system: the ingest endpoint never rejects an event because of what's in it.

A Dropped Log Is Worse Than an Ugly One

Four machines, twenty services, one question

By this point the lab had logs everywhere. Every service wrote its own, on whichever machine it happened to run on, in whatever format seemed reasonable at the time.

Answering "what happened when I asked that question" meant SSH to one machine, find the log, note the timestamp, SSH to another, find its log, try to line them up. It's tedious enough that you skip it, and skipping it means guessing - and I got quite good at confidently guessing wrong.

So: collectors on every machine, forwarding to one aggregator. The aggregator runs on the main server because that's the machine that owns the array and every backup tier, which means the logs land where the storage and the protection already are.

The rule

One decision drives everything else in this design.

The obvious way to build a log ingest endpoint is to validate. Define a schema, check incoming events against it, reject what doesn't conform. That's what you'd do for an API, and it's completely wrong here.

Think about when malformed events arrive. Not on a quiet Tuesday - they arrive when something is broken. A service crashing mid-write, a field that's null for the first time ever, an exception path that logs a different shape. The moments that produce weird log events are exactly the moments you most want the log.

A validating endpoint discards data precisely when the data is most valuable. It's a system that works beautifully when you don't need it.

So ingest never rejects for content. An event that doesn't fit gets coerced - made to fit as best it can - and flagged with a marker saying it was invalid on arrival. It's stored, it's queryable, and it's honest about being malformed.

That flag is what makes the compromise work. I'm not pretending the event was fine; I'm keeping it and labelling it. And searching for the flag turns out to be an excellent way to find bugs, because a service that suddenly starts emitting malformed events is usually a service that has started doing something else wrong too.

A dropped log is worse than an ugly one. It's the rule I'd take to any system that ingests observational data.

Hot and cold, because logs have two lives

Logs get used in two completely different ways.

Something just broke and I want the last few minutes, right now, grepped. Or: I want to know how latency has moved over three months, which is analysis over a large amount of data I'll never read line by line.

Those want different storage. So there are two tiers.

Hot is line-delimited JSON, partitioned by date, kept for about a month. Appendable, greppable with ordinary tools, no special software to read at 2 AM when something's wrong. That last property is worth protecting - a debugging format that needs a working query engine is one more thing that can be broken when you need it.

Cold is Parquet - columnar, compressed, quick to aggregate over. Closed days roll from hot to cold, which compacts them enormously and makes the long-range questions fast.

Date-partitioning the hot tier means retention is a directory operation. Deleting a month means removing directories, not scanning and filtering. Making the expensive maintenance operation into rm -rf of a folder is the sort of small structural choice that keeps a system running unattended for a year.

Enrich at the aggregator, not the collector

A design decision I got right by accident and would now defend on purpose.

Events get annotated with which network segment their host is on - useful context when you're trying to work out whether something is a connectivity problem.

That annotation happens at the aggregator, from a map it holds, rather than on each machine. Which means when the network changes - and in this lab it changes - I update one map in one place. If each collector enriched its own events, a network change would mean touching four machines' configuration and remembering that I had to.

The general principle: derive context where the knowledge lives, not where the data originates. The collector knows what it logged. The aggregator knows what the network looks like. Ask each for what it actually knows.

The one that couldn't reach

A wrinkle that's specific to home networks and worth including because it's the kind of thing that never appears in a reference architecture.

The wall of light panels can't be reached from the main server - it's on a different segment. So alerts destined for it are routed through the workstation, which can see both.

That's inelegant. It's also correct, because the alternative is re-architecting a home network to make a diagram tidier. Home networks are constrained by what the landlord installed and where the router happens to be, and I've come to think the right instinct is to accept the shape and route around it, in one clearly-commented place, rather than pretend the constraint doesn't exist.

What this gives me

The payoff is the one promised two posts ago: search for a request ID and see the whole path of a single spoken question, across four machines and six services, in order, with timing at each hop.

That capability changed how I debug more than any other single thing in the lab. Before it, investigating a slow response was thirty minutes of SSH and guessing. Now it's a search, and the answer is usually visible immediately - and usually not where I'd have looked.

It's also how I found several of the failures described elsewhere in this series. The half-empty index, the routing mistakes. Centralised logs don't fix anything by themselves, but they turn "something is wrong somewhere" into "this specific hop is doing this specific thing", and that's most of the work.

If you're starting

You don't need collectors and tiers and Parquet. You need one place.

If you have three machines, a syslog target or even a shared directory that everything writes into is most of the value. The magic isn't the technology, it's that there's one place to look and the timestamps are comparable. I ran a version of that for a long time before any of the above existed.

Then add one field to every log line: something that identifies the request. That's the whole trick. Everything else here is refinement.

And when you write the ingest, be permissive. Take the ugly event, mark it ugly, and keep it. You'll want it.

Next: the moment a sentence arrives and something has to decide what kind of question it is.

---

The lab so far

This post is part of an ongoing series — the home-lab build written down as it happens, in order:

1. Load-Bearing Docs: How a Homelab Learns to Stop Lying to Itself — Jun 30, 2026
2. 1.7 Tokens Per Second: How the Lab Got Its GPUs — Jul 2, 2026
3. Is Any of This Even Running? Building the Lab's Single Pane of Glass — Jul 4, 2026
4. Zero Redundancy: Backing Up a Striped Array I Chose On Purpose — Jul 6, 2026
5. Two Platforms On Purpose: systemd, k3s, and Refusing to Migrate — Jul 8, 2026
6. Body Language: Giving the Lab a Face on the Wall — Jul 10, 2026
7. An Operator Console, Not a Dashboard — Jul 12, 2026
8. From Wake Word to Soundbar: The Whole Voice Pipeline — Jul 14, 2026
9. A Dropped Log Is Worse Than an Ugly One (you are here) — Jul 16, 2026
10. Where Does This Sentence Go? The Gateway Problem — Jul 18, 2026
11. Mostly Arithmetic: Classifying Intent Without Training Anything — Jul 20, 2026
12. Teaching the Lab to Know Me: Journals Into Structured Data — Jul 22, 2026
13. The Index That Was Quietly Empty — Jul 24, 2026
14. Two Stages and a Refusal: Routing Inside the Memory — Jul 26, 2026
15. Looking Outward: A Small Data Lake for Public Sources — Jul 28, 2026
16. The Coach That Doesn't Use a Model — Jul 30, 2026
17. How Helen Came to Be: Four Voices and a State Machine — Aug 1, 2026
18. A Refusal a Model Can't Be Talked Out Of — Aug 3, 2026
19. Seven in the Morning: What Helen Says, and Why No Model Writes It — Aug 5, 2026
20. Helen Gets a Command Surface — Aug 11, 2026
21. Putting the Lab in a Chat Window: Agent Skills, Flows, and CI/CD for AnythingLLM — Aug 19, 2026

Before it was a series: earthquake data and local LLM reasoning, the Slack integration that followed, CLI vs Open WebUI on context, the AnythingLLM ADR, and the LLM-owned wiki pattern.

---

Credits

Hero photo by Ilya Pavlov on Unsplash.