A few days ago I wrote about how I added OpenTelemetry to my Moodle and sent the traces to a Jaeger at home. At the end of that post I hinted that the natural next step was Grafana: going from the tree (a single trace) to the forest (the trends). Well, I built it. And it got out of hand in the good way: it ended up being the single pane for my whole infrastructure, not just Moodle.

This post isn’t a copy-paste walkthrough. It’s the map: the architecture, the pattern that ties it all together, and the scare in each layer —which is what actually teaches, because every piece had a surprise waiting for me—. If you self-host things, I think you’ll get something out of it. And if you want more detail on any part, I’d be happy to walk you through it: drop me a line anytime.

The idea: one pattern, repeated four times #

All modern self-hosted observability revolves around a stubbornly simple architecture:

[something that produces data] → [Prometheus stores it] → [Grafana draws it]

In the middle-left goes an exporter: a process that exposes metrics in a plain-text format Prometheus understands. There are official exporters for almost everything (the host, databases, queues…), and when there isn’t one, you write it yourself in twenty lines —that last part turned out to be the key to half the project, as you’ll see—.

I already had Prometheus and Grafana running from the previous post (feeding the Moodle metrics). So “adding all the infra” was really just repeating that pattern four times, once per thing I wanted to see:

1. My Moodle traces turned into metrics.
2. The host and the containers: CPU, RAM, disk.
3. Uptime and certificates for all my sites.
4. The real traffic Cloudflare sees.

And on top of everything, alerts to Telegram. Let’s go layer by layer.

Layer 1 — From traces to metrics (and the span scare) #

From the previous post, Moodle’s traces were reaching an OpenTelemetry Collector. The Collector has a lovely trick: a connector called spanmetrics that, without the app emitting a single metric, manufactures RED metrics (requests, errors, duration) from the traces themselves.

1connectors:
2  spanmetrics:
3    histogram:
4      explicit:
5        buckets: [5ms, 25ms, 100ms, 250ms, 1s, 5s, 10s]

The Collector sends the traces to Jaeger and the derived metrics to Prometheus. A p95/p99 latency dashboard for free. Lovely… until I looked at the p99.

The scare: the p99 was pinned at 10 seconds, in red. Panic. Do my pages take ten seconds? No. Two things: 10s was my histogram’s ceiling, and up there were Moodle’s cron tasks (which run in the background and do take seconds), mixed in with the user requests. I tried filtering by span kind (SERVER) and… nothing. It turns out Moodle marks a page’s span as INTERNAL, not SERVER (it reserves SERVER for webservice calls). The right filter was by name:

1histogram_quantile(0.95,
2  sum by (le) (rate(duration_bucket{span_name=~"(GET|POST) .*"}[5m])))

Filtering like that, my pages’ real p99 dropped from 10s to 99ms. Cron, to its own panel. Lesson: a metric without context lies; until you separate “what a user suffers” from “background work”, the number fools you.

Layer 2 — The host and the containers (two macOS traps) #

For machine metrics, the standard is node_exporter. And here, the first trap of running this on a Mac: node_exporter in a container only sees Docker’s virtual machine, not the real macOS. So I installed it natively, with Homebrew, so it reports the Mac mini’s real CPU, RAM and disk. Prometheus reaches it via host.docker.internal:9100.

And the second trap: I wanted to see how much each container uses (Jitsi vs Jaeger vs Grafana…). The standard is cAdvisor, but on OrbStack cAdvisor doesn’t expose the container name, just an unreadable cgroup id. Dead end with the “correct” tool.

This is where the pattern that saved half the project comes in: node_exporter’s textfile collector. You give it a directory, and node_exporter publishes as metrics any text file you drop there. Meaning: if I can generate the data with a script, it’s already a Prometheus source, without writing a real exporter.

1# A script that runs every 30s and writes per-container metrics:
2docker stats --no-stream --format '{{.Name}} {{.CPUPerc}} {{.MemUsage}}' \
3  | ... turn into: docker_container_mem_bytes{name="jaeger"} 275251200

node_exporter picks it up, Prometheus scrapes it, and suddenly I have CPU and memory of each container by name. When the canonical tool doesn’t fit, the textfile collector is your wildcard.

Layer 3 — Is everything up? (uptime + certificates) #

To know if my sites respond from outside, blackbox_exporter: it probes a URL and exposes whether it responded, how long it took, and when its TLS certificate expires. Prometheus hands it the list of sites and it probes them one by one, going out to the internet and back through the tunnel —so it’s an end-to-end check of the real path—.

The scare: two sites showed as “down” that weren’t. One redirects to a protected page that returns 401; another returns 302. Blackbox’s default module only accepts 2xx as “alive”. But a 401 means the server responded (it’s up, just protected). The fix was a more permissive module:

1http_uptime:
2  http:
3    valid_status_codes: [200, 301, 302, 303, 307, 308, 401, 403]
4    follow_redirects: false

Plain-words lesson: “alive” and “healthy” aren’t the same thing. For honest uptime you want “did it respond with anything?”, not “did it respond with a 200?”.

Layer 4 — The real traffic (and a deprecated dataset) #

Everything above measures my origin: what reaches the Mac mini. But in front of it is Cloudflare, which serves a lot from its cache and sees all the traffic —bots included— that my server never even sees. That picture comes from Cloudflare’s analytics API.

I tried two community Cloudflare exporters. Both failed with a permissions error… which had me blaming my token for a while. The token was fine. The problem: the exporters query httpRequests1mGroups, a dataset Cloudflare has deprecated and that the Free plan no longer serves. I confirmed it by asking the API by hand until I found the current dataset, httpRequestsAdaptiveGroups, which does work.

So I went back to the wildcard: a twenty-line script that queries the good dataset over GraphQL and writes the result to the textfile collector.

1{ viewer { zones(filter: {zoneTag: "..."}) {
2    httpRequestsAdaptiveGroups(limit: 5000, filter: {datetime_geq: "..."}) {
3      count sum { edgeResponseBytes } dimensions { cacheStatus edgeResponseStatus }
4} } } }

And with that, a real-traffic dashboard per domain: requests, cache ratio, bandwidth and status codes. No new container. Lesson: before fighting with someone else’s tool, check whether you can talk to the source directly. Often it’s twenty lines.

The cherry — Telegram alerts (without duplicating) #

Pretty graphs are nice, but what makes this useful are the alerts. Grafana alerts natively: you define rules (disk > 90%, a site down, a certificate under 14 days, latency spiking) and route them to a contact. Since I already had a Telegram bot warning me about server things, I reused that same bot: zero new credentials.

And here a detail I almost missed: my old monitoring scripts already warned about some of those things (disk full, web down). If I did nothing, I’d get the same alert twice. The fix wasn’t to turn the scripts off —they do other useful things— but to surgically divert only the alerts Grafana now covers, leaving the rest intact. So each warning arrives once, from the place that gives the most context.

The blind spot I’m left with #

Writing this, I noticed an uncomfortable crack: Grafana, Prometheus, the monitoring scripts and the very Telegram bot that fires the alerts all run on the Mac mini. That is, on the same machine they watch. And there’s a principle worth burning into memory here: a system can’t reliably report its own death. If the Mac mini goes down completely —a power cut, the home network, a kernel panic, the tunnel— no alert reaches me, because whoever should warn me went down with it.

The temptation is to say “well, I’ll move all this to another machine”. But I think that would be overkill. Most of what I measure —host metrics, per-container metrics, Moodle traces, origin traffic— only makes sense running next to what it observes; pulling it out would mean opening ports, exposing metrics over the network and keeping a second server up 24/7 to watch things I already watch fine locally. 95% of the value (performance, trends, degradations) is perfectly covered from the inside.

The only thing that genuinely needs an external, independent observer is the binary question: is the Mac mini as a whole alive?. And for that you don’t need to replicate the stack, a small watcher outside the house is enough. The two options I’m weighing: a dead man’s switch (the Mac mini sends a heartbeat every few minutes to an external service, and it’s that service that warns me if the heartbeat stops arriving) or an external uptime check that probes the site every few minutes. Cloudflare already serves an archived copy if the origin falls, but it doesn’t warn me; that warning is exactly the gap that remains.

I haven’t decided which to build yet —or even whether it’s worth it for now—. But I had to write it down, because it’s the most honest lesson in all of this: no matter how pretty the dashboard looks, it’s still watching itself from the inside. The last mile of observability, the one that confirms the machine hasn’t died, has to live somewhere else.

What I take away #

Three things, after having touched all four layers.

First: the pattern is always the same —exporter → Prometheus → Grafana— and once you internalize it, “adding a new source” stops feeling like a chore. Second, and the most practical: the textfile collector is an underrated wildcard. Half of this build (containers by name, Cloudflare traffic) didn’t come from the “official” tool, but from a twenty-line script dumped to a text file. If you can generate the data, it’s already observable.

And third, the ever-present one when you self-host: every layer keeps a scare for you. The p99 that lied, cAdvisor with no names, the 401 that looks like an outage, the deprecated dataset. None of them were in the happy docs. But each scare, once understood, is exactly what separates “I followed a tutorial” from “I know what I’m doing”.

Now I have a single place —behind a login— where I see Moodle, the host, each container, whether my sites are up, how much real traffic I get and when a certificate expires, and where a Telegram alert fires if something goes sideways. Exactly the “forest” I was missing. From the tree to the forest, yes, but through a fair few scares along the way.

Update (Jul 11) — the tank with no drain #

A day after publishing this, looking at Grafana itself, I noticed something: the Jaeger container’s memory only went up, never down. And it made perfect sense: Jaeger all-in-one stores traces in memory by default, with no limit and no expiry. Every trace that comes in stays until you restart the container. A tank with the tap open and no drain: sooner or later it eats everything.

I fixed it by switching to Badger, Jaeger’s embedded database, which writes to disk with a 72-hour TTL. Now traces persist across restarts —before, a docker restart wiped them all— and expire on their own after three days, so neither RAM nor disk grows without bound.

The lesson is very much in line with the rest of the post: defaults are meant for trying things out, not for leaving them running for weeks. all-in-one with in-memory storage is perfect for a while; for something that runs continuously, you have to give it a drain. And note the irony: it was the Grafana dashboard itself that gave the problem away. Which is exactly what I built it for.