This is the third post in a series I never planned as a series. First I added OpenTelemetry to my Moodle and sent the traces to a Jaeger at home. Then Grafana ended up being the single pane for my whole infrastructure. In neither did I talk about logs —and it turns out that was the part with the most story behind it—. If you self-host things, I think you’ll get something out of this one; and if you want more detail on any part, drop me a line anytime.

The red error that started it all #

One day, looking at my Moodle’s error log, I found this repeated to infinity:

OpenTelemetry: [error] Export failure [exception] Not Found

Not once. On every event. Every login, every page view, every action triggered that error. The instrumentation was trying to export something and getting a 404 back, over and over.

Pulling the thread, the message led me to an unexpected piece: it wasn’t a traces failure (those were flowing perfectly to Jaeger), it was a logs exporter. Moodle’s new instrumentation package, on top of tracing, emitted a log for every event and sent it to an endpoint (/v1/logs) that didn’t exist in my collector. Hence the 404 loop.

I could have silenced it in two lines. But the error was right: I was missing an entire layer of observability.

The third pillar #

Modern observability rests on three kinds of signal, and each answers a different question:

• Traces — where did a request go and where did it slow down? (Jaeger, from the first post).
• Metrics — how much is happening, in aggregate? (Prometheus + Grafana, from the second).
• Logs — what happened, exactly, with names attached?

I had two of three. The red error was, literally, my Moodle trying to hand me the third one while I had no one ready to catch it. So instead of muting the warning, I built the recipient: Loki.

Building Loki (with a drain from day one) #

Loki is to logs what Prometheus is to metrics: it stores them and makes them queryable from Grafana, with its own language (LogQL). You spin up a container and you’re done.

Except this time it came with a lesson learned the scary way. In the previous post I told how Jaeger, with its default storage, was a tank with the tap open and no drain: memory only went up. I wasn’t going to repeat that. So Loki was born with retention from its very first config line:

1limits_config:
2  retention_period: 168h        # 7 days — the drain, from day 1
3compactor:
4  retention_enabled: true

And in the collector, the logs pipeline I was missing: it receives logs over OTLP and forwards them to Loki. With that, the /v1/logs that used to return 404 started answering 200, and my Moodle’s red error went away on its own. I didn’t silence it: I gave it somewhere to go.

1service:
2  pipelines:
3    logs:
4      receivers: [otlp]
5      exporters: [otlphttp/loki]

The scare: what looked like a label but wasn’t #

With logs already reaching Loki, I wanted the obvious thing: to filter them by event type. Every Moodle log carried a bunch of fields —event_data_eventname, event_data_action, event_data_target…— and I happily wrote the most natural query in the world:

{service_name="moodlelms", event_data_action="failed"}

Zero results. But the logs were right there; I could see them with a simpler filter. Half an hour of confusion until I hit the distinction Loki makes that isn’t obvious: there are indexed labels (few, for selecting streams) and structured metadata (everything else, arriving over OTLP). Those event_data_* weren’t labels, they were metadata. And the stream selector, inside the braces, only accepts indexed labels.

The right query doesn’t select by those fields, it filters them in the pipeline:

{service_name="moodlelms"} | event_data_eventname=~".*user_login_failed.*"

Lesson: in Loki’s response both kinds of field show up mixed together, so they look the same. They’re not. One selects, the other filters.

Alerts born from logs #

This is where logs stop being a pretty archive and become useful. That last filter —user_login_failed— is, literally, a failed login attempt. And if I count how many there are in a short window, I’ve got a brute-force detector almost for free:

sum(count_over_time({service_name="moodlelms"}
    | event_data_eventname=~".*user_login_failed.*" [5m]))

A rule in Grafana that fires when that goes above, say, 15 in five minutes, routed to the same Telegram bot that already warned me about disk and certificates. A user mistyping their password generates two or three; fifteen in five minutes is someone running the dictionary. I tested it the brutal way —eighteen failed logins in a row— and the alert hit my phone like clockwork.

With the same technique I built another for sensitive admin activity: alert me if a user is deleted or a role is changed. If it was me, I know; if it wasn’t me, I want to find out.

The lesson: what flows on its own and what you have to go dig for #

And here comes what I really take away from all this, beyond Loki.

Building the brute-force alert was easy. Why? Because a failed login is a Moodle event, and events were already travelling to Loki on their own thanks to the instrumentation. The data was there; I just wrote the question.

Shortly after I wanted an alert for cron tasks that fail. And I ran head-first into the opposite. When a Moodle task fails, it generates no event that reaches Loki, and it doesn’t mark its trace as error either (Moodle catches the exception internally and doesn’t let it out). The only place it’s recorded that “this task is failing” is a field in the database. For that alert neither Loki nor traces helped: I had to write a small script that queries the DB and publishes the value as a metric, and alert on that.

Two alerts, two worlds:

• Brute force → the data flowed on its own. Work: one query.
• Failed tasks → the data had to be manufactured. Work: a whole bridge.

The general rule, the one I’ve burned into memory: observability only sees what the application chooses to emit. Traces, metrics and logs give you a rich picture… of what the app publishes. What stays inside —like a task’s failure state— doesn’t magically appear on any panel: you have to go get it yourself. Knowing how to tell one case from the other is half the craft.

The complete trio #

Now, in a single Grafana, I have all three signals from my Moodle and from my whole infrastructure: traces to see where a request slowed down, metrics for trends, and logs to know what happened exactly, with names. And on top, alerts born from all three.

From the tree (a trace) to the forest (the metrics) to the detail (the logs). The trio the series was missing, and which —of course— started with a red error I almost silenced. Glad I didn’t.