ZIONN
Working together2 min read

What Cloudflare observability needs before an incident, not during one

Logs nobody reads are not observability. The alert that fires before a customer reports the outage is, and it has to be configured before the incident.

ZIONN EngineeringSoftware engineering team

Every Cloudflare account with observability turned on is collecting logs. Most of those logs are never read until after something has already gone wrong, at which point they answer "what happened," which is useful, but a full day late. The actual value of observability is the alert that fires before a customer opens a support ticket, and that alert has to exist before the incident, configured against a threshold someone thought about in advance.

The gap between "we have logs" and "we would know"

Having Workers Logs and Analytics Engine turned on answers one question: can we reconstruct what happened, after the fact. It does not answer the question that actually protects revenue: would anyone have noticed while it was still happening.

SignalCollected by defaultActually alerted on
Error rate on a critical endpointYes, in the logsUsually not, unless someone built it
A spike in 5xx responses from originYesSometimes, if a threshold was ever set
A WAF rule blocking legitimate trafficYes, as a log lineAlmost never — false positives are silent by design
A gradual latency increase over daysYes, in Analytics EngineRarely; gradual trends do not trip a simple threshold
Most accounts are strong on the left column and thin on the right one — which is backwards, since the right column is the one that catches an incident while it is still small.

The pattern in that table is not that the data is missing. It is present in every one of those rows. What is missing is the second step: someone deciding, in advance, what threshold on that data is worth waking a person up for.

Alerting has to be tuned, or it gets ignored

The failure mode on the other side is just as common: alerting on everything, which trains the team to ignore all of it within a month. An alert that fires every day for something that self-resolves is worse than no alert, because it burns the credibility of the alert that matters.

Building this before it is needed

The teams that get real value from observability are the ones who set the thresholds during a calm period, based on what a real incident actually looked like the last time, not the ones scrambling to add a dashboard during the outage itself — the same reasoning behind mapping a system properly during a two-week discovery before committing to what gets built.

We configure logging, alerting, and analytics as part of our Cloudflare implementation practice, sized to what a small on-call team can actually act on, not a dashboard for its own sake. How we handle incidents in our own delivery work is in our security practices.

Check your own case

Where does your team stand on this?

The article describes the pattern. These take your answers and tell you which part applies to you — no signup to see the result.

Related services

  • Cloudflare Implementation

    Security, integrity, monitoring, and performance, implemented directly on the platform your systems already run on.

    Learn more

Keep reading

All articles