Observability tooling is easy to buy and easy to misapply. A common end state is a large monthly bill, dashboards nobody consults, alerts everyone has muted, and an incident where the first twenty minutes go on establishing what is actually wrong. The problem is almost never volume of data. It is that the wrong things were measured, in a form that cannot answer questions.
Start with the four numbers that describe user experience: request rate, error rate, latency at the ninety-fifth and ninety-ninth percentiles, and saturation of whatever your constrained resource is. Percentiles rather than averages, because an average latency of 200 milliseconds is compatible with five per cent of users waiting eight seconds, and those are the users who leave. If you have only these four per service, you can already tell whether something is wrong and roughly where.
Then make logs structured. A log line that a human can read but a machine cannot query is only useful if you already know which server and which minute to look at - which is precisely what you do not know during an incident. Structured fields, with a request identifier propagated through every service and into the logs of each, is what turns an hour of grepping into a single query.
Distributed tracing is the highest-value addition once you have more than a couple of services, because it answers the question metrics cannot: which part of this request was slow. Sample intelligently rather than tracing everything - keep all the errors and slow requests, sample the rest - and instrument with OpenTelemetry so the data is not trapped inside one vendor.
Alerting is where most of the value is lost. Alert on user-visible symptoms, not on machine conditions: error rate above threshold, latency past the level users notice, queue growing faster than it drains. High CPU is not an incident if nobody is affected, and paging someone for it at three in the morning teaches them to ignore the pager. Every alert should have a documented action; if the honest response is to look and do nothing, it is a dashboard, not an alert.
On cost, be deliberate rather than surprised. Observability bills grow with traffic and with cardinality - the number of distinct label combinations - and a single high-cardinality label such as user identifier can multiply a bill overnight. Set retention by usefulness: fourteen days of detailed data and longer aggregates covers almost every real investigation, and the trace from six months ago that you kept has never once been opened.
