The Tool That Was Already There
In 2004 three engineers wrote down what an observability tool must guarantee before anybody may switch it on in production. The Unix line got one, and Linux eventually built its own. The dashboard still gets asked first.
About This Page
This knowledge graph companion was generated from Vivian Voss's article "The Tool That Was Already There" using the kg-generator skill's Business & Market Analysis RDF-Turtle template, then rendered as this HTML infographic using the rdf-infographic-skill. Entity links throughout resolve via the URIBurner SPARQL/DAV resolver against the companion Turtle file.
Technology Stack:
- AI Agent: Claude Code
- Skills: kg-generator, rdf-infographic-skill
- Language Model: Claude Sonnet 5
- Server Platform: Virtuoso
- Knowledge Graph: URIBurner
A Production System Behaves Oddly
Response times have grown a second hump, and nobody knows what from. In most shops somebody opens a web interface, hunts for the right time window, lays two curves on top of one another and begins to guess. If the quantity is not among them, a ticket gets written so it will be emitted from the next deployment onwards — an answer in a fortnight, with luck.
On that same machine sits a tool that could settle the question in forty seconds. It is installed. It costs nothing. It does not ask whether anybody instrumented a field in advance.
What Was Written Down in 2004
In 2001 three kernel engineers at Sun Microsystems — Bryan Cantrill, Michael Shapiro and Adam Leventhal — began DTrace. Twenty-three months to first integration put it at 2003; it shipped with Solaris 10 in March 2005. A year before that, their USENIX 2004 paper, Dynamic Instrumentation of Production Systems, specified the terms.
| When Enabled | "in a unified and absolutely safe fashion" — not handle with care, not test systems only |
|---|---|
| When Disabled | "zero probe effect" — exactly as if DTrace were not present at all |
| How Much | ~30,000 instrumentation points in the kernel alone, on even the smallest system |
Cantrill later put it more casually: if you run a tool and the system dies as a result, you will never be allowed to run that tool again.
Not a Feature, a Decision About Where Things Live
A tool that costs nothing while switched off can sit everywhere — that was precisely the intention. DTrace has been part of the base system since Solaris 10, and wherever it travelled that property travelled with it: FreeBSD took it in, macOS has it. In neither case is it a product you buy; it is something you type — the Unix way of building things, a capability from which you assemble the question yourself.
The Other Answer
DTrace was published under the CDDL; the Linux kernel is GPLv2. The two licences do not sit together, so Linux built its own answer. The Berkeley Packet Filter (1992, Steven McCanne and Van Jacobson) was extended in 2014 by Alexei Starovoitov and Daniel Borkmann into eBPF — ten registers, a 512-byte stack, a JIT compiler, and a verifier — merged into Linux 3.18.
On top sits bpftrace, written by Alastair Robertson with Brendan Gregg among its principal contributors.
What Happens When Somebody Measures Both
Mateusz Piotrowski benchmarked DTrace on FreeBSD against eBPF on Linux at AsiaBSDCon 2024, and concludes it is not possible to say which system is faster — any such comparison "would be unfair and misinformed." One aside: during early experimentation bpftrace took roughly ten minutes to print its collected statistics and quit, where DTrace, when signalled, was done in a few seconds — an open question, not a verdict.
The Bill
| Offer | Price |
|---|---|
| Infrastructure monitoring | $15 per host, per month |
| Application performance monitoring | $31 per host, per month |
| Log ingestion | $0.10 per ingested GB |
| Log indexing | $1.70 per million events |
| The kernel tool | nothing — already installed |
On a Datadog earnings call on 4 May 2023, an analyst worked out from a remark by the finance chief that a large non-recurring bill must be about $65 million. Gergely Orosz went after it and had it confirmed: Coinbase, for the year 2021.
The Limit
A local tool answers local questions — nothing about fleet aggregation, months of history, or following one request through seven services. That skill decays without practice. And dtrace -w is a destructive mode that can modify a running kernel: separately gated, but it exists.
The Point
When the next service misbehaves, the first question is not whether the quantity is in the dashboard. The first question is whether anybody in the room can drive the tool that is already there. If somebody can, the answer costs forty seconds. If nobody can, it costs a fortnight and a line item on an invoice with no known ceiling.
What Happens When a Production System Behaves Oddly
Frequently Asked Questions
Glossary
Knowledge Graph Explorer
Explore Knowledge Graph using SPARQL
Query the companion RDF data using SPARQL. Use the default endpoint or copy queries to your own SPARQL client.