Reading an Apex test failure like the engine does

What a failure discards

Consider the moment an assertion fails. The runtime is holding: the expected value, the actual value, both of their types, the exact source location, the call stack that got there, every SOQL query issued in this transaction and what each returned, every DML statement and how many rows it touched, the governor counters, and the state of every local variable in scope.

What comes out the other side is a string and a line number. Not because that is all anyone wants, but because that is the shape of the interface — an exception carries a message, the runner prints the message, and everything else falls on the floor as the transaction unwinds.

So you go and get it back. Add System.debug around the assertion. Re-run. Read the log. Realise the interesting value was three frames up. Add more debugs. Re-run. In an org that loop is measured in minutes per iteration, which is why Apex debugging feels like archaeology: you are excavating a state that existed, in full, a moment before the message you are staring at was produced.

The framing I have come to prefer is that this is not a debugging problem. It is a reporting problem. The information was never missing. It was discarded, and then you were asked to reconstruct it by hand.

The three things worth handing over

Not all of that captured state is equally useful, and dumping all of it would be its own kind of unhelpful. Three pieces do most of the work.

The operands. "Assertion failed" tells you a comparison did not hold. "Expected 999, got 1" tells you which direction and by how much, and those two facts usually pick the hypothesis for you. Expected 999, got 1 is a query that came back short. Expected 999, got 0 is a query that came back empty, which is a different bug. Expected 'Hot', got null is a field that was never populated. The shape of the mismatch is diagnostic, and it is exactly the part the message format tends to drop.

What ran just before. An assertion failure is the end of a story, and the story is the DML and SOQL that preceded it. Knowing that the insert ran, then the query ran after it, and the query is what came back short collapses a whole branch of the search tree — you can stop suspecting the insert. Each operation with its own file and line turns the failure from a point into a sequence.

Whether this ever passed. A test that has never passed is unfinished work. A test that passed an hour ago and fails now is a regression, and knowing whether the Apex source changed in between separates a regression from non-determinism. That is a genuinely different investigation, and you can be pointed at the right one before you start rather than after twenty minutes.

Evidence versus inference

Once a tool starts reporting more than a message, it acquires a responsibility it did not have before. A stack trace is unhelpful but honest — nobody mistakes it for a complete account. A richer report invites you to treat it as complete, and if it is not, it will quietly send you somewhere wrong.

The distinction that keeps this straight is simple to state and easy to violate: not observed is not the same as did not happen. If a report lists the operations that preceded a failure and the list is empty, there are two possible worlds. Either the test genuinely issued no queries, or the instrumentation did not capture them. Those demand opposite next moves. A report that renders both as an empty list has, in one of those worlds, told you something false.

The same trap sits under every absence. No previous passing run might mean the test never passed, or that history simply does not go back far enough. No recorded cause might mean the failure is inexplicable, or that the engine had nothing to attach. In each case the honest report says which — and it costs a line of output to do.

This is why the design choice I care most about in a failure report is a section listing what it could not establish. Not as a disclaimer, and not as legal hedging, but because it is the part that makes everything above it safe to act on. A tool that presents partial evidence as complete evidence does not merely fail to help. It actively sends you to fix the wrong thing, and it does so with more confidence than the stack trace it replaced.

Why text classification sends you to the wrong place

The problem changes character at scale. One failure is a debugging problem. Two hundred failures on the first run against an unfamiliar codebase is a different question entirely: how many distinct problems is this, actually? Nobody reads two hundred failures. They read the first three and guess.

The obvious move is to group failures by their message text, and it is the wrong move. The reason is that a message describes a symptom, and the mapping from symptoms to causes in Apex is many-to-many in both directions.

Take the most common one. A hundred tests fail with System.NullPointerException: Attempt to de-reference a null object. Grouped by text, that is one problem with a hundred instances, and it reads like a single afternoon of work. It is not. Some of those are a field the test never populated. Some are a missing stub or metadata definition that degraded to null instead of throwing. Some are a genuine null-handling bug in the code under test. Nothing in the message tells those apart — they produce byte-identical text — and merging them creates a group whose single "fix" does not exist.

The inverse error is just as costly. The same underlying cause surfaces with different messages depending on where it happens to bite, so text grouping splits one problem into six, each looking too small to prioritise. Either way the count is wrong, and the count is the entire reason you grouped in the first place.

The alternative is to group by the cause some part of the engine assigned while the failure was happening. A Test.loadData call that referenced a static resource the project does not contain is not a guess about a message — it is a fact recorded at the call site, by the code that went looking for the resource and did not find it. Same for a @testSetup method that threw, and for an assertion that did not hold. These are causes, not symptoms, and they group correctly because they were never inferred.

Which leaves the failures that carry no recorded cause. The temptation is to sort them into whichever group looks plausible, since a report with an unexplained tail feels incomplete. Resist it. Listing them individually is the honest output: each one is genuinely its own investigation, and folding them into a group hides them inside a count where nobody will look again.

Evidence that belongs to the run, not to a test

There is a third category, and it is the one most reporting gets wrong by trying too hard.

Some problems are real, observable, and genuinely not attributable to any particular failure. A missing metadata definition is the standard example. A runner that degrades an absent definition to null or zero rows rather than throwing lets the run continue — which is the right call, because throwing at the first gap means you learn about one problem per run. But the consequence is that the failure surfaces later, somewhere that no longer knows a schema gap caused it.

The gap is worth fixing and worth reporting. Which test it broke is not something the engine can honestly claim. So the right output is a separate section — run-level evidence, deliberately unattributed — rather than a confident-looking arrow from the gap to whichever failure happens to sit nearby. The unattributed version is less satisfying to read and more likely to be true.

The implementation

Everything above is a design argument, so it is fair to point at where it landed. Because Nimbus executes Apex on your machine, in-process, the engine already holds the state described at the top of this post at the moment a test fails. Failure intelligence is the decision to hand it over rather than discard it.

nimbus explain <TestClass.method> assembles one test's outcome into a single report: the exception type and message, the source location, the assertion's expected and actual values side by side, and the SOQL and DML executed immediately before the failure, in order, each with its own file and line.

ZZDiagFailTest.assertionFailsWithOperationTail — FAILED (8ms)

  System.AssertException
  Assertion failed: expected 999, got 1. diagnostics fixture: expected 999 accounts

  at force-app/main/default/classes/ZZDiagFailTest.cls:23

  Assertion
    expected  999
    actual    1

  Last operations before the failure (most recent last)
    dml   insert Account  (1 row)  ZZDiagFailTest.cls:18
    soql  SELECT Id, Name FROM Account WHERE Name = 'Diagnostics Fixture Co'  ZZDiagFailTest.cls:20

  Not established by this explanation
    - Explanation is derived from local execution only; the org remains the final authority.
    - Record-level mutation provenance is not recorded.
    - No comparison against a previous passing run is available.

That closing section is the evidence-versus-inference rule made concrete. It names the boundaries of the feature itself, so an empty operation tail is never mistaken for a test that ran no queries, and a missing comparison against a previous passing run is stated as a missing comparison rather than implied to be an absence of change. When history does hold a run in which the same test passed, the report says when — and whether the Apex source has changed since, which is what separates a regression from non-determinism.

nimbus triage is the same discipline applied to a whole failing suite. It runs your tests and groups the failures by the cause the engine recorded, not by message text. Failures the engine could not classify are listed individually. Missing metadata appears in its own run-level section, unattributed.

Triage — 133 failed of 2357 tests (*)

  Grouped by engine-assigned cause

    assertion.failed             93 test(s)
      An assertion in the test did not hold

    loaddata.resource-not-found  2 test(s)
      Test.loadData referenced a static resource the project does not contain
      fix: Retrieve the static resource, or check the resource name passed to Test.loadData.

  No cause recorded (40) — listed individually, not grouped
    ACCT_IndividualAccounts_TEST.testOwnerLastNameUpdate — Invalid field Owner for Account

  Run-level evidence (2) — not attributed to any failure
    Apex Classes             ABadClass

Both emit versioned JSON — nimbus.explain/v1 and nimbus.triage/v1 — so a script or an agent can depend on the shape instead of parsing prose, and both are exposed to agents through the same construction path the CLI uses, which is what stops a human and an agent reading divergent accounts of one failure. nimbus explain --redact masks assertion operands and SOQL string literals before you paste a report into an issue, keeping the structure diagnosable. nimbus doctor --suite runs the same triage and appends a ranked worklist, largest cause first.

The rule underneath all of it

Report the cause that was recorded, or report that there is none. Never guess a plausible one to fill the gap.

That sounds modest, and it is the constraint that does all the work. It is what makes a group count mean something, what keeps an unexplained failure visible instead of buried, and what lets you act on a report without independently verifying it first. A failure report you have to double-check has saved you nothing — you are back to reconstructing state by hand, only now with a confident-sounding suggestion pointing the wrong way.