Why a dependency graph can’t pick your tests

The temptation

Every codebase past a certain size produces the same wish: I changed one class, run only the tests that touch it. The machinery looks trivial. You already parse Apex to build a dependency graph. Invert the edges, take the transitive closure, intersect with the set of test classes, and you have a selection. It runs in milliseconds, it needs no history, and it works on a cold clone.

The reason to be suspicious is the failure mode. A selector that is too broad wastes time — annoying, visible, self-correcting. A selector that is too narrow returns a green result for code nothing executed. That failure is silent, and it is silent precisely at the moment you are relying on it most. So before shipping selection on top of the graph, we measured it.

The experiment

The ground truth is not a model — it is observed behaviour. A full Nimbus run with coverage writes an impact map recording, per source file, which tests actually executed a line in it. If a test’s execution reached a file, that test depends on the file, whatever the mechanism: a direct call, inheritance, a trigger firing on DML, metadata-driven dispatch.

The prediction is the class-level syntactic graph, inverted and transitively closed: for a changed class, which test classes reach it. Two numbers matter. Recall is the fraction of genuinely-dependent tests the graph would have selected; a miss is a test that really executes the changed file and would not have run. Selectivity is the fraction of the suite skipped, measured because recall alone cannot judge a selector — “run everything” scores a perfect 1.0 and saves nothing.

The first result was too good, which is how we found the bug

Five corpora — EDA, NPSP, Nebula Logger, fflib-apex-extensions, and a small demo project. The first pass came back clean:

Corpus                   Files  Observed deps  Recall  Misses  Selectivity  Precision
EDA                        164           1692  1.0000       0        0.930      0.513
NPSP                       189           5381  1.0000       0        0.493      0.152
nebula-logger               38            548  1.0000       0        0.674      0.598
fflib-apex-extensions       25             92  1.0000       0        0.878      0.568
demo project                12             32  1.0000       0        0.810      0.781

Zero misses across 7,745 dependencies. That number is an artifact and should not be quoted. Breaking recall out by file kind showed why: no trigger file appeared in any corpus’s coverage at all — 71 tracked .trigger files, zero entries. Triggers are exactly where a syntactic graph should fail, because no test references one, and that entire class of dependency was invisible to the ground truth. Perfect recall meant “the graph explains everything the evidence could see”, and the evidence was blind precisely where the graph is weak.

Coverage attribution for triggers was a real bug in the runner, not in the experiment. Fixing it made those dependencies visible, and the measurement was run again.

The second measurement

Corpus   Files evaluated  Triggers skipped  Observed deps  Recall  Files with a miss
EDA                  165                37           2248  0.7527                 33
NPSP                 195                26           5705  0.9432                 43

EDA’s newly-visible dependencies were missed almost entirely: 556 of the 556 that the trigger fix added. Broken out per file, the worst cases are unambiguous — truth is the number of test classes coverage observed executing the file, predicted is the number the graph would have selected:

ACCT_IndividualAccounts_TDTM.cls      recall 0.02   truth 46   predicted 1
SRVC_ContactPrimaryLanguage.cls       recall 0.04   truth 50   predicted 2
SRVC_Contact_PrimaryAffiliations.cls  recall 0.04   truth 49   predicted 2
REL_Utils.cls                         recall 0.05   truth 44   predicted 2
MAPR_CON_PreferredEmailFields.cls     recall 0.08   truth 52   predicted 4

Every one of them is a trigger handler invoked through metadata-driven dispatch. A change to ACCT_IndividualAccounts_TDTM genuinely affects 46 test classes. A graph-based selector would have run one of them and reported success.

Why no amount of parser work fixes this

The mechanism is the TDTM-style pattern that a large share of serious Salesforce codebases are built on, and that every metadata-driven trigger framework reproduces in some form. One thin trigger per object does nothing but hand control to a dispatcher. The dispatcher queries custom metadata records to find out which handler classes are registered for this object and this event, in which order, and instantiates them dynamically.

Follow the source. No test references the handler. No class references the handler. The handler’s name exists in the repository as a string in a custom metadata record — data, not code. There is no edge for a syntactic analysis to follow, because in the source there is no edge. A better parser, a type-resolved graph, a method-level graph: none of them create an edge that the language does not contain. This is not an implementation gap, it is a category boundary.

A second, quieter result points the same way. In the second measurement, 37 files on EDA and 26 on NPSP were skipped — a .trigger path resolves to no class in a class-level graph at all. A changed trigger cannot even be evaluated by static selection, let alone selected for correctly.

What the measurement does not establish

The number is strong; the claim it supports is narrow, and over-reading it is the exact failure the exercise existed to prevent.

• It measures explanation, not prediction. The ground truth only contains paths that already executed. It cannot contain a test that does not currently touch a file but would after the change — a new call, a newly satisfied branch, a trigger newly firing. A release gate asks the second question, and this experiment does not answer it.
• Over-selection was doing real work. Precision ran 0.15–0.78: the graph selects roughly 2–7× the tests that genuinely touch a change. Recall was partly high because the closure is coarse — it absorbs dynamic edges rather than resolving them. A finer, method-level graph would be more selective and, on this evidence, less safe.
• Only Apex files were tested. No schema or metadata change was evaluated, and nothing here argues for treating one as anything other than a full run.
• Selectivity collapses where it would matter most. NPSP skips only 49% of its suite. On densely interconnected codebases the upside of narrowing is modest while the risk profile is unchanged.

The design that survives the number

The conclusion splits cleanly by use case. Read-only explanation — blast radius, orientation on an unfamiliar codebase, cycles, a diagram to argue over in a pull request — survives intact, because over-selection is harmless when nothing correctness-bearing rides on the answer and incompleteness can simply be stated. Test selection from the static graph alone does not survive: silently skipping 45 of the 46 tests that exercise a changed handler is unacceptable even in an inner loop, and doubly so behind a signed release gate.

The interesting part is the third case. Nimbus’s hybrid — the dynamic coverage map combined with the static graph — was previously understood as a map with a static optimisation bolted on. This measurement inverts that reading: the dynamic half is load-bearing, because coverage observes trigger and metadata-dispatch edges that no syntactic analysis can derive. The graph is the supplement.

So the honest interface is to show both answers and refuse to merge them. Ask nimbus graph about a class and it prints the syntactic reading — who references this class in source — and, where a coverage map exists, a second independent answer beside it: which tests a real run observed executing lines in that file. The two are never combined into one number, because a merged figure hides the only thing worth looking at. Where the readings disagree, the disagreement is the finding.

$ nimbus graph ACCT_IndividualAccounts_TDTM

ACCT_IndividualAccounts_TDTM — reachability

  Tests reached syntactically (1)
      ACCT_IndividualAccounts_TEST

  Tests a coverage run actually observed
  executing it (46)
      ACCT_AdministrativeNameRefresh_TEST
      ...

  Observed but NOT reachable syntactically (45)
      — the blind spot, measured

  Not established by this view
    - Syntactic references only. A trigger
      dispatching through custom metadata
      reaches handlers with no reference
      to follow.
    - This is reachability, not a
      test-selection decision.

Everything downstream follows from the same rule. Observed-only edges are drawn dashed in every export so a class reachable only through dispatch looks different at a glance. No label in the CLI, the exports, or any viewer reads “impact” or “affected tests”. Reported cycles are stated as a lower bound, since edges invisible to a syntactic reading may close further loops. And nimbus test --impacted forces a full run when a trigger changes — the same finding, enforced rather than footnoted.

What would change the ruling

Not more of this experiment — a different one. Apply real mutations to the source, then check that the graph-selected set still contains every test whose outcome the mutation changed. That measures prediction rather than explanation, and it is the missing evidence. Until someone runs it, a release gate should keep running the full applicable suite, and any tool that narrows one should be able to show you the number it narrowed on.

The general lesson is not about Salesforce. Any platform where dispatch is configuration rather than code — dependency injection containers, plugin registries, event buses wired in YAML — has the same hole, and the same tempting graph sitting over it. Measure the selector against observed execution before you trust it. If the measurement comes back perfect on the first try, suspect the evidence before you celebrate the tool.