How to find flaky Apex tests

What actually makes an Apex test flaky

"Flaky" is not a cause, it is a symptom: the same test produced different outcomes without the code changing. In Apex, the cause is nearly always one of these.

• Order dependence. Static variables survive across test methods within a single execution context. A test that sets a static flag — a trigger recursion guard, a cached setting, a mock toggle — changes the behaviour of whichever test runs after it. The test passes alone and fails in the suite, or the other way round.
• Time and limit assumptions. System.now() arithmetic that straddles midnight, a month-end boundary, or a DST shift. Business-hours logic that only holds Monday to Friday. And the quiet one: a test sitting just under a governor limit, where one extra query added anywhere in the call path tips it over.
• Data collisions under parallelism. Tests running concurrently that compete for the same unique value, the same custom setting, or the same record. Duplicate-value errors and row-lock errors appear and disappear depending on scheduling.
• Async timing. Anything queued as future, queueable, batch, or scheduled that is asserted on without Test.stopTest() forcing it to complete first. This one usually passes locally and fails under load.
• Iteration-order assumptions. A test asserting on list[0] from a SOQL query with no ORDER BY, or on the first key out of a Map. Ordering that is stable in practice is not ordering that is guaranteed.

Hygiene that removes whole classes of flake

Before you go hunting, remove the causes you can remove outright. These are worth doing regardless of tooling:

• No @isTest(SeeAllData=true). It binds the test to whatever records happen to exist in the org. That data changes; the test does not. Every occurrence is a scheduled flake.
• No hard-coded Ids or usernames. They are org-specific and environment-specific by construction.
• Reset statics. If a class keeps mutable static state, give it a reset path and call it in setup — or make the state instance state.
• ORDER BY on any query you assert positionally. If the order matters to the assertion, say so in the query.
• Bracket async work with Test.startTest() / Test.stopTest(), and assert after stopTest(), not before.

Reproduce it before you fix it

A flake you cannot reproduce is a flake you cannot verify you fixed. Two runs answer most of it — the test alone, repeatedly, and the test inside the full suite:

# Same test, 30 times, alone — catches time, limit, and internal ordering flakes
for i in $(seq 1 30); do nimbus test "OrderServiceTest.testDiscountApplies"; done

# The whole class — catches order dependence between methods
nimbus test "OrderServiceTest"

# The whole suite — catches cross-class static leakage
nimbus test "*"

Local runs are what make this practical: the loop above is a few seconds of wall clock rather than thirty org round-trips. If the failure only appears in the third form, the cause is something another test left behind.

Let history find them for you

Every nimbus test run is recorded under .nimbus/history/ — no flag, no configuration. nimbus history reads it back, and --flaky turns the run list into a flaky-test report:

# Browse recent runs in an interactive TUI
nimbus history

# Flaky report: same test, same conditions, different outcome
nimbus history --flaky

# Widen the window
nimbus history --limit 50

The report is backed by the conditions each run recorded, which is what makes it worth reading: "same test, same conditions, different outcome" is a flake, while a different outcome under different conditions is just a different run. When stdout is not a terminal — a pipe, a CI runner — nimbus history skips the TUI and emits the run list as JSON, so this works without a TTY:

nimbus history --limit 50 | jq '.[] | select(.failedTests > 0)'

Regression, or non-determinism?

That is the question that decides who looks at the failure. nimbus explain answers it from the same history:

nimbus explain OrderServiceTest.testDiscountApplies

The report gives you the exception, the source location, the assertion's expected and actual values, and the SOQL and DML executed immediately before the failure, in order. Then, when history holds a run in which the same test passed, it says when — and whether the Apex source has changed since. An unchanged fingerprint means the code is not what differs, which is exactly the signal that separates a regression from non-determinism.

It also ends with what it could not establish. If no previous passing run is available, the report says so rather than implying nothing changed. See failure intelligence for the full output shape.

Fixes that match the cause

• Order dependence — move the shared state into @testSetup or reset it explicitly; stop asserting on state a previous test created.
• Time assumptions — inject the clock. A method taking a Datetime parameter is testable at midnight, month-end, and DST.
• Limit proximity — bulkify the code under test, not the test. A test that only passes at 99 queries will fail on the next query anyone adds.
• Data collisions — make unique values unique per test (a counter or UUID suffix), not per suite.
• Async timing — assert after Test.stopTest(). If the assertion cannot wait, the test is asserting on the wrong thing.