Flaky Apex tests: where they come from and how to find them

The definition matters more than it sounds

A flaky test is one that produces different outcomes from the same inputs. That word same is doing enormous work, and almost every team gets it wrong in practice. You edit a class, re-run, and four tests go from red to green. Those tests are not flaky. You fixed them. The inputs changed.

This is why "it failed yesterday and passes today" is such a weak signal on its own. Between yesterday and today the source changed, the worker count may have changed, and possibly the machine did too. An outcome flip across any of those is explained by the change, not by the test. Treating every flip as flakiness produces a quarantine list full of tests that were only ever telling the truth.

The useful definition is narrower: same source, same execution conditions, different result. Everything below is a way that "same conditions" turns out to be false when you thought it was true.

1. Order dependence

The classic. A test passes when it runs after another test and fails when it runs first, or the other way around. In Apex the usual carrier is static state, because statics live for the whole transaction and a test class's statics are perfectly happy to carry a value from one method into the next.

@isTest
private class RoutingRulesTest {
    // Survives across test methods in the same transaction.
    private static Boolean rulesLoaded = false;

    private static void ensureRules() {
        if (rulesLoaded) return;
        RoutingRules.load();
        rulesLoaded = true;
    }

    @isTest
    static void routesDomesticLeads() {
        ensureRules();
        System.assertEquals('Domestic', RoutingRules.queueFor('US'));
    }

    @isTest
    static void resetClearsRules() {
        ensureRules();
        RoutingRules.reset();   // leaves rulesLoaded = true, rules actually gone
        System.assertEquals(0, RoutingRules.count());
    }
}

Run routesDomesticLeads first and it loads the rules. Run resetClearsRules first and the second test finds rulesLoaded == true with no rules behind it. Same code, two orders, two results. Any memoisation flag, cached describe result, or "have I already run" guard in production code has the same shape — and production code is where these usually live, because the caching that makes a trigger handler fast is exactly the caching that leaks between tests.

@testSetup interacts with this in a way worth knowing. The platform runs the setup method once and rolls the org back to that state before each test method, so records are restored between methods. Static variables are not records. A test method that mutates a static leaves it mutated for the next one, and the clean data around it makes the surviving state harder to spot, not easier.

2. Time assumptions

Any test that reads the clock has an input you did not write down. The failure modes are boringly predictable and they all arrive at the worst moment:

• Date boundaries. A test that inserts a record with Date.today() and asserts a "due in 30 days" calculation crosses a month boundary, a quarter boundary, or a daylight-saving change and produces an off-by-one.
• Midnight. Anything computing Date.today() twice in one test can straddle midnight. Rare, and it always happens to the person running the suite at 00:00 before a release.
• Now-versus-now. Asserting Datetime.now() equals a storedCreatedDate works until the two land in different milliseconds.
• Leap days and year ends. A suite written in March that first meets February 29th four years later.

The fix is not clever: take the clock as a parameter, or pin it. Compute relative to a fixed Date.newInstance(2026, 3, 1) in the test and let production code accept the date it should work from. A test that cannot fail differently on a Tuesday is worth the extra argument.

3. Data collisions under parallel execution

This is the category that grows as your suite does, because it only appears once tests share a database concurrently. Two shapes account for most of it.

Unique fields. Two tests both create an Account named 'Test Account', or two Contacts with the same external ID. Serially they never meet, because each test's data is rolled back before the next one starts. In parallel they overlap in time and one of them takes a DUPLICATE_VALUE on a unique or external-ID field. Which one loses depends on scheduling, so the same test is fine four runs in a row and red on the fifth.

Row locks. Two tests touching the same parent record — the same Account while inserting child Contacts, the same custom setting, the same shared configuration record — contend for a lock and one of them gets UNABLE_TO_LOCK_ROW. This is the single most common "it only fails in CI" failure, because CI is usually where the parallelism is highest.

Both are fixed the same way: make every test's data unique to that test, and stop reaching for shared records. A test data factory that appends a unique suffix costs one line and removes an entire category of intermittent failure. The broader argument for factories is its own post.

The diagnostic tell for this whole category: the failure rate changes when the worker count changes. If a test is solid at one worker and flips at eight, you are not looking at a flaky test, you are looking at a contention bug — and the parallelism setting is part of the test's inputs whether or not you were treating it that way.

4. Async timing

Async Apex flakiness is usually not really timing — it is a test asserting on work that never ran. @future, Queueable, Batchable and Schedulable all enqueue and return; Test.stopTest() is what flushes the queue and runs the work synchronously. Assert before stopTest() and you are asserting on pre-async state.

The reason this reads as flakiness rather than as a plain bug is that such a test often passes — for the wrong reason. It asserts a field is unchanged, or a count is zero, and that happens to be true at the moment it looks. Then someone reorders the arrange block, or the enqueue moves inside the start/stop pair, and the assertion starts seeing post-async state. Nothing about the test's intent changed. Its result did.

Anything that asserts on AsyncApexJob.Status deserves particular suspicion, since the status a job reports depends on where in the lifecycle you look. The per-flavor patterns are in testing async Apex.

5. Org-state dependence

A test annotated @isTest(SeeAllData=true) reads whatever the org happens to contain. That is not a fixed input — it is a moving one, edited by every admin, every data load, and every other developer in the sandbox. The test passes until someone deactivates the user it queried, renames a record type, or deletes the Account it assumed existed.

The quieter version of this needs no annotation at all. Org defaults leak into tests through custom settings and custom metadata, through the running user's locale and time zone, through picklist values, through record-type availability, through org-wide defaults on sharing. A test that passes in the developer's sandbox and fails in the integration org is usually reading one of these without realising it. The same test is deterministic in each org and non-deterministic across them, which is the most confusing possible way for a flake to present.

The detection discipline

Categories are only useful if you can tell which one you are in. Three habits do most of the work.

Re-run with the conditions, not just the test. "Re-run it" is not a diagnostic. "Re-run it at one worker" is. So is re-running it alone, re-running it in the reverse order of its class, and re-running it with the clock pinned. Each of those changes exactly one suspected input, and a result that moves tells you which category you are in. A result that does not move rules one out, which is just as valuable and much less celebrated.

Quarantine explicitly, with an expiry. A known-flaky test that still runs and still fails trains the team to ignore red, which is a far more expensive problem than the flake. Move it to a named list, keep the list short, and put a date on each entry. An unbounded quarantine list is how a suite stops being trusted.

Treat "it fixed itself" as a data point, not a conclusion. A test that went green with no change to the source is reporting non-determinism. Write down when it happened and what was different — worker count, machine, time of day. Three of those notes usually identify the category without any further investigation. Zero of them means you will re-investigate from scratch next month.

What a runner can do about it

All three habits above are bookkeeping, and bookkeeping is what tools are for. The requirement is specific: to claim a test is flaky rather than fixed, you have to compare two runs that agree on everything except the outcome.

Nimbus records each run to .nimbus/history/ along with the conditions it ran under — a fingerprint of the project's Apex sources, the worker count, the isolation mode, the engine version, and the machine. nimbus history opens an interactive browser over those runs; nimbus history --flaky reports flake suspects.

# Browse recent runs
nimbus history

# Flake report over the recorded runs
nimbus history --flaky

The part that matters is what the report refuses to do. It compares only within a group of runs that are mutually comparable — same source fingerprint, same worker count, same isolation mode, same version, same machine — and it uses the largest such group. Runs that differ in any of those are not compared at all, because an outcome flip across a source change is a fixed test, and an outcome flip across a worker-count change is a contention finding about your configuration rather than about the test.

A test also has to have run in every run of the group to be considered. If it was filtered out of one run by a pattern, its absence is not a result, and a flip across that gap is not a flip. Each suspect gets a flip rate — the fraction of adjacent runs whose outcome differed — and its recent pass/fail sequence, so a test that alternates every run is visibly different from one that flipped once six runs ago.

And when fewer than two comparable runs exist, the report is empty rather than clean. That distinction is the whole point: no evidence is not the same as no flakes, and a tool that blurs the two is worse than no tool, because it retires a suspicion you should still have.

Once you have a suspect

A flake report gives you a name. The next question is what the failing run actually did, and that is a different command: nimbus explain reports one test's failure with the assertion's expected and actual values and the SOQL and DML that ran immediately before it — and, when history holds a run where the same test passed, when that was and whether the Apex source has changed since. An unchanged fingerprint alongside a changed outcome is the cleanest evidence of non-determinism you can get. For run-over-run trends and execution traces, see traces and analytics.

Checklist

• Flaky means same conditions, different outcome. If the source changed, you fixed or broke something — that is not flakiness.
• Statics outlive test methods. @testSetup restores records, not static state.
• Any clock read is an undeclared input. Pin the date or pass it in.
• Failure rate that moves with worker count is contention, not a flaky test — look for shared records and unique-field collisions.
• Async asserted before stopTest() passes for the wrong reason until someone reorders the test.
• SeeAllData and org defaults make a test deterministic per org and non-deterministic across orgs.
• Record the conditions with every result. Without them, no run comparison supports any conclusion.