20,193 Apex tests from 48 open-source projects.
96.7% produce identical results locally.
Nimbus is measured against real Salesforce codebases — cloned unmodified, pinned to a commit, run with the same tests and the same assertions their authors wrote. This page is the record of how that goes, including the part that does not flatter us: every test that still fails, and how many of those failures were re-run on a real org and confirmed as ours.
@isTest method in those projects, none excludedScorecard run Wed Aug 26 01:18:46 CEST 2026 · sha256 45f4ce1dadc0c850
What identical means
A compatibility claim is worth exactly as much as the method behind it. Here is the whole method.
The projects are clones, not samples.
Each row is a public repository checked out at a specific commit, with no source changes. No test is skipped, rewritten, or annotated away, and no project is dropped from the corpus because it scores badly — the worst-scoring rows are in the table below, not omitted from it.
Nimbus runs them with no org attached.
The whole suite runs from source on one machine: the Apex interpreter, SOQL translated to PostgreSQL, an embedded database carrying the Salesforce data model, transaction and savepoint semantics, triggers, and record-triggered flows in their order-of-execution slots. A test passes only when its own assertions pass. Nothing is stubbed to make a number look better.
Every failure is re-run on a real org.
A test failing locally proves nothing on its own — the test may be broken, or the org may be configured in a way the project assumes. So each failure is deployed source-identical to a real Salesforce org and run there. The deploy is the identity proof: it is atomic, so if the source had not compiled identically, nothing would have deployed.
Three verdicts, and only one is ours.
The org passes and Nimbus fails: a confirmed divergence, counted against us and filed as a bug. Both fail: a shared failure — the test is order-dependent, or asserts something only true inside a packaged namespace. The org cannot run it at all: environmental. The last two are reported here, not quietly removed from the denominator.
Every project, every number
Largest suite first. The last column is how many of that project's current failures have been individually re-run on a Salesforce org and confirmed as a Nimbus bug.
| Project | Tests | Passing | Failing | Pass rate | Org-confirmed |
|---|---|---|---|---|---|
| NPSP19646faa | 3,986 | 3,868 | 118 | 97.0% | 110 |
| EDAe5939577 | 2,357 | 2,350 | 7 | 99.7% | 2 |
| nebula-logger05ce16c5 | 1,355 | 1,346 | 9 | 99.3% | 3 |
| fflib-apex-extensions368978bf | 892 | 892 | — | 100.0% | — |
| FormulaShare-DXdc69c49d | 876 | 845 | 31 | 96.5% | 31 |
| dlrs5dbd186b | 831 | 830 | 1 | 99.9% | 1 |
| forceeae7713d24 | 830 | 820 | 10 | 98.8% | 10 |
| rflibac3032b0 | 808 | 681 | 127 | 84.3% | 127 |
| at4dx8c109e4f | 746 | 746 | — | 100.0% | — |
| fflib86bd8791 | 746 | 746 | — | 100.0% | — |
| apex-rollup6b8ae4ca | 745 | 744 | 1 | 99.9% | 1 |
| expressionacbd1374 | 710 | 710 | — | 100.0% | — |
| fflib-apex-mocks24deeebc | 471 | 471 | — | 100.0% | — |
| soql-lib0fdaeb9d | 466 | 466 | — | 100.0% | — |
| PMM4c5f3c79 | 433 | 368 | 65 | 85.0% | 63 |
| amoss3c054ed3 | 424 | 405 | 19 | 95.5% | 19 |
| dml-lib9c02feb3 | 372 | 372 | — | 100.0% | — |
| Apex-Opensource-Library57d09684 | 347 | 270 | 77 | 77.8% | 77 |
| ApexKitc2660a33 | 334 | 291 | 43 | 87.1% | 43 |
| apex-recipesd2ec0b33 | 322 | 322 | — | 100.0% | — |
| nebula-core0a3a20cc | 294 | 228 | 66 | 77.6% | 57 |
| apex-fp61769f99 | 179 | 179 | — | 100.0% | — |
| Volunteers-for-Salesforce40c3d311 | 165 | 153 | 12 | 92.7% | 8 |
| Apex-GraphQL-Clientb29350eb | 162 | 155 | 7 | 95.7% | 7 |
| trigger-actions-framework5c3793d2 | 155 | 155 | — | 100.0% | — |
| apex-mockery320d3a73 | 133 | 133 | — | 100.0% | — |
| sobject-fabricatorbdf66b2e | 124 | 124 | — | 100.0% | — |
| automation-componentsa43d9416 | 122 | 122 | — | 100.0% | — |
| apex-dml-mocking5daf499b | 95 | 76 | 19 | 80.0% | 19 |
| force-dot-com-esapib6eddab9 | 89 | 65 | 24 | 73.0% | 23 |
| apex-test-kit607e6577 | 87 | 86 | 1 | 98.9% | 1 |
| TestDataFactory22487c80 | 83 | 77 | 6 | 92.8% | 6 |
| async-lib5486e899 | 82 | 82 | — | 100.0% | — |
| force-di574d0509 | 61 | 61 | — | 100.0% | — |
| http-mock-lib42bdd066 | 50 | 50 | — | 100.0% | — |
| jsonparse69da320c | 39 | 17 | 22 | 43.6% | 22 |
| test-lib82ebbc9c | 36 | 36 | — | 100.0% | — |
| NebulaQueryAndSearch269a787b | 34 | 34 | — | 100.0% | — |
| apex-domainbuilder26ca2d52 | 34 | 34 | — | 100.0% | — |
| ecarsa5e9cc01 | 23 | 23 | — | 100.0% | — |
| OutboundFunds8acc96b4 | 21 | 21 | — | 100.0% | — |
| cache-manager2e5e55ab | 18 | 18 | — | 100.0% | — |
| ApexValidationRules6a2aa3e9 | 13 | 13 | — | 100.0% | — |
| sfdc-trigger-frameworkb7e36c76 | 13 | 13 | — | 100.0% | — |
| dreamhouse-lwc9810695a | 11 | 11 | — | 100.0% | — |
| streaming-monitor0d35b300 | 11 | 11 | — | 100.0% | — |
| apex-constsff260c18 | 4 | 4 | — | 100.0% | — |
| ebikes-lwcfd29e9fe | 4 | 4 | — | 100.0% | — |
| All projects | 20,193 | 19,528 | 665 | 96.7% | 630 |
Failing is measured locally: how many of that project's tests Nimbus does not currently reproduce. Org-confirmed divergences is a stricter number — it counts only failures that were deployed to a real Salesforce org, run there, observed to pass, and recorded by name. A test drops out of that column the moment it stops failing, so the column shrinks when we fix something rather than staying at whatever a past verification run measured.
Where the two differ, the gap is not a hidden failure. It is either a failure the org reproduced too, or one the org could not run, or one that started failing after the last verification pass. Those are all counted in the failing column and none of them are counted as confirmed.
Rows measured against a private repository are left off the published scorecard; they contribute no tests to the totals above.
How much of this is verified
Not "we ran it and it looked right". Each of these failures was executed on Salesforce, by name, and its verdict written down.
The failing column is the backlog.
A confirmed divergence is not an embarrassment to be managed — it is a work item with a reproduction, a real-org expectation, and a name. That is the entire reason to run a corpus this size against an oracle: it produces a queue of exactly the bugs that matter, ranked by how much real code trips over them. 630 of them are open right now, and they are tracked as issues rather than filtered out of a scorecard.
17 of the failures on this page carry no classification yet: they began failing after the most recent verification pass. They are counted against us in every figure above until an org says otherwise.
Verification passes
- 2026-08-25638 failures classified against the scorecard as it stood then, 628 confirmed as divergences.
- 2026-08-19993 failures classified against the scorecard as it stood then, 969 confirmed as divergences — 3 of the projects it covered no longer fail at all, so nothing from them appears in the table.
Reproduce it
A scorecard nobody can re-run is a claim. Every row above is a public repository at a published commit, and the command is the same one we ran.
brew install nimbus-solution/nimbus/nimbus
# or: curl -fsSL https://testnimbus.dev/install.sh | shfflib-apex-extensions — 892 tests, all of them reproduced locally.
git clone https://github.com/wimvelzeboer/fflib-apex-extensions
cd fflib-apex-extensions
git checkout 368978bf
nimbus test "*"FormulaShare-DX — 876 tests, 31 of them still failing. Check that number too; it is the one we would most like to be wrong about.
git clone https://github.com/LawrenceLoz/FormulaShare-DX
cd FormulaShare-DX
git checkout dc69c49d
nimbus test "*"The manifest lists every project on this page with its repository URL, the exact commit it was measured at, and the counts to expect.
Expect a couple of tests of movement on the largest suites: a handful of record-merge and bulk-DML tests are sensitive to how much of the machine they get, so a run can land a test or two either side of the number here. That is why the manifest carries per-project counts and commits rather than one headline percentage — a difference can be localised to a project and then to a test name, which is how we track it internally too.
A local runtime approximates the platform. This is the measurement of how closely.
Nimbus is not Salesforce and does not claim to be. It is an implementation of the parts of the platform that Apex tests exercise, and like every implementation it is right until it is found to be wrong. The corpus exists to find that out on real code, continuously, before a customer does — and the results are published whether they moved in our favour or not.
The org still owns:
- OWD and role-hierarchy sharing (the with sharing keyword is enforced)
- Approval processes, assignment rules, validation-rule packages
- Lightning UI / browser testing
- Final pre-deployment validation — your org is still the source of truth
Failures are tracked as public issues, not hidden. See the issue tracker.
Why publish this
Fidelity is easy to claim and hard to prove. "Runs your Apex exactly like the platform" is a sentence anyone can type, and we have never found a local Apex runtime — ours included, until now — that published a record you could check it against. So here is ours: named repositories, pinned commits, failing counts, and a verdict from a real org on each failure.
The point of the format is that it can be falsified. We would rather publish 665 failures with their receipts than a round number with none. If a row does not reproduce on your machine, that is a bug report we want.
nimbus test "*"