Coverage tells you which lines executed and nothing about whether the assertions would catch a bug. Mutation testing checks that: Nimbus modifies your code and runs the tests. If they still pass, the test did not check the behavior.
A test that calls a method and asserts nothing achieves 100% coverage of that method: the lines executed, the test passed, the report is green. If the method's logic changed tomorrow, the test would still pass, because it never verified what the method did.
Mutation testing exposes this. Nimbus introduces small deliberate defects into the source, one at a time, and runs the test suite against each mutated version. If the tests pass with the mutant in place, the mutant survived, and a surviving mutant is a test gap.
The result is a mutation score: the percentage of mutants killed by your tests. A 90% mutation score means the tests killed 9 of every 10 mutants.
// This test achieves 100% line coverage.
// It kills zero mutants.
@isTest
static void testDiscount() {
Decimal result = PricingService.applyDiscount(100, 0.1);
System.assertNotEquals(null, result); // ← asserts nothing useful
}
// After mutation: applyDiscount returns 110 instead of 90
// Test still passes. Mutant survived.
// A mutation-killing test:
@isTest
static void testDiscount() {
Decimal result = PricingService.applyDiscount(100, 0.1);
System.assertEquals(90, result); // ← kills the mutant
}Run nimbus test:mutate against a class or test pattern. Nimbus generates mutants for the source class, runs your tests against each one, and reports which mutants survived.
Mutation runs are CPU-intensive, since each mutant requires a full test execution. Nimbus parallelizes across workers and skips equivalent mutants, but runtime grows with the size of the test suite.
The --threshold flag fails the run if the mutation score drops below a target, which gives CI a floor alongside coverage requirements.
# Mutate a specific class
nimbus test:mutate PricingService
# Mutate and run specific tests
nimbus test:mutate PricingService --tests "PricingServiceTest.*"
# Fail if mutation score < 80%
nimbus test:mutate PricingService --threshold 80
# Output formats
nimbus test:mutate PricingService --report mutation.html
nimbus test:mutate PricingService --report mutation.json
# Example output:
# Generating mutants for PricingService... 42 mutants
# Running tests against each mutant...
#
# Killed: 38 (90%)
# Survived: 4 (10%)
# Timeout: 0
#
# Surviving mutants:
# PricingService.cls:34 operator >= → > (survived)
# PricingService.cls:51 return null (survived)
# PricingService.cls:67 && → || (survived)
# PricingService.cls:89 statement removed (survived)Nimbus applies mutations to the AST rather than by text substitution, so every mutant is syntactically valid Apex and compiles.
Relational operators are flipped: >, <, >=, <=, ==, !=. A mutant survives if your tests pass either way.
Boolean conditions are negated. Catches tests that never exercise the false branch of a condition.
Return values are replaced with null, zero, empty string, or an empty list depending on type.
Side-effecting statements are removed entirely. Catches tests that do not assert on every state change.
+, -, *, / are swapped. Useful for catching missing assertions on calculated values.
&& and || are swapped. Catches conditions where one side is never independently false.
Run mutation tests in CI as a quality gate. The --threshold flag returns a non-zero exit code if the score drops below the target, so the pipeline fails the same way it does for a failing test.
Mutation testing every class on every commit is expensive. A practical pattern is to mutate only the changed classes, on PRs to main.
# .github/workflows/mutation.yml
- name: Mutation test changed classes
run: |
CHANGED=$(git diff --name-only origin/main \
| grep '.cls$' \
| xargs -I{} basename {} .cls \
| grep -v Test \
| tr '\n' ' ')
if [ -n "$CHANGED" ]; then
nimbus test:mutate $CHANGED --threshold 80
fiCoverage tells you which lines ran. Mutation testing tells you whether the assertions would catch a bug. nimbus test:mutate is a Pro feature.