ServiceNow ATF: 25 Green Tests That Proved Nothing

The short version#

• Coverage counts, assertion counts, a passing badge — none of them are evidence a behavior actually fired; treat the color as a claim you still have to verify.
• The suite that can't fail is worse than no suite, because it converts an open question into false confidence your client ships on.
• The one part a client can't self-serve is the discipline that makes a test go red on purpose — which is exactly where the margin sits.

What was actually broken beneath the green?#

Beneath the passing badge sat seventeen distinct defects⁵ — the kind a green suite is structurally incapable of catching. The outbound business rules that were supposed to fire the dispatch were left inactive or set to run async, so the handoff never reliably fired or logged⁵. Green, but the core behavior of the app simply did not happen.

The transform map was worse, because it mapped only sys_id and silently dropped every field that mattered — close_code, close_notes, work start and end⁵. Records moved. The payload was hollow. A test that only checks a row landed would pass this every single time and tell you nothing about the fields a field-service dispatcher actually reads.

Then the parts that had nothing to do with logic at all. A plaintext OAuth secret sat in the update set⁵, and the whole thing shipped as a bloated dump of roughly 2,000 rows⁵ — the update-set equivalent of tipping a drawer onto the floor and calling it packed. None of it was hidden. It was just never asked a question that could return a wrong answer.

Why does AI test-gen keep producing green suites?#

Because all twenty-five tests asserted nothing⁴ — they exercised the code, then checked no outcome, which is the anti-pattern Martin Fowler named "assertion-free testing" years before any of this was AI-built¹. A test that runs a flow but verifies no result cannot fail on a broken behavior. It can only fail on a crash. So the suite was doing exactly what it was written to do.

What changed is the scale. AI test-gen mirrors historical happy-path usage, so it writes the flow it has seen a thousand times and skips the assertion that would confirm the flow actually did something — Autonoma documented the same pattern outside ServiceNow, coverage climbing while assertions verify nothing². In ATF specifically, the assert steps are the only mechanism that confirms a test succeeded versus merely ran³. Drop them and every test collapses into "did the page load."

Here is the why-now. AI-built work is now the default lane for a lot of ServiceNow delivery, and because the generator reaches for happy-path shape by reflex, it manufactures green-but-empty suites at a rate no human backlog ever could. Each quarter you trust the color, more silently broken apps ship behind a passing badge — the FSM dump was not a freak, it was the pattern industrialized.

We already have ATF — isn't that enough?#

No — owning ATF changes nothing, because the framework faithfully reports whatever assertions you write, and when you write none it reports green with total confidence. ATF is a real thing teams invest real time in, and that investment is worth defending. But a framework is not a discipline. The assert steps are the only place a behavior gets checked³, and ATF has no opinion about whether you put any there.

That is the part the objection misses. You can stand up ATF, wire the scheduler, run the suite nightly, and still be exactly where the FSM app was — fully instrumented, fully passing, fully hollow. The tooling was never the gap. A team with mature ATF and a generator writing assertion-free tests ends up in the same place as a team with no tests at all, except now the emptiness wears a badge.

Think of it as two suites that look identical on the dashboard and behave nothing alike when a behavior breaks:

So the honest question is not whether you have ATF. It is whether anyone on the build wrote a test designed to fail — and whether the person doing it knows which failures matter enough to design for. That judgment does not come with the license.

How do you build a suite that fails when it should?#

You write tests that assert the outcome, then add negative controls engineered to turn the suite red the moment a behavior breaks. That is what the FSM rebuild replaced the hollow twenty-five with: ten real asserting ATF tests plus negative controls⁴, each one designed around a specific way the dispatch could silently fail rather than around the happy path it was supposed to walk.

The difference is where the test points. An asserting test confirms close_notes actually arrived on the target record; the negative control deliberately breaks the transform and demands the suite go red — if it stays green, the test itself is broken and you learn that before your client does. Same framework, opposite intent.

This is not a one-off heroic pass. Across 25+ certified ServiceNow Store apps^{xpertappdev.com}, roughly 40% of the test cases we design are negative-path by design⁶ — cases that exist to prove the app rejects, drops, and errors the way it should. A generator reaching for historical usage does not write those, because they never happened in the logs it mirrors.

That gap is the whole point. Designing the failures worth catching — and knowing which ones matter enough to gate a release on — is the review layer a client cannot self-serve, and it is exactly the work we spend this newsletter pulling apart week over week.

References#

1. Assertion Free TestingMartin Fowler · martinfowler.comCanonical named source: tests that run code but assert nothing give false confidence↩
2. AI-Generated Tests That Pass But Don't Assert AnythingAutonoma AI · getautonoma.comCorroborates the AI-specific pattern: coverage climbs while assertions verify nothing↩
3. ServiceNow ATF: Implementation & Best Practices 2025EchelonAI · echelonai.comServiceNow-native context: assert types are how ATF confirms a test succeeded vs merely ran↩
4. Founder accountFirst-party · not externally published↩
5. XpertApps assessmentFirst-party · not externally published↩
6. XpertApps internal recordsFirst-party · not externally published↩

Published 03 August 2026 · More issues from The Delivery Edge