AI / Agentic
When an AI Test Copies the Bug: Any Tag Versus Every Tag
By Vivek Jalondhara · 4 min read
Published
Use a CRM tag filter to expose correlated implementation and test mistakes, with explicit expected records and a discriminating fixture.
A generated test can call the same helper as the implementation to calculate its expected result. The assertion then confirms that a function agrees with itself. The code may be tidy, the test may pass, and the product rule may still be wrong.
Consider an illustrative SmackDab CRM note filter. Selecting urgent and customer should show notes containing both tags. This all-tags rule is a proposed exercise requirement. It provides a precise way to examine shared assumptions without claiming a defect in the actual CRM.
Locate the first ambiguous word
The request filter by selected tags omits the relationship between selections. One reasonable interpretation uses any selected tag; another requires every tag. An agent that chooses the first interpretation may also write tests expecting the first interpretation. More generated assertions do not resolve that missing decision.
Define the rule before choosing an array method. Here, a note with both urgent and customer qualifies. A note with only urgent does not. Clearing the selection removes the tag restriction for already authorised notes. These statements describe the desired behaviour independently of JavaScript.
Create a fixture that separates the interpretations
Use four notes: A has urgent and customer, B has urgent, C has customer, and D has neither. For the two-tag selection, the expected result is A. A fixture containing only A and D is insufficient because both any-tag and all-tag implementations return A.
This is why fixture design matters more than its size. B and C are the witnesses that distinguish the competing rules. Give each note an uncomplicated identifier and compute the expected identifiers manually. Avoid names such as validNote that assume the conclusion the test is supposed to establish.
Recognize a circular expected value
A weak test filters the fixture with the production predicate, stores that result as expected, and compares it with the service response. It can exercise service wiring, but it cannot independently establish that the predicate implements all-tags semantics.
Sharing fixture builders and harmless data shapes can reduce repetition. Sharing the actual decision rule across the implementation and oracle removes the disagreement the test needs to detect. Name this limitation clearly instead of treating every imported helper as equally dangerous or equally safe.
Use properties without overclaiming their coverage
Changing the order of selected tags should not change the result. Duplicating a selected tag should not change it either. Those are useful invariants, but an incorrect any-tag implementation also satisfies both. A property can be correct while failing to discriminate the bug under investigation.
Pair those checks with explicit examples and an empty selection. ECMAScript specifies that every returns true for an empty array, while some returns false. In this exercise the former matches the clear-filter rule, but that language behaviour is not a substitute for deciding what the product should do.
Carry the meaning across the request boundary
A correct in-memory function does not establish that the API applies the same rule. A database query may use any-tag semantics while a component preview uses all-tags semantics. Exercise the public list request with the same discriminating fixture and inspect the returned identifiers.
Keep authorisation outside this toy predicate: apply the tag restriction only to notes the actor can access. Check pagination against the filtered result, rather than filtering an arbitrary page locally. These integration checks answer different questions from whether the small JavaScript predicate is logically correct.
Challenge the assertion, then assess the remaining uncertainty
Temporarily substitute some for every in a disposable run. The two-tag case must fail, and the empty-selection case should also expose the semantic change. Restore the intended implementation immediately. This experiment establishes sensitivity to that specific mistake, not universal test quality.
The tradeoff is maintaining a few independently reasoned expectations rather than deriving everything automatically. That effort is concentrated where alternative meanings diverge. Tests remain drafts requiring review when an agent writes them; the expected result needs a source other than the implementation's current behaviour.
Make the expected identifiers explicit
This complete JavaScript example uses controlled fixtures and Node's strict assertions. Inputs have already been validated as arrays of tag strings. The helper illustrates filtering only; it neither fetches notes nor checks access.
Illustrative example · JavaScript · Simplified for explanation
import assert from 'node:assert/strict';
const notes = [
{ id: 'A', tags: ['urgent', 'customer'] },
{ id: 'B', tags: ['urgent'] },
{ id: 'C', tags: ['customer'] },
{ id: 'D', tags: [] },
];
const select = tags => notes
.filter(note => tags.every(tag => note.tags.includes(tag)))
.map(note => note.id);
assert.deepEqual(select(['urgent', 'customer']), ['A']);
assert.deepEqual(select([]), ['A', 'B', 'C', 'D']);- Clarify all-tags semantics
- Create a discriminating fixture
- Assert explicit identifiers
- Change the predicate to check test sensitivity
Conceptual flow for the example above.
Tradeoff to keep in mind
The fixture proves a narrow semantic distinction. It does not verify server filtering, authorisation, input limits, or ordering; those need evidence at the boundaries where they are implemented.