AI / Agentic
Testing AI-Generated Code with an Independent Oracle
By Vivek Jalondhara · 5 min read
Published
Verify generated changes with behavior-based assertions, failure cases, boundary tests, and deliberate checks that the tests can fail.
Generated code and tests can agree about the wrong behavior. Useful assertions follow a reviewed requirement or existing contract, rather than simply repeating the implementation. The important question is where the expected result came from.
This walkthrough uses an illustrative SmackDab CRM change: enforce an existing edit permission when a record update arrives through the API. It is a testing exercise grounded in the project's access-control domain, not a claim about a discovered issue or its private test suite.
Write observable outcomes before implementation
Define the rule in terms of actors and effects. A user with the edit permission can update an allowed record in their workspace. A user without it cannot. A user in another workspace cannot, even if they hold the same permission there. A rejected request leaves the record and related write history unchanged.
These examples form the independent oracle: the source of expected behavior. Product rules still need review. Keep field restrictions, sharing rules, and ownership exceptions visible; one role check cannot express the entire policy.
Test the boundary where the bug would matter
A unit test can verify a small policy function but cannot establish that an HTTP handler calls that function before writing. Add a focused integration case at the API boundary: submit a disallowed update and verify both the response and the unchanged persisted record.
Use a separate workspace fixture with a record identifier that could otherwise look valid. This catches queries that filter only by record ID. Keep the setup understandable; an elaborate fixture factory can accidentally grant permissions that the test intends to withhold, making the scenario pass for the wrong reason.
Check negative paths and side effects
A denial response after a database write is still a failed permission boundary. Assert that the title remains unchanged and that no successful-update event was emitted. If the operation schedules notifications, verify the rejected request did not enqueue one. The observable effect matters more than which private helper was called.
Include a permitted update so a function that denies every request cannot satisfy the suite. Add malformed input and missing-record cases when the change affects those paths. Define whether inaccessible and absent records share a response according to the API contract, and avoid assertions that accidentally require information disclosure.
Prove the important assertion can fail
Temporarily remove the workspace condition in a local, disposable version of the policy and run the relevant test. The cross-workspace case should fail. Remove the permission condition and the no-permission case should fail. Restore the correct implementation immediately; the intentionally broken version is a diagnostic tool, not a proposed patch.
This small mutation check is more informative than a green result alone. If the test stays green, inspect its setup and assertions before adding more cases. A coverage percentage can show that a line ran, but cannot show that the assertion would notice that line making the wrong decision.
Use mocks without replacing the rule under test
Mock unrelated network delivery to keep a permission test deterministic. Do not mock the authorization function to return false and then claim the test proves the permission logic. That verifies the caller's denial path, which may be useful, but answers a narrower question and should be named accordingly.
For asynchronous failures, await the operation and use an assertion that checks the expected error or response. Node's test runner supports asynchronous tests, and its strict assertion module provides explicit rejection checks. Unawaited work can escape the test's intended lifetime, leaving a reassuring result that never examined the failure.
Review the change around the test
Read the handler, policy, query, and side-effect path together. Check for weakened assertions, broader permissions, or unnecessary dependencies. Run relevant static checks and affected tests; choose additional verification for a plausible regression, not to accumulate green commands.
Generated tests are useful drafts that still need review. Small policy tests make failures easy to understand, while boundary tests catch missing wiring and persistence mistakes. The tradeoff is keeping enough integration coverage to verify consequential behavior without duplicating every unit case across a slow, fragile end-to-end suite.
Exercise both halves of a small edit policy
This complete Node.js example covers workspace ownership and edit permission. Save it as a .test.mjs file and use Node's test runner. Cases cover a permitted update, missing permission, and another workspace.
Illustrative example · JavaScript · Simplified for explanation
import { test } from 'node:test';
import assert from 'node:assert/strict';
function canEdit(actor, record) {
return actor.workspaceId === record.workspaceId
&& actor.permissions.includes('records:edit');
}
const record = { workspaceId: 'alpha' };
const cases = [
['allowed', 'alpha', ['records:edit'], true],
['no permission', 'alpha', [], false],
['other workspace', 'beta', ['records:edit'], false],
];
for (const [name, workspaceId, permissions, expected] of cases) {
test(name, () => {
assert.equal(canEdit({ workspaceId, permissions }, record), expected);
});
}- Review expected behavior
- Generate a bounded implementation
- Test permitted and rejected effects
- Break the policy locally to check assertions
- Review connected code and restore passing checks
Conceptual flow for the example above.
Tradeoff to keep in mind
This runnable example verifies a deliberately small policy. It does not prove the API uses it, that database access is scoped correctly, or that field and sharing rules are complete; those need focused tests at their respective boundaries.