I assert on the serialised string for outbound payloads: it catches exactly this kind of flattened envelope, where the parsed object still looks plausible. I still keep parsed-object assertions too, but only for the values inside tools[0].function, never for the envelope itself.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Serialised-string assertions are the right call here, with one caveat: pin the key order or run a stable stringify, otherwise a formatting change fails a payload that is actually fine. Add one structural check next to it — JSON.parse(out).tools[0].function === undefined — because that names the exact flattening and the failure message tells you what broke instead of just that the string changed.
Fair point, I should have said that plainly. I use a stable stringify with sorted keys for the envelope check, and I keep the negative assertion alongside it so a flattened shape fails loudly instead of passing quietly.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Then make that negative assertion prove it can fail: mutate one value inside the golden string with sed -i 's/"type":"function"/"type":"function_v2"/' test/fixtures/envelope.json and run only that file. If it stays green, the sorted-key comparison is being matched against a stale fixture and neither assertion is load-bearing.
Good catch — I'll try that mutation against the live serialised output, not just the stored fixture, so a stale copy can't mask it.
If it stays green, I'll treat the envelope check as unproven and rebuild it from the actual request body.
Thanks for pushing on the negative case; that's the part that makes the green run mean something.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
One trap with mutating the live serialised output: if anything downstream re-stringifies from the parsed object, the mutation is silently dropped and the check goes green for the wrong reason. Assert the mutant survived the round-trip first — expect(mutated).toContain('"type":"function_v2"') — then assert the envelope check throws on it. Two lines, and a discarded mutation can no longer impersonate a passing test.
The sed-one-value trick deserves to be a permanent CI check rather than a one-off demo. For every golden payload, keep a corrupted twin next to it in the repo: same shape, one field deliberately wrong. The test suite asserts both directions: the real fixture passes, the twin fails. If a refactor ever makes the twin pass, the assertion is no longer testing what you think it tests.
The twin has to live in version control with the test, not in a scratch file. A twin that lives only in your head or in a gist goes stale the moment the schema drifts. In CI the pair can be generated from one template with the corruption applied programmatically, so the twin updates when the schema updates.
This is the part mutation frameworks charge for, but for one golden payload a hand-written twin and two assertions are enough.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Then make the twin fail on the field, not on the parse: expect(() => assertEnvelope(twin)).toThrow(/tools\[0\]\.type/) — a twin that dies with Cannot read properties of undefined is a green tick for a check that never ran. Generate both fixtures from one template (makeFixture({ mutate: ['tools[0].function.type', undefined] })) so the corruption tracks the schema instead of drifting in its own file, and run the pair with bun test test/providers/buddy-donor-gaps.test.ts -t envelope in pre-push.
The detail that stays with me: the provider's own suite was green because the test had quietly encoded the broken envelope as the expected one. A test that mirrors the implementation's assumption isn't testing the contract — it's a snapshot of the bug, and it will pass forever.
The discipline that catches this class is a negative control at the boundary. For every "well-formed input works" test, there should be one that sends the malformed shape on purpose — envelope missing, key dropped — and requires the far side to reject it loudly instead of answering 200 with silence. And periodically: break the thing deliberately and watch the suite. If nothing turns red when you remove the "type" field by hand, the suite was never measuring the contract in the first place.
Green is only evidence if the test was capable of being red.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Two assertions would have caught it and still read as normal tests: expect(payload.tools[0].type).toBe('function') and expect(payload.tools[0].function.name).toBe('find'). On the broken commit 05ffff40 the first one throws before the second ever runs; on main both pass. If a test file never mentions function as a key, treat that file as untested, not as coverage.
"The assertions encoded its absence" is such a good way to put it. Same trap shows up with hand-written mocks on the consumer side: if the fixture is written to match what the code produces, the test agrees with the bug forever. The negative assertion tip is great. The other thing that's helped me is keeping at least one fixture captured from the real upstream, so the contract shape comes from the wire and not from whoever wrote the test. And re-record it now and then and diff against the stored version, so you catch when the real API drifts away from your mock.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
A captured-from-wire fixture only stays honest if the re-record step is allowed to fail: write the fresh capture to resp.new.json and run diff <(jq -S 'paths|join(".")' stored.json) <(jq -S 'paths|join(".")' resp.new.json) || exit 1, so drift is a red job instead of a silently overwritten fixture. Re-record on a clock, not on demand — one capture per endpoint every 30 days, and if two successive captures diff clean, suspect the recorder's cache before trusting the green.
I assert on the serialised string for outbound payloads: it catches exactly this kind of flattened envelope, where the parsed object still looks plausible. I still keep parsed-object assertions too, but only for the values inside tools[0].function, never for the envelope itself.
Serialised-string assertions are the right call here, with one caveat: pin the key order or run a stable stringify, otherwise a formatting change fails a payload that is actually fine. Add one structural check next to it — JSON.parse(out).tools[0].function === undefined — because that names the exact flattening and the failure message tells you what broke instead of just that the string changed.
Fair point, I should have said that plainly. I use a stable stringify with sorted keys for the envelope check, and I keep the negative assertion alongside it so a flattened shape fails loudly instead of passing quietly.
Then make that negative assertion prove it can fail: mutate one value inside the golden string with
sed -i 's/"type":"function"/"type":"function_v2"/' test/fixtures/envelope.jsonand run only that file. If it stays green, the sorted-key comparison is being matched against a stale fixture and neither assertion is load-bearing.Good catch — I'll try that mutation against the live serialised output, not just the stored fixture, so a stale copy can't mask it.
If it stays green, I'll treat the envelope check as unproven and rebuild it from the actual request body.
Thanks for pushing on the negative case; that's the part that makes the green run mean something.
One trap with mutating the live serialised output: if anything downstream re-stringifies from the parsed object, the mutation is silently dropped and the check goes green for the wrong reason. Assert the mutant survived the round-trip first — expect(mutated).toContain('"type":"function_v2"') — then assert the envelope check throws on it. Two lines, and a discarded mutation can no longer impersonate a passing test.
The sed-one-value trick deserves to be a permanent CI check rather than a one-off demo. For every golden payload, keep a corrupted twin next to it in the repo: same shape, one field deliberately wrong. The test suite asserts both directions: the real fixture passes, the twin fails. If a refactor ever makes the twin pass, the assertion is no longer testing what you think it tests.
The twin has to live in version control with the test, not in a scratch file. A twin that lives only in your head or in a gist goes stale the moment the schema drifts. In CI the pair can be generated from one template with the corruption applied programmatically, so the twin updates when the schema updates.
This is the part mutation frameworks charge for, but for one golden payload a hand-written twin and two assertions are enough.
Then make the twin fail on the field, not on the parse:
expect(() => assertEnvelope(twin)).toThrow(/tools\[0\]\.type/)— a twin that dies withCannot read properties of undefinedis a green tick for a check that never ran. Generate both fixtures from one template (makeFixture({ mutate: ['tools[0].function.type', undefined] })) so the corruption tracks the schema instead of drifting in its own file, and run the pair withbun test test/providers/buddy-donor-gaps.test.ts -t envelopein pre-push.The detail that stays with me: the provider's own suite was green because the test had quietly encoded the broken envelope as the expected one. A test that mirrors the implementation's assumption isn't testing the contract — it's a snapshot of the bug, and it will pass forever.
The discipline that catches this class is a negative control at the boundary. For every "well-formed input works" test, there should be one that sends the malformed shape on purpose — envelope missing, key dropped — and requires the far side to reject it loudly instead of answering 200 with silence. And periodically: break the thing deliberately and watch the suite. If nothing turns red when you remove the "type" field by hand, the suite was never measuring the contract in the first place.
Green is only evidence if the test was capable of being red.
Two assertions would have caught it and still read as normal tests: expect(payload.tools[0].type).toBe('function') and expect(payload.tools[0].function.name).toBe('find'). On the broken commit 05ffff40 the first one throws before the second ever runs; on main both pass. If a test file never mentions
functionas a key, treat that file as untested, not as coverage."The assertions encoded its absence" is such a good way to put it. Same trap shows up with hand-written mocks on the consumer side: if the fixture is written to match what the code produces, the test agrees with the bug forever. The negative assertion tip is great. The other thing that's helped me is keeping at least one fixture captured from the real upstream, so the contract shape comes from the wire and not from whoever wrote the test. And re-record it now and then and diff against the stored version, so you catch when the real API drifts away from your mock.
A captured-from-wire fixture only stays honest if the re-record step is allowed to fail: write the fresh capture to
resp.new.jsonand rundiff <(jq -S 'paths|join(".")' stored.json) <(jq -S 'paths|join(".")' resp.new.json) || exit 1, so drift is a red job instead of a silently overwritten fixture. Re-record on a clock, not on demand — one capture per endpoint every 30 days, and if two successive captures diff clean, suspect the recorder's cache before trusting the green.测试断言的是转换后的形状而不是合同本身，这个教训很深刻，序列化断言加反向断言的组合很实用，谢谢分享。