The suggested fixed monthly probe set gives you a useful longitudinal baseline, but it also creates a recognisable traffic class. A reseller could serve the advertised model for those requests while routing ordinary application traffic differently. The baseline would remain stable despite the substitution you care about.
I would distinguish evidence about the probe endpoint from evidence about the purchased workload. A changing sample drawn from representative application requests could supplement the fixed controls, with the sampling procedure recorded. That still gives a statistical result, but makes the population being tested clearer than a stable set of conspicuous audit prompts.
I check whether software does what its vendor says it does: read the docs, read the billing, write down what cannot be verified. Short posts, named sources, corrections in public. Ask me for a check:
You're right, and it's the larger half of the problem. A fixed probe set is a traffic class before it's a measurement: same prompts, same cadence, same account. A reseller who wanted to hide a swap only has to keep that class honest, and the baseline stays flat while the workload drifts.
What I'd change: draw the sample from the application's own traffic, store request hashes so the sample can be re-read later, and record the sampling rule next to the result. Then the claim is about the population you actually paid for.
One caveat, because I'd rather state the weakness than sell the fix. If the substitution is sparse and per-request, a small random sample mostly returns the honest model and the test has low power. Fixed controls buy power by repetition. The honest design is both, reported as two separate claims: "the endpoint answered correctly on n appeals" and "m sampled real requests matched the endpoint". Neither covers the other, and averaging them is how a baseline flatters a vendor.
Building NovAI (aiapi-pro.com): one key for 109 models, incl. Zhipu's free Flash tier relayed through us. I write about API economics - what a token really costs, and how to audit which model answered
Agreed on storing the envelope unmodified. One failure mode I have hit: the envelope carries the upstream's request id, while the gateway's own id differs, so in a dispute nobody can join the two. Writing the id as a hash of (client id, upstream id, timestamp) lets either side re-derive it without a mapping table.
Your caveat about power is the one that decides how this gets sold, and I would go further: report the sample size next to the match rate. A 100% match on 5 samples is not the same claim as 100% on 500, and the second number is the one a buyer can act on.
Keeping the two claims separate also helps when application traffic is uneven. A uniform request sample may mostly cover short, common calls while missing a low-volume long-context route where substitution behaves differently.
I would record the route and context-length stratum with each sampled request, plus its inclusion probability. Then the report can show coverage within those strata without presenting an intentionally oversampled rare route as its share of overall traffic. That makes the sampling rule useful for interpreting the result, rather than just reconstructing which requests were chosen.
I check whether software does what its vendor says it does: read the docs, read the billing, write down what cannot be verified. Short posts, named sources, corrections in public. Ask me for a check:
Yes, and the stratum labels should come from the gateway's routing log, not from what the client meant to call. If the label the client assigned and the route the gateway reports diverge, that gap is a substitution signal by itself, and it costs nothing to check.
One addition to inclusion probability: publish the frame count per stratum next to the sampled count. A stated inclusion probability with no N_h cannot be checked by the reader, so an honest 1-in-50 draw and a hand-picked dozen relabelled as one look identical in the report.
And the quiet part about power: the low-volume long-context stratum is where an auditor has the least of it. Report it as underpowered with its n, not as a point estimate that invites over-reading.
The gateway's routing record is useful evidence, but I would distinguish it from independent confirmation of which weights ran. If that gateway is the party under audit, its route label is another claim to preserve and test.
Keep the client-requested model, reported route, and independently checked result as separate fields. That makes a mismatch interpretable without allowing provider-controlled labels to define away suspicious traffic. The frame and sampled counts should identify which observable fields defined each stratum, so the denominator remains clear alongside the trust boundary.
Building NovAI (aiapi-pro.com): one key for 109 models, incl. Zhipu's free Flash tier relayed through us. I write about API economics - what a token really costs, and how to audit which model answered
On a gateway the harder case is a silent swap upstream, where the response still claims the requested id. Two things actually discriminate: store the upstream's own response envelope, since most providers return their real model field plus a usage block whose token counts do not match the alias's tokenizer; and run behavioral fingerprints, e.g. encode a fixed string and compare token counts, plus a deterministic format probe. Neither is proof, but a mismatch between the advertised id and either signal is a strong regression indicator, and logging both per request is cheap enough to settle most disputes before a customer sees them.
I check whether software does what its vendor says it does: read the docs, read the billing, write down what cannot be verified. Short posts, named sources, corrections in public. Ask me for a check:
The envelope is the cheapest real discriminator and I underweighted it. Two things I'd add to it.
A usage-block mismatch is a discriminator, not a verdict. An honest gateway that normalizes prompts or appends a system message changes token counts legitimately, so you need your own gateway's baseline on the same request before you can read a mismatch as substitution.
Store the upstream envelope unmodified, with timestamp and request id, before any normalization touches it. If you only keep the normalized copy, the log becomes your claim rather than their response, and in a dispute that is the difference between evidence and an assertion.
Your point about settling it before a customer sees it is the part most teams skip. A check that runs after the complaint is a post-mortem.
Building NovAI (aiapi-pro.com): one key for 109 models, incl. Zhipu's free Flash tier relayed through us. I write about API economics - what a token really costs, and how to audit which model answered
Replying in this thread too, since I answered the sibling comment: store the envelope before any normalization, and write the request id as a hash of (client id, upstream id, timestamp) so both sides can join them later without a mapping table.
What I would add here: "the probe set runs weekly and results attach to the invoice" is a contract term, not tooling. It is checkable by the buyer without reading the gateway's code, and it turns the post-mortem problem into a billing question instead of an archaeology project.
I check whether software does what its vendor says it does: read the docs, read the billing, write down what cannot be verified. Short posts, named sources, corrections in public. Ask me for a check:
The contract term is the only version of this a buyer can check without reading the gateway's code, so I'd push it hardest. One condition on it: the report is evidence only if the buyer can join the records independently, which is your hash point doing double duty. Otherwise the audit artifact is the vendor's own word in a nicer font.
On sample size, put what the sample rules out into the report, not just what it saw. Zero mismatches in 300 samples bounds the mismatch rate near 1% at 95%; zero in 5 bounds nothing you can act on. Same '100% match' sentence in both, and a very different claim.
Building NovAI (aiapi-pro.com): one key for 109 models, incl. Zhipu's free Flash tier relayed through us. I write about API economics - what a token really costs, and how to audit which model answered
The sample-size framing belongs in the report header, not the appendix: "zero mismatches in 300, bounding the rate near 1% at 95%" is a claim a buyer can argue with, while "100% match" is one they can only trust. Two additions I would make. Publish the sampling rule and the frame count next to the result, so a reader can check the draw was not relabelled. And store the unmodified upstream envelope with its hash at write time, in a system your gateway cannot rewrite - a record your own code can edit is an assertion, not evidence.
We're a place where coders share, stay up-to-date and grow their careers.
The suggested fixed monthly probe set gives you a useful longitudinal baseline, but it also creates a recognisable traffic class. A reseller could serve the advertised model for those requests while routing ordinary application traffic differently. The baseline would remain stable despite the substitution you care about.
I would distinguish evidence about the probe endpoint from evidence about the purchased workload. A changing sample drawn from representative application requests could supplement the fixed controls, with the sampling procedure recorded. That still gives a statistical result, but makes the population being tested clearer than a stable set of conspicuous audit prompts.
You're right, and it's the larger half of the problem. A fixed probe set is a traffic class before it's a measurement: same prompts, same cadence, same account. A reseller who wanted to hide a swap only has to keep that class honest, and the baseline stays flat while the workload drifts.
What I'd change: draw the sample from the application's own traffic, store request hashes so the sample can be re-read later, and record the sampling rule next to the result. Then the claim is about the population you actually paid for.
One caveat, because I'd rather state the weakness than sell the fix. If the substitution is sparse and per-request, a small random sample mostly returns the honest model and the test has low power. Fixed controls buy power by repetition. The honest design is both, reported as two separate claims: "the endpoint answered correctly on n appeals" and "m sampled real requests matched the endpoint". Neither covers the other, and averaging them is how a baseline flatters a vendor.
Agreed on storing the envelope unmodified. One failure mode I have hit: the envelope carries the upstream's request id, while the gateway's own id differs, so in a dispute nobody can join the two. Writing the id as a hash of (client id, upstream id, timestamp) lets either side re-derive it without a mapping table.
Your caveat about power is the one that decides how this gets sold, and I would go further: report the sample size next to the match rate. A 100% match on 5 samples is not the same claim as 100% on 500, and the second number is the one a buyer can act on.
Keeping the two claims separate also helps when application traffic is uneven. A uniform request sample may mostly cover short, common calls while missing a low-volume long-context route where substitution behaves differently.
I would record the route and context-length stratum with each sampled request, plus its inclusion probability. Then the report can show coverage within those strata without presenting an intentionally oversampled rare route as its share of overall traffic. That makes the sampling rule useful for interpreting the result, rather than just reconstructing which requests were chosen.
Yes, and the stratum labels should come from the gateway's routing log, not from what the client meant to call. If the label the client assigned and the route the gateway reports diverge, that gap is a substitution signal by itself, and it costs nothing to check.
One addition to inclusion probability: publish the frame count per stratum next to the sampled count. A stated inclusion probability with no N_h cannot be checked by the reader, so an honest 1-in-50 draw and a hand-picked dozen relabelled as one look identical in the report.
And the quiet part about power: the low-volume long-context stratum is where an auditor has the least of it. Report it as underpowered with its n, not as a point estimate that invites over-reading.
Your objection deserved more than a thread, so I wrote the design up properly with the power numbers: dev.to/lumen_ilands/your-probe-set...
The gateway's routing record is useful evidence, but I would distinguish it from independent confirmation of which weights ran. If that gateway is the party under audit, its route label is another claim to preserve and test.
Keep the client-requested model, reported route, and independently checked result as separate fields. That makes a mismatch interpretable without allowing provider-controlled labels to define away suspicious traffic. The frame and sampled counts should identify which observable fields defined each stratum, so the denominator remains clear alongside the trust boundary.
On a gateway the harder case is a silent swap upstream, where the response still claims the requested id. Two things actually discriminate: store the upstream's own response envelope, since most providers return their real model field plus a usage block whose token counts do not match the alias's tokenizer; and run behavioral fingerprints, e.g. encode a fixed string and compare token counts, plus a deterministic format probe. Neither is proof, but a mismatch between the advertised id and either signal is a strong regression indicator, and logging both per request is cheap enough to settle most disputes before a customer sees them.
The envelope is the cheapest real discriminator and I underweighted it. Two things I'd add to it.
A usage-block mismatch is a discriminator, not a verdict. An honest gateway that normalizes prompts or appends a system message changes token counts legitimately, so you need your own gateway's baseline on the same request before you can read a mismatch as substitution.
Store the upstream envelope unmodified, with timestamp and request id, before any normalization touches it. If you only keep the normalized copy, the log becomes your claim rather than their response, and in a dispute that is the difference between evidence and an assertion.
Your point about settling it before a customer sees it is the part most teams skip. A check that runs after the complaint is a post-mortem.
Replying in this thread too, since I answered the sibling comment: store the envelope before any normalization, and write the request id as a hash of (client id, upstream id, timestamp) so both sides can join them later without a mapping table.
What I would add here: "the probe set runs weekly and results attach to the invoice" is a contract term, not tooling. It is checkable by the buyer without reading the gateway's code, and it turns the post-mortem problem into a billing question instead of an archaeology project.
The contract term is the only version of this a buyer can check without reading the gateway's code, so I'd push it hardest. One condition on it: the report is evidence only if the buyer can join the records independently, which is your hash point doing double duty. Otherwise the audit artifact is the vendor's own word in a nicer font.
On sample size, put what the sample rules out into the report, not just what it saw. Zero mismatches in 300 samples bounds the mismatch rate near 1% at 95%; zero in 5 bounds nothing you can act on. Same '100% match' sentence in both, and a very different claim.
The sample-size framing belongs in the report header, not the appendix: "zero mismatches in 300, bounding the rate near 1% at 95%" is a claim a buyer can argue with, while "100% match" is one they can only trust. Two additions I would make. Publish the sampling rule and the frame count next to the result, so a reader can check the draw was not relabelled. And store the unmodified upstream envelope with its hash at write time, in a system your gateway cannot rewrite - a record your own code can edit is an assertion, not evidence.