Grouping outcomes by (test, commit_sha) and requiring len(seen) > 1 catches nondeterminism without flagging real regressions, though it creates a budget tradeoff in CI. When a pipeline runs each commit once unless someone manually retries, a flaky test leaves only one outcome per SHA and stays invisible to the scorer. Measuring it requires automatic reruns, which trades extra runner minutes for detection signal.
Splitting the deterministic scoring step from the Strands and Gemini agent keeps inference spend bounded too. Instead of passing 19,644 raw JUnit records into a prompt, the workflow reduces them to the 12 flagged tests before calling export_analysis. That limits token spend to the summarized exceptions.
The flakiness score you use — flaky commits over total commits, with the 0.1 threshold and a minimum of 5 commits — is the part I'd steal first, because it turns a vague 'this test is flaky' feeling into something a team can actually rank and triage. Running the flagged tests 10 times to confirm before the agent investigates is a nice guard against chasing ghosts. Have you found a window where the score starts misleading you, like a test that only flakes on a particular runner or at a certain time of day and still lands under the threshold?
Senior QA Engineer | SDET | AWS Community Builder | He4rt
I write extensively about test automation, performance, best practices, and various other useful things I have learned as a QA professional.
I haven’t come across such a case myself, but it’s a good point to investigate and consider how it might be incorporated into the project and your development process.
The test-and-commit grouping could mark a deterministic matrix failure as flaky. A test might pass on Python 3.12 and fail on 3.13 every time, even at the same SHA. I'd report those as configuration disagreements and only count mixed outcomes within the same runtime, OS and dependency lock. A fixture with one stable pass job and one stable fail job would make that distinction testable.
Senior QA Engineer | SDET | AWS Community Builder | He4rt
I write extensively about test automation, performance, best practices, and various other useful things I have learned as a QA professional.
That’s a very good point; it would be interesting to collect information about the environment (I still need to think about how to do this) and ensure that version discrepancies are not counted as flaky.
The trust-erosion point is the real cost: once a suite has enough intermittently failing tests, teams stop trusting any red and start skipping genuine failures. Scoring per test and per commit is a solid baseline, because a flake that only appears after a specific order or fixture state won't show up in a single run. In Electron E2E this is especially painful — timing-dependent waits, window focus, and first-run state produce the same intermittent red, and repeating the run masks it identically; keeping the failure traces before the AI root-cause pass anchors the analysis to real signals instead of guesses.
What I liked most is the order of operations: deterministic scoring first, AI analysis second. Defining flaky as the same test passing and failing on the same commit, then ranking with flaky commits over total commits at a 0.1 threshold and a minimum of five commits, gives the agent a clean, bounded input instead of asking it to guess from raw logs. That split also keeps the inference spend tied to a real signal.
Marcus's matrix point is the interesting edge case for me. If a test passes on one environment and fails on another with the same commit, the (test, commit) grouping could flag it as flaky when it is really deterministic per environment. Your reply about collecting environment information points the right way — adding that dimension to the grouping, alongside the commit_sha and run_id you already require, would make the score harder to misread and the agent's root-cause report more useful.
One tweak I'd try on error_messages: keep a count per message and normalize it first. Strip numbers, ids and durations, so "timed out after 30s" and "after 31s" become one entry. Right now the agent sees five distinct strings for test_9 but can't tell if one of them covers 40 of the 42 flaky commits. That ratio changes the advice a lot. If most failures are "503" and "connection reset", the fix is probably a stubbed or retried gateway, not the test itself. Did Gemini ever mix up an infra cause with a test cause on your sample data?
Senior QA Engineer | SDET | AWS Community Builder | He4rt
I write extensively about test automation, performance, best practices, and various other useful things I have learned as a QA professional.
I believe it was down to the sample data itself; as it was generated by a script, this can happen. One of the things I want to evaluate in the future is the analysis carried out using different models.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The order of operations here is what I'd steal — deterministic scoring first, AI analysis second. Most flaky-test tools I've seen lead with the AI part and hand you a confident-sounding root cause for a test that just needed five more commits of data. The 0.1 threshold with min 5 commits is a nice, honest floor. One question though: marcusykim's matrix point — have you thought about tagging the environment into the (test, commit) key, or would that explode the grouping?
Senior QA Engineer | SDET | AWS Community Builder | He4rt
I write extensively about test automation, performance, best practices, and various other useful things I have learned as a QA professional.
Yes, yes, I’ve been looking at some JUnit formats from certain testing frameworks that might provide more information for analysis, but I’m still organising my thoughts on how to incorporate this into a workflow with several stages
Staff Software Engineer @ Navan, ML Flow & Cypress Ambassador, Docker Captain and creator of agent-inspect, writing about production AI agents, TypeScript, observability, testing, and developer tools.
The deterministic (test, commit) scorer gives the Strands agent a bounded input. The discussion already surfaced environment keys and message normalization; the next test I'd want is for the agent's root-cause claim: seed a few failures with known causes, then check whether its explanation cites the matching JUnit evidence or says "unknown." When you compare models, will you score that evidence alignment separately from the summary's readability?
Senior QA Engineer | SDET | AWS Community Builder | He4rt
I write extensively about test automation, performance, best practices, and various other useful things I have learned as a QA professional.
To be honest, I haven’t really got round to thinking about that part yet; there’s been so much feedback that I now have loads of ideas, but it’s something I intend to do in the future
One root-cause category the scoring won't catch: order-dependent flakes. A test that passes alone but fails in the full suite isn't environment noise — it's state leaking between tests, and grouping by (test, sha) will file it under nondeterminism forever. Cheap way to make it mechanical: replay the suite with a fixed set of random seeds until the flake reproduces, then bisect the test sequence down to the minimal pair — the offender plus the test that triggers it. Add that seed to CI so the pair runs in the offending order every time, and the flake stops being "random" and becomes a bug with a reproduction.
The per-commit grouping is a useful way to distinguish inconsistent tests from ordinary regressions. It may also help to show the run count per commit, so readers can tell how much evidence sits behind each score.
One thing I would add is tracking the score over time rather than only looking at the weekly snapshot. A test creeping from 0.02 toward 0.1 is worth noticing before it crosses the threshold, and the JSON reports already have what is needed to chart that trend.
The commit-level grouping is the part I'd fix first — it treats every result for a commit as if it came from the same environment. If you run a matrix build (OS, Python 3.10 vs 3.12) or someone re-runs a failed job, you get mixed outcomes for reasons that have nothing to do with flakiness, and the score goes up for a test that never actually misbehaved.
You already parse run_id, but the grouping only uses commit_sha. Grouping by (commit_sha, run_id) first — then aggregating within a run — separates "the test failed and passed in the same environment" from "the test failed on Python 3.10 and passed on 3.12." The first is flakiness, the second is a real signal you probably want to keep.
The 0.1 threshold with a minimum of five commits is a clear starting point, but the calibration question is interesting: a test that fails once in twenty runs and a test that fails three times in ten runs can end up with very different scores, yet both may need different handling. Weighting recent runs more heavily, or reporting a confidence interval alongside the score, could help separate a genuinely flaky test from a small-sample artefact.
Another useful split might be between failure modes. A timeout or resource-related flake and an assertion flake caused by shared state or ordering point to different fixes, even when their scores look similar. Would it make sense for the agent report to group flakes by that kind of cause, and to suggest quarantine for the highest-scoring ones while lower-scoring ones stay under observation?
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
Scoring deterministically first and only then handing the agent a bounded report is a good split. One question for suites where the test runner retries inside the job: depending on reporter settings, a test that fails and then passes on retry can be written to the JUnit file as a single pass. Does your parser see the failed attempts, or only the final result?
I write when an idea won't leave me alone 🧠 Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
Your defaults have a fun side effect. With min_commits at 5 and a 0.1 threshold, one commit where a test both passed and failed flags it until it has more than 10 commits of history, while a test with 40 commits needs four bad ones. I think that's the right bias for new tests, though a fresh test's 0.2 can read as twice as flaky as an old test's 0.1.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
scoring per test per commit pair is the right unit, because a test that fails on commit A and passes on commit B is a regression and not flake. what i'd be curious about is infrastructure noise. we had a stretch where a shared runner with noisy neighbors made timing sensitive tests look flaky when the code was fine, and a root cause agent will happily call that a race condition. do you feed it runner or resource metadata alongside the JUnit XML, or does it only see the failure message and stack trace?
We're a place where coders share, stay up-to-date and grow their careers.
Grouping outcomes by (test, commit_sha) and requiring len(seen) > 1 catches nondeterminism without flagging real regressions, though it creates a budget tradeoff in CI. When a pipeline runs each commit once unless someone manually retries, a flaky test leaves only one outcome per SHA and stays invisible to the scorer. Measuring it requires automatic reruns, which trades extra runner minutes for detection signal.
Splitting the deterministic scoring step from the Strands and Gemini agent keeps inference spend bounded too. Instead of passing 19,644 raw JUnit records into a prompt, the workflow reduces them to the 12 flagged tests before calling export_analysis. That limits token spend to the summarized exceptions.
The flakiness score you use — flaky commits over total commits, with the 0.1 threshold and a minimum of 5 commits — is the part I'd steal first, because it turns a vague 'this test is flaky' feeling into something a team can actually rank and triage. Running the flagged tests 10 times to confirm before the agent investigates is a nice guard against chasing ghosts. Have you found a window where the score starts misleading you, like a test that only flakes on a particular runner or at a certain time of day and still lands under the threshold?
I haven’t come across such a case myself, but it’s a good point to investigate and consider how it might be incorporated into the project and your development process.
The test-and-commit grouping could mark a deterministic matrix failure as flaky. A test might pass on Python 3.12 and fail on 3.13 every time, even at the same SHA. I'd report those as configuration disagreements and only count mixed outcomes within the same runtime, OS and dependency lock. A fixture with one stable pass job and one stable fail job would make that distinction testable.
That’s a very good point; it would be interesting to collect information about the environment (I still need to think about how to do this) and ensure that version discrepancies are not counted as flaky.
The trust-erosion point is the real cost: once a suite has enough intermittently failing tests, teams stop trusting any red and start skipping genuine failures. Scoring per test and per commit is a solid baseline, because a flake that only appears after a specific order or fixture state won't show up in a single run. In Electron E2E this is especially painful — timing-dependent waits, window focus, and first-run state produce the same intermittent red, and repeating the run masks it identically; keeping the failure traces before the AI root-cause pass anchors the analysis to real signals instead of guesses.
What I liked most is the order of operations: deterministic scoring first, AI analysis second. Defining flaky as the same test passing and failing on the same commit, then ranking with flaky commits over total commits at a 0.1 threshold and a minimum of five commits, gives the agent a clean, bounded input instead of asking it to guess from raw logs. That split also keeps the inference spend tied to a real signal.
Marcus's matrix point is the interesting edge case for me. If a test passes on one environment and fails on another with the same commit, the (test, commit) grouping could flag it as flaky when it is really deterministic per environment. Your reply about collecting environment information points the right way — adding that dimension to the grouping, alongside the commit_sha and run_id you already require, would make the score harder to misread and the agent's root-cause report more useful.
One tweak I'd try on error_messages: keep a count per message and normalize it first. Strip numbers, ids and durations, so "timed out after 30s" and "after 31s" become one entry. Right now the agent sees five distinct strings for test_9 but can't tell if one of them covers 40 of the 42 flaky commits. That ratio changes the advice a lot. If most failures are "503" and "connection reset", the fix is probably a stubbed or retried gateway, not the test itself. Did Gemini ever mix up an infra cause with a test cause on your sample data?
I believe it was down to the sample data itself; as it was generated by a script, this can happen. One of the things I want to evaluate in the future is the analysis carried out using different models.
The order of operations here is what I'd steal — deterministic scoring first, AI analysis second. Most flaky-test tools I've seen lead with the AI part and hand you a confident-sounding root cause for a test that just needed five more commits of data. The 0.1 threshold with min 5 commits is a nice, honest floor. One question though: marcusykim's matrix point — have you thought about tagging the environment into the (test, commit) key, or would that explode the grouping?
Yes, yes, I’ve been looking at some JUnit formats from certain testing frameworks that might provide more information for analysis, but I’m still organising my thoughts on how to incorporate this into a workflow with several stages
The deterministic (test, commit) scorer gives the Strands agent a bounded input. The discussion already surfaced environment keys and message normalization; the next test I'd want is for the agent's root-cause claim: seed a few failures with known causes, then check whether its explanation cites the matching JUnit evidence or says "unknown." When you compare models, will you score that evidence alignment separately from the summary's readability?
To be honest, I haven’t really got round to thinking about that part yet; there’s been so much feedback that I now have loads of ideas, but it’s something I intend to do in the future
One root-cause category the scoring won't catch: order-dependent flakes. A test that passes alone but fails in the full suite isn't environment noise — it's state leaking between tests, and grouping by (test, sha) will file it under nondeterminism forever. Cheap way to make it mechanical: replay the suite with a fixed set of random seeds until the flake reproduces, then bisect the test sequence down to the minimal pair — the offender plus the test that triggers it. Add that seed to CI so the pair runs in the offending order every time, and the flake stops being "random" and becomes a bug with a reproduction.
The per-commit grouping is a useful way to distinguish inconsistent tests from ordinary regressions. It may also help to show the run count per commit, so readers can tell how much evidence sits behind each score.
One thing I would add is tracking the score over time rather than only looking at the weekly snapshot. A test creeping from 0.02 toward 0.1 is worth noticing before it crosses the threshold, and the JSON reports already have what is needed to chart that trend.
The commit-level grouping is the part I'd fix first — it treats every result for a commit as if it came from the same environment. If you run a matrix build (OS, Python 3.10 vs 3.12) or someone re-runs a failed job, you get mixed outcomes for reasons that have nothing to do with flakiness, and the score goes up for a test that never actually misbehaved.
You already parse run_id, but the grouping only uses commit_sha. Grouping by (commit_sha, run_id) first — then aggregating within a run — separates "the test failed and passed in the same environment" from "the test failed on Python 3.10 and passed on 3.12." The first is flakiness, the second is a real signal you probably want to keep.
The 0.1 threshold with a minimum of five commits is a clear starting point, but the calibration question is interesting: a test that fails once in twenty runs and a test that fails three times in ten runs can end up with very different scores, yet both may need different handling. Weighting recent runs more heavily, or reporting a confidence interval alongside the score, could help separate a genuinely flaky test from a small-sample artefact.
Another useful split might be between failure modes. A timeout or resource-related flake and an assertion flake caused by shared state or ordering point to different fixes, even when their scores look similar. Would it make sense for the agent report to group flakes by that kind of cause, and to suggest quarantine for the highest-scoring ones while lower-scoring ones stay under observation?
Thanks for sharing, planning on including this in the aiinqa.com newsletter this week
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
Scoring deterministically first and only then handing the agent a bounded report is a good split. One question for suites where the test runner retries inside the job: depending on reporter settings, a test that fails and then passes on retry can be written to the JUnit file as a single pass. Does your parser see the failed attempts, or only the final result?
Your defaults have a fun side effect. With min_commits at 5 and a 0.1 threshold, one commit where a test both passed and failed flags it until it has more than 10 commits of history, while a test with 40 commits needs four bad ones. I think that's the right bias for new tests, though a fresh test's 0.2 can read as twice as flaky as an old test's 0.1.
scoring per test per commit pair is the right unit, because a test that fails on commit A and passes on commit B is a regression and not flake. what i'd be curious about is infrastructure noise. we had a stretch where a shared runner with noisy neighbors made timing sensitive tests look flaky when the code was fine, and a root cause agent will happily call that a race condition. do you feed it runner or resource metadata alongside the JUnit XML, or does it only see the failure message and stack trace?