Hunt down and fix flaky tests in CI
The coding agent reruns the suite to identify flaky tests, traces each one to a race, shared state or timing assumption, and opens a fix with an explanation.
- Agent
- Claude Code
- Difficulty
- Advanced
- Setup time
- About 20 min to set up
- Published
How it works
Point the agent at the repository and the CI history.
It reruns the suspected tests in a loop to confirm flakiness.
For each flaky test it finds the root cause.
It commits a fix per test with a short write-up.
The prompt
Replace anything in [brackets] with your own details.
Our CI has been failing intermittently. Look at the last 30 CI runs, list the tests that failed at least once but also passed on retry, and confirm each by running it 50 times locally. For every confirmed flaky test, find the root cause (shared state, timing, ordering, network) and fix it without adding sleeps or retries. Make one commit per test explaining the cause.Reported result
CI pass rate went from 82% to 99% in a week.
As reported by the source. Results depend on your setup and data.