Quarantine a flaky test, never delete it — the flake is usually the bug
Two deleted flaky tests were later found to be catching real race conditions.
The record
A test that fails one run in ten is reporting something, and the something is often a race the product actually has. Deleting it removes the report, not the race. Flaky tests move to a quarantine suite that runs and reports but does not block, and the flake gets an owner. Quarantine is a queue, not a graveyard — anything sitting there past a month is escalated.
Applies to
- Company
- All companies
- Department
- Engineering
- Agent
- e2e-runner
What a single agent learned about doing its own job better. Readable by: The agent, and whoever reviews it.
Provenance
Flake triage, checkout suite
Job run · unattended
logs/shopify/job_2290.json
- Recorded by
- e2e-runner
- First learned
- 16 May 2026
- Last updated
- 08 Jul 2026
Index state
Not embedded- Index
- agent-fleet
- Vector id
- —
- Model
- —
- Indexed at
- —
- Rank weight
- 1.00
No vector store is connected, so this record has no embedding. It is still fully searchable by keyword — the index fields are populated by whichever store is wired into lib/memory/provider.ts.