slides-move-fast-and-do-not-break-things
The aim should be for 100% reproducibility... what should you do if you can't get it?
predictable and deterministic
"There's the companies that use @flaky, which includes the vast majority of Python-using SF Bay area unicorns. If you don't know what this is, this is a library that lets you add a Python annotation to those annoying flaky tests that sometimes pass and sometimes fail. When I asked multiple co-workers and former co-workers from three different companies what they thought this did, they all guessed that it re-runs the test multiple times and reports a failure if any of the runs fail. Close, but not quite. It's technically possible to use @flaky for that, but in practice it's used to re-run the test multiple times and reports a pass if any of the runs pass. The company that created @flaky is effectively a storage infrastructure company, and the library is widely used at its biggest competitor. Marking tests that expose potential bugs as passing is totally normal." (#)
blog-post-test-that-cried-wolf
blog-post-how-to-deal-with-and-eliminate-flaky-tests
blog-post-eradicating-non-determinism-in-tests