Flaky Tests at Google

Jeff Listfield has published an interesting read on where does tests’ flakiness at Google come from.

The main takeaway from that article is that the larger the binary size and more RAM the test consumed, the more likely it was to be flaky.

So, try to avoid creating large tests, and you should have less flaky tests!

It is worth noting that one way of addressing the flakiness of certain types of tests is using stubbing, mocking or service virtualization. When used properly, it can help stabilize the environment. It has to be assessed on a case-by-case basis if that is the right thing to do. For example, system tests will often be run in isolation, but end to end or smoke tests will require the whole stack to be deployed.