What is local testing?
comment-run-tests-before-committing
This blog post talks a little bit about the issues associated with slow tests: blog-post-just-say-no-to-more-end-to-end-tests
CORALINE: Noel, we've talked a lot about legacy code and we started touching on test times. And I just have to bring up the fact that when you end up with a long in the tooth Rails application it's not unusual in my experience for local testing not to happen at all and throwing everything over the wall to CI.
NOEL: Yeah.
CORALINE: And that of course ruins the whole point of testing as a feedback loop as part of your design and development. So, what are some strategies [inaudible] that?
NOEL: The best strategy for dealing with that is not to get in that situation in the first place. But of course, once you're in that situation that's horrible advice.
CORALINE: Barring time travel, yes.
NOEL: Yeah, barring time travel. Yeah, I worked at a place and I wasn't a direct developer on that at this place that I'm talking about. But I certainly worked at a place that had all of their testing happen on a very, very powerful CI server farm and it took 30, 40 minutes on the server farm. Nobody had any idea how long it took on a local laptop because nobody in their right mind would ever try to run the test suite on their local machine.
The Hydra guy, Nick Gauthier said:
"Continuous integration is great. Working, testing, And then committing tests and code. Always works real nice. The problem is when you're working for 30 minutes and then our test take 30 minutes it's just not really a very efficient process. So at that point a lot of people say hey let's build the CI server to run a test for us. The CI servers are helpful for running code in production environments but this usually ends up getting you out of sync with your usual testing process so your feedback loop is still 30 to 40 minutes behind."
The other thing that I don't like about relying on CI is that you're pushing untested code out into the world (if you're committing on master and relying on CI to catch the issues). Other developers could pull that bad master commit before the build breaks. People get around this by cutting branches just to see CI pass.
Aaron Patterson talks a little about what should be done
But what if we could predict which tests are likely to fail after we've changed the code? If we run just those tests, then we could send a pull request and let Travis run the thousands of unrelated tests in the whole suite.
41:45 - slow tests are a big cause of false negative test failures because you don't run them all locally and you push to CI to see what broke. Integration tests are often the cause of these.