At a certain point we started expecting to see tests in our Gems and started looking at test-less software with a wary eye.
We should increase our expectations of open source projects.
A Vagrantfile with a single script that installs from a known open source base (like Ubuntu 14.04) and runs the tests to completion. A pre-configured Vagrant box is not good enough... it must install from sources, and the installation process should be well documented.