Some caches can be slow to upload and download from, resulting in slower overall build time. Also, each caching step doubles the number of conditional branches that you need to verify after making a change to your CI script.
I would try caching in each case and measure difference in build time.
I have been leaning more towards having developers run tests locally, produce a text file with a hash based on source files of the last test run, commit the text file into repo, and have the PR builds just verify that last test run matches source code. Then have PR merge build do a full build as a last minute check of reproducibility.
Yeah I’ve noticed that getting and updating cache can take 1-2 minutes, depending on the size of the cache. But given a fairly large list of dependencies it’s almost always worth it.
As for changes in CI scripts: Can’t you set up cache to re validate when it detects changes in key files, such as package.json.
The idea of running Tests locally and uploading proof is interesting. But I’d be worried about having uncontrollable environments and side effects. Centralized CI allows for a “source of truth”. Or how do you manage your developer environments?
Centralized CI allows for a “source of truth”. Or how do you manage your developer environments?
I suspect most people use CI to just ensure all devs are running the tests...for that case, I would just use the check I mentioned, along with scripts that encourage runnings tests locally with minimum friction. As mentioned, I still would use CI as a final check after PR review is complete, but not on each change in a PR.
The next level is people trying to ensure the tests run from a clean state. You can also encourage this with scripts to clean local environment state. Generally, most languages have some sort of sandbox or dependency management, so builds are relatively isolated and reproducible.
The next level would be people ensuring the build works on multiple operating systems or on a blessed operating system. At that point, CI can help if devs use macos and blessed operating system is linux.
A final level is when people want to speed up their integration tests by split and running in parallel. For this, I would use CI.
As mentioned, I still would use CI as a final check after PR review is complete, but not on each change in a PR.
For me it's the other way around, I want automatic tests to pass before I review the PR, because developer time is more valuable then CI time.
You can also encourage this with scripts to clean local environment state.
Sure you can clean the git state and and build directory, however this could still mean that developers run different compiler versions, or different HW architecture (Intel vs Apple Silicon), which essentially makes very hard, if not impossible to have reproducible builds and tests. Which is why I prefer having a fast CI setup as gatekeeper and never release anything that has not been built by the CI server.
I just vote for fast local tests (I would probably wrap git push with an alias that ran unit tests && git push instead and ask everyone to use the alias), along with a final check before merging. If you keep thinking through all the scenarios that make CI tests slower than local dev, you will ultimately notice that there are multiple scenarios where CI will always become the bottleneck in your code workflow. One scenario is when you modify the version of one of your external dependencies. The tests will run quickly locally, but CI will have a busted cache and need to fetch all the dependencies again and possibly upload the new set to a cache. This will happen on a regular basis. Developers will complain that CI can become slow sometimes.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Some caches can be slow to upload and download from, resulting in slower overall build time. Also, each caching step doubles the number of conditional branches that you need to verify after making a change to your CI script.
I would try caching in each case and measure difference in build time.
I have been leaning more towards having developers run tests locally, produce a text file with a hash based on source files of the last test run, commit the text file into repo, and have the PR builds just verify that last test run matches source code. Then have PR merge build do a full build as a last minute check of reproducibility.
Yeah I’ve noticed that getting and updating cache can take 1-2 minutes, depending on the size of the cache. But given a fairly large list of dependencies it’s almost always worth it.
As for changes in CI scripts: Can’t you set up cache to re validate when it detects changes in key files, such as package.json.
The idea of running Tests locally and uploading proof is interesting. But I’d be worried about having uncontrollable environments and side effects. Centralized CI allows for a “source of truth”. Or how do you manage your developer environments?
I just meant that you have to verify this logic works as you expect, which usually means running the ci build twice to try out the two branches.
I suspect most people use CI to just ensure all devs are running the tests...for that case, I would just use the check I mentioned, along with scripts that encourage runnings tests locally with minimum friction. As mentioned, I still would use CI as a final check after PR review is complete, but not on each change in a PR.
The next level is people trying to ensure the tests run from a clean state. You can also encourage this with scripts to clean local environment state. Generally, most languages have some sort of sandbox or dependency management, so builds are relatively isolated and reproducible.
The next level would be people ensuring the build works on multiple operating systems or on a blessed operating system. At that point, CI can help if devs use macos and blessed operating system is linux.
A final level is when people want to speed up their integration tests by split and running in parallel. For this, I would use CI.
For me it's the other way around, I want automatic tests to pass before I review the PR, because developer time is more valuable then CI time.
Sure you can clean the git state and and build directory, however this could still mean that developers run different compiler versions, or different HW architecture (Intel vs Apple Silicon), which essentially makes very hard, if not impossible to have reproducible builds and tests. Which is why I prefer having a fast CI setup as gatekeeper and never release anything that has not been built by the CI server.
I just vote for fast local tests (I would probably wrap git push with an alias that ran unit tests && git push instead and ask everyone to use the alias), along with a final check before merging. If you keep thinking through all the scenarios that make CI tests slower than local dev, you will ultimately notice that there are multiple scenarios where CI will always become the bottleneck in your code workflow. One scenario is when you modify the version of one of your external dependencies. The tests will run quickly locally, but CI will have a busted cache and need to fetch all the dependencies again and possibly upload the new set to a cache. This will happen on a regular basis. Developers will complain that CI can become slow sometimes.