Caddy alludes to this but due to reasons there is no example. And it took me a long while to figure it out.
If you run this command (on Windows at least)
caddy file-server --domain localhost
The first time Caddy will add a cert to your machine and then you can run https://localhost in your browser. YAY!
Again it was not obvious from the documentation in any way shape or form that this is the right way to do it. I am not convinced this is what they intended but it works.