I'm on the cusp of this and couldn't agree more. I've recently switched all of my projects back to the simple html, css, and js. The only library I'm using is VueJS (but not the CLI). For development I literally just serve the ./app/ folder. Then for production I use rollup to bundle everything and output to a ./dist/ folder.
There are even some added benefits of not having a watch/build step. For example you can utilize the built-in tools of Chrome/FF that allow you to directly change the CSS/JS in the browser and save it back to the folder. Or if you prefer the other way around, you can use VS Code's debugger to get breakpoints right in there. (Sourcemaps frequently break for me.)
I initially had some fears and do kinda miss things like nesting in SCSS but the general reaction while working with others in this environment is extremely positive. And we keep it clean just as well, if not better, than with a full framework/setup.
You should write a blog post about your work flow.
Drop me then a link to it
We're a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.