Uncle Bob, one of the biggest promoters of Test Driven Developement, wrote up a really good blog post about the problems you encountered here: blog.cleancoder.com/uncle-bob/2017...
The point people miss with TDD you need to do some kind of upfront design, then use the tests and code to validate your assuptions. You also need to design your tests to avoid all the extra work of reworking your tests when you refactor something. Using an interface or api layer between your code and tests allows for refacroring the code and not have to constantly rewrite tests. Also important to keep the tests just as clean as the production code.
I fail to see the necessity for steps 2 to 5 in your process? Why do you have to decide if the code you have written in step 2 needs refactoring, if you haven't written any code between step 1 and 2? The test is supposed to fail in the first step before you go about writing code to make the test pass. You would then need to run your test suite to identify and fix any unwanted side effects.
It is also important, in my opinion to correctly design the interface of the units in your system, because it facilitates teamwork.
Given that it's an iterative process, I'm referring to subsequent times round the loop.
Without doing detailed up front design of every class in your system, it's likely that you will write a class in one iteration and then change it in a future iteration. Possibly quite drastically.
The biggest problem I've got here is doing this in a compiled language. As soon as you write a new test for an as-yet-unwritten unit, your entire build fails to compile. This can make it unnecessarily difficult to determine what has and hasn't broken by the changes you're making in order to make the new test pass.
If instead you write the Unit Test - and I do only mean Unit Tests here - afterwards then you know that your code all compiles and that all existing tests still pass. You know the interface to your new code. You can test it and make sure it works, and if it fails or if the interface isn't as good as you first thought it's still easy to tweak.
Of course, the single most important thing here is - this is just my opinion. Everyone is different. There is no single perfect way to do this. If writing all of the tests first works for you then do not stop doing that. Do whatever works best for you to create great things :-)
Just curious what language and tools are you using? I have never ran into these issues using TDD. I use Visual Studio and it tells me right away what the compiler errors are. And if I am missing a class or function I just use the refactoring tools to create or rename them at will. Sounds a lot of issues might be tool related.
We're a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.