"Exploring the Art of Code Refactoring: Best Practices and Real-World Experiences"

Indeed need to think on balance between the immediate benefits of new features and the long-term benefits of a clean codebase....

The question is, when is the right time to refactor code?

The right time to refactor code is when it becomes necessary. Pre-optimization contributes to many failures with software projects.

What are the best practices for successful code refactoring?

• Clarity on the primary objective (ex: Delivering a feature)
• Having a solid understanding of how the existing code is used - you don't want your refactor creating a ton of bugs because you weren't aware of a certain use-case.
• Avoid scope-creep: Get your original refactor done and then revisit if you identified something else that should be refactored.
• Read all the F*cking Manuals. A LOT of pattern opinions exist in regards to programming, and most of them are valid - in specific contexts. I prefer to lay down code in a way that requires the least amount of documentation on my part - which typically means following the patterns documented by the libraries I use.
• DON'T RECREATE THE WHEEL. You may think you're clever writing implementing your own pattern instead of using what comes out-of-the-box but you're likely adding tech debt that someone will bitch about long after you've departed.

how do we strike a balance between the immediate benefits of new features and the long-term benefits of a clean codebase?

Accept that perfection is a vision, not a destination. If you care about your work, you'll always set out to create clean code that doesn't need to be refactored, but you can't predict the future. For example: I wrote very clean AngularJS for several years in the 2010s, but all of that code became tech debt once Angular shifted to TypeScript.