AI Tools and Refactoring

Previously in the series, we saw that AI tools have limitations. We also
discussed maximizing our productivity by using them to write code - without
shooting ourselves in our legs.

In this post, we'll cover one more piece of the puzzle: how they can (or can't)
help us to refactor.

This post was heavily influenced by thoughtworks' excellent podcast episode
Refactoring with AI.

The Goal of Refactoring

If we want to boil it down, refactoring has a simple goal: to make it easier to
change the code.

People usually make this identical to improving code quality. Most of the time,
this is indeed what we want to achieve. Higher-quality code is easier to
understand, reason about, and, as a result, modify.

However, there is a case where we decrease code quality during refactoring.
Sometimes, we need to restructure the code to introduce new behavior later. For
example, consider we have a Dog class:

class Dog {
  String communicate() {
    return "woof";
  }

  String move() {
    return "run";
  }
}

We realize that we want to have other types of pets, so we introduce a Pet
superclass:

abstract class Pet {
    abstract String communicate();
    abstract String move();
}

class Dog extends Pet {
  @Override
  String communicate() {
    return "woof";
  }

  @Override
  String move() {
    return "run";
  }
}

The code got unnecessarily complicated - for the current feature set. However,
now it's much easier to introduce cats to the codebase:

class Cat extends Pet {
  @Override
  String communicate() {
    return "meow";
  }

  @Override
  String move() {
    return "climb";
  }
}

In the rest of the article, we'll focus on code quality improvement because
that's the primary motivation for refactoring.

Measuring Code Quality

We want to quantify code quality objectively. Otherwise, it'd be hard to reason
whether refactoring improved it.

CodeScene has a metric called code health. Adam
Tornhill, the CTO and founder of the company, explains it in the following way:

The interesting thing with code quality is that there's not a single metric
that can capture a multifaceted concept like that. I mean, no matter what
metric you come up with, there's always a way around it, or there's always a
counter case. What we've been working on for the past six to seven years, is
to develop the code health metric.

The idea with code health is that, instead of looking at a single metric, you
look at a bunch of metrics that complement each other. What we did was that,
we looked at 25 metrics that we know from research that they correlate with an
increased challenge in understanding the code. The code becomes harder to
understand.

What we do is, basically, we take these metrics, there are 25 of them, stick
them as probes into the code, and pull them out and see what did we find.
Then, you can always, weigh these different metrics together, and you can
categorize code as being either healthy, or unhealthy. My experience is
definitely that when code is unhealthy, when it's of poor quality, it's never
a single thing. It's always a combination of factors.

Improving Code Quality With AI

Now that we can measure code quality, we can benchmark AI tools' capabilities.
Fortunately, Adam did the heavy lifting for us:

Basically, the challenge we were looking at was that, we have a tool that is
capable of identifying bad code, prioritizing it, but then of course, you need
to act on that data. You need to do something with the code, improve it. This
is a really hard task, so we thought that maybe generative AI can help us with
that. What we started out with was a data lake with more than 100,000 examples
of poor code. We also had a ground truth, because we had unit tests that
covered all these code samples.

We knew that the code does, at least, what the test says it does. What we then
did was that we benchmarked a bunch of AI service as like Open AI, Google,
LLaMA from Facebook, and instructed them to refactor the code. What we found
was quite dramatic, in 30% of the cases, the AI failed to improve the
code. Its code health didn't improve, it just wrote the code in a different
way, but the biggest drop off was when it came to correctness, because in
two-thirds of the cases, the AI actually broke the tests, meaning, it's not
the refactoring, it has actually changed the behavior. I find it a little bit
depressing that in two-thirds of the cases, the AI won't be able to refactor
the code.

What does that mean to us? We can't use AI to refactor the code?

Fortunately, that's not the case. Even though we can't unquestionably trust AI
to improve our code, we can utilize it in a controlled fashion:

• Detection: Static code analyzers are very efficient but have limited capabilities. They identify bad practices as patterns - which are sometimes hard to detect. AI tools can deal with more complex cases - especially if they work with the abstract syntax tree instead of the code itself.
• Suggestions: Once we detect a problem, we can suggest solutions. Generative AI can shine in that, too.
• Localized refactoring: If we restrict the scope of the refactoring, tools have to work with a much less complex problem space. Less complexity means less room for errors - namely, breaking the tests.

The commonality in these techniques is that we have control over what is
happening in the codebase - which is crucial for today's AI tools.

Conclusion

The tools that are available today can't reliably improve code quality. Or even
ensure correctness. For these reasons, tests and manual supervision are still
essential when we use AI tools.

We should take Adam's results with a grain of salt, though. The tools he tried
weren't optimized for refactoring. We saw that code quality and correctness are
quantifiable through code health and the number of failing tests. Once AI tool
creators optimize their products toward these metrics, we can expect them to
improve - significantly. I can't wait to see it.