An iterative validation of a new implementation

Sometimes I need to replace or refactor a function in a program. In this post I want to share a process I’ve used to make such change. The goal is to improve our confidence in the end result.

Suppose you have a function f we need to migrate. For the sake of simplicity, we will assume f will never raise an exception.

def f(input : String)
  # ... stripped ...
end

There are no unit tests, but said function is widely used through the whole suite of tests.

First step, leave the original function and define a wrapper.

def original_f(input : String)
  # ... stripped ...
end

def f(input : String)
  original_f(input)
end

Second step, define a new implementation:

def new_f(input : String)
  # something new we hope will work
end

Third step, run both implementations and detect invalid behaviors of new_f with respect to original_f. These invalid behaviors can be either crashes or unexpected results.

To detect them we are going to change the wrapper with a bit of code that you will probably not like at all. Relax, it will be gone at the end.

def f(input : String)
  original_result = original_f(input)

  begin
    new_result = new_f(input)
  rescue e
    e.inspect_with_backtrace(STDOUT)
    File.write("bug.txt", input)
    exit(1)
  end

  if original_result != new_result
    File.write("bug.txt", input)
    puts "\n\nUnexpected result: #{input.inspect}.\n  Expected: #{original_result.inspect}\n       Got: #{new_result.inspect}\n\n"
    exit(1)
  end

  original_result
end

If there is an unexpected result the program will stop. Immediately. This can be done differently but, for the sake of simplicity, we are stopping at the first invalid behavior.

We run the whole suite of tests and, if there is a crash, a bug.txt file will be created with the input that caused the invalid behavior.

We can use a smaller program to work on the input that causes the crash.

def t(input)
  original_result = original_f(input)
  new_result = new_f(input)

  if original_result != new_result
    pp! input, original_result, new_result
  end
end

# Some trivial cases, maybe
t("")
t("abc")
t("   abc    ")
t("   ")

# The input that caused the crash
t(File.read("bug.txt"))

# You could also use a test framework, of course.

This way we can work on new_f until the identified case is fixed.

Iterate the process of running the whole suite of tests until there is no crash.

Now you have a new_f that works as original_f. At least, to the extent the suite of tests needs. Note that we are not talking about only unit tests of f.

Drop the "ugly" code to compare both implementations. Drop the original_f. We don't need them anymore. Leave only the new_f as the implementation of f.

def f(input : String)
  # something new we hope will work
end

You are done! 🎉

This process was explained with Crystal but it can be adapted easily to other languages.

Programming is a tool. Yet, as a tool, it can be used as a process. We made temporal additions to our codebase as part of the process. There is no need for the code you write to always be permanent in the code base.

There are more advanced processes and tools in software verification. This is a small example that might motivate you to look into them.