Introducing Skunk: Combine Code Quality and Coverage to Calculate a Stink Score

Two weeks ago I had the opportunity to speak at Solidus Conf 2019.I presented Escaping the Tar Pitfor the first time and I got to talk about a few metrics that we can use toquickly assess code qualityin any Ruby project.

In this article I'd like to talk about Skunk: A Stink Score Calculator!I'll explain why we need it, how it works, and the roadmap for this new tool.

Every month we get contacted by leads (potential clients) who want to work withus on their Rails upgrade projects. Given that wehave some basic requirements for all of our new client projects, we want tocarefully analyze every project before we commit to working on it.

We analyze two very important aspects:

1. Code Coverage
2. Code Quality

For Code Coverage we like to use SimpleCov.For Code Quality we like to use RubyCritic.Both tools give us a few signals which tell us a story about the health of aRails application. We want to answer these questions:

• Is it a dumpster fire?
• Are we going to get ourselves stuck in the tar pit?
• Is it a project that is easy to maintain?

Skunk is a Ruby gem that will combine codequality metrics from Reek;Flay;Flog; andSimpleCov to calculate a Stink Scorefor a file or set of files.

Skunk is a library built on top ofRubyCritic. It uses the cost valuefor each module:

module RubyCritic
  class AnalysedModule
    def cost
      @cost ||= smells.map(&:cost).inject(0.0, :+) +
                (complexity / COMPLEXITY_FACTOR)
    end
  end
end

The cost is a combination of smells and complexity:

• Smells: They come from static code analysis performed by Flog; Flay; and Reek.
• Complexity: It comes from Flog's total ABC metric

After determining that the cost, Skunk penalizes modules which lack code coverageby multiplying their cost by a factor directly related to the lack of coverage:

module RubyCritic
  # Monkey-patches RubyCritic::AnalysedModule to add a stink_score method
  class AnalysedModule
    PERFECT_COVERAGE = 100

    # Returns a numeric value that represents the stink_score of a module:
    #
    # If module is perfectly covered, stink score is the same as the
    # `churn_times_cost`
    #
    # If module has no coverage, stink score is a penalized value of
    # `churn_times_cost`
    #
    # For now the stink_score is calculated by multiplying `churn_times_cost`
    # times the lack of coverage.
    #
    # For example:
    #
    # When `churn_times_cost` is 100 and module is perfectly covered:
    # stink_score => 100
    #
    # When `churn_times_cost` is 100 and module is not covered at all:
    # stink_score => 100 * 100 = 10_000
    #
    # When `churn_times_cost` is 100 and module is covered at 75%:
    # stink_score => 100 * 25 (percentage uncovered) = 2_500
    #
    # @return [Float]
    def stink_score
      return churn_times_cost.round(2) if coverage == PERFECT_COVERAGE

      (churn_times_cost * (PERFECT_COVERAGE - coverage.to_i)).round(2)
    end
  end
end

After doing all these calculations, we get a Stink Score for the files we are evaluating:

$ skunk
running flay smells
.............
running flog smells
.............
running reek smells
.............
running complexity
.............
running attributes
.............
running churn
.............
running simple_cov
.............
New critique at file:////skunk/tmp/rubycritic/overview.html
+-----------------------------------------------------+----------------------------+----------------------------+----------------------------+----------------------------+----------------------------+
| file | stink_score | churn_times_cost | churn | cost | coverage |
+-----------------------------------------------------+----------------------------+----------------------------+----------------------------+----------------------------+----------------------------+
| lib/skunk/cli/commands/default.rb | 166.44 | 1.6643999999999999 | 3 | 0.5548 | 0 |
| lib/skunk/cli/application.rb | 139.2 | 1.392 | 3 | 0.46399999999999997 | 0 |
| lib/skunk/cli/command_factory.rb | 97.6 | 0.976 | 2 | 0.488 | 0 |
| test/test_helper.rb | 75.2 | 0.752 | 2 | 0.376 | 0 |
| lib/skunk/rubycritic/analysed_module.rb | 48.12 | 1.7184 | 2 | 0.8592 | 72.72727272727273 |
| test/lib/skunk/cli/commands/status_reporter_test.rb | 45.6 | 0.456 | 1 | 0.456 | 0 |
| lib/skunk/cli/commands/base.rb | 29.52 | 0.2952 | 3 | 0.0984 | 0 |
| lib/skunk/cli/commands/status_reporter.rb | 8.0 | 7.9956 | 3 | 2.6652 | 100.0 |
| test/lib/skunk/rubycritic/analysed_module_test.rb | 2.63 | 2.6312 | 2 | 1.3156 | 100.0 |
| lib/skunk.rb | 0.0 | 0.0 | 2 | 0.0 | 0 |
| lib/skunk/cli/options.rb | 0.0 | 0.0 | 2 | 0.0 | 0 |
| lib/skunk/version.rb | 0.0 | 0.0 | 2 | 0.0 | 0 |
| lib/skunk/cli/commands/help.rb | 0.0 | 0.0 | 2 | 0.0 | 0 |
+-----------------------------------------------------+----------------------------+----------------------------+----------------------------+----------------------------+----------------------------+

Stink Score Total: 612.31
Modules Analysed: 13
Stink Score Average: 0.47100769230769230769230769231e2
Worst Stink Score: 166.44 (lib/skunk/cli/commands/default.rb)

The most important signals here are:

• Average Stink Score per module
• Most complex files with little to no code coverage

We now know where we stand. We can clearly see the state of the application interms of code coverage and project complexity. We can now answer this question: "Which are the most complex files with the least coverage?"

We can use the Stink Score to guide us in our refactoring efforts:

• How can I pay off technical debt and invest in the future of my application?
• If I were to write tests to decrease the stink score, which files could Iwrite tests for?
• If I were to refactor some of the most complex files, which files with goodcode coverage could I refactor?

Caveats

Skunk expects you to have a .resultset.json file in the coverage directorywithin the directory that you are evaluating. It uses the data within that fileto calculate the code coverage percentage for each module.

That means that you will have to run your test suite with SimpleCov enabled before you call skunk.

Total Stink Score is not a useful metric within a single project, as the totalwill continue to grow as you add more features to your application. It iscertainly a useful metric if you use it to compare two projects.

Known Issues

The calculation of the Stink Score is not 100% accurate. It is comparing amodule's code coverage and a module's complexity. It should be a method-basedcalculation: It should calculate the complexity of a method, the code coverageof the same method, then calculate the Stink Score per method.

Finally, the Stink Score of a module should be the sum of all the Stink Scoresin the module.

Roadmap

Assessing code quality for an application shouldn't stop at the applicationlevel. The Stink Score of our application is composed by two Stink Scores:

• Stink Score of your application
• Stink Score of your dependencies

Right now Skunk will only calculate Stink Score for your application code. Inthe future it should consider your dependencies as well, generating a StinkScore for each individual dependency.

The best way to assess progress in your project is to keep track of the StinkScore average over time. Is that number going up? Is it going down? How muchdoes your pull request change the Stink Score average? Right now Skunk doesnot support this, so you will have to do it manually.

Final Thoughts

I know that "stink" is a negative word to judge an application's technical debtand it might lead you down a negative path. By all means I don't want the StinkScore to be used in a witch hunt, to point fingers at code authors, or in anegative way in your team.

I seriously hope that you can use the Stink Score as the compass to move yourteam in the right direction. You should be able to use the Stink Score as acompass to gradually pay off technical debt:

• Writing tests which increase code coverage will improve the Stink Score
• Refactoring complex files will improve the Stink Score

Skunk will show you your location in the map of technical debt. It will alsoshow you a few paths to take to get to a better place. You will be able toprioritize the paths and pick one to pay off technical debt.

What do you think about this new metric for technical debt? Would you use itnext time you need to evaluate legacy code?

Please let me know in the comments below or come talk to me atRubyConf 2019 (I'll be speaking aboutthis topic at the conference)