re: My Tests Are Being Maintained by Artificial Intelligence VIEW POST

re: I would struggle to trust the results of something like this. I defer to my bible of unit testing - Osherove's The Art of Unit Testing: UPDATED AN...

This isn't a unit test, it's a end 2 end integration test. So the unit test definition doesn't apply here :) Just sayin #noharm


I was hoping that the reason Osherove included the bold sentence in his definition would be clear to everyone, but apparently that needs to be explained to people who are more interested in semantics.

The principle can and should be abstracted to any automated test and even beyond computer science. One-Factor-at-a-Time (OFAT) is a paradigm of experiment design that sees use in almost every branch of science and engineering. The primary arguments against OFAT are that:

1) It fails to identify interaction effects that result from combined inputs in a multi-factor system.
2) It's inefficient in situations where data is costly.

Point 2 doesn't apply to automated testing because the cost of acquiring more data is simply waiting for your tests to run.

Point 1 makes an interesting case for why I would argue against self-changing tests. In the case of self-changing tests we have two factors that vary - the code-under-test and the tests themselves. We could state that we don't want there to be an interaction between the tests and the code because we don't want our code to perform one way in testing and another way when given to users.

But how could you check that no such interaction exists? You would need to have a representative sample of the range of values that both factors could assume, and analyze the results of the varying combinations. In the case of self-changing tests, you have no ability to make the system try out various values and present you with its findings for a given value of code-under-test, nor do you have any way to conceptualize what range of parameters the system is considering changing.

At least when you write tests yourself you can form some idea of what the range is of possible test parameters and make use of your knowledge of the domain to tease out any possible interactions between your test set and the code-under-test.

code of conduct - report abuse