DEV Community

Discussion on: Nondeterminism In Testing - How To Do It Wrong

Collapse
 
frantzen profile image
Lars Frantzen

Hi @amorganpd , thanks for your comment. You are fully right that there are several definitions of (non)deterministic systems. I am not too happy with the Wikipedia definition for the purpose of testing, since "randomness" is a bit vague here. It does e.g. not differentiate between choices done by the environment (external choice) or choices performed by the system (internal choice). This distinction is very clear when you model systems e.g. via process algebras / Labelled Input Output Transition Systems.

But there is no need to get too formal here, a proper notion (for testing) of a deterministic system says that whenever you provide an input (sequence) to the system, you always observe the same outputs. At the end the "randomness" or "internal choice" of the system is just observed randomness. In reality it will never be random, but dependent on internal details you abstract away from when modeling the system.

Another common definition of a nondeterminisitic test case is a test case that sometimes passes and sometimes fails (see Martin Fowler ). This is a different, but related topic. Such a test case is also referred to as a flaky test. Being flaky could caused by two things:

  1. the system really shows different behaviour when running the test several times, even though the environment and system setup is unchanged (so it is nondeterministic in the sense of this post)
  2. the test case is not able to bring the system in each run always in the same state for each step it takes

The flaky notion of a nondeterministic test case as used by Fowler refers to the latter meaning, not to the first meaning I use in this post.

So, the notion of nondeterminism is quite overloaded unfortunately. So adding something like your suggestion of "state dependency" could definitely help.

Collapse
 
amorganpd profile image
amorganPD • Edited

For sure, I was even having a discussion with a colleague the other day in regards to deterministic behavior and had to align on whether or not we were discussing the time domain or just the result of the output.

If we are talking about result / output, I would simply define it as: If the output is always repeatable, given the same inputs, then it is deterministic.