DEV Community

Discussion on: Anatomy of test automation

Collapse
jessekphillips profile image
Jesse Phillips

Benjamin and Alan, could you both clarify "maximalist description describes pretty much an end to end or system integration test." and "next level for testers would look like."

I did not view this as end-to-end testing but rather integration testing at all layers.

Alan you've introduced a concept of testing levels, I don't think this article established a leveling system but could be wrong.

Collapse
alanmbarr profile image
Alan Barr

I read into it. I work on Kubernetes infrastructure things now and problems I would have highlighted do not exist anymore. There are more different emergent problems and I wonder if people are aware that the skills in this kind of role might change dramatically in the next few years.

Thread Thread
dowenb profile image
Benjamin Dowen Author

I tend to find some problems I think have been solved by the industry, reoccurring. What I mean is, there are lots of solved problems that are unsolved for teams and companies that are not at the same level of maturity.

This isn't a bad thing as such, sometimes solving the be same problems again we come up with better and better solutions.

Thread Thread
jessekphillips profile image
Jesse Phillips

Well now I'm curious, what you feel is solved and what are new challenges. I have some thoughts on the subject of Kubernetes, but would like to know where you are going with it?

Thread Thread
dowenb profile image
Benjamin Dowen Author

It starts to get a bit complicated to talk in general terms at this point without discussing teams I've worked with and their problems. And I to want to avoid naming names.

As a general trend, Docker is making my life steadily easier in terms of setting up isolated test environments that have controlled test data.

But, if I hit something that cannot easily live in Docker for licensing or technical reasons, then I'm back to hosted environments I'm not in control of.

I test a lot of APIs so mocks fill in some of these gaps.

But now I've got a problem that my Integration Tests are not always triggered by builds or before merged and reporting and monitoring isn't trivial. And if something goes wrong, debugging is harder.

So you make gains in some areas, but end up with gaps. That with static environments were solved problems. Like the SUT staying around to debug.

All the new problems are solvable. It just all comes down to time, and sometimes licensing or infrastructure costs.

Thread Thread
alanmbarr profile image
Alan Barr • Edited

Most of the software problems stem from human communication issues. Conway's law. It is a belabored topic but ever-present and essential.

Thread Thread
alanmbarr profile image
Alan Barr

Anything related to server setup, repeatability, scalability. I wouldn't spend a lot of time unless I'm aware of non-functional requirements depending on your environment. I think that in a large enough business with shifting business models it might be nice to have someone thinking through and telling me: "This concept doesn't make sense anymore". That's fanciful, I'm also bored to death with UI automation ;)

Thread Thread
dowenb profile image
Benjamin Dowen Author

As our microservices architecture matures and working with skilled developers, I find fewer straight code bugs.

Problems with configuration and deployment remain plentiful as do bigger picture issues.

Being able to explore, learn and exercise the full stack end to end, from concept to production is privilege I enjoy as a quality expert.

I test and automate very few UIs. So there is that at least.

Thread Thread
jessekphillips profile image
Jesse Phillips

I love the repeatability of containers. Having the ability to stand up something for testing and each tester having their own. So many manual install fails.

I am definitely still very busy though.

Collapse
dowenb profile image
Benjamin Dowen Author

I was contrasting system integration testing, that I consider End to End testing to be part of, with what I would call partial automation, or using tools and code to assist Exploratory Testing.

I admit I casualty threw the term End to End in there without any discussion. Really it's a whole topic or its own.

I discussed End to End Testing over at The Club:

club.ministryoftesting.com/t/where...

And this is a summary of what I found out:

  • End to end testing is typically a system test that follows a user or data journey through the system
  • The scope of the test highly depends on the system under test
  • There will be a vast number of possible paths through the system
  • Judicious selection of as few critical paths as possible is important to avoid high maintenance cost
  • Where possible it is preferable to have multiple smaller tests that cover sub systems and system integration over more E2E tests
  • Context is king and no one size fits all definition of E2E testing covers all usages