This is not a post to discuss and compare the low-level capabilities of each web automation library or framework. Here we plan to lay out some foundational concerns that you should keep in mind when choosing the right tool for the job.

Enjoy!

Stepping with the "right" foot in the door for Web Automation

Proposing a Web Automation Initiative 👔

There comes the time that that you want to start your web automation initiative. That requires a quite smart deal of preparation.

First you have got to convince your team that this is worthwhile.

In the same manner you have to make the decision of which web automation strategy and which library or framework you should consider based on your application use cases.

Let us go over some foundational concerns to help guide that decision and effort.

Getting In The Mood 🤖

The first best thing is that you arrived at the point of discussing something related to web automation 👏

In most discussions around web automation, especially when planning to introduce a new initiative or re-evaluate your technology stack, the topics that can quickly drive the discussions are cross-browser compatibility, programming language API and community backing that available tools offer.

To not get confused here, by browser compatibility we mean the range of available browsers a web automation task can run seamlessly on without any change required.

Aim For Efficiency 🚀

For modern software products and businesses, efficiency is a vital factor.

That makes it all the more important for us to be pragmatic. That means evaluate solutions that

1. Minimize wasted effort
2. Do just enough to complete the task at hand safely and with the expected quality levels

For our context we need to pick the right web automation tool serving our own needs.

 -How many people in the team have experience with web automation testing ?
 -Do we have to run the suite in different browsers ?
 -Do we need to support Internet Explorer ?

These are just some standard and common questions, but we will go over the process on how you can evaluate your requirements and tradeoffs even in more complex situations.
Let's go! 🏄

Evaluating Your Reality

The first step for exploring the best possible solution is to rule out the cases you do not even need to consider such decision.

They are probably self explanatory but compiling down a table with rough estimations to get you out of this situation pretty quickly:

These situations as coarse grained as they might seem, can at least point you to a first solid direction. Efficiency metrics and team feedback will guide you for the rest of the decisions you have to make down the line depending on the path you chose initially.

The Messy Middle of Web Automation 🌀

Since you reached this part of the article, means we can borrow this amazing phrase Scott Belsky to describe a common but certainly messy situation you need to wrestle with. The Messy Middle.

This phrase will be used to describe the state that at some point in time your web automation endeavors are bound to reach.

One of the characteristics of this state is choice overload 🤯

As browser technology is progressing rapidly, the ecosystem and available tools for web automation are growing along with it, especially in the End to End testing league. The absolutely wrong path to take is start exploring the vast ecosystem without any guidance from you actual goals, data and expectations. These main drivers as you can understand are highly dependent on the scenario that brought you to this stage from the start.

This Messy Middle most commonly can occur in one of the following scenarios:

• Trying to introduce the concept of End to End testing to a team (or in a whole organization).
• Re-evaluating the state of a legacy web automation codebase that bears the weight of time. Flakiness, Configuration & Setup mayhem, takes hours to run etc...

Above we list these two specific scenarios as they share the common requirements:

• Low resource cost
• High impact
• A twist of wow factor

In both cases you absolutely need to convince and inspire your team, that is why we need to invest up front on the things that matter.

Look Further Into Requirements Not Options

As technologists, and software enthusiasts at that, we love diving in and analyzing a plethora of options that we have to solve a particular problem. It feels like a playground with thousands of toys to pick up and play.

We want to try the latest, touch the bleeding edge and feel that we are joining the innovation march, leaving expectations as an afterthought that we will tackle after we make our choices.

-Should we assemble our own minimal framework or start with something like Gauge or WebdriverIO ?
-Should we integrate tests using Cucumber or try it together with TestNG ?
-Might it be a good case to try the Screenplay pattern ?

At that time, which is bound to happen, take a step back and bring again into your mind the word "Pragmatic". We are operating under the umbrella of a business and we have so many edges that our choices will have an effect on.

From our peers and seniors but finally to the customer, we ought as professionals to make our decisions as informed as possible. Not that the technology questions are unimportant, on the contrary, since they are important they should not be taken lightly.

Gather Real User Data 👨‍👩‍👧‍👦

There is no better indicator about your real requirements than from the people that actually use your product.

For most use cases, the main difference you first need to consider is how cross-browser compatible your solution needs to be.

You are trying to get a foot in the door but if the proposed path does not cover 30% of user technology specifications, it is not the "right" foot you are getting into with. It is totally reasonable to look for the option that would allow broader amount of coverage, but you have to act smart here, you need to base "broader" on something, which is not 100% of the browser market.

This step is skipped for most developers as they become allergic to step out of the comfort zone of development tools 😥

Data analytics tools such as Google Analytics, can provide you with fine-grained information about which browser/browser version/device combination your actual users are preferring to navigate your product with.

Now you can start working your case with reality as your guide.
Special Hint: Look also for the conversions/combination metric! If Internet Explorer users are just 5% but bring 15% of the conversions, it matters a lot!

It will be clear as day that if your userbase is composed by a considerable amount of traffic from Internet Explorer, you should go for a solution that supports Selenium. On the other hand, if your application is mostly used on Safari, you should give Playwright its fair chance against Puppeteer.

Thankfully though, there are free and open source options that want to lift the burden of this choice from our shoulders as much as possible. Frameworks like WebdriverIO and CodeceptJS provide the capabilities to switch between Selenium, Puppeteer or even Playwright(only for CodeceptJS) as your automation core library.

Evaluate Team Experience & Preferences 🤝

Presenting your case in a team that already has experience with web automation is always a good thing. In the best case scenario you will be having more eyes on the target along with more technical knowledge from your peers.

On the other side of the spectrum, a team that had a bad experience with similar efforts in the past, can be seen as a source of feedback for what went wrong and ways to avoid the same mistakes.

⭐ Do not forget that this situation can be fertile ground to change perspectives for the better (that is why you should continue reading)

Bring The Excitement 🏂

Talking about perspectives, it might be by far one of the strongest factors on the success of the initiative, you should strive to excite all the more people from your team.

Try to create a group of "followers", people that will support your claim that this change will have positive impact for the team as a whole. In the same note, having your peers excited will have a high chance of them being willing to collaborate with you and contribute in the effort. This can only lead to much better results.

Keep in mind that more experienced managerial stuff, will not naturally prefer a "one-man" initiative as they will need to account for the bus factor of a new system.

On the other hand, as any solid initiative, it needs to pull its own weight, that is why you must strive to require as few slices of people as possible in order to make this a reality.

As an example, if you get the feeling that the CI pipeline is convoluted or it requires significant involvement of Operations people, try to choose a solution that makes the process as lightweight and minimal as possible.

👉 Minimizing the effort from other individuals and the changes required in other parts of the system, is sure to increase the chance the initiative is accepted and not thrown in a "low priority" bucket.

Maximum Output - Low Effort

Independently of the results of your preparation, since hopefully you did not skip any corners, it is time to get the wheels spinning. Together with your team you have made the choices, keeping in mind all the needs of the application, and in order to actually start the initiative you need to present to management the incremental steps towards your desired end goal.

Special Hint: Do not come empty handed on this aspect, expecting the question to be asked by someone else, be prepared with the correct amount of effort that can fit into the work schedule that your team is following.

"Incremental" can mean different things for different people, but we need to have this down for our initiative so that we can produce the maximum amount of output in low effort iterations. What better way to illustrate such a case than with an example.

Strap in!

---

Example: Monolith with Multiple Frontend Applications 🤹

The Goal

Let's suppose your case is a monolithic application (Django, Rails etc.) with multiple frontend apps for distinct parts of your product (Data dashboard, Signup flow, Landing pages).

The application does not have any End to End tests as of yet, but has a simple test pipeline that spins up a local instance of the monolith, runs the backend tests and the different tests for the frontend that happen to be run using jest for all applications.

In this scenario your end goal is to create a containerized suite of End to End tests with Puppeteer that run in every PR merge of the application and tests all the parts at once.

Some additional features of the suite that everybody agreed on include:

• Screenshots on failure cases
• Statistics of past and current test runs
• Slack notifications when the suite fails to build a PR

First Iteration Proposal

For the first iteration, you should aim for the least amount of setup required and happy path coverage of couple of the main flows that are most valuable to your business.

Our first targets can look like:

• The landing page is loaded successfully and the CTA is visible within the viewport.
• The dashboard application is loaded successfully and the main 3 routes can be clicked through and render the main graphs of the user expects to see.

Common Mistake: Setup For the First Tests ❌

One of the main mistakes that individuals make when introducing a web automation initiative is to try to setup the "final" view of the pipeline integration right at the start of the initiative.

This is where many trip up and the whole initiative gets discarded when you depend on other people to kickstart it. Some examples where you will need to take time from your DevOps team...

• Nobody in the team knows how to setup Jenkins for the Selenium Server to run in parallel
• Your Docker setup is failing for "reasons" 🐳
• You are not feeling comfortable modifying the Jenkinsfile on your own

Without going deeper here into the reasons and remediation of this, you have to go with the most unobtrusive setup as possible.

The Unobtrusive Setup 👻

In our case this means integrating the first scenarios inside the existing frontend unit testing suite. Just the plain npm test that your frontend apps are already using in one of the build hooks (hopefully 😅) is more than enough.

The application is already running the tests in the pipeline, so you can easily go with the solution of adding jest-puppeteer to your project and add your two scenarios then and there on their respective frontend repositories.

No further containerization is required
No additional commands on the build pipeline
No further effort, just your two test scenarios

First Iteration Complete 🎖

With the scenario described above that would take:
- One additional NPM package
- 100 or so lines of code
- Some seconds added on the pipeline total build time

With that we have completed the first iteration of our initiative. Every time our application goes through the simple test pipeline, we make sure that two of the main parts of our product are not broken. This is no small guarantee for the business and you manage to deliver it in a really short timespan.

Hopefully you have surprised a couple of people that did not know the power you can have with End to End tests giving a little effort.

Full steam ahead for the next iteration! 🚄

Next Iteration!

Actually, for the next iteration, you are on your own.

You have already something going on and that is enough to take some time from each iteration cycle and enhance it. Keep showing the value and advocate the confidence that it gives while shipping code into production.

Do not forget that value is expressed differently in the ears of different people.

• Remind developers that they can be confident that they did not break something important while trying new features refactorings 🛠
• Show the product owners that time-to-release will keep decreasing as QA has less manual work to do for each feature ⏲
• Indicate to managers and stakeholders that the main revenue flows are safe while the product is becoming better and more feature-rich 💰

You got this! 🔥

Closing

Thank you for taking the time over yet another not so short writeup, which I hope you enjoyed and found useful ideas about how to tackle a web automation initiative, both in the technical and in the non-technical aspects 🙇‍♂️

From personal experience, committing and going through the process of gathering data, evaluate current requirements and analyze past experiences from your team is probably the most important decision you could have made.

Not because it will make the effort more likely to succeed, which can never be guaranteed, but for the sole reason that it will arm you with the feeling of innate confidence and the joy of solving a real problem for your business, not just a tech test-drive that might have a chance to fit the requirements.

Cross-posted from The Home of Web Automation

Resources to explore:
- Getting started with Puppeteer
- Getting started with Playwright
- Google Analytics technology report

Image by Mateusz Dach