DEV Community

Building a new type of testing framework

Nir Gazit on March 30, 2023

Hi Community! Looking forward to hearing your thoughts on this. Jest-opentelemetry was created out of a pain that @galklm and I felt when working ...

Read full post

JoelBonetR 🥇 • Mar 30 '23 • Edited

Looking good!
I think that the best thing the project can have is the option to build your own "observers" through an easy API, this way you don't need to hurry a lot to add tones of integrations while allowing people to add their "recipes".

I can imagine it like

expectTrace(t)
    .toSendCustomGetRequest(args)
    .withHeaders('key', 'value');

expectTrace(t)
    .toSendCustomPostRequest('args')
    .withRecipient('someone@gmail.com')
    .withBody('Your order is ready!')

This way you can test custom/propietary email services, any tipe of calls and other nice stuff.

Plus if you show up with a nice forum where people discusses and adds recipes, the better! 😁

Edit: I just checked the repo, it does just that 😂😂😂😂

10/10

Nir Gazit • Mar 30 '23

Haha yes! :)
We opened a slack workspace and we have GitHub discussions so you're welcome to join.

viyashdoss • Apr 4 '23

Tal Borenstein • Mar 30 '23

Super interesting!

Nir Gazit • Apr 3 '23

The idea is so that you can write tests that verify the system is working as expected using traces. Then, when the code changes - you can verify the system is still working as expected. How is that debugging / incident management?

Or - how do you go and write these types of tests today?

Sebastian Wessel • Apr 3 '23

I agree @nirga
Also, when the code did not change, but new versions of third party providers are rolled out, you need to test your system.

There are a lot of use cases, where this tool will help and improve.
When you have event driven, message based or similar asynchronous and cluttered structures, many tests are mostly done manually.
Also, if a lot of (black boxed) third party solutions involved, testing becomes hard.
Additionally, for someone like me, who is building a framework which integrates into a lot of different message brokers and stuff, it will become helpful.
I want one test, which validates the flow, without writing tests, mocks, stubs for every single message broker implementation.

This tool is not a replacement for other tools - it's one more tool in your toolbox. And from my point of view a tool, which helps to automate manual tests.

Gal Kleinman • Apr 3 '23

You couldn't describe the problems we aim to solve better, @sebastian_wessel!

Leading engineering @ Fiverr before, where the architecture is highly distributed (200+ microservices) and heavily async/event-driven (Kafka as a message broker), we experienced the same pains you described firsthand.

Testing flows combined of many "side-effects" taking place asynchronously in multiple services/3rd party components becomes a nightmare and usually ends up testing manually since at least I'm not familiar with a convenient way/framework of doing that automatically.

Femi Akinyemi • Apr 3 '23

The impressive capabilities of this framework have captured my attention and piqued my interest in utilizing it for my next project. Awesome🙌

Nir Gazit • Apr 4 '23

Amazing! We’re looking for contributors who can suggest or even implement more services! Let me know if there’s anything specific and we can try and work on it.

Sebastian Wessel • Apr 3 '23

this looks very promising and I guess I will try it out for my own project.

Nir Gazit • Apr 2 '23

Can you elaborate or give an example? Note that we’re not working on debug logging but tracing which is stable and represents what the different microservices actually “did”.

Michael • Apr 3 '23

I'm not clear how this is different than something like PACT, unless its because you need something in NodeJS type environments, there are PLENTY of back end frameworks maybe you aren't looking for the right ones.

Seems like the errors you are finding could be checking at a Unit level with the right kind of mocking, or with Contract Testing (like PACT), or with better communication of dependencies among your Teams. Communication does wonders without the need for additional code.

Very cool idea though, always nice to fill a gap you see in your environment.

Nir Gazit • Apr 4 '23

Thanks! Can you give an example? I don’t see how unit tests can replace integration tests. Both are complimentary to each other (unit tests are easier to run, but can’t test the whole system).

Michael • Apr 4 '23

Never said Unit Tests can replace Integration Tests, I am saying you are waiting too far along to verify the communication with your services, and you could be doing this with contract testing. In your example you show nothing more than endpoints changing what is supposed to be trusted communication. All you do is create different terms for the same thing.

Not saying this is bad, after all not every framework covers every Team's situation and knowledge base, so there will probably be someone who could use it. It's just the way this reads is you have covered something no one else has, and that's not true.

Nir Gazit • Apr 4 '23

Got it, that makes sense! But imagine you have Kafka in the middle (like I showed in the example) - how can you cover that with contract testing for example? How can you make sure that whenever a user registers, we send them an e-mail (given that there's an event UserRegistered fired on Kafka, which then the email-service catches to send the e-mail).