DEV Community

Cover image for Building a new type of testing framework
Nir Gazit
Nir Gazit

Posted on • Updated on • Originally published at traceloop.com

Building a new type of testing framework

Hi Community!
Looking forward to hearing your thoughts on this.

Jest-opentelemetry was created out of a pain that @galklm and I felt when working on complex systems with multiple side effects, some of which happen asynchronously. People were changing the code of different services frequently, but weren't always aware of the dependencies across the system. So often, things would break - resulting in us up at 4am debugging the issue and reverting some recent deployments.

Complex Architecture

We were basically missing a way to make sure that all side effects always happen, even when we change the code of different microservices.

Most testing frameworks out there are testing the frontend but for some reason, only a few focus on backend and integration tests.

Validating side-effects with telemetry data

Enter OpenTelemetry. Don't worry if you're not familiar with it (although it's the 2nd largest CNCF project!) - it's just a standard protocol for reporting traces, logs and metrics from production services.
We realized it's a great way to observe all the side effects as they happen within the system. We can send a request to the system and then validate side-effects using the OpenTelemetry data coming out of the system.

Jest-OpenTelemetry Architecture

We wanted to make it easy to validate any side effect. So you should be able to validate that an e-mail gets sent by writing:

expectTrace(t)
    .toSendEmailWithSendGrid()
    .withRecipient('someone@gmail.com')
    .withBody('Your order is ready!')
Enter fullscreen mode Exit fullscreen mode

Behind the scenes, this should look for a span (=part of a trace) with a POST request to SendGrid with the appropriate parameters.

Next Steps

We try to constantly add more high-level constructs so more side effects on systems (like Snowflake, Stripe, BigQuery, Segment, and others) can be tested.

What are your thoughts about the need for such a framework? Have you ever needed to write integration tests in that way? What would you expect to see here?

Feel free to comment here or at our Github page https://github.com/traceloop/jest-opentelemetry

Top comments (17)

Collapse
 
joelbonetr profile image
JoelBonetR ๐Ÿฅ‡ • Edited

Looking good!
I think that the best thing the project can have is the option to build your own "observers" through an easy API, this way you don't need to hurry a lot to add tones of integrations while allowing people to add their "recipes".

I can imagine it like

expectTrace(t)
    .toSendCustomGetRequest(args)
    .withHeaders('key', 'value');

expectTrace(t)
    .toSendCustomPostRequest('args')
    .withRecipient('someone@gmail.com')
    .withBody('Your order is ready!')
Enter fullscreen mode Exit fullscreen mode

This way you can test custom/propietary email services, any tipe of calls and other nice stuff.

Plus if you show up with a nice forum where people discusses and adds recipes, the better! ๐Ÿ˜

Edit: I just checked the repo, it does just that ๐Ÿ˜‚๐Ÿ˜‚๐Ÿ˜‚๐Ÿ˜‚

10/10

Collapse
 
nirga profile image
Nir Gazit

Haha yes! :)
We opened a slack workspace and we have GitHub discussions so you're welcome to join.

Collapse
 
viyash profile image
viyashdoss

oh

Collapse
 
talboren profile image
Tal Borenstein

Super interesting!

Collapse
 
femi_dev profile image
Femi Akinyemi

The impressive capabilities of this framework have captured my attention and piqued my interest in utilizing it for my next project. Awesome๐Ÿ™Œ

Collapse
 
nirga profile image
Nir Gazit

Amazing! Weโ€™re looking for contributors who can suggest or even implement more services! Let me know if thereโ€™s anything specific and we can try and work on it.

Collapse
 
sebastian_wessel profile image
Sebastian Wessel

this looks very promising and I guess I will try it out for my own project.

Collapse
 
mfurmaniuk profile image
Michael

I'm not clear how this is different than something like PACT, unless its because you need something in NodeJS type environments, there are PLENTY of back end frameworks maybe you aren't looking for the right ones.

Seems like the errors you are finding could be checking at a Unit level with the right kind of mocking, or with Contract Testing (like PACT), or with better communication of dependencies among your Teams. Communication does wonders without the need for additional code.

Very cool idea though, always nice to fill a gap you see in your environment.

Collapse
 
nirga profile image
Nir Gazit

Thanks! Can you give an example? I donโ€™t see how unit tests can replace integration tests. Both are complimentary to each other (unit tests are easier to run, but canโ€™t test the whole system).

Collapse
 
mfurmaniuk profile image
Michael

Never said Unit Tests can replace Integration Tests, I am saying you are waiting too far along to verify the communication with your services, and you could be doing this with contract testing. In your example you show nothing more than endpoints changing what is supposed to be trusted communication. All you do is create different terms for the same thing.

Not saying this is bad, after all not every framework covers every Team's situation and knowledge base, so there will probably be someone who could use it. It's just the way this reads is you have covered something no one else has, and that's not true.

Thread Thread
 
nirga profile image
Nir Gazit

Got it, that makes sense! But imagine you have Kafka in the middle (like I showed in the example) - how can you cover that with contract testing for example? How can you make sure that whenever a user registers, we send them an e-mail (given that there's an event UserRegistered fired on Kafka, which then the email-service catches to send the e-mail).

Collapse
 
jarrodhroberson profile image
Jarrod Roberson

There are tools that can do impact analysis across code bases. That is what you should be doing. Not trying to create an "assert" query language acrosss debug logging after the fact.

Collapse
 
nirga profile image
Nir Gazit

Can you elaborate or give an example? Note that weโ€™re not working on debug logging but tracing which is stable and represents what the different microservices actually โ€œdidโ€.

Collapse
 
jarrodhroberson profile image
Jarrod Roberson

"Building a new type of testing framework" is "debugging" the failures you find ...after the fact; the bugs and errors have already happened and any data affected is already corrupt, this is not "testing" or "debugging" this is at best naรฏve incident management.

Thread Thread
 
nirga profile image
Nir Gazit

The idea is so that you can write tests that verify the system is working as expected using traces. Then, when the code changes - you can verify the system is still working as expected. How is that debugging / incident management?

Or - how do you go and write these types of tests today?

Thread Thread
 
sebastian_wessel profile image
Sebastian Wessel

I agree @nirga
Also, when the code did not change, but new versions of third party providers are rolled out, you need to test your system.

There are a lot of use cases, where this tool will help and improve.
When you have event driven, message based or similar asynchronous and cluttered structures, many tests are mostly done manually.
Also, if a lot of (black boxed) third party solutions involved, testing becomes hard.
Additionally, for someone like me, who is building a framework which integrates into a lot of different message brokers and stuff, it will become helpful.
I want one test, which validates the flow, without writing tests, mocks, stubs for every single message broker implementation.

This tool is not a replacement for other tools - it's one more tool in your toolbox. And from my point of view a tool, which helps to automate manual tests.

Thread Thread
 
galklm profile image
Gal Kleinman

You couldn't describe the problems we aim to solve better, @sebastian_wessel!

Leading engineering @ Fiverr before, where the architecture is highly distributed (200+ microservices) and heavily async/event-driven (Kafka as a message broker), we experienced the same pains you described firsthand.

Testing flows combined of many "side-effects" taking place asynchronously in multiple services/3rd party components becomes a nightmare and usually ends up testing manually since at least I'm not familiar with a convenient way/framework of doing that automatically.