Hi Community!
Looking forward to hearing your thoughts on this.
Jest-opentelemetry was created out of a pain that @galklm and I felt when working on complex systems with multiple side effects, some of which happen asynchronously. People were changing the code of different services frequently, but weren't always aware of the dependencies across the system. So often, things would break - resulting in us up at 4am debugging the issue and reverting some recent deployments.
We were basically missing a way to make sure that all side effects always happen, even when we change the code of different microservices.
Most testing frameworks out there are testing the frontend but for some reason, only a few focus on backend and integration tests.
Validating side-effects with telemetry data
Enter OpenTelemetry. Don't worry if you're not familiar with it (although it's the 2nd largest CNCF project!) - it's just a standard protocol for reporting traces, logs and metrics from production services.
We realized it's a great way to observe all the side effects as they happen within the system. We can send a request to the system and then validate side-effects using the OpenTelemetry data coming out of the system.
We wanted to make it easy to validate any side effect. So you should be able to validate that an e-mail gets sent by writing:
expectTrace(t)
.toSendEmailWithSendGrid()
.withRecipient('someone@gmail.com')
.withBody('Your order is ready!')
Behind the scenes, this should look for a span (=part of a trace) with a POST request to SendGrid with the appropriate parameters.
Next Steps
We try to constantly add more high-level constructs so more side effects on systems (like Snowflake, Stripe, BigQuery, Segment, and others) can be tested.
What are your thoughts about the need for such a framework? Have you ever needed to write integration tests in that way? What would you expect to see here?
Feel free to comment here or at our Github page https://github.com/traceloop/jest-opentelemetry
Top comments (15)
Looking good!
I think that the best thing the project can have is the option to build your own "observers" through an easy API, this way you don't need to hurry a lot to add tones of integrations while allowing people to add their "recipes".
I can imagine it like
This way you can test custom/propietary email services, any tipe of calls and other nice stuff.
Plus if you show up with a nice forum where people discusses and adds recipes, the better! 😁
Edit: I just checked the repo, it does just that 😂😂😂😂
10/10
Haha yes! :)
We opened a slack workspace and we have GitHub discussions so you're welcome to join.
oh
Super interesting!
The idea is so that you can write tests that verify the system is working as expected using traces. Then, when the code changes - you can verify the system is still working as expected. How is that debugging / incident management?
Or - how do you go and write these types of tests today?
I agree @nirga
Also, when the code did not change, but new versions of third party providers are rolled out, you need to test your system.
There are a lot of use cases, where this tool will help and improve.
When you have event driven, message based or similar asynchronous and cluttered structures, many tests are mostly done manually.
Also, if a lot of (black boxed) third party solutions involved, testing becomes hard.
Additionally, for someone like me, who is building a framework which integrates into a lot of different message brokers and stuff, it will become helpful.
I want one test, which validates the flow, without writing tests, mocks, stubs for every single message broker implementation.
This tool is not a replacement for other tools - it's one more tool in your toolbox. And from my point of view a tool, which helps to automate manual tests.
You couldn't describe the problems we aim to solve better, @sebastian_wessel!
Leading engineering @ Fiverr before, where the architecture is highly distributed (200+ microservices) and heavily async/event-driven (Kafka as a message broker), we experienced the same pains you described firsthand.
Testing flows combined of many "side-effects" taking place asynchronously in multiple services/3rd party components becomes a nightmare and usually ends up testing manually since at least I'm not familiar with a convenient way/framework of doing that automatically.
The impressive capabilities of this framework have captured my attention and piqued my interest in utilizing it for my next project. Awesome🙌
Amazing! We’re looking for contributors who can suggest or even implement more services! Let me know if there’s anything specific and we can try and work on it.
this looks very promising and I guess I will try it out for my own project.
Can you elaborate or give an example? Note that we’re not working on debug logging but tracing which is stable and represents what the different microservices actually “did”.
I'm not clear how this is different than something like PACT, unless its because you need something in NodeJS type environments, there are PLENTY of back end frameworks maybe you aren't looking for the right ones.
Seems like the errors you are finding could be checking at a Unit level with the right kind of mocking, or with Contract Testing (like PACT), or with better communication of dependencies among your Teams. Communication does wonders without the need for additional code.
Very cool idea though, always nice to fill a gap you see in your environment.
Thanks! Can you give an example? I don’t see how unit tests can replace integration tests. Both are complimentary to each other (unit tests are easier to run, but can’t test the whole system).
Never said Unit Tests can replace Integration Tests, I am saying you are waiting too far along to verify the communication with your services, and you could be doing this with contract testing. In your example you show nothing more than endpoints changing what is supposed to be trusted communication. All you do is create different terms for the same thing.
Not saying this is bad, after all not every framework covers every Team's situation and knowledge base, so there will probably be someone who could use it. It's just the way this reads is you have covered something no one else has, and that's not true.
Got it, that makes sense! But imagine you have Kafka in the middle (like I showed in the example) - how can you cover that with contract testing for example? How can you make sure that whenever a user registers, we send them an e-mail (given that there's an event
UserRegisteredfired on Kafka, which then the email-service catches to send the e-mail).