As developers, we strive to write error-free code, but no one actually does so because...well, bugs. In order to catch those pesky bugs before they wreak havoc on our applications, we rely on automated testing. While positive tests ensure our code works as intended, negative tests play a crucial role in validating that our applications are robust enough to handle unexpected input and edge cases.
I'm working on Pythagora, an open source tool that writes automated integration tests by itself (well, with a bit of help from GPT-4) without you, the dev, having to write a single line of code. Basically, you can get from 0 to 80% code code coverage within 30 minutes (video).
We just created a feature that automatically generates negative tests from the entire test suite with a single command. While building that feature, I researched what are different ways one can break an API server to test if it handles errors gracefully so here it is - a comprehensive list of ways with which you can break your server, if it doesn't handle errors properly.
1. Empty or missing required fields
{
"endpoint": "/api/users",
"body": {
"username": "",
"email": ""
},
"method": "POST"
}
2. Invalid field values - exceeding character limits
{
"endpoint": "/api/users",
"body": {
"username": "ThisIsAnIncrediblyLongUsernameThatExceedsTheCharacterLimit"
},
"method": "POST"
}
3. Invalid field values - malformed data
{
"endpoint": "/api/users",
"body": {
"email": "invalid-email@"
},
"method": "POST"
}
4. Extra or irrelevant keys in the payload
{
"endpoint": "/api/users",
"body": {
"username": "validuser",
"extra_key": "irrelevant_value"
},
"method": "POST"
}
5. Incorrect or invalid HTTP methods
{
"endpoint": "/api/users/123",
"body": {},
"method": "POST"
}
6. Invalid endpoint paths
{
"endpoint": "/api/nonexistent_endpoint",
"body": {},
"method": "GET"
}
7. Query parameters instead of using the request body in POST requests
{
"endpoint": "/api/users?username=testuser",
"body": {},
"method": "POST"
}
8. Missing or invalid authentication headers (e.g., API keys)
{
"endpoint": "/api/users",
"body": {},
"method": "GET",
"headers": {
"Authorization": "Invalid API_KEY"
}
}
9. Incorrect data structure - array instead of an object
{
"endpoint": "/api/users",
"body": [
"username": "testuser",
"email": "test@example.com"
],
"method": "POST"
}
10. Incorrect data structure - object instead of an array
{
"endpoint": "/api/users",
"body": {
"users": {
"username": "testuser",
"email": "test@example.com"
}
},
"method": "POST"
}
11. JSON formatting issues - invalid Unicode characters
{
"endpoint": "/api/users",
"body": {
"username": "test\uFFFFuser"
},
"method": "POST"
}
12. Duplicate keys in the payload
{
"endpoint": "/api/users",
"body": {
"username": "testuser",
"username": "duplicate"
},
"method": "POST"
}
13. Invalid or unsupported content types (e.g., sending XML instead of JSON)
{
"endpoint": "/api/users",
"body": "<user><username>testuser</username><email>test@example.com</email></user>",
"method": "POST",
"headers": {
"Content-Type": "application/xml"
}
}
14. Exceeding payload size limits
{
"endpoint": "/api/users",
"body": {
"large_data": "A very large data string that exceeds the server's payload size limit..."
},
"method": "POST"
}
15. Invalid or expired authentication tokens
{
"endpoint": "/api/users",
"body": {},
"method": "GET",
"headers": {
"Authorization": "Bearer expired_token"
}
}
16. Using special characters in field values
{
"endpoint": "/api/users",
"body": {
"username": "test!@#$%^&*()-user"
},
"method": "POST"
}
17. Sending nested objects instead of simple key-value pairs
{
"endpoint": "/api/users",
"body": {
"user": {
"username": "testuser",
"email": "test@example.com"
}
},
"method": "POST"
}
18. Sending data in the wrong data type (e.g., string instead of integer)
{
"endpoint": "/api/users",
"body": {
"age": "25"
},
"method": "POST"
}
19. Sending null values for required fields
{
"endpoint": "/api/users",
"body": {
"username": null
},
"method": "POST"
}
20. Using reserved keywords in field names
{
"endpoint": "/api/users",
"body": {
"class": "user"
},
"method": "POST"
}
21. Sending incomplete or malformed multipart file uploads
{
"endpoint": "/api/upload",
"body": {
"file": "incomplete_file_data"
},
"method": "POST",
"headers": {
"Content-Type": "multipart/form-data"
}
}
22. Incorrect or missing URL encoding for special characters
{
"endpoint": "/api/users?username=test user",
"body": {},
"method": "GET"
}
23. Sending the request body in GET requests
{
"endpoint": "/api/users",
"body": {
"username": "testuser"
},
"method": "GET"
}
24. Invalid date or time formats
{
"endpoint": "/api/users",
"body": {
"birthdate": "01-25-1990"
},
"method": "POST"
}
25. Using non-ASCII characters in field names
{
"endpoint": "/api/users",
"body": {
"üsername": "testuser"
},
"method": "POST"
}
26. Sending deeply nested objects
{
"endpoint": "/api/users",
"body": {
"user": {
"profile": {
"details": {
"nested": "too_deep"
}
}
}
},
"method": "POST"
}
27. Using non-printable or control characters in field values
{
"endpoint": "/api/users",
"body": {
"username": "test\u0008user"
},
"method": "POST"
}
28. Sending the same field multiple times with different values
{
"endpoint": "/api/users",
"body": {
"username": "testuser",
"username": "different"
},
"method": "POST"
}
29. Missing or invalid content-length headers for request bodies
{
"endpoint": "/api/users",
"body": {
"username": "testuser"
},
"method": "POST",
"headers": {
"Content-Length": "invalid"
}
}
30. Using spaces or special characters in field names
{
"endpoint": "/api/users",
"body": {
"user name": "testuser"
},
"method": "POST"
}
31. Sending invalid or malformed JSONP callbacks
{
"endpoint": "/api/users?callback=invalid(callback)",
"body": {},
"method": "GET"
}
32. Sending the payload as a single string instead of key-value pairs
{
"endpoint": "/api/users",
"body": "username=testuser&email=test@example.com",
"method": "POST"
}
33. Sending boolean values as strings (e.g., "true" instead of true)
{
"endpoint": "/api/users",
"body": {
"active": "true"
},
"method": "POST"
}
34. Using non-standard HTTP methods (e.g., PATCH, CONNECT)
{
"endpoint": "/api/users/123",
"body": {
"username": "updateduser"
},
"method": "PATCH"
}
35. Sending unsupported HTTP version numbers
{
"endpoint": "/api/users",
"body": {},
"method": "GET",
"httpVersion": "HTTP/3.0"
}
36. Sending multiple authentication headers (e.g., both API key and token)
{
"endpoint": "/api/users",
"body": {},
"method": "GET",
"headers": {
"Authorization": "Bearer token_value",
"API-Key": "api_key_value"
}
}
37. Sending unnecessary or invalid CORS headers
{
"endpoint": "/api/users",
"body": {},
"method": "GET",
"headers": {
"Access-Control-Allow-Origin": "*"
}
}
38. Sending conflicting query parameters and request body data
{
"endpoint": "/api/users?username=testuser",
"body": {
"username": "different_user"
},
"method": "POST"
}
39. Using non-standard characters in authentication header values
{
"endpoint": "/api/users",
"body": {},
"method": "GET",
"headers": {
"Authorization": "Bearer t@ken_value"
}
}
40. Sending negative numbers for fields that should only accept positive values
{
"endpoint": "/api/users",
"body": {
"age": -25
},
"method": "POST"
}
41. Sending timestamps in the future or past beyond expected range
{
"endpoint": "/api/users",
"body": {
"birthdate": "01-25-1800"
},
"method": "POST"
}
42. Using HTML, JavaScript, or SQL code in field values to attempt code injection
{
"endpoint": "/api/users",
"body": {
"username": "<script>alert('test')</script>"
},
"method": "POST"
}
43. Using different character encodings in the payload (e.g., UTF-8, UTF-16)
{
"endpoint": "/api/users",
"body": {
"username": "téstuser"
},
"method": "POST",
"headers": {
"Content-Type": "application/json; charset=UTF-16"
}
}
44. Sending arrays with mixed data types
{
"endpoint": "/api/users",
"body": {
"values": [1, "string", true]
},
"method": "POST"
}
45. Sending field values as arrays or objects instead of simple data types (e.g., string, number)
{
"endpoint": "/api/users",
"body": {
"username": ["testuser"]
},
"method": "POST"
}
That's it. I hope this list gave you new ideas to test and protect your server.
If you found this post valuable, it would mean the world to me if you could support us by starring Pythagora Github repo.
And, if you try it out, please let us know your feedback, we're happy to hear it.
Top comments (16)
Wow! This is super cool. Appreciate ya sharing all this research. Pythagora sounds like a really cool tool!
Thank you so much Michael! Did you maybe try Pythagora? I'd love to hear your feedback.
To be honest, I haven't.
I'm a Community Manager here with admittedly very limited technical abilities. 😅 That said, I do feel like I have a decent understanding of what's going on here — you've created Pythagora to automatically generates negative tests to make sure that whatever is being built can handle unexpected inputs and edge cases. And in the process of doing that ya researched a bunch of ways one can break an API server, which is what ya have shared here.
So yeah, no use from me here (yet! ... hey, there's always a chance I get more techie in the future), but I do really think this is a great idea!
Ah, got it. Btw, it's great that you understood what's happening here without being a QA or a dev. I'd say this is advanced for a non-tech person. Btw, do you have any devs around you who can try out Pythagora? As you can imagine, we're super early so any support/feedback would mean the world to me.
Thanks a bunch, ya explained it well which helps!
I feel you on the struggle of starting small and trying to reach folks. So, while I can't commit on behalf of others, I will definitely throw the suggestion out there to our devs on the team via Slack and see if anybody is interested. 🙂
And now I'm gonna put my DEV Community Manager hat on for a sec... One thing you might consider with Pythagora is setting up a free organization on DEV for it. You can check out our organization info page to learn about some of the perks of using an org and if you wanna get a sense of what it's like, you can see the DEV Team org page here — just click through some of our posts and you'll see how our org branding appears around them. Using orgs helps with general brand recognition because as folks are reading your posts and learning about Pythagora, they'll also see your logo in places and you have the option to point them to your website/repo/etc. I'll not blab about orgs for too much longer, haha, you can check out the info page and feel free to hit me up if you have any questions!
I really wish you and the rest of the Pythagora team the best! It really does sound like y'all are building something cool.
Oh nice, did it! Thanks for this, I didn't know dev.to has orgs.
Re sharing Pythagora, thank you so much!!! This really means a lot. Wherever you can share it, I'd be more than thankful.
Cool article, simple & clear.
@zvone187 Reading the comments it looks like you're keen for feedback & even though I haven't tried it yet (I will at some point), my first thoughts are that the name Pythagora is an interesting choice given that the tool generates tests "for your Node.js app"
If I didn't have the context that this article provides, I might have overlooked the repo if I had discovered it another way because I thought it was a tool for a Python codebase.
Hi Mike,
Thanks! I'm glad you like it.
Re name, yes, you're completely right. I didn't think too long when I was choosing the name - I just took what had a domain available. It will come to it's place when we support Python.
Btw, yes, I'm definitely eager to hear feedback so I'd appreciate so much to hear yours as well. Also, if you get stuck anywhere, let me know, I'd be happy to help.
I think most of this type of problem can be easily managed by joi package.
Oh nice, good one. Never used it but it seems like it should handle many of these. Btw, were you able to see Pythagora (github.com/Pythagora-io/pythagora)? Since you're familiar with automated testing, I'm curious to hear your opinion on it. Would you use generated tests for your projects?
Pythagora sounds interesting will going to take a look soon.
Thanks for sharing 🙂
Oh awesome, I'm looking forward to hearing what you think.
Excellent !
Thanks Ricardo! Are you used to building negative tests in your team?
Great Articule,
thanks for sharing, good cases to keep in mind.
Thank you so much Juan!