If you know a bit of python, you probably know the very popular library requests created by Kenneth Reitz.
You might have noticed that your CI tool (Travis or other) was complaining about something if you use requests directly in your project or rely on another library that uses requests.
By opening the logs of your CI tool, they would show something like this:
Traceback (most recent call last):
(...)
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict:
(idna 2.8 (/home/travis/build/.../.../.tox/py35/lib/python3.5/site-packages),
Requirement.parse('idna<2.8,>=2.5'), {'requests'})
(...)
Okay, at first glance there was some issue within requests and idna. So what happened?
The IDNA library
What is this library for?
According to the library readme, this library offers "[s]upport for the Internationalised Domain Names in Applications (IDNA) protocol as specified in RFC 5891 and also provides support for Unicode Technical Standard 46.
What happened?
On the 4th of December, the IDNA library gets an update and a bump of version to 2.8. Since requests were specifying that this library needs to have a version between 2.5 to 2.7 as soon as the library got updated Travis got upset.
An update to requests
A week later, on December the 10th, requests get updated and its version bumped to 2.21.0. With this update, requests accept the new version of the IDNA library.
Travis and other CI tools are now happy once more and you can get your PR merged again.
The fix
Fixing this issue is quite easy. Simply specify the version of idna that requests needs within your project. This will make Travis pull that version of idna and the requirements of requests won't conflict anymore.
On your requirements.txt or any other file you use to manage your dependencies adding the following will fix the failing tests.
idna == 2.7
You can apply this with any conflicting libraries that you might have until things get fixed and you are able to use the most up to date versions.
Conclusion
Does this all mean that the world of Python was indeed broken? Not at all, but if you use requests on your project, then for a week your CI tool wasn't happy(unless you pin the idna version).
In opsdroid we had to wait until requests got updated so we could merge two PRs which itself is not a big deal.
This episode made me realise how important dependencies are within projects and how something can break without being directly your fault or anyone's fault really.
I would love to hear how do you keep track with your dependencies and updates to these dependencies. Do you just merge new updates or do some testing beforehand?
I'd like to thank to @rhymes for showing me how home assistant solved this issue.
Top comments (9)
Hi Fabio,
I'm not sure I understood what happened, I'm probably missing something.
idna gets updated to 2.8, requests asks for 2.7 max. PyPI keeps old versions so it should have just matched with idna 2.7, no? Did they delete idna 2.7 from pypi?
Maybe there was another dependency asking for idna > 2.7 that conflicted with requests's requirements? That's usually when I get these problems, when two dependencies asks for incompatible versions of the same lib.
Hello, rhymes thank you for your comment, I was a bit puzzled as well even because I thought we weren't using requests on our project (but there was an old
requests_mockdependency that was unused).For some reason Travis kept complaining that requests need idna up to 2.7, this happened as soon as idna got updated to 2.8. Maybe Travis checks the latest version of all dependencies and if there is some discrepancies it complains?
Weird, especially if you could install the dependencies on your local machine.
It seems you weren't the only one with that problem github.com/home-assistant/home-ass...
🧐
Yeah I checked aiohttp to see if things where okay but the badge showed that the build was breaking, when I check Travis it complained about the same thing.
Funny enough it seems that home-assistant seem to have fixed the issue by pinning the old idna version, I haven't thought about it since we weren't using idna directly. I will keep that in mind haha
--EDIT--
Also, it seems that aiohttp fixed the issue by following the same strategy.
That's a good strategy. You add the indirect dependency to your code, pin it, wait until your direct dependency solves the issue and then remove the indirect one. Until the next time then :D
BTW do you use aiohttp directly?
Yeah I never thought about doing that, I think I will update my post with this for the future haha
Yeah we use aiohttp directly why?
Because it would be great if you wrote an article on using aiohttp day by day :)
Great idea I will work something out :D
This is such an oddity in programming