DEV Community

loading...

Python urldecode on command line

Kamal Mustafa
Python/Django Developer at GetOTP (otp.dev), fast and simple to integrate OTP API with ready made UI
・1 min read

I have some logs that contain url with encoded characters and I need to extract some data out of the urls. Searching around lead me to this stackexchange's answer:-

alias urldecode='python3 -c "import sys, urllib.parse as ul;print(ul.unquote_plus(sys.argv[1]))"'
Enter fullscreen mode Exit fullscreen mode

Then you can use it as:-

urldecode '%22status%22%3A%22SUCCESS%22'
"status":"SUCCESS"
Enter fullscreen mode Exit fullscreen mode

Now that's perfect. But the data that I want to decode must be passed through unix pipe, like this:-

grep 'status: 504' /var/log/local0* | awk '{ print $10}' | xargs urldecode
Enter fullscreen mode Exit fullscreen mode

But this won't work, as xargs won't see the alias. So we alias xargs as well:-

alias xargs='xargs '
Enter fullscreen mode Exit fullscreen mode

But that however only work for the first line. How to make xargs execute for each line?

alias xargs='xargs -L 1 '
Enter fullscreen mode Exit fullscreen mode

Now we can run urldecode for each line we piped from grep!

Discussion (2)

Collapse
rrampage profile image
Raunak Ramakrishnan • Edited

We can also do it without xargs by using sys.stdin to read from pipe
e.g:

yes 'Mozilla%2F5.0%20%28Macintosh%3B%20U%3B%20Intel%20Mac%20OS%20X%2010.6%3B%20en' |\
 head -n 10 |\
 python3 -c "import sys, urllib.parse as ul; [print(ul.unquote_plus(l), end='') for l in sys.stdin]"
Collapse
k4ml profile image
Kamal Mustafa Author

Thanks for the tips!