DEV Community

Cover image for How I Built A Python Web Framework And Became An Open Source Maintainer

How I Built A Python Web Framework And Became An Open Source Maintainer

Florimond Manca on December 23, 2018

This post was originally published at blog.florimond.dev. It's been a while since I've written a blog post. Nearly two months, actually. So, where...
Collapse
 
rhymes profile image
rhymes • Edited

Wow great work and article, Florimond.

Reinventing the wheel is not a bad thing per se and you've demonstrated it (even if it's true that historically Python has had too many web frameworks :P)

I need to spend some more time on async Python, I'll definitely try Bocadillo. The amount of work you pulled off in less than two months is astounding!

Thank you for telling your story as a new open source maintainer.

I don't know if @ben has read this article already but I'm tagging him because he should :-)

Collapse
 
florimondmanca profile image
Florimond Manca • Edited

Brilliant; thanks for the very kind words! πŸ™Œ I’ve found async Python strangely exciting so far. At first, it all looks like black magic until you realize there is a higher-level metaphor which the whole event loop and coroutines stuff is merely an implementation of. Watching talks on the origins of async definitely helps there. :) If you’re about to deep dive, let me know how that goes!

Collapse
 
rhymes profile image
rhymes

I have a bit of experience with async programming in general through Twisted (and JS). Twisted was the first framework I've used and it introduced me to deferreds (promises), an "async/await" mechanism, a single thread main loop (based on the various kqueue/epoll/completion ports) and (at the time) a sizable community.

I haven't kept up much with the new async framework in Python to be honest, that's why I said I need to catch up.

I still have some hesitation, not because of Python, but because I'm not 100% sold on the concurrency model (even though I recognize its marvels). Async programming forces your entire app (the client app you mount in the framework) to be async and mostly single threaded. This means that every IO library you plug in has to be async.

You can't do async from sync which means that slowly everything you write has to be modeled on the async framework.

It's the same exact problem Node has, it's nothing wrong per se, it's just that "async/await" seem simple keywords but they bring with them cognitive load (well, all concurrency paradigms do) and some "existential questions" πŸ˜…

It is true though that async is a valid alternative, especially in Python (which I believe is the whole reason why Twisted started doing async back in 2002 :D). Good things come back.

Thread Thread
 
florimondmanca profile image
Florimond Manca

Absolutely β€” as some say, async is "all-in".

As you pointed out, I don't think it is specific to how Python does async. It's really two separate views on how things should even run. It does lead to having to basically rebuild everything from scratch just so that it is made non-blocking, which is frankly a rather high barrier to entry.
Luckily in Python, projects like aio-libs are already on their way to building an async equivalent to our otherwise familiar synchronous world. :)

Reminds me of times when I used Kafka for stream processing and I had a hard time interacting with batch-processing systems. Maybe there's a parallel?

  • A batch system is essentially blocking the whole pipeline instead of letting other (streaming) systems consume its changes one after the other.
  • A synchronous function blocks the whole event loop because it doesn't let other (async) functions use the CPU or parts of its outputs while it is running.

Maybe it's this cooperative concurrency model that creates the separation between the sync and async worlds?

I like it though, because I think it's also how we humans work. Our brain is basically a single-threaded concurrent systems that can deal with multiple things at once (to a certain extent) but only ever do one thing at a given time.

Hmm…

"existential questions"

Here we are? :P

Thread Thread
 
rhymes profile image
rhymes

Absolutely β€” as some say, async is "all-in".

Yeah, and that's the divide right there. Async is basically a low level framework. If you adopt it, everything has to be modeled after it. The same way some people criticize some frameworks because they model your application instead of the other way around. Just food for thought.

Luckily in Python, projects like aio-libs are already on their way to building an async equivalent to our otherwise familiar synchronous world. :)

A part of me rejoices, another part of me wonders why we're reimplementing everything from scratch for the nth time :-D

I like it though, because I think it's also how we humans work. Our brain is basically a single-threaded concurrent systems that can deal with multiple things at once (to a certain extent) but only ever do one thing at a given time.

But is it truly? :-) Could it be a bunch of communicating sequential processes instead?

Thread Thread
 
florimondmanca profile image
Florimond Manca

But is it truly? :-) Could it be a bunch of communicating sequential processes instead?

I think it kind of is at a physiological/unconscious level. But I know that's not how my conscious self works. 😁

Collapse
 
florimondmanca profile image
Florimond Manca • Edited

Hey, great to hear! Glad you enjoyed it. :)

Collapse
 
giorgosk profile image
Giorgos Kontopoulos πŸ‘€ • Edited

Very inspirational.

I wish I was able to start my open source ideas but I never seem to find the time. There so little time and so many things to do ...

Collapse
 
florimondmanca profile image
Florimond Manca

True that it can be hard to find the time. That said, open source doesn’t have to take a lot of time. :) A few hours here and there or on weekends can go a long way. As I said in the article, it’s not a sprint! A lot of the work is deciding and clarifying what you want to do, and mapping out what your project should do. Then you can work towards that in small steps. :)

Collapse
 
buet17 profile image
buet17

I have a tense about your write up. I am learning oon Python.
Please feel free contact with me at nour.buet@gmail.com

Collapse
 
luishck profile image
Luis J. Centeno

As a Spanish speaker I feel hungry when read Bocadillo.
I loved reading your post!

Collapse
 
florimondmanca profile image
Florimond Manca

Thanks! The Spanish translation is exactly the origin of the name. πŸ₯ͺ