I'm writing this post to illustrate my personal learning experience around runtime speed, High and low-level programming languages, and their pros and cons.
I work as a junior system administrator for the university manage around 600 Linux machines, 7000 users, 350 Unix groups, and a handful of net groups. A few months ago, we decided to migrate our directory services from Network Information Services(NIS) to Light Weight Directory Access Protocol(LDAP). I was asked to write a migration script and test it on the database.
I was excited to write such a critical piece of infrastructure and worked hard using the PythonLdap API to get all the correct fields set up in LDAP. After a few weeks, my script was ready for production, migrating all the users and their data fields, the Unix groups, and net groups to the new directory server. The idea was that the script would keep LDAP and our old database in sync. If something changed on LDAP, the changes would be reflected on NIS and vise-versa.
Speed
The script worked great. Except for that, it took about 5min to iterate through both databases. I looked into optimizations, checked my data structures, and reduced runtime by about 1min. Not fast enough. We would call the script to run every time a change is made to the NIS database to update LDAP.
My boss told me he wanted me to try and rewrite the script in C. I was bummed out about this at first because I had spent so much time working on the python script. It felt like all my work debugging and optimizing the script was for nothing. However, after some time, I got excited about the opportunity to dive deeper into a low-level programing language. C is essential and still widely used in industry.
Lessons Learned
Python compiles at runtime. On average C runs about 45 times faster than python because it is a compiled low-level language. Python is an interpreted language, meaning one line at a time is translated into machine code. Python has some advantages in ease of use and syntax, but C allows the developer to become familiar with memory architectures and delicate grained nuances of systems like buffer overflows.
Both languages have their place in the world. If you are building a lightweight automation script, python works just fine. But if you want to write a piece of network infrastructure, you should write in a language with a lot of granularity and speed. If you are pursuing a Career in Networking or cyber Security, I recommend including both a scripting language and a low-level language in your tool kit.
My recommendations for scripting languages
My recommendations for low level languages:
- C Note: The official documentation is not very user friendly but there are plenty of open resources available
- Rust Note: Rust is a newer precompiled language but it becoming widely adopted very quickly. It has some security advantages over C and is favored highly.
- Depending on what you are doing Assembly is also very granular
Top comments (8)
Please take a look on pypy 😉
I also don't recommend Rust. My preference goes to Go which is way easier for approximately the same performances if you only need speed, coroutines and good memory management. For specific problems, like trying to work on new Linux kernel or for specific memory management, Rust is ok.
Excellent, thanks for the feedback! I'll check out pypy. I am mainly interested in Rust because it's relevant to my thesis and cyclic redundancy checks. I am also following a few of the rust cryptography crates on GitHub, which has a lot of community involvement. I don't know very much about go, but I will start looking into it anyway because we have a GO SDK for hedera that I need to be acquainted with. Either way, super grateful for your input. The community here is fantastic :relaxed
Yes, I am not trying to criticize this or that language without reason. I just noticed a tendency to see the community using tools by fashion effect and I find it very unfortunate for technologies that are losing visibility while they are much more suitable for many projects.
And indeed, the community on this site is really cool 😎
I can see where you're coming from. Programing can be a vast field and finding the right tool for the job can seem like aa daunting task. Finding a needle in the hey stack at times. It's a good learning experience though!
Did you look into cython? It might of helped to speed things up.
I've heard of this. I'll have to take a deeper look into it. Thanks for the recommendation.
Didn't you compile the code through compile library? That makes a .pyc extension file that's is more faster than runtime interpreted python source code.
I did give this a try. I'm going to attempt to implement some of the recomended libraries and see if that makes a big difference :)