When in comes to Web Development nothing beats JavaScript. But sometimes we have to do a bit more demanding task, for example analyzing big pile of data. In that case Python might be a superior option. But that's just one function of our website. Do we want to switch to Python just because of that one feature? Probably not.
So what if we could build our backend mostly using NodeJS and only use Python when we have to.
That would be awesome right? We can use child process in Node.JS to run a python script when needed.
const spawn = require('child_process').spawn
app.get("process_data", (req, res) => {
spawn('python3', ['script.py'])
})
# script.py
doSometing()
And if we want we can pass data to our python script also.
const spawn = require('child_process').spawn
app.get("process_data", (req, res) => {
const msg = "Hello"
spawn('python3', ['script.py', msg])
})
In Python in order to be able to read the data you must import the sys module.
import sys, json
def main():
msg = sys.argv[1]
doSometing(msg)
if __name__ == '__main__':
main()
Now instead on passing data while spawning the Python process, lets send data in stream.
const spawn = require('child_process').spawn,
const py = spawn('python3', ['script.py'])
const data = {
msg: "Hello"
}
py.stdin.write(JSON.stringify(data)) //we have to send data as a string, so we are using JSON.stringify
py.stdin.end()
import sys, json
def main():
lines = sys.stdin.readlines()
data = json.loads(lines)
doSometing(data['msg'])
if __name__ == '__main__':
main()
Finally we can send response back to our nodejs from the python script
const spawn = require('child_process').spawn
const py = spawn('python3', ['cscript.py'])
py.stdout.on('data', function(res){
let data = JSON.parse(res.toString())
console.log(data)
})
import sys
# You will have your own implementation of get data. In this case lets assume it returns a dict/json
res = getData()
print(json.dumps(data))
sys.stdout.flush()
So this article has come to an end. But make sure you check out my other articles.
Top comments (15)
i was working on a similar project that i needed to pass data from node js to a python script, i think you may use flask better and create an endpoint to send and receive data between them, it can help you in using asynchronous functions
great approach
Or just use Python and Flask :) A much more pleasant way to develop web apps than Node :) Everything web doesn't have to be JavaScript
Or aiohttp which is faster and has native asyncio support.
You have to make async process queue in Python side for this approach. In other case it will work wrong because node.js has shared context among clients and Python instance going to be shared because you execute it in global context. Other option is execute Python in callback or use messaging queue like rabbitmq.
I agree partially with the other comments, an easier and maybe more elegant approach could be to use just python with a framework in the backemd, but I find your article very useful for existing legacy project in nodejs where you need to run something in python (i.e. data processing, ml flows, etc), because sometimes it is more convenient and faster than to create a microservice. Thanks for the article, keep writing 🙌.
Many thanks 💓
related to this topic ... filebus uses a file watcher to notify each-other when either NodeJS sent something to process, or Python did ... have a look 👋
Or you can run apache kafka and get an extremely fast, scalable and stable communication tunnel between your processes instead of shit described above.
Kafka is much more harder to maintain, and can increase the product cost.
So maybe the "shit" described above is can fit better than your "shit"
I think when sending data TO the python process, you'll need to write to
stdout(on the node side) rather thanstdinGenerally you write to stdout and read from stdin
Wonderful idea
Very interesting. Had no idea this can happen
I was also thinking about this from the last few weeks. Nice to have it here. 🥰
Glad you found it.