Organiser of the Edinburgh Language Exchange and The Edinburgh Open Tech Scene |
Full Snack Developer 🥪, Ramen guzzler 🍜, quiche murderer 🥧. A friendly cat.
So the current bash shell actually causes two calls to the python script indeed ; each one's output actually writes to a named pipe (typically held in RAM) rather than to disk.
I suspect the overhead of calling the two python processes from a bash script would be significantly outweighed by writing to two on-disk temp files, which diff then needs to read... I was partway through trying to implement the suggestion when I realised this... 🤦♂️
I did have a look at difflib to see if there's a way of passing the chunks to it gradually, but no, it expects a pre-loaded entirety of data in the form of a list of strings. On reflection, I do believe it is part of the diffing algorithm to look at "future" data further down so as to compute the context. Worst-case scenario, the context will be searched for on the whole file...
Generally is a sensible option, because diff is normally used with source files in the small kilobytes range....
--
Try running
# Print the start datedate# Run a no-op that takes the file-descriptors for FILE1 and FILE2 simultaneously# The completion times for each will write to stderr only after completion
: <(bexplode.py FILE1 >/dev/null;echo"File 1: $(date)">&2) <(bexplode.py FILE2 >/dev/null;echo"File 2: $(date)">&2)echo"Please wait - processing in background ..."
It should show that the total time will only be the time of the largest file.
The main culprit is likely diff, loading both fully, and then running the diffing algorithm...
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
So the current bash shell actually causes two calls to the python script indeed ; each one's output actually writes to a named pipe (typically held in RAM) rather than to disk.
I suspect the overhead of calling the two python processes from a bash script would be significantly outweighed by writing to two on-disk temp files, which
diff
then needs to read... I was partway through trying to implement the suggestion when I realised this... 🤦♂️I did have a look at
difflib
to see if there's a way of passing the chunks to it gradually, but no, it expects a pre-loaded entirety of data in the form of a list of strings. On reflection, I do believe it is part of the diffing algorithm to look at "future" data further down so as to compute the context. Worst-case scenario, the context will be searched for on the whole file...Generally is a sensible option, because diff is normally used with source files in the small kilobytes range....
--
Try running
It should show that the total time will only be the time of the largest file.
The main culprit is likely diff, loading both fully, and then running the diffing algorithm...