DEV Community

Giannis Ftaras
Giannis Ftaras

Posted on

Recursive wget and file transfer

We can simply transfer files between two PCs via the network without having to use USB sticks or other “archaic” methods.

We first cd to the directory we want and then run a simple python HTTP server.

Python2: python -m SimpleHTTPServer <port_number>
Python3: python3 -m http.server <port_number>

On the other machine we then use wget to recursively retrieve all the files from the remote directory.

wget -r -np -R "<condition>" <remote_url>

-r Recursively retrieve all files and directories from the remote machine.
-np No Parent flag. Do not ascend to the parent directory when retrieving recursively
-R Reject all files that match the conditions

Example: wget -r -np -R "index.html*"

Why reject index.html?

When we use wget recursively the command downloads automatically an index.html file for each directory downloaded from the remote machine. With this flag we keep our downloads tidy and clean without any unnecessary files.

Top comments (1)

kwabenasapong profile image

How will you download the website if it requires authentication using a username, password and an authenticity token? I tried the following below but I get stuck on the sign-in page;

!/usr/bin/env bash

code=wget -qO- https://urlname/sign_in service= | cat | grep 'name="lt"' | cut -d"_" -f2
wget --save-cookies cookies.txt \
--keep-session-cookies \
--post-data 'username=$username&password=$password&lt=$hidden_code&_eventId=submit' \
--delete-after \

wget --load-cookies cookies.txt \