Recently, we needed to download a file from a Docker container. Since the file was not inside a bind mount, we couldn't directly access the remote server to get the file on the file system.
So we quickly found the docker cp utility to copy the file on our host and voila, we had the file on our computer. ☺️
docker cp CONTAINER_NAME:/opt/service/FILENAME_TO_OBTAIN .
docker cp command, where the name of the container is known.Unfortunately, we didn't know the exact name of the container, as containers created by docker compose are, by default, suffixed with a randomly generated hexadecimal string, such as .eq6l28g5mwsenh0gvag7zcg0j. 😶🌫️
And because we deployed our stack in swarm mode, we couldn't use the container_name option as it is ignored in this mode...
So, to obtain the full name of the container, we used the docker ps command, which lists the containers, along with its --format option to get only its full name.
docker ps --format='{{.Names}}' | grep "CONTAINER_NAME_PREFIX"
docker ps command, where we know the prefix of the container name.If we combine these commands, we can have only one command to run to download our file.
docker cp "$(docker ps --format='{{.Names}}' | grep "CONTAINER_NAME_PREFIX")":"/opt/service/FILENAME_TO_OBTAIN" .
It worked fine on our local computer, but the real file we were interested in was on a remote server... And this is where it all gets complicated for us. 🤮
Constraints
- We wanted an automated solution where we could just provide some information and we would get the file. 🤖
- The container was hosted on a random server from a pool of Docker machines. 🌈
- We had to be privileged users to run certain Docker commands. 👑
After a lot of guesswork and mistakes, we finally managed to get a working solution under these constraints, but it took longer than expected. Let's dive into some of the issues we encountered. 😌
Issues
How to run sudo commands on a remote server
In my opinion, this step is the most complicated part of our final script. Please be patient with me as I explain some of the things we have done below. 🤢
# Define and export some variables that will be used in this part
export SERVER=SERVER_HOSTNAME
export FILENAME=FILENAME_TO_OBTAIN
export CONTAINER_PREFIX=ranb2002_system
export PASSWORD=PASSWORD_TO_BE_ROOT
# Create a template where the environment variables will be replaced
TEMPLATE=$(
envsubst <<'EOF'
# Authenticate as sudo and pass the password without indentation
sudo -S whoami
$PASSWORD
# Delete the previous file if it exists on the server
rm -f "${FILENAME}"
# List all Docker containers on this server
CONTAINERS=$(sudo -S docker ps --format='{{.Names}}')
# Check if we have a container for our application on this server
CONTAINER_NAME=$(printf '%s\n' "${CONTAINERS[@]}" | grep "${CONTAINER_PREFIX}")
# Otherwise, continue to the next server
if [-z "${CONTAINER_NAME[0]}" ]; then
echo "The container with name ${CONTAINER_PREFIX} was not found on this server ($(hostname))... :("
exit
fi
echo "The container with name ${CONTAINER_NAME[0]} was found on this server ($(hostname))! :)"
# Copy the file from the container to the current host
sudo -S docker cp "${CONTAINER_NAME[0]}":"/opt/service/${FILENAME}" .
EOF
)
# Connect to the server and run all commands from the template
ssh -T "${USER}@${SERVER}" <<< "$TEMPLATE"
- To run multiple commands via SSH, we use a here document structure that lists all the commands we want to run on the remote host.
- Since the variables were not replaced inside this structure, we encompassed it with the envsubst command.
- To make the shell variables accessible to
envsubst, we had toexportthem. - Since we wanted to capture the container name in a variable, we ended up splitting the commands because our one-line command didn't work well with
sudo. - Since my user was not able to run Docker commands on this server, we had to gain root privileges. We have tried many (many) attempts and we managed to get a working solution by outputting our password to the first
sudo -Scommand. The following commands don't need to have the password again. 🥺 - For some reason still unknown to me, the value of the
CONTAINER_NAMEvariable is an array... So, to access it, we had no choice but to useCONTAINER_NAME[0]... 🤷
As you may have noticed, we didn't need to pass our password when running the
sshcommand, because we are using SSH public keys. If you want to install SSH keys on your servers, check out my article on how to automate ssh-copy-id! 😉
How to transfer the file from the remote server to our host
This part was easy to find, using sshpass and scp commands.
# Copy the file from the remote server to our host
sshpass -p "${PASSWORD}" scp "${USER}@${SERVER}":"/home/${USER}/${FILENAME}" .
sshpass to suppress the password prompt when connecting to the remote server.Since we had previously installed an SSH public key on the server and we are using the same username on our host and the server, we can rewrite the previous command as follows.
scp "${SERVER}":"~/${FILENAME}" .
Simpler, right? 😉
How to find the server where the application container runs
To browse over a cluster of servers, we have provided the list as an array, along with a simple switch case. It's always useful when we need it to have a switch case example in Bash , so I'm putting it here for future reference. 😋
case "${ENVIRONMENT}" in
DEV)
SERVERS=(
"DEV_SERVER_1_HOSTNAME"
"DEV_SERVER_2_HOSTNAME"
)
;;
TEST)
SERVERS=(
"TEST_SERVER_1_HOSTNAME"
"TEST_SERVER_2_HOSTNAME"
)
;;
*)
SERVERS=(
"PRODUCTION_SERVER_1_HOSTNAME"
"PRODUCTION_SERVER_2_HOSTNAME"
)
;;
esac
for SERVER in "${SERVERS[@]}"; do
# All previous commands
done
Unfortunately, since we are not able to know on which server we need to transfer the file from, we have to run all the previous commands on each server... 😵💫
So, we get a lot of scp: /home/ranb2002/[...]: No such file or directory error because only one server has the wanted file... 😒
How to ensure the arguments are provided
Since our script could fail if some arguments were not supplied or were incorrect, we added some conditions to verify these cases.
Note : if the password is incorrect or if the user is not a privileged user, the script will still fail because we haven't taken some time to figure out how to prevent this.
#!/bin/bash
echo "Usage:"
echo " ./scripts/copy_config.sh 'PASSWORD' ENVIRONMENT"
echo " where ENVIRONMENT is DEV, TEST, DEMO or PROD."
echo ""
# Make sure we have your password
if [-z "${1}"]; then
echo "You must provide your password!"
exit
fi
# Export the password to an environment variable
export PASSWORD="${1}"
# Make sure we have the environment
if [-z "${2}"]; then
echo "You must provide the environment!"
exit
fi
export ENVIRONMENT="${2}"
# Make sure the environment provided is supported
ENVIRONMENTS=("DEV" "TEST" "DEMO" "PROD")
if [[! " ${ENVIRONMENTS[*]} " == *" ${ENVIRONMENT} "* ]]; then
#
echo "The environment provided is incorrect!"
exit
fi
Bonus : if for some reason you need to transform an uppercase value to its lowercase equivalent, here is the Bash command you can use:
LOWER_ENVIRONMENT=$(echo "${ENVIRONMENT}" | awk '{print tolower($0)}'). 😜
Final solution
If we put together all the pieces together, our final script looks like below. 🧩
#!/bin/bash
echo "Usage:"
echo " ./scripts/copy_config.sh 'PASSWORD' ENVIRONMENT"
echo " where ENVIRONMENT is DEV, TEST, DEMO or PROD."
echo ""
# Make sure we have your password
if [-z "${1}"]; then
echo "You must provide your password!"
exit
fi
# Export the password to an environment variable
export PASSWORD="${1}"
# Make sure we have the environment
if [-z "${2}"]; then
echo "You must provide the environment!"
exit
fi
export ENVIRONMENT="${2}"
# Make sure the environment provided is supported
ENVIRONMENTS=("DEV" "TEST" "DEMO" "PROD")
if [[! " ${ENVIRONMENTS[*]} " == *" ${ENVIRONMENT} "* ]]; then
#
echo "The environment provided is incorrect!"
exit
fi
LOWER_ENVIRONMENT=$(echo "${ENVIRONMENT}" | awk '{print tolower($0)}')
export DOCKER_STACK="stack-${LOWER_ENVIRONMENT}_service"
export FILENAME="FILENAME_TO_DOWNLOAD"
case "${ENVIRONMENT}" in
DEV)
SERVERS=(
"DEV_SERVER_1_HOSTNAME"
"DEV_SERVER_2_HOSTNAME"
)
;;
TEST)
SERVERS=(
"TEST_SERVER_1_HOSTNAME"
"TEST_SERVER_2_HOSTNAME"
)
;;
*)
SERVERS=(
"PRODUCTION_SERVER_1_HOSTNAME"
"PRODUCTION_SERVER_2_HOSTNAME"
)
;;
esac
for SERVER in "${SERVERS[@]}"; do
TEMPLATE=$(
envsubst <<'EOF'
# Authenticate as sudo and pass the password with no indentation
sudo -S whoami
$PASSWORD
# Delete the previous file if it exists
rm -f "${FILENAME}"
# List all Docker containers on this server
CONTAINERS=$(sudo -S docker ps --format='{{.Names}}')
# Check if we have a container for our application on this server
CONTAINER_NAME=$(printf '%s\n' "${CONTAINERS[@]}" | grep "${DOCKER_STACK}")
# Otherwise, continue to the next server
if [-z "${CONTAINER_NAME[0]}" ]
then
echo "The container with name ${DOCKER_STACK} was not found on this server ($(hostname))... :("
exit
fi
echo "The container with name ${CONTAINER_NAME[0]} was found on this server ($(hostname))! :)"
# Copy the file from the container to the current host
sudo -S docker cp "${CONTAINER_NAME[0]}":"/opt/service/${FILENAME}" .
EOF
)
# Connect to the server and run all commands from the template
ssh -T "${SERVER}" <<< "$TEMPLATE"
# Copy the file from the server to our computer
# shellcheck disable=SC2140
scp "${SERVER}":"~/${FILENAME}" .
done
Hope this post is useful to you! Stay safe! 🍻
Top comments (0)