Automating ISO Customization with Jenkins and Shell Script

Welcome back gentle reader, I am Silent-Mobius, aka Alex M. Schapelle, your mechanical floating point calculator that will guide you with topics of open-source code and software.

In one of my previous articles we talked about Customizing Debian Based ISO's, in it we describe the steps for customizing Debian based operational system ISO, while disassembling ISO files, editing and configuring automated installation with nocloud, which we covered in other article, adding repositories, tools and software for future use, and then packing it back to new ISO file for future use.

Why Not Cubic ?

Cubic is Ubuntu based Graphical Tool for manual ISO manipulation, on which we have written previously short tutorial

Short answer: from short over view of the tool, it can not be used on terminal, meaning --> it can not be embedded inside CI/CD that we are aiming at.

Self Check

After re-reading the article, in an attempt to update the article, yes - quarterly we check our own articles to see if there is a case to adding/updating with new data, several things popped out to me:

• Doing everything manually is tire some
• Dependencies are confusing
• Steps are not clear
• Reproducing the procedure without article is some what hard.

Just like any other mechanic calculator, once reached all zeros, I'll have to restart counting numbers, so why not to do it with more suitable tools ?
Here's my choice:

• Bash Script : Cause supported by any *nix system
• Jenkins : Cause of the automation
• Debian : Cause IMPO there is no better *nix based system (RedHat kind of disappointed me) The tools I've chosen are basic, yet provide clear basics for SysOps, DevOps, SysAdmin and anyone who can use *nix based systems.

The Great Plan

To plan our work we'll use steps from previous essays, with clear directions on what we want to create and eventually implement those direction with previously mentioned tools. Automation is the key to getting the software working, thus we'll combined Jenkins and bash script creating dual element tool for creating automated ISO editor.
The plan itself:

• Write the steps
• Setup up development environment
• Create Tool for editing ISO
• Setup CI/CD for automating dynamic values for our tool

Write the steps

ISO stands for International Organization for Standardization which is governing body that develops and publishes standardization in all technical and nontechnical fields other than electrical and electronic engineering, which is handled by the IEC. ISO number 9660 is a file system for optical disc media, which was used for arranged file information in a dense, sequential layout... AKA ISO format Operational System.
To put it to human term: ISO is just a way the tree folder structure is compressed and arranged. Sound like zip file, right ?
But that is enriching information, the steps to handle ISO file are as follow:

• Disassemble ISO file by extracting it to folder with 7z tool
• Un-squash the squash-sf file that holds the install-able file system with unsquashfs from squashfs-tools package
• Modify files, users, groups, binaries and configuration with your required tools and utilities
• Clean up the changes to reduce size of ISO
• Make squash file to replace initial squash-fs file with mksquash from squashfs-tools
• Repack extracted folder with xirroso

Seems to me that we have here something that looks like a plan, wouldn't you agree ? Let's go on by setting a working environment with required dependencies and dedicated explanations.

Setup up development environment

Some Assumptions/My Setup :

• We are using Debian based Linux distribution
• Any text editor is fine as long as you know how to operate it.
• We'll need couple ISO's of Debian and Ubuntu to test all we are working with.
• We have short list of packages that we'll need to have installed:
  • git
  • p7zip-full
  • squashfs-tools
  • genisoimage
  • fakeroot
  • pwgen
  • whois
  • xorriso
  • isolinux
  • binutils

So let's install them, so not miss it future:

sudo apt-get install -y p7zip-full git genisoimage fakeroot pwgen whois xorriso isolinux binutils squashfs-tools

And while at it, let us get ISO files of the Linux Distributions that we are gonna work on:

curl -L https://mirror.isoc.org.il/pub/ubuntu-releases/22.04.3/ubuntu-22.04.3-live-server-amd64.iso -o ~/Projects/ogun/ubuntu-22.04.3-live-server-amd64.iso

curl -L https://gemmei.ftp.acc.umu.se/debian-cd/12.1.0-live/amd64/iso-hybrid/debian-live-12.1.0-amd64-standard.iso -o ~/Projects/ogun/debian-live-12.1.0-amd64-standard.iso

IMHO, it is always good to have dedicated folder where we havoc our way into those ISO's. Let's call our small project with catchy code name of OGUN, a spirit that appears in several African religions, mostly as diety of blacksmiths and techonologists, and if it does not make sense to you then no need to dive into it.

While inside the folder, let us save our code and files with git version control on our gitlab repository to keep track of things during our explorations:

mkdir -p Projects/ogun
git init
git config user.name 'silent-mobius'
git config user.email 'alexm@otomato.io'
git remote add origin https://gitlab.com/silent-mobius/ogun.git
echo "Project Ogun" > README.md
# adding gitignore for future use
echo "\*.iso" > .gitignore
echo "iso" >> .gitignore
echo "squashfs-root" >> .gitignore
git add README.md .gitignore
git commit -m "adding README file"
git push -u origin https://gitlab.com/silent-mobius/ogun.git

Back to the topic : Disassemble

As mentioned, ISO is just a way the tree folder structure is compressed and arranged. Sounds like zip file...
What can we do with zip files: zip and unzip. One of the easiest open source tool for zip/unzip files, available for all platforms, is p7zip.

To unzip or to be precise export ISO file we can use:

7z x -y debian-live-12.1.0-amd64-standard.iso -oiso

The command be translated as follows

• 7z : Binary for 7zip
• x : eXtract files with full paths
• -y : Assume Yes on all queries
• -o : Set Output directory

If you are working with my direction, then under our project folder we'll be able to find new folder named iso that will include the content of the Debian ISO we have previously downloaded.

The same command can be transcribed as shell script function:

function disassemble_iso(){ 
        local IN=$1 # full path is required ...
        local OUT=$2
            7z x -y $IN -o$OUT
    }

[!] Incoming Notice : you promised tool, why just running commands?
Indeed, no need to just write commands, lets build something with them:

• I'll be writing the full script with descriptions in the next segment, and till then we'll cover the steps and how we can automate them in shell functions...
• Lets create script named ogun.sh that will be converted as shell tool.
• I prefer to use headers in my script, so in case something not clear, ask in comments below
• My shell scripts are written in functional programming pattern, thus if not clear, you are welcome to ask questions, in comments and open PR/MR in gitlab repository.

[!] Note: I might rewrite the tool in new programming language, thus if you'll search for shell/bash script equivalent, I'll be saving branch with dedicated name for you to use.

Un-squash the filesystem

Squashfs is a compressed read-only file system and it includes compressed files, inodes and directories under one file. Most of Linux based distributions are using it during the install process and it is the core of what you'll be using in your hardware.

Thus we need to open the sqyashfs filesystem, in order to make necessary changes. Usually it is done with unsquashfs tool and most common way to use it will look like this:

cp iso/live/filesystem.squashfs . # include the dot at the end
sudo unsquashfs filesystem.squashfs
rm -rf filesystem.squashfs
ls -l

• We are removing the original filesystem.squashfs file, so we won't be confused later on.
• Once unsquashed, ls command should show us new folder named squashfs-root, yet you can rename to anything else with -d option in case you would like to use different name (from my experience while writing this article, it is not suggested)

To translate it in to shell function it could look like this:

function disassemble_squashfile(){ 
        local IN=$1 # full path to squashfs file
        local OUT=$2 # where to export the file
            mv "$IN" "$OUT"
            unsquashfs "./${IN##*/}"
            rm -rf "./${IN##*/}"
    }

• [!] I'll probably modify variable names later to be more readable - I'll present it all when we get to it.

Let's Modify the unsquashfs filesystem

When it comes to modification, it is really up to the modifier him/her selves to decide what and how they want the changes. Yet I'll still share my insight:

• DO NOT CREATE YOUR OWN USER: the install process will remain the same, thus no point adding your own user
  • However you can setup your custom configurations to user skeleton of the system
    • Set your changes in .profile or .bashrc in squashfs-root/etc/skel/ folder.
    • Configure custom prompt variable.
    • Add your aliases.
    • Extend PATH variable.
    • Install packages and binaries.
    • if you use vim, then add .vimrc configuration file.
    • Not my preference, yet you can add ssh-keys in here.
• Less talking more coding:
  • When doing changes remember that you are changing your root folder, meaning that we need to login in to read-only filesystem with chroot command, which runs a specified commands with a specified root directory (squashfs-root in our case)

sudo chroot squashfs-root # prompt should change

cd /etc/skel

echo 'set nocompatible
set number
colorscheme desert
if has("syntax")
  syntax on
endif
' > .vimrc

echo 'PATH=$PATH:/usr/local/bin:/usr/local/sbin' >> .bashrc #extending PATH variable

echo "PS1='\[\033[01;32m\]\u\[\033[00m\]@\[\033[01;32m\]\h \[\033[00m\]\w\[\033[01;34m\] [$(git symbolic-ref --short HEAD 2>/dev/null)]\[\033[00m\]\n$ '" >> .bashrc

echo "
alias cl='clear'
alias cp='cp -v'
alias g='git'
alias ga='git add'
alias gc='git clone'
alias gp='git push'
alias gpo='git push -u origin'
alias k='kubectl'
alias kd='kubectl describe'
alias kg='kubectl get'
alias kgn='kubectl get nodes'
alias kgp='kubectl get pods'
alias l='ls'
alias ll='ls -l'
alias ls='ls --color=auto'
alias mv='mv -v'
alias vi='vim'
" >> .bashrc 

echo 'nameserver 8.8.8.8' > /etc/resolv.conf # squashfs does not recongnized the network automatically

echo '
deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware
deb http://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security/ bookworm-security main contrib non-free non-free-firmware
deb http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
' > /etc/apt/sources.list


apt-get update

apt-get install -y build-essential task-cinnamon-desktop plank arp-scan guake plank geany meld moka-icon-theme wireshark ethtool arp-scan nmap python3-nmap vlc nala macchanger nmap etherape git gitg gnupg2 curl cmake ipython3 vim libvirt-daemon p7zip-full plymouth plymouth-themes remmina libvirt-daemon-system virtinst bridge-utils python3-networkmanager python3-networkx vagrant vagrant-libvirt python3-netmiko python3-netifaces python3-netaddr gir1.2-gtop-2.0 glade terminator vlc virt-manager bash-completion darkslide python-landslide jq yq tomlq pipx

# All binary installation will require manual changes because by default they are installed to user root

curl -sL get.sdkman.io | bash # this is installed into /root folder in squashfs-root system

cp -r ~/.sdkman/ /etc/skel/ # we are copying the system to spread on whole system

echo '#THIS MUST BE AT THE END OF THE FILE FOR SDKMAN TO WORK!!!
export SDKMAN_DIR="$HOME/.sdkman"
[[ -s "$HOME/.sdkman/bin/sdkman-init.sh" ]] && source "$HOME/.sdkman/bin/sdkman-init.sh"' >> /etc/skel/.bashrc # adding sdkman init script to path

curl -sL install.python-poetry.org | python3 -

curl -sL get.docker.com | bash

curl -L https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 -O /usr/local/bin/minikube

chmod +x /usr/local/bin/minikube 

#installing vscodium 
wget -qO - https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/master/pub.gpg | gpg --dearmor | dd of=/usr/share/keyrings/vscodium-archive-keyring.gpg

echo 'deb [ signed-by=/usr/share/keyrings/vscodium-archive-keyring.gpg ] https://download.vscodium.com/debs vscodium main' | tee /etc/apt/sources.list.d/vscodium.list

apt-get update && apt-get install -y codium

• Yes, indeed I love installing packages and binaries
• All the commands above are more or less the changes I use for creating my custom Debian ISO's
• And I admit: It does look like mess, much less how it can be automated with bash script, yet here is a function I use myself:

function configure_squashfs_folder(){
local CHROOT=$1
      curl -sL https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/master/pub.gpg -o "$CHROOT/usr/share/keyrings/vscodium-archive-keyring.gpg"
      curl -sL https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 -O $CHROOT/usr/local/bin/minikube
      chmod +x $CHROOT/usr/local/bin/minikube
      echo -e '\n
deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware \n
deb http://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware \n
deb http://security.debian.org/debian-security/ bookworm-security main contrib non-free non-free-firmware \n
deb http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware \n
' > $CHROOT/etc/apt/sources.list

chroot $CHROOT /bin/bash -c "
     echo 'nameserver 8.8.8.8' > /etc/resolv.conf
     apt-get update
     DEBIAN_FRONTEND=noninteractive apt-get install -y apt-get install -y build-essential task-cinnamon-desktop plank arp-scan guake plank geany meld moka-icon-theme wireshark ethtool arp-scan nmap python3-nmap vlc nala macchanger nmap etherape git gitg gnupg2 cmake ipython3 vim libvirt-daemon p7zip-full plymouth plymouth-themes remmina libvirt-daemon-system virtinst bridge-utils python3-networkmanager python3-networkx vagrant vagrant-libvirt python3-netmiko python3-netifaces python3-netaddr gir1.2-gtop-2.0 glade terminator vlc virt-manager bash-completion darkslide python-landslide jq yq tomlq pipx
"
}

• I do agree that function looks mess as well
  • Yet I do promise we'll clean it up with more general script
• BTW: we are all done... (are we though?)

Every mess needs Cleanup

All the changes we made will increase the size of the squash-filesystem, and like after long evening of cooking with kids (which I don't have), we do have something to show, like they have on nailed it, yet one thing that we are not shown is the clean up that happens after the show and which is what we need to do, in order to ensure clean filesystem. Mostly useful steps would be to remove temporary files, removing caches and packages, and of course, clear the history.
So, while we are still in chrooted environment :

# We still need to be in chrooted environment
cd /var
du -h # If there is anything too big --> rm -rf that
rm -rf cache/apt/apt/*
rm -rf lib/apt/*
history -c

We came, we saw, we Resquash

Once the clean up is done we need to to recompress everything back to what is was before, and thus we do it by exiting squahs chroot environment and use mksquashfs to generate new read-only filesystem to use for our ISO.

exit # to exit from chroot
mksquashfs ./squashfs-root/ filesystem.squashfs -comp xz -b 1M -noappend

• ./squashfs-root/ : the folder which we updated.
• filesystem.squashfs: the name we are giving to new filesystem.
• mkdquashfs: tool to create and append to squashfs filesystems
• -comp : select compression algorythm: we use xz to maximlize on compression
• -b : set data block to 1 in our case.
• -noappend : do not append to existing filesystem.

[!] Note: This will use all cores on your CPU, just be paitient with it, by the end of it, the file will be generated and we have to return it to its rightful place:

mv filesystem.squashfs iso/live/filesystem.squashfs

This will place the filesystem.squashfs file back where we took it from and now all is left is

The shell script representation of is pretty simple and looks mostly like other functions we wrote up until now:

OUT_FOLDER="iso/live/"
function reassemble_squashfile(){
    local IN=$1
    local OUT=$2
        mksquashfs $IN $OUT -comp xz -b 1M -noappend
        mv $OUT "$OUT_FOLDER/$OUT"
    }
# [!] Note: the variable OUT_FOLDER is set outside of the function and should be used as a global variable

Recap for the REPACK

So to sum it up:
We have iso folder that includes new read-only filesystem.squashfs file with all the changes, tools and what not. All we need is Love ... that's different song... All we need is to repack the iso folder to convert is back to ISO.
To do so, we'll sign the changes on the iso folder with new md5sha256 signiture - we do not want to recieve warnings about security issues while installing the system.

md5sum iso/.disk/info > iso/md5sum.txt
sed -i "s|iso/|./|g"  iso/md5sum.txt

And to save it as a bash shell function:

ISO_FOLDER="iso/"
function sign_md5(){
        md5sum $ISO_FOLDER/.disk/info > $ISO_FOLDER/md5sum.txt
        sed -i "s|$ISO_FOLDER/|./|g" $ISO_FOLDER/md5sum.txt
    }

Now - to repack it all, we'll be using xorriso tool and isolinux library which in terms of command looks like this:

xorriso -as mkisofs -r -V "Custom  Debian12 Install" -J -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot                -boot-load-size 4 -isohybrid-mbr /usr/lib/ISOLINUX/isohdpfx.bin -boot-info-table -input-charset utf-8 -eltorito-alt-boot -e boot/grub/efi.img -no-emul-boot -isohybrid-gpt-basdat -o "custom-iso-deb12.iso" ./iso

• xorriso : Xorriso is a program which copies file objects from POSIX compliant filesystems into Rock Ridge enhanced ISO 9660 filesystems and performs session-wise manipulation of such filesystems.
• -as : Xorriso can emulate other software for creating ISO filesystems. Emulation enables other features to be enabled in one tool
• mkisofs: One of the most used format which xorriso can emulate. below options are combinations of xorriso and of mkisofs .
• -r:
• -V: short for add tag message
• -J: amount of cores to use for job, uses most in case no value is provided
• -b: short for -boot_image
• -c: short for execute command
• -no-emul-boot:
• -boot-load-size:
• -isohybrid-mbr: add isohybrid headers for bootloader
• -boot-info-table: update boot table

• input-charset:

• eltorito-alt-boot: add eltorito alternative boot for bootloader

• -e:

• isohybrid-gpt-basdat: add isohybrid-gpt headers for bootloader

• -o: output the generated data into file named with value provided

After few minutes of running it should generate ISO file with all the needed changes and ready to be test, used, deployed and once again modified for future use-cases.

The issue with that long and complicated command is that it not as easy to know whether you need all those features in your ISO file. There can be lots of cases that command will be much less complicated, yet also vice versa, much much more complicated. Thanks to dbkinghorn efforts one way to know what features you might need is to check how the initial ISO was created by running:

xorriso -indev debian-live-12.1.0-amd64-standard.iso -report_el_torito as_mkisofs

The command will output the setup the arguments for building an ISO file, which you can copy paste and use for your own build.

As a shell function the command it depends on the ISO that you are using, in this example for Debian the command provided above is fine, and for Ubuntu 20.04 it will work as well, in case of Ubuntu 22.04 it shall fail, so the function, for now at least, will look like this:

ISO_FOLDER="iso/"
function reassemble_iso(){
    local system_version=$1
        if [[ ! ${system_version} ]] || [[ ${#system_version} -gt 2 ]];then
            deco $_msg_reassemble_error 
            exit 1
        else
            if [[ ${system_version} -eq 22 ]];then
                xorriso -as mkisofs  -r   -V 'Unattended Custom Install'  --grub2-mbr ./BOOT/1-Boot-NoEmul.img -partition_offset 16   --mbr-force-bootable \
                -append_partition 2 28732ac11ff8d211ba4b00a0c93ec93b ./BOOT/2-Boot-NoEmul.img   -appended_part_as_gpt \
                -iso_mbr_part_type a2a0d0ebe5b9334487c068b6b72699c7   -c '/boot.catalog'   -b '/boot/grub/i386-pc/eltorito.img' \
                -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info   -eltorito-alt-boot   -e '--interval:appended_partition_2:::' \
                -no-emul-boot -o "iso-deb${system_version}.iso" $ISO_FOLDER
            else
                xorriso -as mkisofs -r -V "Unattended Custom Install" -J -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot\
                -boot-load-size 4 -isohybrid-mbr /usr/lib/ISOLINUX/isohdpfx.bin -boot-info-table -input-charset utf-8 -eltorito-alt-boot\
                -e boot/grub/efi.img -no-emul-boot -isohybrid-gpt-basdat -o "iso-deb${system_version}.iso" $ISO_FOLDER
            fi
        fi    
    }

Discouraging - I agree, but we'll find our way around it later.

Just to mid-sum it up :

• We went and discovered how Linux Distribution ISO is structured.
• Learned that we can disassemble it.
• Studied to modify the filesystem.
• Dwell on how to reassemble same ISO.
• Grasped important parts of ISO structs.
• Wrote automation with bash script functions

Thus, all is left for us, is assembly of the Tool...
Which we will do in next article
Until that time comes, remember: Do Try To Have Some Fun