Speed up Jenkins with npm build

#jenkins #speedup #ci

Our team have many projects delivered by Jenkins. Every component need to be assembled by webpack which write in React.js or Vue.js.

Each build will spend 5 or 10 minutes to finish, and most time spend on npm install. But we all know that if package.json were not change, node_modules directory should not change as well.

So that we checksum package.json with MD5, and package whole node_modules directory as cache. If the hash string does not changed, which implied nothing changed, just unpack cached node_modules directory and then execute npm install, all things done.

Here is a simple example

PKG_SUM=$(md5sum package.json | cut -d\  -f 1)
NPM_TARBALL=node_modules-${PKG_SUM}.tgz
NPM_TARBALL_MD5SUM=${NPM_TARBALL}.md5sum
NPM_TARBALL_CACHE=${HOME}/.cache/npmtarball
S3_NPM_TARBALL=s3://deployment/npmtarball

[[ ! -e $NPM_TARBALL_CACHE ]] && mkdir -p $NPM_TARBALL_CACHE

function downloadNpmTarball(){
    pushd $NPM_TARBALL_CACHE
        if [ ! -f ${NPM_TARBALL} ];then
            s3cmd get ${S3_NPM_TARBALL}/${NPM_TARBALL} ${NPM_TARBALL}
            s3cmd get ${S3_NPM_TARBALL}/${NPM_TARBALL_MD5SUM} ${NPM_TARBALL_MD5SUM}
            md5sum -c ${NPM_TARBALL_MD5SUM} || rm -f ${NPM_TARBALL} ${NPM_TARBALL_MD5SUM}
        fi
    popd
}

function checkNpmMod() {
    downloadNpmTarball
    TARBALL=${NPM_TARBALL_CACHE}/${NPM_TARBALL}
    [[ -f $TARBALL ]] && tar xzf $TARBALL
}

function uploadNpmMod() {
    TARBALL=${NPM_TARBALL_CACHE}/${NPM_TARBALL}
    if [ ! -f  ${TARBALL} ];then
        if [ -d node_modules ];then
             tar zcf ${TARBALL} node_modules || return 1
        fi
        pushd $NPM_TARBALL_CACHE
             md5sum $NPM_TARBALL > ${NPM_TARBALL_MD5SUM}
             s3cmd put $NPM_TARBALL $S3_NPM_TARBALL/$NPM_TARBALL
             s3cmd put $NPM_TARBALL_MD5SUM $S3_NPM_TARBALL/$NPM_TARBALL_MD5SUM
        popd
    fi
}
checkNpmMod
npm install
uploadNpmMod

Top comments (6)

Vitalii Dasaev • Jul 16 '20

I created such script to check md5sum of package.json in Jenkins:

stage('NPM Build') {
  steps {
    sh '''
    node -v && npm -v
    '''
    // rm -rf node_modules
    sh '''
    CACHE_FOLDER=${HOME}/.cache/md5
    echo "EXECUTOR_NUMBER: ${EXECUTOR_NUMBER}"
    MD5_FILE_NAME=package-json_${EXECUTOR_NUMBER}.md5sum

    [ -d ${CACHE_FOLDER} ] || mkdir -p ${CACHE_FOLDER}
    ls ${CACHE_FOLDER}

    if [ -f ${CACHE_FOLDER}/${MD5_FILE_NAME} ];then
      cp ${CACHE_FOLDER}/${MD5_FILE_NAME} ${MD5_FILE_NAME}
      md5sum package.json
      cat ${MD5_FILE_NAME}
      md5sum -c ${MD5_FILE_NAME} || npm ci
    else
      echo "No md5sum backup"
      npm ci
    fi

    echo "create new md5sum backup"
    md5sum package.json
    md5sum package.json > ${MD5_FILE_NAME}
    cp ${MD5_FILE_NAME} ${CACHE_FOLDER}
    '''
    sh '''
    npm run ngcc
    '''
    sh '''
    npm run build
    '''
  }
}

T. Todua • Mar 4 '22

For the sake of whatever, please create a Jenkins plugin which will facilitate life of hundreds of others. it's untold overkill everytime doing npm install such unnecessarily

Gois • Sep 3 '18

How much fast is to cache on S3? Download, unzip, zip and upload time... My team have a similar problem due to npm install time

Guixing Bai • Dec 17 '18

If you are using aws ec2, it will be very fast.

Gois • Dec 17 '18

I wrote a script to get MD5 only from dependencies and devDependencies, cause we do a patch bump on every merge to develop branch (we work with master and develop in a fork flow). Here is the gist.

Guixing Bai • Dec 18 '18

Great!

DEV Community

Speed up Jenkins with npm build

Top comments (6)

Read next

Python

What is Computer Vision? (2)

Easy Loan App | True Balance

Mastering the Prototype Design Pattern: A Comprehensive Guide