by Udaybhaskar Sarma Seetamraju
ToSarma@gmail.com
Dec 31 2023

Highest-level Context

If you are into “Shift-Left” (whether re: Testing, Security, or Replicating-problems-on-developer-laptop, etc ..), then this article is for you.

For the very first time that you switch to an M1-chipset based MacBooks (from intel-chip based MacBooks) .. Productivity is significantly impacted when doing development/testing/troubleshooting “locally” on your laptop. Out-of-scope of this article is supporting those switching from Windoze.

Towards enabling up to 5x developer-productivity by allowing developers to robustly SIMULATE the Cloud-environment on a laptop — I have the following series of articles re: M1-chipset based MacBooks:

1. Running AWS CodeBuild locally on MacBook-M1.
   • Running Containers based on older Ubuntu 20.04 (released in the year 2020) as well as on the newer Ubuntu 22.04 (released in the year 2022)
   • Running Containers based on arm64-based Linux
2. Running AWS Glue locally on MacBook-M1. Various scenarios covered like: you do Not have “aws credentials” on your Laptop (forcing you to mock all the AWS API calls like S3 GET, Glue-Catalog queries, etc..)
3. (This) New Security-related Best-Practices when creating arm64/aarch64 Docker-Images on a MacBook-M1.

Summary

You’ve been issued a MacBook-M1 laptop (by your own choice or otherwise). You may have prior experience working on Intel-based MacBooks; but, that is of little help once you start “local” Container-based development on it. Basic commands like “npm” and “maven/gradle” as well as “curl/wget” stop working for local Dockerfile builds, with the weirdest in-decipherable errors that leave you and the rest of your development simply cannot figure out.

You may ask: Why even bother with arm64/aarch64 docker images, especially when we can set the following ENV-Variables and successfully emulate x86/amd64 chipsets on MacBook-M1?

export DOCKERPLATFORM="linux/amd64"
export DOCKER_DEFAULT_PLATFORM="${DOCKERPLATFORM}"
export TARGETPLATFORM="${DOCKER_DEFAULT_PLATFORM}"

My response: Sooner or later you’ll hit a situation like this -> One single Neo4j ver4.x container in amd64-emulation mode takes up 50% of cpus + 8GB/50% of RAM on your MacBook-M1.

Benefit #1: If you’d like to have your software work on linux/arm64, which invariably is cheapest compute on cloud.

Benefit #2: In addition, this document hopefully will bring some significant changes to the way you look at “free software” (as mostly un-trustworthy). Hope this will help you to be in harmony with MacOS and thereby avoid fighting with the Corporate Security-team (so that you can efficiently continue with your software development).

Advanced-User: Short summary

1. Use your Laptop’s browser (Chrome, ideally) and download the entire “Cert-Chain” of SSL-Public-Certs for maven.apache.org and for npm.org or .. whatever url/website is failing on your MacBook-M1.
2. Store all these downloaded SSL-certs into a new folder in your project.
3. See also: “should you store these in Git?” in sub-section below (preceding the Appendix).
4. For Java, use “keytool -importcert” commands to import these (within the Dockerfile).
For NodeJS/npm, use “npm config set cafile” command to import the “unified” CER file (within the Dockerfile).

Problem Statement: What you experience on your MacBook-M1

1. curl commands (within Dockerfile) fail during “docker build” commands on MacBook-M1.
2. “mvn” Maven commands (within Dockerfile) fail on MacBook-M1 — see sample error below.
3. “npm” commands (within Dockerfile) fail on MacBook-M1 — see sample error below.
4. “dockerfile-maven-plugin” uses a runtime based on x86 architecture and will NOT run on Apple M1 (aarch64).

Sample “mvn/maven” ERROR (for a spring-boot app):

Non-resolvable parent POM for groupId.subgroupId.myapp:myapp:0.0.1:
Could not transfer artifact org.springframework.boot:spring-boot-starter-parent:pom:3.0.2
from/to central (https://repo.maven.apache.org/maven2):
transfer failed for https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/3.0.2/spring-boot-starter-parent-3.0.2.pom
and 'parent.relativePath' points at wrong local POM @ line __,
column 13: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target
..
Root Cause ERROR: UnsafeLegacyRenegotiation option for SSL connections
 is now disabled by default in OpenSSL 3.0
..

Sample “npm” error:

qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped
Trace/breakpoint trap
ERROR: executor failed running [/bin/sh -c npm install --production]: exit code: 133

What is the underlying root-cause?

MacBook-M1-chip based laptops use the latest Apple MacOS.
Latest MacOS uses SSL v3.x.
Latest MacOS exclusively relies on SSL-v3.x (if you rely on Xcode’s tools and on home-brew).

FYI - SSL-v3 is officially renamed to “TLS v1.2”.
It is the latest and greatest and apparently, much much much more secure (than SSL-v2.x which is now considered legacy and a security-risk).
For those who are a tiny bit curious about why SSL-v3.0/TLS-v1.2 is a big-deal, take a quick look at Amazon’s decision to implement their own custom TINY-version of TLS - as published in “Continuous Formal Verification of Amazon s2n”

SSL v2.x has been around for so long, and .. “most” (in quotes) of its Ciphers it supports are considered legacy.
Translating that into developer language .. ..

• AlpineLinux.org website uses “legacy” SSL-Cert CIPHERS.
• maven.apache.org — same issue.
• npm.org — same issue.
• .. etcetera ..

Bottomline .. ..

Since Docker relies on the laptop’s O/S for all Cryptography-related activities, Docker is missing support for the legacy SSL-v2.x on the MacBook-M1.

Really BAD-idea(s) & Hit-or-Miss workaround(s)

(A) Choose “mirrors” that use latest SSL-Cert ciphers.

Example: switch to Alpine-MIRROR: https://mirror.math.princeton.edu/pub/alpinelinux/v3.17/

Unfortunately, it’s very hard to find “good” mirrors (that use SSL-v3.x/TLS-v1.2).

(B) Get rid of encryption-in-motion! ugh!

Example: replace “httpS” with plain “http” within “Dockerfile”.

https curl http://archive.apache.org/dist/tomcat/tomcat-"$TOMCAT_MAJOR"/v"$TOMCAT_VERSION"/bin/apache-tomcat-"$TOMCAT_VERSION".tar.gz

Robust and Permanent Fixes

Step 1: Download the SSL-Certs - the complete “Hierarchy”

Since I have many easy-to-follow screenshots re: this, and since I want to keep this article “short” .. ..

.. Please jump to the only Appendix at the end of this article.

Step 2: For Java Maven

Inside Dockerfile, run the following commands.

Important for Java-only: 1st import the TOP-MOST CA-Cert;
Then, finally, import (bottommost in chain) web-site SSL-Cert.

The “if” condition (below) is just a suggestion - so that the keytool is ONLY run for MacBook-M1 chipset.
Keep it, if you’d like to have your software work on linux/arm64, which invariably is cheaper compute on AWS.

RUN if [ "${TARGETPLATFORM}" == "linux/aarch64" ] || [ "${TARGETPLATFORM}" == "linux/arm64" ]; then \
      keytool -importcert -noprompt -trustcacerts \
        -cacerts -storepass changeit   -keystore "${JAVA_HOME}/lib/security/cacerts"   \
        -alias ROOT-1A               -file "${APP_SRC_FOLDER}/etc/ROOT-1A.cer"; \
      keytool -importcert -noprompt -trustcacerts \
        -cacerts -storepass changeit   -keystore "${JAVA_HOME}/lib/security/cacerts"   \
        -alias SSLCA-1A.cer          -file "${APP_SRC_FOLDER}/etc/SSLCA-1A.cer";   \
      keytool -importcert -noprompt -trustcacerts \
                 -storepass changeit   -keystore "${JAVA_HOME}/lib/security/cacerts"   \
        -alias org.apache.maven.repo -file "${APP_SRC_FOLDER}/etc/org.apache.maven.repo.cer"; \
    fi

Java’s KeyTool command: https://docs.oracle.com/en/java/javase/17/docs/specs/man/keytool.htm

Step 2: For npm

ISSUE: We have multiple Cert-files. But, “npm” can only accept ONE SINGLE file.

SOLUTION:

1. Combine multiple Cert-files into a single file.
2. Here’s bash-commands for that.
3. Note: The files should be listed in the ORDER in which they were downloaded - as shown in this article.

FILES_IN_PROPER_SEQ=(nodejs.org.cer
Cloudflare-ECC-CA-3.cer
LEVEL2-ROOTCA-1A.cer TOPMOST-ROOT-1A.cer
)

TMPFILE="./etc/my-unified-certs.cer"
echo '' > "${TMPFILE11}"                  ### Empty out the file (if it exists)

for f in ${FILES_IN_PROPER_SEQ[@]}; do
    CERTFILE="${CERTS_FOLDER}/${f}"
    if [ ! -f "${CERTFILE}" ]; then
        echo "ERROR: File '${CERTFILE}' does not exist !"
        exit 9
    fi
    cat "${CERTFILE}" >> "${TMPFILE11}"
    echo '' >> "${TMPFILE11}"
done

NEXT: Inside Dockerfile, run the following commands.

COPY      /usr/local/share/ca-certificates/.

RUN  if [ "${TARGETPLATFORM}" == "linux/aarch64" ] || [ "${TARGETPLATFORM}" == "linux/arm64" ]; then \
        npm config set cafile   ${APP_SRC_FOLDER}/etc/my-unified-certs.cer  \
     fi

Step 3 (Optional) - dockerfile-maven-plugin won't run on Apple M1-chip

Try official “substitute” plugins like Eclipse JKube plugin: Kubernetes Maven Plugin or OpenShift Maven Plugin . Refer to  Migration Guide for more details.

Open Questions, Concerns and Challenges

Should you store these SSL-Public-Certs in Git?

Risks as viewed by Enterprise Security Team(s) and my responses.

1. Risk: As a broad/generic statement, Certs and Secrets should Not be put into Git.
   • Counter-Argument #1: These are PUBLIC-Key SSL-Certs. Meant for the whole world to know (or have a copy)!
   • Counter-Argument #2: By checking them into Git, perhaps .. .. Security will know what Apps are impacted if the corresponding website is ever hijacked, or the CA-Root is compromised; Security can then alert the developers - to find a workaround (or, at least pause all Builds temporarily).
2. Risk: Not a good idea to download + store the CA-Root and the Intermediate-Root Certs.
   • Counter-Argument - Benefit: So that we do Not have to “blindly” trust just one “standalone” certificate. This significantly reduces chances that Laptop will Not be downloading malware.
3. Challenge: This is a Laptop-issue and Not a CI/CD or DevOps issue. Nor related to application-design.
So, why is this part of the codebase?
   • Wham!
   • KO’ed.
   • Hard to have a counter-argument.
   • But here’s a pathetic “excuse”!!
     • There are “developers” and then there are “developers”.
     • You know EXACTLY what I mean.
     • Putting this in the Git-repo makes life convenient for the leads and for the senior members of any development team.

APPENDIX

Step 1: Download the SSL-Certs - the complete “Hierarchy”

Note: The screenshots are for “maven.apache.org".
Note: Please repeat the following for the website/url that fails to download on your MacBook-M1.

Now repeat steps 4.a and 4.b (steps shown above) ..
.. for the two OTHER entries in the “Certificate Hierarchy” (as indicated below).

End of Article.