Automating Code Signing with multiple E-Business Suite instances - Part 3

As you are probably aware by now, since June 1st 2023 all well-known / public Certificate Authorities (CA) no longer provide Code Signing Certificates using pure software based private keys (see https://www.linkedin.com/posts/johannes-michler-099892ab_code-signing-key-storage-requirements-will-activity-7090432157688492032-jGvC).

Since I prefer using such a trusted / public CA to sign Java Applets (that are still crucial for Oracle E-Business Suite or Oracle Forms) I've recently had a look into how we can now sign those Java JAR files. Part 1 of this blog series introduced the topic and an available "Cloud Based" Code Signing Certificate provider: Signing EBS/Forms - Part 1

In a second part I covered how the code signing can be done on an E-Business Suite Application Server running on Oracle Linux 7 on Oracle Cloud Infrastructure (OCI).

This third post will look how we can further automate this by installing the Certum tools onto the E-Business Suite Cloud Manager VM. First, we'll cover the latest changes from Certum, then we'll look into some scripts that can be used on multiple E-Business Suite Application servers to send the .jar files for signing to that central signing instance.

Back in the previous post, I've complained about the incomplete translation of the Certum tools still revealing a lot of polish error messages. While it seems this is fixed at least partially, I realized that the 2.9.9 versions available over there https://files.certum.eu/software/SimplySignDesktop/Linux-RedHat/ leads to fatal crashes (segmentation fault). That is why for now I stuck with the 2.9.8 release.

When installing the SimplySignDesktop tool according to the official documentation it is necessary to do so globally/as the root user. Since I didn't like the tool to modify my cloud manager VM in that massive way, I've investigated what the installer actually does. With that I was able to get the tool running with a way less privileged user (that I call certum). Run the following as root:

yum install https://rpmfind.net/linux/epel/8/Everything/x86_64/Packages/s/stalonetray-0.8.3-15.el8.x86_64.rpm
yum install libxslt.x86_64 pulseaudio-libs-glib2.x86_64 libwebp.x86_64 xkeyboard-config
useradd certum
sudo su – certum
mkdir .ssh
vi .ssh/authorized_keys
# add the SSH public key(s) of your oracle@ebs-appserver
chmod 700 .ssh
chmod 600 .ssh/authorized_keys

Then connect a SSH Session with X-Forwarding as certum:

wget https://files.certum.eu/software/SimplySignDesktop/Linux-RedHat/2.9.8-9.1.6.0/SimplySignDesktop-2.9.8-9.1.6.0-x86_64-prod-centos.bin
sh SimplySignDesktop-2.9.8-9.1.6.0-x86_64-prod-centos.bin --target /home/certum/
cp /home/certum/SSD-2.9.8-dist/SimplySignDesktop.xml /home/certum/

Create a /home/certum/provider_simplysign.cfg file as follows:

name=SimplySignDesktop/SimplySignPKCS
library=/home/certum/SSD-2.9.8-dist/SimplySignPKCS_64-MS-1.0.20.so
slot=-1

Furthermore, create a script startGUI.sh as follows:

export LD_LIBRARY_PATH=/home/certum/SSD-2.9.8-dist/
export QT_QPA_PLATFORM_PLUGIN_PATH=/home/certum/SSD-2.9.8-dist/plugins
export OPENSSL_CONF=/etc/ssl/
stalonetray &
/home/certum/SSD-2.9.8-dist/SimplySignDesktop

Finally start the Script and sign in with a one-time-token.

Do a test as follows (in new SSH Session):

/home/certum/SS-9.1.6.0-dist/jre/bin/keytool -list -keystore NONE -storetype PKCS11 -providerclass sun.security.pkcs11.SunPKCS11 -providerArg /home/certum/provider_simplysign.cfg -v

This will provide an alias, in our case: 4F4F410D1234A9110B16DA9C83BD6F59

Furthermore, create a /home/certum/mychain.pem file as described in the previous episode.

On the E-Business Apps-Server first create a ~/sign_1.sh script as follows:

folderstamp=$(date +%Y-%m-%d-%H:%M)
mkdir -p /home/oracle/sign_bkp/${folderstamp}
jar=$1
# Remove Signature from jar files created through ADADMIN in EBS
echo " ** Removing EBS signature from: ${jar} "
cp -i ${jar} /home/oracle/sign_bkp/${folderstamp}/
zip -d ${jar} 'META-INF/*.SF' 'META-INF/*.RSA'
scp ${jar} certum@10.1.2.199:/tmp/signing-dummy.jar

ssh certum@10.1.2.199 "/home/certum/SS-9.1.6.0-dist/jre/bin/jarsigner -keystore NONE -tsa \"http://time.certum.pl\" -certchain /home/certum/mychain.pem -storetype PKCS11 -providerClass sun.security.pkcs11.SunPKCS11 -providerArg /home/certum/provider_simplysign.cfg -storepass 12345 /tmp/signing-dummy.jar 4F4F410D1234A9110B16DA9C83BD6F59"

scp certum@10.1.2.199:/tmp/signing-dummy.jar ${jar}

The script first creates a backup of the jar, then un-signs the .jar files and copies it to the cloud-manager VM (in my case with IP 10.1.2.199). There the jar is signed and finally the signed .jar is copied back to the E-Business Suite Apps Tier.

This allows signing a single .jar file; the script may be helpful when applying a patch with "options=nojarsigning". Then in there should be a file such as /u01/install/APPS/fs_ne/EBSapps/log/adop/176/20240327_132920/apply/mastebsapp01/36177213/log/jarlist.txt containing all the .jar files that require re-signing.

For the initial signing the procedure in the previous episode can be combined with the copying of the .jar to the Cloud Manager VM.

As an alternative to signing "just" the files in $NE_BASE/EBSapps/log/adadmin/log/jarlist.txt I found it useful to just sign all .jar files under $JAVA_TOP. For this the following script proved helpful:

folderstamp=$(date +%Y-%m-%d-%H:%M)
mkdir -p /home/oracle/sign_bkp/${folderstamp}
# Select the jar files from jarlist.txt
for jar in $(find $JAVA_TOP/oracle/apps -name \*.jar)
do
# Remove Signature from jar files created through ADADMIN in EBS
echo " ** Removing EBS signature from: ${jar} "
cp -i ${jar} /home/oracle/sign_bkp/${folderstamp}/
zip -d ${jar} 'META-INF/*.SF' 'META-INF/*.RSA'
scp ${jar} certum@10.1.2.199:/tmp/signing-dummy.jar
ssh certum@10.1.2.199 "/home/certum/SS-9.1.6.0-dist/jre/bin/jarsigner -keystore NONE -tsa \"http://time.certum.pl\" -certchain /home/certum/mychain.pem -storetype PKCS11 -providerClass sun.security.pkcs11.SunPKCS11 -providerArg /home/certum/provider_simplysign.cfg -storepass 12345 /tmp/signing-dummy.jar 4F4F410D1234A9110B16DA9C83BD6F59"
scp certum@10.1.2.199:/tmp/signing-dummy.jar ${jar}
done

It is helpful to first to a check if the $jar is already signed as follows:

result=`jarsigner -verify -certs ${jar}| tr -d '[:space:]'`
if [[ "jarverified." != "$result" ]]
then
echo ${jar} needs re-sign; $result
# put the signing here
fi

Using above scripts, it is amazingly easy to sign all .jar files both initially as well as after applying a patch. The version using a "find" on $JAVA_TOP may sign "a bit more than needed", but in my experience that does not do any harm.

I am still hoping that Oracle will provide a way to "Hook" a script such as sign_1.sh into the signing process called during patching or through adadmin. This would probably be announced in "Signing EBS Jar Files With HSM (Hardware Security Module) - (Doc ID 2806640.1)".