Customizable CloudBoot Images

OnApp CloudBoot compute resources allow you to add custom drivers. This feature is needed when hardware is not supported by the kernel version used by OnApp. If required, you can also change existing modules. The procedures for adding new modules and changing existing ones are similar with the exception of the fourth stage.
The process consists of the following stages:

Environment installation
Compile/Build custom driver (the recommended way)
Compile/Build custom driver (using tools we provide)
Proceed with the image customization

I. Environment installation

In order to get the environment the same as OnApp CloudBoot compute resources run, please install corresponding template into your OnApp cloud (templates are available via Template Server):

The templates are for OnApp versions >= 5.0. There are no templates available for other CentOS versions and virtualization variants (like CentOS5 XEN, and CentOS7 KVM). It is strongly recommended that you create a VS and run them as KVM guests.

II. Compile/Build custom driver (the recommended way)

It is recommended to get and install source RPM (.src.rpm) of the certain driver version provided by driver maintainer. It will ensure that you rebuild driver against CloudBoot's kernel with correct settings and options.

The steps below might not match for 100% your driver building procedure, but they are a part of general understanding how to rebuild kernel modules from source RPM.
Let's go step by step through megaraid sas driver version 06.904.09.00 for Dell PERC H310/H710/H710P/H810/SPERC8/SPERC8 External/H330/H730/H730P/H830 SAS RAID Controllers building for CentOS6 KVM ramdisk, which runs CentOS 6.7 and kernel version 2.6.32-642.6.2.el6.x86_64.

Visit vendors home page and try to download file(s), RPM sources package(s) or any other provided sources of your hardware driver. In our case the web page is http://www.dell.com/support/home/us/en/4/Drivers/DriversDetails?driverId=RFX3H&fileId=3646341357&osCode=RH60&productCode=poweredge-r730&languageCode=en&categoryId=SF and the file with sources is - https://downloads.dell.com/FOLDER04095093M/1/UnifiedDriver_6.904.09_RHEL6.tar.gz
```
# wget https://downloads.dell.com/FOLDER04095093M/1/UnifiedDriver_6.904.09_RHEL6.tar.gz
```
CODE
The file needs to be downloaded into the installed guest with the corresponding build environment.

Try to locate src.rpm within the downloaded file (if the file is not the RPM package itself):

# tar ztf UnifiedDriver_6.904.09_RHEL6.tar.gz | grep src.rpm
Kmod-megaraid_sas_rhel6.5 x86_64/megaraid_sas-06.904.09.00_rhel6.5-2.src.rpm
Kmod-megaraid_sas_rhel6.6 x86_64/megaraid_sas-06.904.09.00_rhel6.6-2.src.rpm
Kmod-megaraid_sas_rhel6.7 x86_64/megaraid_sas-06.904.09.00_rhel6.7-2.src.rpm
Kmod-megaraid_sas_rhel6.8 x86_64/megaraid_sas-06.904.09.00_rhel6.8-2.src.rpm

CODE

Extract the required .src.rpm from the archive:

# tar zxf UnifiedDriver_6.904.09_RHEL6.tar.gz Kmod-megaraid_sas_rhel6.8\ x86_64/megaraid_sas-06.904.09.00_rhel6.8-2.src.rpm

CODE

Install the .src.rpm:

# rpm -ivh  Kmod-megaraid_sas_rhel6.8\ x86_64/megaraid_sas-06.904.09.00_rhel6.8-2.src.rpm
   1:megaraid_sas           ########################################### [100%]

CODE

At this point you are ready to rebuild the source RPM of the driver against the required kernel version (CentOS6 KVM example, kernel version 2.6.32-642.6.2.el6.x86_64):
```
# rpmbuild --define='kernel_version 2.6.32-642.6.2.el6.x86_64' -bi /root/rpmbuild/SPECS/megaraid_sas.spec
```
CODE
The build could fail with the following error:
```
+ make -C /usr/src/kernels/2.6.32-573.el6.x86_64 M=/root/rpmbuild/BUILD/megaraid_sas-06.904.09.00_rhel6.7/obj/default
make: *** /usr/src/kernels/2.6.32-573.el6.x86_64: No such file or directory.  Stop.
error: Bad exit status from /var/tmp/rpm-tmp.8EzUVv (%build)
```
CODE
This particular error means that the kernel version is hardcoded in the .spec. It is required to modify the megaraid_sas.spec file to fix the error.
Edit the file and remove the %global kernel_version 2.6.32-642.el6.x86_64 line from it:
```
# vim /root/rpmbuild/SPECS/megaraid_sas.spec
```
CODE
Please be aware that the errors and RPM's macro can differ, so you should personally take care of fixing them and passing correct values to rpmbuild utility. In our case kernel_version was the issue reason.

Locate the kernel module file .ko. As soon as the rpmbuild succeeds, run:

# find /root/rpmbuild/BUILDROOT -name \*.ko
/root/rpmbuild/BUILDROOT/megaraid_sas-06.904.09.00_rhel6.8-2.x86_64/lib/modules/2.6.32-642.6.2.el6.x86_64.debug/extra/megaraid_sas/megaraid_sas.ko
/root/rpmbuild/BUILDROOT/megaraid_sas-06.904.09.00_rhel6.8-2.x86_64/lib/modules/2.6.32-642.6.2.el6.x86_64/extra/megaraid_sas/megaraid_sas.ko

CODE

The lib/modules/2.6.32-642.6.2.el6.x86_64/extra/megaraid_sas/megaraid_sas.ko is the kernel module, with which you can customize your CentOS7 KVM CloudBoot ramdisk/image.

III. Compile/Build custom driver (using tools we provide)

If your are not able to locate .src.rpm provided by the driver's vendor, please use our tools to build RPM package(s) with the driver for your hardware from sources.
Let's go step by step through i40e driver version 1.6.42 for PCIe* Intel® 40 Gigabit Ethernet Network Connections building for CentOS6 KVM ramdisk, which runs CentOS 6.7 and kernel version 2.6.32-642.6.2.el6.x86_64.

Visit vendors home page and try to download sources of your hardware driver. In our case the web page is https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-Intel-40-Gigabit-Ethernet-Network-Connections-Under-Linux- and the sources file - i40e-1.6.42.tar.gz
Check if the file matches our naming requirements, is packed with tar+gzip and allocates "main" Makefile:
- the driver sources file should be named like DRIVER_NAME-DRIVER_VERSION.tar.gz. Otherwise, you should unpack the sources and move them into newly created DRIVER_NAME-DRIVER_VERSION directory. Then please repack the whole directory into the DRIVER_NAME-DRIVER_VERSION.tar.gz ;
- allocate the "main" Makefile.
```
# tar ztf i40e-1.6.42.tar.gz | grep Makefile
i40e-1.6.42/src/Makefile
```
  CODE
  Our file is located in the src directory.
Start building the RPM package with our tools:

Please check for new tools versions by running # yum update onapp-ramdisk-kmod-build.
```
# cd /onapp/onapp-ramdisk-kmod-build
# ./onapp-ramdisk-kmod-build.sh -n i40e -v 1.6.42 -C src
```
CODE
-C src is the option to specify the "main" Makefile location.

The following package is created if the build succeeded:

/root/rpmbuild/RPMS/x86_64/onapp-ramdisk-centos6-kvm-kmod-i40e-1.6.42-1.x86_64.rpm

CODE

It contains the following files to customize CentOS6 KVM ramdisk/image:

# rpm -qpl /root/rpmbuild/RPMS/x86_64/onapp-ramdisk-centos6-kvm-kmod-i40e-1.6.42-1.x86_64.rpm
/tftpboot/export/centos6/kvm/lib/modules
/tftpboot/export/centos6/kvm/lib/modules/2.6.32-642.6.2.el6.x86_64
/tftpboot/export/centos6/kvm/lib/modules/2.6.32-642.6.2.el6.x86_64/extra
/tftpboot/export/centos6/kvm/lib/modules/2.6.32-642.6.2.el6.x86_64/extra/earlyboot-i40e.sh
/tftpboot/export/centos6/kvm/lib/modules/2.6.32-642.6.2.el6.x86_64/extra/i40e
/tftpboot/export/centos6/kvm/lib/modules/2.6.32-642.6.2.el6.x86_64/extra/i40e/i40e.ko

CODE

Run the tool with -h for help and more options:

# ./onapp-ramdisk-kmod-build.sh -h
Usage: ./onapp-ramdisk-kmod-build.sh -n NAME -v VERSION [-b BUILD] [-r KVERSION] [-o RPMBUILD_OPTS] [-C MAKEFILE_PATH] [-d DISTRO] [-t VIRT] [-h|--help]

                           -d DISTRO	: Distributive and its version.
                                          The default is: centos6
                                          Supported are: centos5 centos6 centos7
                           -t VIRT	: Virtualization name.
                                          The default is: kvm
                                          Supported are: xen, kvm

                           -n NAME	: Driver short name
                           -v VERSION	: Driver version
                           -b BUILD	: RPM package build number. The default is: 1

                           -r KVERSION	: Kernel version to compile driver's kmog against. The default is: 2.6.32-642.6.2.el6.x86_64
                           -o RPMBUILD_OPTS : Additional options for the 'rpmbuild' utility. The default are: --define='__find_requires %{nil}' --define='release 1' --define='kversion 2.6.32-642.6.2.el6.x86_64' --define='distro centos6' --define='virt kvm'
                           -C MAKEFILE_PATH : Path to the driver's sources main Makefile - the directory where 'make' utility is called.
					      By default isn't set. Usefull values are: src, ...
                                              The path could be determined by running (remove leading KMOD_NAME-KMOD_VERSION dir): tar ztf KMOD_NAME-KMOD_VERSION.tar.gz | grep Makefile 

                           -h|--help : print this info

CODE

Put the onapp-ramdisk-centos6-kvm-kmod-i40e-1.6.42-1.x86_64.rpm into your Control Panel box and install it there:
```
# rpm -Uvh onapp-ramdisk-centos6-kvm-kmod-i40e-1.6.42-1.x86_64.rpm
```
CODE

IV. Proceed with the image customization

To add a custom CloudBoot driver:

Check if there is a file called ''earlyboot.sh'' in the overlay folder on the CP for each compute resource. For example, for a compute resource with the MAC address 00:11:22:33:44:55, look whether the /tftpboot/images/centos5/diskless/snapshot/00-11-22-33-44-55/overlay/earlyboot.sh folder exists. This script will be run before the onapp SAN Controller service is run and as such can be used for adding custom drivers.
Whilst the path mentions centos5, in practice this same script is used for all compute resource types.
You need to add an executable bit for the earlyboot.sh script for it to become executable. This is done with the help of the chmod command:
chmod +x earlyboot.sh

CODE
You can then check whether the executable bit has been added to the earlyboot.sh file ('x' indicates the executable file):
ls -la earlyboot.sh -rwxr-xr-x 1 root root 90 Feb 25 19:06 earlyboot.sh

CODE
Include drivers in the same overlay folder.
Reference the files in the earlyboot.sh script as if the overlay folder is the root folder.

Insert drivers for each compute resource in the overlay folder as described.

An example overlay file for the megaraid driver:

When adding a new module:

[root@cp-onapp overlay]# cat earlyboot.sh
#!/bin/sh
PATH=/sbin:/usr/sbin:/bin:/usr/bin
export PATH
insmod /megaraid/megaraid_sas.ko

CODE

When changing an existing module:

[root@cp-onapp overlay]# cat earlyboot.sh
#!/bin/sh
PATH=/sbin:/usr/sbin:/bin:/usr/bin
export PATH
rmmod megaraid_sas 
insmod /megaraid/megaraid_sas.ko

CODE

The megaraid driver folder is actually in the overlay folder, not the root:

ls -lR
.:
total 8
-rwxr-xr-x 1 root root   37 Apr 15 16:47 earlyboot.sh
drwxr-xr-x 2 root root 4096 Jul  9 12:21 megaraid


./megaraid:
total 2000
-rw-r--r-- 1 root root 2048000 Jul  9 12:22 megaraid_sas.ko

CODE

The driver will then be inserted and can be used immediately after you reboot the compute resource. Otherwise, to use the driver you need to load it manually and restart the services.