Customizable CloudBoot Images
OnApp CloudBoot compute resources allow you to add custom drivers. This feature is needed when hardware is not supported by the kernel version used by OnApp. If required, you can also change existing modules. The procedures for adding new modules and changing existing ones are similar with the exception of the fourth stage.
The process consists of the following stages:
- Environment installation
- Compile/Build custom driver (the recommended way)
- Proceed with the image customization
I. Environment installation
In order to get the environment the same as OnApp CloudBoot compute resources run, please install corresponding template into your OnApp cloud (templates are available via Template Server):
The templates are for OnApp versions >= 5.0. There are no templates available for other CentOS versions and virtualization variants (like CentOS5 XEN, and CentOS7 KVM). It is strongly recommended that you create a VS and run them as KVM guests.
II. Compile/Build custom driver (the recommended way)
It is recommended to get and install source RPM (.src.rpm) of the certain driver version provided by driver maintainer. It will ensure that you rebuild driver against CloudBoot's kernel with correct settings and options.
The steps below might not match for 100% your driver building procedure, but they are a part of general understanding how to rebuild kernel modules from source RPM.
Let's go step by step through megaraid sas driver version 06.904.09.00 for Dell PERC H310/H710/H710P/H810/SPERC8/SPERC8 External/H330/H730/H730P/H830 SAS RAID Controllers building for CentOS6 KVM ramdisk, which runs CentOS 6.7 and kernel version 2.6.32-642.6.2.el6.x86_64.
Visit vendors home page and try to download file(s), RPM sources package(s) or any other provided sources of your hardware driver. In our case the web page is http://www.dell.com/support/home/us/en/4/Drivers/DriversDetails?driverId=RFX3H&fileId=3646341357&osCode=RH60&productCode=poweredge-r730&languageCode=en&categoryId=SF and the file with sources is - https://downloads.dell.com/FOLDER04095093M/1/UnifiedDriver_6.904.09_RHEL6.tar.gz
# wget https://downloads.dell.com/FOLDER04095093M/1/UnifiedDriver_6.904.09_RHEL6.tar.gz
CODEThe file needs to be downloaded into the installed guest with the corresponding build environment.
Try to locate src.rpm within the downloaded file (if the file is not the RPM package itself):
# tar ztf UnifiedDriver_6.904.09_RHEL6.tar.gz | grep src.rpm Kmod-megaraid_sas_rhel6.5 x86_64/megaraid_sas-06.904.09.00_rhel6.5-2.src.rpm Kmod-megaraid_sas_rhel6.6 x86_64/megaraid_sas-06.904.09.00_rhel6.6-2.src.rpm Kmod-megaraid_sas_rhel6.7 x86_64/megaraid_sas-06.904.09.00_rhel6.7-2.src.rpm Kmod-megaraid_sas_rhel6.8 x86_64/megaraid_sas-06.904.09.00_rhel6.8-2.src.rpm
CODEExtract the required .src.rpm from the archive:
# tar zxf UnifiedDriver_6.904.09_RHEL6.tar.gz Kmod-megaraid_sas_rhel6.8\ x86_64/megaraid_sas-06.904.09.00_rhel6.8-2.src.rpm
CODEInstall the .src.rpm:
# rpm -ivh Kmod-megaraid_sas_rhel6.8\ x86_64/megaraid_sas-06.904.09.00_rhel6.8-2.src.rpm 1:megaraid_sas ########################################### [100%]
CODEAt this point you are ready to rebuild the source RPM of the driver against the required kernel version (CentOS6 KVM example, kernel version 2.6.32-642.6.2.el6.x86_64):
# rpmbuild --define='kernel_version 2.6.32-642.6.2.el6.x86_64' -bi /root/rpmbuild/SPECS/megaraid_sas.spec
CODEThe build could fail with the following error:
+ make -C /usr/src/kernels/2.6.32-573.el6.x86_64 M=/root/rpmbuild/BUILD/megaraid_sas-06.904.09.00_rhel6.7/obj/default make: *** /usr/src/kernels/2.6.32-573.el6.x86_64: No such file or directory. Stop. error: Bad exit status from /var/tmp/rpm-tmp.8EzUVv (%build)
CODE
Edit the file and remove the %global kernel_version 2.6.32-642.el6.x86_64 line from it:# vim /root/rpmbuild/SPECS/megaraid_sas.spec
CODEPlease be aware that the errors and RPM's macro can differ, so you should personally take care of fixing them and passing correct values to rpmbuild utility. In our case kernel_version was the issue reason.
- Locate the kernel module file .ko. As soon as the rpmbuild succeeds, run:
# find /root/rpmbuild/BUILDROOT -name \*.ko /root/rpmbuild/BUILDROOT/megaraid_sas-06.904.09.00_rhel6.8-2.x86_64/lib/modules/2.6.32-642.6.2.el6.x86_64.debug/extra/megaraid_sas/megaraid_sas.ko /root/rpmbuild/BUILDROOT/megaraid_sas-06.904.09.00_rhel6.8-2.x86_64/lib/modules/2.6.32-642.6.2.el6.x86_64/extra/megaraid_sas/megaraid_sas.ko
CODEThe lib/modules/2.6.32-642.6.2.el6.x86_64/extra/megaraid_sas/megaraid_sas.ko is the kernel module, with which you can customize your CentOS7 KVM CloudBoot ramdisk/image.
III. Compile/Build custom driver (using tools we provide)
If your are not able to locate .src.rpm provided by the driver's vendor, please use our tools to build RPM package(s) with the driver for your hardware from sources.
Let's go step by step through i40e driver version 1.6.42 for PCIe* IntelĀ® 40 Gigabit Ethernet Network Connections building for CentOS6 KVM ramdisk, which runs CentOS 6.7 and kernel version 2.6.32-642.6.2.el6.x86_64.
- Visit vendors home page and try to download sources of your hardware driver. In our case the web page is https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-Intel-40-Gigabit-Ethernet-Network-Connections-Under-Linux- and the sources file - i40e-1.6.42.tar.gz
Check if the file matches our naming requirements, is packed with tar+gzip and allocates "main" Makefile:
- the driver sources file should be named like DRIVER_NAME-DRIVER_VERSION.tar.gz. Otherwise, you should unpack the sources and move them into newly created DRIVER_NAME-DRIVER_VERSION directory. Then please repack the whole directory into the DRIVER_NAME-DRIVER_VERSION.tar.gz ;
allocate the "main" Makefile.
# tar ztf i40e-1.6.42.tar.gz | grep Makefile i40e-1.6.42/src/Makefile
CODEOur file is located in the src directory.
Start building the RPM package with our tools:
Please check for new tools versions by running # yum update onapp-ramdisk-kmod-build.
# cd /onapp/onapp-ramdisk-kmod-build # ./onapp-ramdisk-kmod-build.sh -n i40e -v 1.6.42 -C src
CODE-C src is the option to specify the "main" Makefile location.
The following package is created if the build succeeded:
/root/rpmbuild/RPMS/x86_64/onapp-ramdisk-centos6-kvm-kmod-i40e-1.6.42-1.x86_64.rpm
CODEIt contains the following files to customize CentOS6 KVM ramdisk/image:
# rpm -qpl /root/rpmbuild/RPMS/x86_64/onapp-ramdisk-centos6-kvm-kmod-i40e-1.6.42-1.x86_64.rpm /tftpboot/export/centos6/kvm/lib/modules /tftpboot/export/centos6/kvm/lib/modules/2.6.32-642.6.2.el6.x86_64 /tftpboot/export/centos6/kvm/lib/modules/2.6.32-642.6.2.el6.x86_64/extra /tftpboot/export/centos6/kvm/lib/modules/2.6.32-642.6.2.el6.x86_64/extra/earlyboot-i40e.sh /tftpboot/export/centos6/kvm/lib/modules/2.6.32-642.6.2.el6.x86_64/extra/i40e /tftpboot/export/centos6/kvm/lib/modules/2.6.32-642.6.2.el6.x86_64/extra/i40e/i40e.ko
CODERun the tool with -h for help and more options:
# ./onapp-ramdisk-kmod-build.sh -h Usage: ./onapp-ramdisk-kmod-build.sh -n NAME -v VERSION [-b BUILD] [-r KVERSION] [-o RPMBUILD_OPTS] [-C MAKEFILE_PATH] [-d DISTRO] [-t VIRT] [-h|--help] -d DISTRO : Distributive and its version. The default is: centos6 Supported are: centos5 centos6 centos7 -t VIRT : Virtualization name. The default is: kvm Supported are: xen, kvm -n NAME : Driver short name -v VERSION : Driver version -b BUILD : RPM package build number. The default is: 1 -r KVERSION : Kernel version to compile driver's kmog against. The default is: 2.6.32-642.6.2.el6.x86_64 -o RPMBUILD_OPTS : Additional options for the 'rpmbuild' utility. The default are: --define='__find_requires %{nil}' --define='release 1' --define='kversion 2.6.32-642.6.2.el6.x86_64' --define='distro centos6' --define='virt kvm' -C MAKEFILE_PATH : Path to the driver's sources main Makefile - the directory where 'make' utility is called. By default isn't set. Usefull values are: src, ... The path could be determined by running (remove leading KMOD_NAME-KMOD_VERSION dir): tar ztf KMOD_NAME-KMOD_VERSION.tar.gz | grep Makefile -h|--help : print this info
CODEPut the onapp-ramdisk-centos6-kvm-kmod-i40e-1.6.42-1.x86_64.rpm into your Control Panel box and install it there:
# rpm -Uvh onapp-ramdisk-centos6-kvm-kmod-i40e-1.6.42-1.x86_64.rpm
CODE
IV. Proceed with the image customization
To add a custom CloudBoot driver:
Check if there is a file called ''earlyboot.sh'' in the overlay folder on the CP for each compute resource. For example, for a compute resource with the MAC address 00:11:22:33:44:55, look whether the
/tftpboot/images/centos5/diskless/snapshot/00-11-22-33-44-55/overlay/earlyboot.sh
folder exists. This script will be run before the onapp SAN Controller service is run and as such can be used for adding custom drivers.Whilst the path mentions centos5, in practice this same script is used for all compute resource types.
You need to add an executable bit for the
earlyboot.sh
script for it to become executable. This is done with the help of thechmod
command:chmod +x earlyboot.sh
CODEYou can then check whether the executable bit has been added to the
earlyboot.sh
file ('x' indicates the executable file):ls -la earlyboot.sh -rwxr-xr-x 1 root root 90 Feb 25 19:06 earlyboot.sh
CODEInclude drivers in the same overlay folder.
Reference the files in the earlyboot.sh script as if the overlay folder is the root folder.
Insert drivers for each compute resource in the overlay folder as described.
An example overlay file for the megaraid driver:
When adding a new module:
[root@cp-onapp overlay]# cat earlyboot.sh #!/bin/sh PATH=/sbin:/usr/sbin:/bin:/usr/bin export PATH insmod /megaraid/megaraid_sas.ko
CODEWhen changing an existing module:
[root@cp-onapp overlay]# cat earlyboot.sh #!/bin/sh PATH=/sbin:/usr/sbin:/bin:/usr/bin export PATH rmmod megaraid_sas insmod /megaraid/megaraid_sas.ko
CODE
The megaraid driver folder is actually in the overlay folder, not the root:
ls -lR .: total 8 -rwxr-xr-x 1 root root 37 Apr 15 16:47 earlyboot.sh drwxr-xr-x 2 root root 4096 Jul 9 12:21 megaraid ./megaraid: total 2000 -rw-r--r-- 1 root root 2048000 Jul 9 12:22 megaraid_sas.ko
CODEThe driver will then be inserted and can be used immediately after you reboot the compute resource. Otherwise, to use the driver you need to load it manually and restart the services.