Upgrade Guide for Cloud with CloudBoot and Static (5.5 to 6.0)

This guide explains how to upgrade OnApp Cloud 5.5 to 6.0 for the cloud with the mixed CloudBoot and static servers configuration. Follow the procedure listed below in the correct order to upgrade your cloud. Please follow the complete procedure of the upgrade process. All the packages (Control Panel, CloudBoot, and compute resources) must belong to the same major version to ensure the best performance of your cloud.

If you do not complete an upgrade procedure, you may face issues with standard functions such as VS creation, VS migration etc.


Upgrade Control Panel Server


Follow the procedure at Upgrade Guide (5.5 to 6.0) to upgrade your Control Panel server. 

Upgrade Static Compute Resources


At first upgrade your static compute resources.

  1. Make sure your compute resource is visible and online in the Control Panel.

  2. If you have CentOS 7 Xen resources, remove the centos-release-xen-46 package: 

    bash# rpm -e centos-release-xen-46
  3. Download and install the latest OnApp YUM repository file:

    bash# rpm -Uvh http://rpm.repo.onapp.com/repo/onapp-repo-6.0.noarch.rpm
  4. Upgrade OnApp compute resource installer package:

    bash# yum update onapp-hv-install
  5. If required, update your server OS components using the corresponding installer (Xen or KVM) with the '-y' parameter given:

    Xen installer example

    bash# /onapp/onapp-hv-install/onapp-hv-xen-install.sh -y

    KVM installer example

    bash# /onapp/onapp-hv-install/onapp-hv-kvm-install.sh -y
  6. Run the OnApp compute resource installer script, depending on the compute resource: 

    • Run the OnApp compute resource installer script for Xen compute resources:

      bash# /onapp/onapp-hv-install/onapp-hv-xen-install.sh 

      The full list of Xen installer options: 

      Usage: 

      /onapp/onapp-hv-install/onapp-hv-xen-install.sh [-c CONFIG_FILE] [-v HV_VERSION] [-p API_VERSION] [-a] [-y] [-t] [-s] [-x] [-h]

      Where:

      -c CONFIG_FILE
      Custom installer configuration file. Otherwise, the pre-installed one is used.
      -v HV_VERSION
      Custom compute resource Tools version
      -p API_VERSION
      Custom StorageAPI version
      -a
      Do NOT be interactive. Process with automatic installation.
      -y
      Update OS packages (except for OnApp provided) with 'yum update'.
      -t

      Initiate Recovery templates and ISO(s), which are used to provision FreeBSD guests, download. The download is initiated if the '-a' option is used.

      -s
      Skip packages management: install, remove, upgrade. None of the packages are installed/updated/removed.
      - x

      Skip xen*, kernel*, and libvirt* (compute resource related RPM) packages management. Useful for update (not for fresh install).

      -h
      Print this info



    • Run the OnApp compute resource installer script for KVM compute resources:

      bash# /onapp/onapp-hv-install/onapp-hv-kvm-install.sh 

      The full list of KVM installer options: 

      Usage: 

      /onapp/onapp-hv-install/onapp-hv-kvm-install.sh [-c CONFIG_FILE] [-a] [-y] [-t] [-s] [-x] [-v HV_VERSION] [-p API_VERSION] [-h]

      Where:

      -c CONFIG_FILE
      Custom installer configuration file. Otherwise, the pre-installed one is used.
      -a
      Do NOT be interactive. Process with automatic installation.
      -v HV_VERSION
      Custom compute resource Tools version
      -p API_VERSION
      Custom StorageAPI version
      -t

      Initiate Recovery templates and ISO(s), which are used to provision FreeBSD guests, download. The download is initiated if the '-a' option is used.

      -y
      Update OS packages (except those provided by OnApp for a compute resource) with 'yum update'. Useful for update (not for fresh install).
      -s
      Skip packages management: install, remove, upgrade. Useful for update (not for fresh install).
      - x

      Skip kvm, kernel, and libvirt (compute resource related RPM) packages management. Useful for update (not for fresh install).

      -h
      Print this info



  7. Gracefully reboot static compute resources from UI. To do so, log in to Control Panel and go to Settings > Compute Resources. Click the specific compute resource. When the page loads, click Tools > Reboot Compute resource. For KVM compute resources only: the kernel package update is a part of the "Upgrade Static Compute Resources" default procedure. If the kernel package was upgraded, reboot KVM compute resources to use the new kernel package.

    If you do not have the /home/mq/onapp/messaging/credentials.yml file on your compute resources and you plan to deploy an Accelerator, run the following command on the CP server:

    • For all compute resources:

      rake hypervisor:messaging:configure
    • For certain compute resources only:

      rake hypervisor:messaging:configure['11.0.50.111 11.0.50.112']

      To perform the configuration for a number of compute resources, separate their IPs with a space.

      For information on manual configuration for Accelerator, refer to RabbitMQ Configuration for Accelerator.



Upgrade Static Backup Servers


After you upgraded static compute resources, proceed to static backup servers upgrade.

  1. Download the OnApp repository:

    bash# rpm -Uvh http://rpm.repo.onapp.com/repo/onapp-repo-6.0.noarch.rpm
  2. Update the package:

    bash# yum update onapp-bk-install
  3. If required, update your server OS components using installer with the '-y' parameter given:

    bash# /onapp/onapp-bk-install/onapp-bk-install.sh -y
  4. Run the installer. It is recommended to download Base, Load Balancer, and CDN templates while running the installer. You may rerun the installer later with the -t option.

    bash# sh /onapp/onapp-bk-install/onapp-bk-install.sh 

    The full list of installer options:

    Usage:

    /onapp/onapp-bk-install/onapp-bk-install.sh [-c CONFIG_FILE] [-a] [-y] [-t] [-b] [-v BK_VERSION] [-p API_VERSION] [-h]

    Where: 

    -c CONFIG_FILE 

    Custom installer configuration file. Otherwise, the pre-installed one is used.

    -a 

    Do NOT be interactive. Process with automatic installation.

    - v BK_VERSION Custom backup tools version
    - p API_VERSION Custom StorageAPI version
    -y 

    Update OS packages (except for the ones provided by OnApp) with 'yum update'.

    - t

    Initiate the Load Balancer Virtual Appliance, OnApp CDN Appliance, and Application Server Appliance templates download. The download is initiated if the '-a' option is used.

    - b Initiate the Base templates download
    -h 

    Print this info


Upgrade CloudBoot Packages




  • Create a backup of the /tftpboot directory in case the storage packages rollback is needed.
  • Power off Windows virtual servers before proceeding to the following procedures.


To upgrade the OnApp Storage packages:

  1. Upgrade the repo:

    CP_host# rpm -Uvh http://rpm.repo.onapp.com/repo/onapp-repo-6.0.noarch.rpm
    
  2. Update the onapp-store-install package:

    bash# yum update onapp-store-install
    
  3. Update the following packages:

    bash# yum update onapp-ramdisk-tools
    bash# yum update onapp-ramdisk-centos6-xen
    bash# yum update onapp-ramdisk-centos6-kvm
    bash# yum update onapp-ramdisk-centos7-kvm
    bash# yum update onapp-ramdisk-centos7-default
  4. Install the following new package:

    bash# yum install onapp-ramdisk-centos7-xen

    After the packages installation, go to the Control Panel's Settings menu > Configuration and click the Save Configuration button.


    THE FOLLOWING STEPS (5-6) APPLY ONLY TO XEN-BASED COMPUTE RESOURCES: 

  5. Increase the Dom0 value for a Xen CloudBootcomputeresource up to 3072 Mb. To increase the Dom0 memory for all Xen compute resources, edit the Dom0 value in the /tftpboot/pxelinux.cfg/template-xen file on the CP server.

    To increase dom0 memory for a single Xen compute resource, edit the /tftpboot/pxelinux.cfg/xx-xx-xx-xx-xx-xx file, where you have to replace the x's with your compute resource's management NIC MAC address.

  6. Power off all the virtual servers that run on a Xen compute resource.

  7. Run the script:

    CP_host# /onapp/onapp-store-install/onapp-store-install.sh

    Be aware thatthe disk-less nodes passwordis the root password for the CloudBoot compute resources. By default it is blank.


    Whenrun in the interactive mode, enter the required information.


Upgrade CloudBoot Backup Servers




Make sure to update CloudBoot packages on your Control Panel server before proceeding to the upgrade of CloudBoot backup servers.

CloudBoot backup servers are CloudBoot KVM compute resources that can be used as backup servers. The CloudBoot backup server upgrade procedure is almost the same as the CloudBoot compute resource upgrade. Follow the instructions provided in this section to upgrade CloudBoot backup servers in your cloud.

Once you have upgraded the CloudBoot dependencies, you have to reboot your CloudBoot compute resource to update the Cloud Boot RPM. You do not need to perform any backup server upgrade operations usingconsole.

To do so:

  1. Go to your Control Panel Settings > Compute Resources menu.

  2. Click the label of the CloudBoot compute resource the backup server is based on.

  3. On the compute resource details screen, click the Actions button, then click Reboot Compute resource.

  4. A new screen will open asking for confirmation before reboot:

    • Are you sure you want to reboot this compute resource? Confirm that you want the compute resource to reboot.

  5. When you're certain you want to proceed with the reboot, click the Reboot button.

  6. Repeat these steps for all CloudBoot backup servers in your cloud.

  7. Once all are rebooted, proceed to CloudBoot compute resourcesupgrade.


Upgrade CloudBoot Compute Resources


Depending on the infrastructure, scale and needs of your cloud we suggest the following methods of upgrading CloudBoot compute resources:

Simple Reboot

This method is the simplest method technically. It also ensures all tools are updated. However, it will result in some limited downtime
(its duration depends on how many virtual servers are running on each compute resource).

Migrate and Reboot

For Xen only the cold migrate option is available.

This method involves migrating all virtual servers off each CloudBoot compute resource in turn.
The compute resource can then be safely rebooted, picking up the upgraded Integrated Storage and CloudBoot packages. 
Virtual servers that do not support hot migrate will have to be stopped.

In case you have applied any custom configuration to your CloudBoot servers, it is recommended to recheck that this customization does not break new cloud boot image version. For this, reboot a compute resource and run Storage Health Check and Network Health Check. Make sure that Vdisks hosted on a compute resource are redundant and healthy before rebooting a CloudBoot compute resource.

For more information about upgrade scenarios, refer to the OnApp IS Upgrade Paths 5.0 to 6.0 or 5.5 to 6.0.

If you are using the auto healing functionality for Integrated Storage, make sure to disable it before an upgrade.


Simple Reboot

Follow the below procedure to upgrade the CloudBoot compute resources withreboot:

1. Upgrade CloudBoot Packages.

2. When the CloudBoot packages upgrade is complete, stop all virtual servers which reside on the CloudBoot compute resources.

3. Reboot all CloudBoot compute resources.

Once the compute resources are booted, the upgrade is complete. Before starting all Virtual Servers please ensure that the diagnostics page does not report any issue. In case of any issue, please clik repair button to resolve it, then continue with starting Virtual Servers.

Note that virtual servers cannot be stopped simultaneously, but must be stopped in sequence. This can result in considerable downtime if there are a large number of virtual servers.


Migrate and Reboot

Use this procedure if you prefer migrating all virtual servers to anothercomputeresource and conducting overall upgrade of your CloudBoot and Integrated Storage. Virtual servers that do not support hot migrate will have to be stopped.

Once you have upgraded the CloudBoot packages, you have to reboot your CloudBoot compute resources to update them.

To do so:

  1. Run the following command from the Control Panel server terminal to display the list ofcomputeresources with their IP addresses. Make a note of the list of IPs:

    CP_host# liveUpdate listHVs

    If the command liveUpdate is not available, then it may be located in thesbin directory instead (cd /usr/local/sbin).

  2. Migrate all the virtual servers from the CloudBoot compute resource to another computeresource. Follow the instructions described in the Migrate Virtual Server section of the Admin guide tomigrate virtual servers.

    Before you migrate, reboot, or start upvirtual servers, please ensure that grub2.img is present on the destination compute resource in the /onapp/tools directory. To add the file to allcompute resources, please run the following commands on the Control Panel server before the corresponding actions:

    cd /onapp/;wget http://templates.repo.onapp.com/Linux/grub2.img
    for i in `cat /onapp/configuration/dhcp/dhcpd.conf | grep fixed | sed 's/;//' | awk '{print $2}'`; do echo -n "$i -> "; scp /onapp/grub2.img $i:/onapp/tools/; done
  3. After that, go to your Control Panel Settings menu.

  4. Click the Compute Resources icon.

  5. Click the label of the CloudBoot compute resource you have migrated all VSs from.

  6. On the compute resource details screen, click the Actions button, then click Reboot Compute resource.

    Rebooting acomputeresource assigned to a data store with a single replica (single-replica compute resource) or degraded virtual disks may result in data loss.


  7. A new screen will open asking for confirmation (via two check boxes) before reboot:

    • Stop all virtual servers that cannot be migrated to another compute resource? Check this box if you want VSs that cannot be migrated to be powered off. When acompute resource is scheduled for a reboot, OnApp will first attempt to hot migrate all VSs it hosts. If hot migration is not possible for a VS, OnApp will attempt to cold migrate that VS. With this box checked, ifcold migration fails, the VS will be stopped so the reboot may proceed. If you don't check this box, OnApp will attempt to hot and then cold migrate all VSs hosted by thecompute resource being rebooted – but will stop the migration process if any VS cannot be migrated.
    • Are you sure you want to reboot this compute resource? A simple confirmation to confirm that you want thecompute resource to reboot.

      Before the reboot, please ensure that allvdisksare fully synced and redundant. If some of them are not fully synced, the virtual server, that isownerof a degraded (or non-redundant)vdisk, canlooseaccess to thevdisk. It can be manifested as IO errors during writes or reads to/from thevdiskinside the virtual server.

  8. When you're certain you want to proceed with the reboot, click the Reboot button.

  9. Repeat these steps for all CloudBoot compute resources in your cloud.


Local Read Policy


Enabling Local Read on a compute zone ensures that the locally stored copy of the data will always be used for reads. This significantly reduces read latency and improves overall storage performance by reducing load on the SAN network. However, in order to use this policy every compute resource must have sufficient physical drives to be able to store the number of stripes specified in the data store. E.g. in a 2R4S data store there must be at least 4 physical disks on the compute resource to use local read.


Changes to Local Read Policy Enforcement 


Originally, when this policy was introduced OnApp did not enforce the requirement for the minimum number of drives. Consequently, some users who set the policy having insufficient drives may see the following error message:


Fatal: OnApp::Actions::Fatal Storage API Call failed: {"result"=>"FAILURE", "error"=>"Local reads have been enabled on the zone - members required per host: 4, required hosts: 2, available hosts: 0"}

The solution is to either add additional drives to that compute resource and then add them to the data store or to disable read local.