Install Compute Resources

Please do not create mixed compute zones. Do not add CloudBoot and static compute resources to one compute zone, as well as Xen and KVM compute resources to one compute zone. The reason is that Xen virtual servers cannot be migrated to a KVM compute resource and vice versa.

After you successfully installed the Control Panel server, you can follow the next procedures to set up Xen or KVM compute resources:

Install CloudBoot Compute Resources



Follow this installation to enable CloudBoot for your compute resources. The CloudBoot compute resource installation enables dynamic boot of compute resource servers without any persistent installation requirements. The servers must support and have PXE boot enabled on the Network Interface Card (set up in BIOS if not already enabled by default). See Configure CloudBoot Settings in BIOS for details. We strongly recommend you to deploy one or more backup servers for backups and virtual servers provisioning when using CloudBoot.

  1. Enable CloudBoot on the Control Panel:

    • Go to Settings > Configuration > System > CloudBoot.

    • Scroll down to the CloudBoot section and check the Enable box.

  2. Enable Storage on the Control Panel:
    • Go to Settings > Configuration > System OnApp Storage.
    • Scroll down to the OnApp Storage section and check the Enable OnApp Storage box. 

    • Tick the Use Local Read Path checkbox to minimize the network throughput dependency for read heavy workloads.

  3. Enter IP addresses for static content target and Control Panel server CloudBoot interface. Enter the relevant IPs in Settings > Configuration > System CloudBoot. Static content, such as CloudBoot images, kernels, virtual server templates, can be hosted on a standalone NFS server if you want. The default configuration is to install everything on the Control Panel server.
  4. Add CloudBoot IP addresses for compute resources:
    • Create a CloudBoot IP Net:
      Settings > Compute Resources > CloudBoot IPs tab > New IP Net.
    • Add an IP range to the new IP Net:
      Settings > Compute Resources > CloudBoot IPs > Actions icon next to the IP net Add New IP Range.
    • Add CloudBoot IP Addresses:
      Settings > Compute Resources > CloudBoot IPs > Create IP Address.
  5. Power on servers and allow them to boot the default image. Add servers to the Control Panel by selecting MAC addresses and assigning IP address via Settings > Compute Resources > Add New CloudBoot Compute Resource.

    If you want to expose drives in compute resources to OnApp Storage, our integrated storage platform, then you must select them at this point. For more information on setting up and configuring CloudBoot, see the CloudBoot Compute resources section of the Admin guide. 
    To increase dom0 memory for all new Xen compute resources,  edit the dom0 value in the /tftpboot/pxelinux.cfg/template-xen file on the CP server. To increase dom0 memory for a single Xen compute resource, edit the /tftpboot/pxelinux.cfg/xx-xx-xx-xx-xx-xx file, where you have to replace the x's with your compute resource's management NIC MAC address.

  6. CloudBoot compute resources mount the following locations automatically at boot:

    • /tftpboot/images/centos7/ramdisk-default/ to /.ro
      The path may vary depending on the compute resource template used.
    • /data to /onapp/tools/recovery

    The NFS server from which these are mounted is defined by the Static Config Target parameter (see Edit System Configuration section for details). You can set the default Control Panel server IP to any other server. This change will affect all CloudBoot compute resources.

    The following paths must be available in the static config target to make it possible to use CloudBoot:

    • /tftpboot/export
    • /data
    • /tftpboot/images

    Compute resources will use local templates (mounted from Static Config target) during the server provisioning if the Use SSH File Transfer configuration setting is disabled or the template has null backup_server_id.

  7. If you do not have a Dedicated Backup Server in place, please use Custom Config to mount /onapp/templates and /onapp/backup from your Control Panel server or another NFS export.
  8. After you have installed CloudBoot compute resource, proceed to the Configure CloudBoot Settings in BIOS section.

    If you do not have a dedicated backup server, you must mount your template and backup repositories to compute resources. If your template and backup repositories are located on the Control Panel server, you can mount them as follows:

    Add locations to /etc/exports on the Control Panel server:

    /onapp/templates 192.168.10.0/24(rw,no_root_squash)
    /onapp/backups 192.168.10.0/24(rw,no_root_squash)

    Add locations to Custom Config on a compute resource and run them manually on the command line (in this example, we are mounting from 192.168.10.101):

    mkdir -p /onapp/backups && mount -t nfs 192.168.10.101:/onapp/backups /onapp/backups
    mkdir -p /onapp/templates && mount -t nfs 192.168.10.101:/onapp/templates /onapp/templates

If you want to deploy Accelerator, run the following steps.

  1. Run the following command:

    su onapp
    cd /onapp/interface
  2. Run the following command on the Control Panel server:

    • For all compute resources:

      rake hypervisor:messaging:configure
    • For certain compute resources only:

      rake hypervisor:messaging:configure['11.0.50.111 11.0.50.112']

      To perform the configuration for a number of compute resources, separate their IP addresses with a space. The command above should be run after every reboot. However, you can avoid the necessity to run the command repeatedly after every reboot by coping the following information (using your parameters) from /home/mq/onapp/messaging/credentials.yml to the custom config:

       echo "---
      host: 10.0.50.4  # RABBITMQ SERVER IP/FQDN
      port: 5672      # RABBITMQ CONNECTION PORT(default: 5672)
      vhost: '/'
      user: accelerator-example # RABBITMQ USER NAME
      password: 'e{y31?s8l' #RABBITMQ ACCESS PASSWORD
      queue: 'hv-10.0.50.102' # hv-[IP Address of Compute Resource]
      exchange:
        name: 'acceleration'
        type: 'direct'
        durable: True" > /home/mq/onapp/messaging/credentials.yml
      chown -R mq:mq /home/mq
      service onapp-messaging restart

    For information on manual configuration for Accelerator, refer to RabbitMQ Configuration for Accelerator.



Configure CloudBoot Settings in BIOS



Your BIOS settings may vary from the example provided in this section.

To use PXE boot, you have to make sure it is enabled in BIOS.

  1. Select the required Ethernet card supporting PXE as a boot device:










  2. Go to the Advanced settings > PCI/PnP Configuration.










  3. In the Advanced settings, select the first/primary Onboard LAN/NIC Option ROM and press Enter.

    Use up and down arrow keys to set Option ROM settings to enabled and press Enter.

    Press the Escape key to return to the Advanced menu.











  4. Set a local disk as a second boot device.


Configure InfiniBand



Your hardware must meet the following requirements for the Ethernet mode utilization:

  • VPI enabled switches (including a proper license key)
  • VPI adapter cards (HCAs)

There are the following limitations for InfiniBand:

  • Be aware that InfiniBand is not supported for CloudBoot CentOS 7 due to the lack of integrated OFED stack.
  • InfiniBand in Ethernet mode is supported only for CentOS 6 KVM nodes.
  • InfiniBand is supported only for the SAN network, not PXE boot.

To enable the InfiniBand mode:

  1. Go to Control Panel > Settings > Configuration on the OnApp CP.
  2. Move the Enable InfiniBand Boot slider in the CloudBoot section.
  3. Click the Save Configuration button.


Install Static Compute Resources



Before You Proceed

  • Install base CentOS packages on the local drive before compute resource installation, depending on which virtualization method you choose:
    • Xen 4 compute resources: CentOS 6.x x86/64 or CentOS 7.x x86/64 
    • KVM compute resources: CentOS 6.x x64 or CentOS 7.x x86/64
  • We recommend installing CentOS from the minimal CentOS ISO for static compute resources.
  • Disable CPU power-saving features in BIOS before you proceed to the compute resource installation.
  • If you are not using a dedicated backup server in your cloud setup, configure NFS server with the following options to preserve files owner and group settings during template unpacking on NFS storage:  
    • no_root_squash
    • no_all_squas
  • Pay attention that smart and baremetal servers cannot be installed using the static compute resource installation method.
  • If you plan to deploy baremetal servers, you can enable recovery mode for baremetal servers.

To install a compute resource:

  1. Add a compute resource to your cloud using the OnApp Control Panel: Settings > Compute resources > Add New Compute Resource.
    Make sure that the compute resource is visible in the Control Panel and at this point is shown as inactive.

  2. Update your server:

    bash# yum update
  3. Download the OnApp repository:

    bash#> rpm -Uvh http://rpm.repo.onapp.com/repo/onapp-repo-6.0.noarch.rpm
  4. Install the OnApp compute resource installer package:

    bash#> yum install onapp-hv-install 
  5. Edit custom configuration for a compute resource. Custom values must be set before the installer script runs.

    If you deploy Xen to a server running CentOS 6, it is important to specify a number for XEN_DOM0_MAX_VCPUS. We recommend to set the value to 2 if the compute resource has 12 cores or less; or 4 if the compute resource has more than 12 cores.

    #vi /onapp/onapp-hv.conf
     The full list of custom values for a compute resource.
    • OnApp HV tools custom version

      HV_VERSION=""
    • OnApp StorageAPI custom version

      API_VERSION=""
      
    • Enable monit - tool for managing and monitoring Unix systems

      ENABLE_MONIT=1
    • Default server to sync time on the compute resource

      NTP_TIME_SERVER='pool.ntp.org'
    • Xen HV (Domain-0) related configuration

      XEN_DOM0_MEM_MIN=409600
      XEN_DOM0_MEM_DEVISOR=48
      XEN_DOM0_MAX_VCPUS=""
      XEN_DOM0_VCPUS_PIN_ENABLE=0
      XEN_DOM0_SCHEDULER_WEIGHT=65535
      XEN_DOM0_SCHEDULER_CAP=200
      # 4.2.x and higher versions only
      XEN_DOM0_SCHEDULER_RATELIMIT_US=100
      XEN_DOM0_SCHEDULER_TIMESLICE_MS=5
    • The number of loopback devices created

      LOOPBACKS=128
    • The maximum size of the connection tracking table.

      The value can't be greater than 65536 if the total memory of Xen Domain-0 or KVM is less than 1Gb.
      The value can be doubled (or even more, depending on the memory amount).
      NET_IPV4_NETFILTER_IP_CONTRACK_MAX=""
    • The divisor to calculate the hash table. The recommended value is 8.

      hashsize = nf_conntrack_max / 8
      CONTRACK_TO_HASHSIZE=8
    • Outdated Xen compute resource's (Domain-0) configuration parameters

      XEN_DOM0_MEM_OVERHEAD_MIN=262144
      P_TO_VCPUS=4
  6. Run the OnApp compute resource installer script, depending on the compute resource: 

    • For Xen

      bash#> /onapp/onapp-hv-install/onapp-hv-xen-install.sh 
       The full list of Xen installer options.

      Usage: 

      /onapp/onapp-hv-install/onapp-hv-xen-install.sh [-c CONFIG_FILE] [-v HV_VERSION] [-p API_VERSION] [-a] [-y] [-t] [-s] [-x] [-h]

      Where:

      -c CONFIG_FILE
      Custom installer configuration file. Otherwise, the pre-installed one is used.
      -v HV_VERSION
      Custom compute resource Tools version
      -p API_VERSION
      Custom StorageAPI version
      -a
      Do NOT be interactive. Process with automatic installation.
      -y
      Update OS packages (except for OnApp provided) with 'yum update'.
      -t

      Initiate Recovery templates and ISO(s), which are used to provision FreeBSD guests, download. The download is initiated if the '-a' option is used.

      -s
      Skip packages management: install, remove, upgrade. None of the packages are installed/updated/removed.
      - x

      Skip xen*, kernel*, and libvirt* (compute resource related RPM) packages management. Useful for update (not for fresh install).

      -h
      Print this info
    • For KVM

      bash#> /onapp/onapp-hv-install/onapp-hv-kvm-install.sh 
       The full list of KVM installer options.

      Usage: 

      /onapp/onapp-hv-install/onapp-hv-kvm-install.sh [-c CONFIG_FILE] [-a] [-y] [-t] [-s] [-x] [-v HV_VERSION] [-p API_VERSION] [-h]

      Where:

      -c CONFIG_FILE
      Custom installer configuration file. Otherwise, the pre-installed one is used.
      -a
      Do NOT be interactive. Process with automatic installation.
      -v HV_VERSION
      Custom compute resource Tools version
      -p API_VERSION
      Custom StorageAPI version
      -t

      Initiate Recovery templates and ISO(s), which are used to provision FreeBSD guests, download. The download is initiated if the '-a' option is used.

      -y
      Update OS packages (except those provided by OnApp for a compute resource) with 'yum update'. Useful for update (not for fresh install).
      -s
      Skip packages management: install, remove, upgrade. Useful for update (not for fresh install).
      - x

      Skip kvm, kernel, and libvirt (compute resource related RPM) packages management. Useful for update (not for fresh install).

      -h
      Print this info
  7. Configure the compute resource for your cloud. This step is also required for the SNMP statistics receiver configuration:

    bash#> /onapp/onapp-hv-install/onapp-hv-config.sh -h <CP_HOST_IP> -p <HV_HOST_IP> -b <HV_BSNET_IP>
     The full list of compute resource (both Xen and KVM) configuration options.

    Usage: 

    /onapp/onapp-hv-install/onapp-hv-config.sh [-h CP_HOST_IP] [-p HV_HOST_IP] [-b HV_BSNET_IP] [-f FTS_IP] [-l LVM_HOST_ID] [-a|-i [USER:PASSWD]] [-s] -?

    Where:

    -h CP_HOST_IP

    FQDN or IP Addresses (separated by a comma) of the management server, which should receive all status reports and are authoritative for this compute resource. Used by snmpd and snmptrapd.

    -p HV_HOST_IP

    FQDN or IP Address of server (the compute resource) which will serve all stats-related and other requests send by the CP_HOST_IP. Used by snmpd, snmptrapd, and StorageAPI.

    -b HV_BSNET_IP

    Compute resource's IP Address from Backup Servers' network. Used to bind the SCSI target daemon.

    -f FTS_IP

     File Transfer Server FQDN or IP address, used for daily cron update recovery ISO by recovery.sh

    If unsure, set the Control Panel server's management IP as CP_HOST_IP and FILE_TRANSFER_SERVER_IP.

     -l LVM_HOST_ID

    The lvmlockd sanlock host_id. The ID should be unique for each compute resource in the cloud. It's value for Static compute resources are between 257 and 2000.

    -a

     Install AoE

    -s

     Install sshfs

    -?

     Print this help info

  8. Reboot the compute resource to complete the installation:

    bash#> shutdown -r now
  9. Generate SSH keys that OnApp requires for you to access various elements of the cloud. The script provided will generate and transfer keys as necessary. The script needs to be run on your Control Panel server. It will overwrite any keys that already exist so if you have custom keys already installed, you will need to add them again after running the script. The script will ask you for your login details to various servers during the execution. Please follow the onscreen instructions.
  10. If you install a new cloud instance, connect  to your Control Panel server via SSH, download, and run the script:

    bash#> wget http://downloads.repo.onapp.com/install-all-keys.sh
    bash#> /bin/sh install-all-keys.sh
  11. If you are add additional compute resources to an existing cloud, update the authorized_keys file by running the following script on the Control Panel server:

    bash#> ssh-copy-id -i /home/onapp/.ssh/id_rsa.pub root@HV_HOST_IP
  12. If you do not have a dedicated backup server, you must mount your template and backup repositories to compute resources. If your template and backup repositories are located on the Control Panel server, you can mount them as follows:

    Add locations to /etc/exports on the Control Panel server and then reboot.

    /onapp/templates 192.168.10.0/24(rw,no_root_squash)
    /onapp/backups 192.168.10.0/24(rw,no_root_squash)

    Add locations to /etc/rc.local on the compute resource and run them manually on the command line (in this example, we are mounting from 192.168.10.101).

    mkdir -p /onapp/backups && mount -t nfs 192.168.10.101:/onapp/backups /onapp/backups
    mkdir -p /onapp/templates && mount -t nfs 192.168.10.101:/onapp/templates /onapp/templates
  13. Mount ISO locations:

    To build virtual servers from ISO images, it is required to mount and share the location where the ISOs are stored at the Control Panel server with all the compute resources. When virtual servers are booted from ISOs, the ISO image is taken from the compute resource server. The location is preconfigured in onapp.yml config file:

    • iso_path_on_cp - specifies the location where ISOs are stored on the Control Panel server. By default, the location is /data. You can change it to any other suitable location. Make sure that this location is shared with the specified iso_path_on_hv location.
    • iso_path_on_hv - specifies the location where ISOs are located on the compute resource servers. By default, the location is /data.  You can change it to any other suitable location with the onappowner user and read/write access. Make sure that this location is mounted to the specified iso_path_on_cp location.

    CloudBoot compute resources mount the /data location automatically at boot to the /onapp/tools/recovery on a compute resource.

    ISOs can be hosted on a dedicated server at any desired location with an arbitrary name if you want. In this case, it is necessary to mount the ISOs location on this server to the Control Panel iso_path_on_cp directory and all the compute resources iso_path_on_hv locations. This can be a backup server to avoid the excess usage of the Control Panel resources.

  14. Reboot static compute resources.


    To edit the configuration for Accelerator manually, perform the following steps:

  1. Copy file:

    cp /home/mq/onapp/messaging/credentials{_example,}.yml
  2. Open vi /home/mq/onapp/messaging/credentials.yml and check the following details:

    ---
    host: 10.0.50.4  # RABBITMQ SERVER IP/FQDN
    port: 5672  	# RABBITMQ CONNECTION PORT(default: 5672)
    vhost: '/'  	
    user: accelerator-example # RABBITMQ USER NAME
    password: 'e{y31?s8l' #RABBITMQ ACCESS PASSWORD
    queue: 'hv-10.0.50.102' # hv-[IP Address of Compute Resource]
    exchange:
      name: 'acceleration'
      type: 'direct'
      durable: True 
  3. Change an owner:

    chown -R mq:mq /home/mq
  4. Run the following commands:

    service onapp-messaging start
    monit monitor onapp-messaging