After upgrading to version 6.0, hot migration fails for VSs with the following message:
Running: virsh migrate --live --unsafe byyjnxgstgywrm qemu+ssh://192.168.163.136:22/system?no_verify=1 tcp:192.168.163.136 error: internal error: process exited while connecting to monitor: 2018-10-30T11:02:45.478516Z qemu-kvm: unable to map backing store for guest RAM: Cannot allocate memory Fatal: Execution of virsh migrate --live --unsafe byyjnxgstgywrm qemu+ssh://192.168.163.136:22/system?no_verify=1 tcp:192.168.163.136 failed
OnApp version 6.0
If you do not have any VSs migrated, the issue can be fixed in the following way:
Check the value of
HugePages_Totalon the destination compute resource:
# cat /proc/meminfo | grep HugePages_Total HugePages_Total: 0CODE
These values indicate that the huge pages are disabled and hot migration is impossible.
Reserve some resources for the huge pages. Before VS migration, reserve the huge pages for the qemu process:
• Identify the size of a huge page by checking
/proc/meminfofile. For example:
# cat /proc/meminfo | grep Hugepagesize Hugepagesize: 2048 kBCODE
• Calculate the number of the huge pages to be reserved using the formula:
HPAGES = $((($GUESTs_SIZE_IN_MB)/ 2) + 10)CODE
GUEST_SIZE_IN_MB - the amount of memory size in MB required for VSs migration.
You need to migrate two virtual servers with 1 GB of RAM. Divide the amount of the huge page memory that you need by the size of a huge page:
HPAGES = $(((1024+1024)/ 2) + 10) = 1029CODE
• Allocate the memory as huge pages by adding the number of the huge pages to the
# echo 1029 >/proc/sys/vm/nr_hugepagesCODE
Check if the huge pages values are configured accordingly:
# cat /proc/meminfo | grep -i huge AnonHugePages: 112640 kB HugePages_Total: 1029 HugePages_Free: 1029 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kBCODE
You have already migrated some VSs, but need to migrate one more. Check what is already used and add that number to it. In this case, do the following:
Check for the current free huge pages:
#cat /proc/meminfo | grep HugePages_Free HugePages_Free: 802CODE
HugePages_Freeby 2 (2 is the number of Mb of a single page).
A virtual server is migrated with no issues if RAM does not exceed 1600 Mb.
If the virtual server RAM exceeds this value, the number of
HugePages_Totalmust be increased:
# cat /proc/meminfo | grep HugePages_Total HugePages_Total: 1029 #cat /proc/meminfo | grep HugePages_Free HugePages_Free: 802CODE
Calculate the new
Example: VS with 2Gb RAM, require 1024 of HugePages_Free but we have 802 1024-802=222 - is the number of pages that we need to add to our reserveCODE
Set a new value:
New value for HugePages_Total = HugePages_Total + 222 +10 = 1261 Where 10 HugePages reserved for some factors.CODE
# echo 1261 >/proc/sys/vm/nr_hugepages # cat /proc/meminfo | grep HugePages_Total HugePages_Total: 1261CODE
You can specify how large the pool of huge pages can grow in the
/proc/sys/vm/nr_overcommit_hugepages file if more huge pages than
/proc/sys/vm/nr_hugepages are requested by applications. Specifying any non-zero value in this file indicates the number of "surplus" huge pages from the kernel's normal page pool that the
hugetlb subsystem is allowed to obtain when the persistent huge page pool is exhausted. As these surplus huge pages become unused, they are freed back to the kernel's normal page pool.
Check the value of huge pages on the destination compute resource:
# cat /proc/sys/vm/nr_overcommit_hugepages 0CODE
These values indicate that the huge pages overcommit is disabled.
Identify the size of a huge page by checking the
Hugepagesizeentry in the
Example: # cat /proc/meminfo | grep Hugepagesize Hugepagesize: 2048 kBCODE
- Set the huge page overcommit to a large value (large enough to meet your peak huge page requests). Thus, you can have a dynamic huge page allocation in your environments.
You have a compute resource with 32 Gb (32768 Mb) of RAM, so you can set the huge page overcommit value to almost 80% of free RAM (approximately 26000 Mb). Divide this value by 2 (2 is the number of Mb of a single page). Use the following command:
# echo 13000 > /proc/sys/vm/nr_overcommit_hugepageCODE
In previous versions, the VSs could be booted with the enabled huge pages. In OnApp 5.10 version, huge pages were removed and the resources cannot be allocated on the destination compute resource. To enable transparent huge pages for CentOS 6 and CentOS 7 CloudBoot KVM compute resources, run the following command in the custom config file:
echo always > /sys/kernel/mm/transparent_hugepage/enabled