OnApp introduces auto healing -  an auto-scheduling option to repair degraded vDisks. This functionality can be used only in case there are no serious issues with Integrated Storage. The following conditions should be met (it can be checked using the compute zone diagnostics): 

  • No disks with partial memberlist found 
  • No disks with no stripe replicas found 
  • No disks with no redundancy found
  • No partially online disks found 
  • No disks in other degraded states found
  • No partial nodes found 
  • No inactive nodes found 
  • No nodes with delayed ping found 
  • No nodes with high utilization found 
  • No out of space nodes found 
  • No inactive controllers found 
  • No unreferenced NBDs found 
  • No reused NBDs found 
  • No dangling device mappers found
  • No disks with inactive cache
  • No stale cache volumes

It is recommended to disable auto healing before the Integrated Storage upgrade.



Configure Auto Healing for Data Store


To enable auto healing for a data store:

  1. Go to your Control Panel > Admin > Settings menu.
  2. Click the Data Stores icon. You'll see a list of the data stores in your system.
  3. Click the Actions button next to the data store you want to change, then click Edit.
  4. Move the Auto Healing slider to the right to enable auto healing. 

  5. Click the Save Data Store button to finish.

To disable auto healing for a data store:

  1. Go to your Control Panel > Admin > Settings menu.
  2. Click the Data Stores icon. You'll see a list of the data stores in your system.
  3. Click the Actions button next to the data store you want to change, then click Edit.
  4. Move the Auto Healing slider to the left to disable auto healing. 
  5. Click the Save Data Store button to finish.

AutoHealing script is performed every 1 hour on each IS data store marked as auto healing. Auto healing repairs disks one by one for each data store, where it is enabled. There are several conditions, when auto healing will not proceed:

  • If diagnostics fails
  • If there are active repair or rebalance transactions
  • If there are no degraded disks in a data store
  • Auto healing will not try to repair disk if its last repair transaction is failed in the last 24 hours




Emails about Auto Healing Events


You will receive the following email notifications about auto healing process:

  • Hourly emails about degraded vDisks
  • If auto healing is impossible because of issues with Integrated Storage, you will receive an email with the following text: "Degraded vdisks found, but there are problems with Integrated Storage and Auto Healing will not start until you log in and investigate/repair the problems. "
  • If auto healing is running, you will receive an email with the following text: "Degraded vdisks found" and auto healing will start processing the list of degraded vDisks