dmsetup remove Failure

Question

What can I do if device-mapper: remove ioctl failed: Device or resource busy shows when running dmsetup remove?

Environment

OnApp 5.x,6.x

Answer

Follow these steps to find the process preventing the dm entry from being removed:

1. SSH into compute resource shown in the error log.

2. Check the device-mapper hierarchy to see which backup is utilizing the -real snapshot in the error log. Run the following command:

   dmsetup ls --tree | grep -B5 onapp--xxxxxxxxxxxxxx-zzzzzzzzzzzzzz-real

   CODE

   Please keep in mind that if you don't see the root backup device, you may need to increase the value for the -B switch to something higher than 5.

   Example:

   [root@10.0.0.97 ~]# dmsetup ls --tree | grep -B5 onapp--y0jmntchnsldn8-yz2zvp2xygmg72-real
   onapp--y0jmntchnsldn8-backup--c00fxwrzcy1jhq (253:12)
    |-onapp--y0jmntchnsldn8-backup--c00fxwrzcy1jhq-cow (253:21)
    |  `-mpath2 (253:6)
    |     |- (8:64)
    |     `- (8:32)
    `-onapp--y0jmntchnsldn8-yz2zvp2xygmg72-real (253:20)

   CODE

3. Determine what process is using the top dm entry in the hierarchy with lsof.

   - onapp--xxxxxxxxxxxxxx-backup--yyyyyyyyyyyyyy

   CODE

   Run the following command:

   lsof /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy     

   CODE

   Example:

   [root@10.0.0.97]# lsof /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq 
   COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
   tgtd    5991 root   15u   BLK 253,12      0t0 331375079 /dev/mapper/onapp--y0jmntchnsldn8-backup--c00fxwrzcy1jhq

   CODE
   

   If there is no backup server, lsof might show nothing here. Check /proc/mounts/ to see if the backup is mounted. If it is mounted, umount it, and proceed with clearing the dmsetup entries.

   
   

   4. Since the backup is being utilized by tgtd, check if there are any iSCSI targets active. Run:

   tgtadm --lld iscsi --mode target --op show

   CODE

   Example:

   [root@10.0.0.97]# tgtadm --lld iscsi --mode target --op show
   Target 1: iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq
       System information:
           Driver: iscsi
           State: ready
       I_T nexus information:
       LUN information:
           LUN: 0
               Type: controller
               SCSI ID: IET     00010000
               SCSI SN: beaf10
               Size: 0 MB, Block size: 1
               Online: Yes
               Removable media: No
               Readonly: No
               Backing store type: null
               Backing store path: None
               Backing store flags: 
           LUN: 1
               Type: disk
               SCSI ID: IET     00010001
               SCSI SN: beaf11
               Size: 644245 MB, Block size: 512
               Online: Yes
               Removable media: No
               Readonly: No
               Backing store type: rdwr
               Backing store path: /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq
               Backing store flags: 
       Account information:
       ACL information:
           10.0.50.99

   CODE

   5. Check the backup server for related ISCSI sessions and log out of them:

   1. Login to the backup server shown under "ACL information" from the previous step and run iscsiadm -m session.

   2. Compare the target name from the previous step and logout of the corresponding ISCSI sessions using iscsiadm -m node -u -T <target name>.

   3. If there are no corresponding sessions on the backup server, you can skip to step 6.

Example:

[root@10.0.0.99]# iscsiadm -m session

tcp: [171] X.X.X.X:XXXX,1 iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq

[root@10.0.0.99]# iscsiadm -m node -u -T iqn.2015-03-09:onapp.com:backup-c00fxwrzcy1jhq

Make sure the -T command is -T. If the iscsiadm command does not receive the -T after the -u, it will log out of all iscsi targets.

6. Delete the iSCSI target. Go back to the compute resource and run tgtadm --lld iscsi --op delete --mode target --tid *

Where: - tid stands for Target ID, which is shown when running the command in step 4.

Example:

[root@10.0.0.97]# tgtadm --lld iscsi --op delete --mode target --tid 1

7. Remove the dm entries for the backup and the -real snapshot.

For entries that end with X1, you should run this first: kpartx -d -p X /dev/mapper/backup-XXXXXXXXXX

Run the following commands:

dmsetup remove /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy

dmsetup remove /dev/onapp-xxxxxxxxxxxxxx/backup-yyyyyyyyyyyyyy-cow

dmsetup remove /dev/mapper/onapp--xxxxxxxxxxxxxx-zzzzzzzzzzzzzz-real

Example:

[root@10.0.0.97]# dmsetup remove /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq
[root@10.0.0.97]# dmsetup remove /dev/onapp-y0jmntchnsldn8/backup-c00fxwrzcy1jhq-cow
[root@10.0.0.97]# dmsetup remove /dev/mapper/onapp--y0jmntchnsldn8-yz2zvp2xygmg72-real

Do not run dmsetup remove on a non-backup or non-real device. That can cause data loss for the VS.