Backup Failed: iscsciadm Could Not Login to All Portals

Issue

Backups fail with the following log output:

Remote Server: 37.72.173.162
Running: iscsiadm -m node -o new -T "iqn.2015-04-11:onapp.com:backup-cih8vr6t2to5cu" -p 10.0.1.1
New iSCSI node [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.0.1.1,3260,-1 iqn.2015-04-11:onapp.com:backup-cih8vr6t2to5cu] added

Running: iscsiadm -m node -T "iqn.2015-04-11:onapp.com:backup-cih8vr6t2to5cu" -l
Logging in to [iface: default, target: iqn.2015-04-11:onapp.com:backup-cih8vr6t2to5cu, portal: 10.0.1.1,3260] (multiple)

iscsiadm: Could not login to [iface: default, target: iqn.2015-04-11:onapp.com:backup-cih8vr6t2to5cu, portal: 10.0.1.1,3260].

iscsiadm: initiator reported error (8 - connection timed out)

iscsiadm: Could not log into all portals

Fatal: Execution of iscsiadm -m node -T "iqn.2015-04-11:onapp.com:backup-cih8vr6t2to5cu" -l failed
Executing Rollback...

CODE

Environment

All OnApp versions

LVM/SAN datastore

Dedicated backup server

Resolution

To resolve the issue, restart tgtd on the virtual server's compute resource. It can be done with service tgtd restart. If tgtd is stuck, kill the old processes and then restart it. Make sure to use restart and not start. restart will clean up the old stuff from the previous running tgtd processes.

[root@sjhypervisor1 ~]# ps aux | grep tgtd
root     13240  0.0  0.0 103296   836 pts/10   S+   12:10   0:00 grep tgtd
root     28097  0.0  0.0  54836  1152 ?        Ss   02:10   0:03 tgtd
root     28098  0.0  0.0  19916   424 ?        S    02:10   0:01 tgtd
[root@sjhypervisor1 ~]# kill -9 28097 28098
[root@sjhypervisor1 ~]# /etc/init.d/tgtd restart
Stopping SCSI target daemon: not running                   [FAILED]
Starting SCSI target daemon:                               [  OK  ]
[root@sjhypervisor1 ~]#

CODE

Cause

The backup server needs to log in to the iSCSI targets on this compute resource. These targets are temporarily created for the backups.