SHIFT

--- Sjoerd Hooft's InFormation Technology ---

User Tools

Site Tools


fcppartnerpathmisconfigured
Differences

This shows you the differences between two versions of the page.

Link to this comparison view

fcppartnerpathmisconfigured [2013/04/18 20:10] (current)
sjoerd created
Line 1: Line 1:
 += Fix: FCP Partner Path Misconfigured =
  
 +This is a small manual on fixing the error
 +<​code>​
 +FCP Partner Path Misconfigured
 +</​code>​
 +which could show in the AutoSupport report created in NetApp/​N-series filers. It means that a particular LUN is being accessed over the partner path through the interconnect link rather than through the preferred primary (optimized) path. For more information see the resources.
 +\\
 +Finding and resolving the issue is a matter of a few steps:
 +# Identify the LUN which is being accessed over the partner path
 +# Identify the host initiator for the LUN
 +# Identify the correct target WWPN of the filer
 +# Solve the path selection error
 +
 += Identify the LUNs =
 +
 +The only way to see which LUN is the source of the error is by checking the data traffic over the partner path. It's possible to see the LUNs statistics but you should zero them first so events in the past don't interfere with the troubleshooting process: \\
 +This is the command to zero the LUN statistics:
 +<​code>​
 +filer01> lun stats -z
 +</​code>​
 +After issuing the command you should wait for a couple of minutes.
 +\\
 +Than check the statistics so far. Note that a few partner ops as well as a few kbytes are normal for multipathing operations but not too much:
 +<​code>​
 +filer01> lun stats -o
 +...<​cut>​...
 +    /​vol/​DATA_Volume_0/​F1A13_DATA ​ (0 hours, 1 minute, 18 seconds)
 +        Read (kbytes) ​  Write (kbytes) ​ Read Ops  Write Ops  Other Ops  QFulls ​ Partner Ops Partner KBytes
 +        66672           ​105168 ​         2480      23267      44         ​0 ​      ​12957 ​      42990
 +...<​cut>​...
 +</​code>​
 +
 +The affected LUN is "/​vol/​DATA_Volume_0/​F1A13_DATA"​
 +
 += Identify the host Initiators =
 +
 +== Identify the Initiator Group ==
 +
 +Now you have to identify the host which is accessing the LUN. To do so you first check the initiator group which has access, followed by checking the members for the identified initiator group:
 +\\
 +First find the initiator group which accesses the LUNs:
 +<​code>​
 +filer01> lun show -v /​vol/​DATA_Volume_0/​F1A13_DATA
 +        /​vol/​DATA_Volume_0/​F1A13_DATA ​ 800.1g (859047985152) ​ (r/w, online, mapped)
 +                Serial#: W-CqCJRe/​kaJ
 +                Share: none
 +                Space Reservation:​ disabled
 +                Multiprotocol Type: vmware
 +                Maps: ESXServers_datacenter=21
 +</​code>​
 +
 +The LUN is accessed by hosts in the initiator group "​ESXServers_datacenter"​
 +
 +== Identify Member Servers for Initiator Group ==
 +
 +Now check which hosts are in the initiator group:
 +<​code>​
 +filer01> igroup show -v ESXServers_datacenter
 +    ESXServers_datacenter (FCP):
 +        OS Type: vmware
 +        Member: 21:​01:​00:​1b:​32:​b0:​33:​00 (logged in on: 4b, 4a, vtic)
 +        Member: 21:​00:​00:​1b:​32:​90:​33:​00 (logged in on: 4d, 4c, vtic)
 +        Member: 21:​00:​00:​1b:​32:​90:​35:​00 (logged in on: 4d, 4c, vtic)
 +        Member: 21:​01:​00:​1b:​32:​b0:​35:​00 (logged in on: 4b, 4a, vtic)
 +        Member: 21:​00:​00:​1b:​32:​90:​35:​01 (logged in on: 4d, 4c, vtic)
 +        Member: 21:​01:​00:​1b:​32:​b0:​35:​01 (logged in on: 4b, 4a, vtic)
 +        Member: 21:​00:​00:​1b:​32:​90:​35:​01 (logged in on: 4d, 4c, vtic)
 +        Member: 21:​01:​00:​1b:​32:​b0:​35:​01 (logged in on: 4b, 4a, vtic)
 +        Member: 21:​01:​00:​1b:​32:​b0:​35:​0f (logged in on: 4b, 4a, vtic)
 +        Member: 21:​00:​00:​1b:​32:​90:​35:​0f (logged in on: 4d, 4c, vtic)
 +</​code>​
 +
 +> Note: Only members that are also clogged in on "​vtic"​ (interconnect link) can access the LUN over the partnerpath. The order in which they are listed is not relevant when the host utilities are installed.
 +
 +== Reduce host initiators list ==
 +
 +Since the member list of the initiator group is quite extensive you can reduce the number of hosts by checking which ones are sending Read/Write IO over the partner path:
 +<​code>​
 +filer01> lun config_check -A
 +
 +The following FCP Initiators are sending Read/Write i/o over the
 +FCP Partner Paths during the last 10 seconds
 +WWPN                      Partner'​s Port       ​ops ​        ​bytes ​     ostype
 +21:​00:​00:​1b:​32:​90:​35:​0f ​              ​4c ​     1453       ​5047296 ​     vmware
 +
 +Proxy ops exceeded 4 out of 10 times
 +Proxy bytes exceed 6 out of 10 times
 +
 +The following values have been changed from
 +their recommended values
 +lun.use_partner.cc.warn_limit ​   Recommended:​300 ​        ​Changed:​10
 +lun.use_partner.cc.bytes ​        ​Recommended:​2457600 ​    ​Changed:​512000
 +No Problems Found
 +</​code>​
 +
 +So the server accessing LUN F1A13_DATA over it's partner path has a WWN "​21:​00:​00:​1b:​32:​90:​35:​0f"​. I manually found the responsible host in vCenter but there are scripts available if needed. The corresponding host is: esx03
 +
 += Identify the primary storage controller node = 
 +
 +Now you need to know what are the primary paths. First we need the cfmode. The cfmode determines if LUNs can be accessed over their partner path:
 +<​code>​
 +filer01> fcp show cfmode
 +fcp show cfmode: single_image
 +</​code>​
 +In the active-active storage controller configuration,​ primary versus secondary ports will depend on the cluster failover mode (cfmode) employed. The single_image cfmode allows each FC target port to be used for primary path access to LUNs hosted on the local node and partner path access for LUNs hosted on the partner node.
 +\\
 +Now find the WWPN for the local FC adapters:
 +<​code>​
 +filer01> fcp show adapters
 +Slot:                    4a
 +Description: ​            Fibre Channel Target Adapter 4a (Dual-channel,​ QLogic 2432 (2464) rev. 3)
 +Adapter Type:            Local
 +Status: ​                 ONLINE
 +FC Nodename: ​            ​50:​0a:​09:​80:​89:​7a:​cc:​db (500a0980897accdb)
 +FC Portname: ​            ​50:​0a:​09:​81:​89:​7a:​cc:​db (500a0981897accdb)
 +Standby: ​                No
 +
 +Slot:                    4b
 +Description: ​            Fibre Channel Target Adapter 4b (Dual-channel,​ QLogic 2432 (2464) rev. 3)
 +Adapter Type:            Local
 +Status: ​                 ONLINE
 +FC Nodename: ​            ​50:​0a:​09:​80:​89:​7a:​cc:​db (500a0980897accdb)
 +FC Portname: ​            ​50:​0a:​09:​82:​89:​7a:​cc:​db (500a0982897accdb)
 +Standby: ​                No
 +
 +Slot:                    4c
 +Description: ​            Fibre Channel Target Adapter 4c (Dual-channel,​ QLogic 2432 (2464) rev. 3)
 +Adapter Type:            Local
 +Status: ​                 ONLINE
 +FC Nodename: ​            ​50:​0a:​09:​80:​89:​7a:​cc:​db (500a0980897accdb)
 +FC Portname: ​            ​50:​0a:​09:​83:​89:​7a:​cc:​db (500a0983897accdb)
 +Standby: ​                No
 +
 +Slot:                    4d
 +Description: ​            Fibre Channel Target Adapter 4d (Dual-channel,​ QLogic 2432 (2464) rev. 3)
 +Adapter Type:            Local
 +Status: ​                 ONLINE
 +FC Nodename: ​            ​50:​0a:​09:​80:​89:​7a:​cc:​db (500a0980897accdb)
 +FC Portname: ​            ​50:​0a:​09:​84:​89:​7a:​cc:​db (500a0984897accdb)
 +Standby: ​                No
 +</​code>​
 +As these are local FC ports, these ports are in potential all primary path to the LUNs. 
 +\\
 +Conclusion, if the host is not using these ports for its active paths the traffic is routed over the partner path.
 +
 += ESX solving =
 +
 +As you can see in the screenshot the path was indeed configured to go over the partner path:
 +
 +{{fcppartnerpathmisconfigured01.jpg}} \\
 +
 +After running this command on the host the issue was resolved, first query, solve and than query again:
 +
 +<​code>​
 +[root@esx03 ~]# /​opt/​ontap/​santools/​config_mpath --query --name naa.60a98000572d4371434a52652f6b614a
 +Verifying firewall port is open...................................DONE
 +
 +Determining Primary/​Proxy paths for LUN=naa.60a98000572d4371434a52652f6b614a
 +   ​PATH=vmhba3:​C0:​T2:​L21 is a PROXY path
 +   ​PATH=vmhba3:​C0:​T3:​L21 is a PROXY path
 +   ​PATH=vmhba4:​C0:​T0:​L21 is a PRIMARY path
 +   ​PATH=vmhba4:​C0:​T1:​L21 is a PRIMARY path
 +   ​PATH=vmhba3:​C0:​T0:​L21 is a PRIMARY path
 +   ​PATH=vmhba4:​C0:​T2:​L21 is a PROXY path * Preferred
 +   ​PATH=vmhba3:​C0:​T1:​L21 is a PRIMARY path
 +   ​PATH=vmhba4:​C0:​T3:​L21 is a PROXY path
 +
 +[root@esx03 ~]# /​opt/​ontap/​santools/​config_mpath --primary --loadbalance --policy fixed --name naa.60a98000572d4371434a52652f6b614a
 +Verifying firewall port is open...................................DONE
 +
 +Determining Primary/​Proxy paths for LUN=naa.60a98000572d4371434a52652f6b614a
 +   ​PATH=vmhba3:​C0:​T2:​L21 is a PROXY path active
 +   ​PATH=vmhba3:​C0:​T3:​L21 is a PROXY path active
 +   ​PATH=vmhba4:​C0:​T0:​L21 is a PRIMARY path active
 +   ​PATH=vmhba4:​C0:​T1:​L21 is a PRIMARY path active
 +   ​PATH=vmhba3:​C0:​T0:​L21 is a PRIMARY path active* Preferred
 +   ​PATH=vmhba4:​C0:​T2:​L21 is a PROXY path active
 +   ​PATH=vmhba3:​C0:​T1:​L21 is a PRIMARY path active
 +   ​PATH=vmhba4:​C0:​T3:​L21 is a PROXY path active
 +
 +
 +PSP Policy for LUN naa.60a98000572d4371434a52652f6b614a is now set to VMW_PSP_FIXED
 +Preferred path has been set for vmhba4:​C0:​T0:​L21 for LUN naa.60a98000572d4371434a52652f6b614a
 +
 +The path policy has been set for your configuration.
 +
 +[root@esx03 ~]# /​opt/​ontap/​santools/​config_mpath --query --name naa.60a98000572d4371434a52652f6b614a
 +Verifying firewall port is open...................................DONE
 +
 +Determining Primary/​Proxy paths for LUN=naa.60a98000572d4371434a52652f6b614a
 +   ​PATH=vmhba3:​C0:​T2:​L21 is a PROXY path
 +   ​PATH=vmhba3:​C0:​T3:​L21 is a PROXY path
 +   ​PATH=vmhba4:​C0:​T0:​L21 is a PRIMARY path * Preferred
 +   ​PATH=vmhba4:​C0:​T1:​L21 is a PRIMARY path
 +   ​PATH=vmhba3:​C0:​T0:​L21 is a PRIMARY path
 +   ​PATH=vmhba4:​C0:​T2:​L21 is a PROXY path
 +   ​PATH=vmhba3:​C0:​T1:​L21 is a PRIMARY path
 +   ​PATH=vmhba4:​C0:​T3:​L21 is a PROXY path
 +
 +</​code>​
 +> Note: Not using the --name parameter will set or query all LUNs
 +
 +The result also shows in the vSphere client:
 +
 +{{fcppartnerpathmisconfigured02.jpg}} \\
 +
 += Resources =
 +
 +[[https://​kb.netapp.com/​support/​index?​page=content&​id=3010111&​actp=search&​viewlocale=en_US&​searchid=1310045898129|NetApp Knowledgebase Article]] \\
 +
 +{{tag>​vmware fix netapp performance}}
fcppartnerpathmisconfigured.txt ยท Last modified: 2013/04/18 20:10 by sjoerd