How to replace a PCIE card in Sun Cluster
I’ve just returned from a quick trip to Nizhniy Novgorod where I had to replace a PCIE card, it was an ethernet adapter (e1000g). Not a big deal, sure, but the tricky part lied in the fact that this card was a part of Sun Cluster setup so I had deliberately chosen a slippery path called DR or dynamic reconfiguration. Actually, it’s not that difficult at all as it might seem at first. Let’s make the long story short…
In my case it was in Intel-based ethernet adapter (e1000g) with two ports: one for the public network (e1000g2) and the other for the private interconnect between clusters (e1000g3). To be able to DR the adapter it must be properly unconfigured and taken out of the OS’s control. Otherwise cfgadm would complain saying the device is busy and won’t let you to remove the card.
- First, lets migrate the public IP address to another interface frm the same IPMP group.
# if_mpadm -d e1000g2
- Next, take the public interface out from IPMP group. And don’t forget to unmplub it – it’s mandatory.
# ifconfig e1000g2 group "" # ifconfig e1000g2 unplumb
- Make a record of the current cluster interconnect configuration:
# clintr show # clintr status
- Disable the cluster’s interconnect cable:
# clintr disable node1:e1000g3,node2:e1000g3
- Now you could safely tamper with the PCIE card through the list of the well-know Solaris commands:
# cfgadm -c unconfigure iou#1-pci#3 # cfgadm -c disconnect iou#1-pci#3
- Physically replace the card and add it back into the Solaris OS:
# cfgadm -c connect iou#1-pci#3 # cfgadm -c configure iou#1-pci#3
- Add back the previously removed interconnect cable:
# clintr enable node1:e1000g3,node2:e1000g3
- If required plumb and failover the IP for the second port (e1000g2). In my case though, that was done automatically as soon as I configured the PCIE card. Just run the following commands if you have to do that manually. I used sc_ipmp0 as a IPMP group name but in your case it could be different.
# ifconfig e1000g2 plumb # ifconfig e1000g2 group sc_ipmp0 # if_mpadm -r e1000g2
Hope it would help someone.
Cheers.