ICE/ICN Troubleshooting

Topic ID: id_SL16404369-1365992

Overview

This document describes the troubleshooting guide for the following error conditions. This is only a supplement to existing troubleshooting procedures.

VRE configuration is successful but iVRF diagnostic fails.
VRE configuration failed.

Topic ID: id_SL16404372-1365992

Diagram

Figure 1. Connection between ICE, ICN and Host PC

Topic ID: id_SL16404374-1365992

Precondition

Topic ID: id_SL16404375-1365992

Network Path Check

Before starting troubleshooting, make sure the connection between ICE, ICN and Host PC is good

Note:

You can also disconnect Ethernet cable from host, connect it to laptop, and ping ICE/ICN instead of performing step 2 to step 5.

Setup your laptop IP address different with the node you will test on ICE/ICN. (10.0.1.50 for example).
Disconnect Ethernet cable from suspected node on ICE/ICN.
Connect laptop instead and ping the host from there.
If OK, reconnect Ethernet cable to suspected node, and disconnect its other end from 8-Port Ethernet Switch.
Connect it to laptop directly and ping the suspected node.

Topic ID: id_SL16404380-1365992

Speed Mode Check

All the Ethernet cable should be running at 1000 Mbps mode. 100/1000 LED on Ethernet switch front panel should be green. If anyone of them is orange, change it with a new cable.

Topic ID: id_SL16404401-1365992

Available ICE/ICN Diagnostics

Depending on the fault condition, use the appropriate diagnostics to help isolate the root cause.

Table 1. Available ICE/ICN Diagnostics
Diag Folder	Diag Name	Purpose
Acquisition/Pulse Generation	ICE Board Level Diagnostics	ICE BLD is an ICE internal main part diagnostic tool, however, it will not diagnostic PCI-E outside connection.
ICN Quick Health	ICN Hardware Health	ICN Quick Health Check, is a shorter form of diagnostics that would cover the ICN H/W, Communication and PCIe adapter card Health status.
Check	ICN Hardware Health
ICN Network Connection	Host-ICN Ethernet Path	The purpose of this diagnostic is to verify network connectivity between the host and an ICN. This diagnostic helps identify network failure points between a host and an ICN by sending an ICMP (ping) packet from the host to the ICN. The ICN is expected to acknowledge the packet by responding back to the host.
	Host-ICN Ethernet Stress Diagnostic	The purpose of this diagnostic is to stress the Ethernet network path between the host and ICN to verify that it meets the minimum throughput requirements of 50 MB/sec.
	Host-ICN IPMI Path Diagnostic	The purpose of this
		diagnostic is to verify the
		IPMI remote communication link from the host to the ICNs. The IPMI interface is used for remote management of the ICNs.
		This diagnostic uses the
		VRE configuration file /w/config/vre.cfg that specifies the IP numbers or the ICNs IPMI Ethernet interface.
	Host-ICN Remote Shell	The purpose of this diagnostic is to verify remote shell connectivity to the ICN. This diagnostic attempts to execute a command on the ICN via the remote shell protocol.
ICN Hardware Diagnostics	ICN Hard Disk Stress	The purpose of this diagnostic is to test the disk throughout for both of the hard disks on an ICN. Some application uses Hard disks to store acquired data for further processing. This diagnostic essentially verifies the ICN hard disk performance is fast enough to keep up with the incoming data during acquisition.
ICN Miscellaneous Diagnostics	ICN Reconstruction Diagnostics	VRE Reconstruction Diagnostics includes the following items:
		• Host ICN Time Sync Diagnostic
		• ICN Image Reconstruction
		• ICN Process Verification
	ICN Sensors Display	This diagnostic displays the sensor values for all ICNs installed in the system. Three categories of sensor values are displayed: Fan Speeds, Temperatures, and Voltages. The display format includes the ICN number, sensor name, value and units, minimum value, maximum value, critical value, and whether critical values were exceeded.
	iVRF Communication Diagnostics	ICN-ICE Datapath: Tests the communication data path between ICN and iVRF at the ICE across the PCIe cable. This datapath is used to pass iVRF filtered MR data from ICE to ICN for image reconstruction, and to pass control information from ICN kernel module to iVRF. This diagnostic essentially verifies the ICN PCIe switch, the PCIe cable, and the PCIe switch on the ICE daughter board and the PCIe switch on the mother board, and the iVRF memory all work reliably to support the communication between ICN and ICE.

Topic ID: id_SL16404403-1365992

ICE/ICN Troubleshooting

Topic ID: id_SL16404404-1365992

VRE Configuration Successful But iVRF Diagnostic Failed

VRE configuration is successful but iVRF diagnostic is failed. Check PCIe cable, ICE BLD, Dolphin link LED, and iVRF Diagnostics.

Topic ID: id_SL16404406-1365992

5.2 VRE Configuration Failed

Under this condition, system cannot boot up, so service browser and diagnostic cannot be running.

VRE configure could be classified into six milestones, specific trouble shooting actions could be adopted according to which milestone dose the fault appears.

Following table shows the six milestones during VRE configuration.

Table 2. Milestones
Milestone	Description
1	Starting VRE Installation and Configuration at: Mon May 15 12:20:08 CST 201
	RDS Configuration: Disabled
	Copying OS contents from VRE OS disk to host machine is needed
	mount: block device /dev/sr0 is write-protected, mounting read-only
	Copying VRE OS contents
	VRE OS content successfully copied
	Ejecting OS media
2	Performing the OS integrity check for VRE OS
	OS integrity check completed successfully for VRE OS
	IVRF system detected
	VERBOSE reporting activated. Printing debug information.
	Entering pre-detect phase:
	MGD subnet is 10.0.1.0 / 255.255.255.0
	Entering ICN detection phase:
	Detect via NMAP
	ICN discovery phase completed, detected following ICNs:
	10.0.1.223 [ 84:7B:EB:D7:74:F4 ]
	icn1, bmc ipaddr: 10.0.1.223, bmc macaddr: 84:7B:EB:D7:74:F4
	Received BMC IP Address: 10.0.1.223
	Determining ICN Vendor
	Dell ICN: Bios upgrade not required
	Launching ICN Remote Console in the background....requires license
	Entering ICN BMC validity check phase:
	vrevalidtest: Checking ICN BMC validity...
	We matched a valid ICN BMC configuration !
3	Entering pre_install phase
	Entering install phase:
	Inserted logfile marker for NFS mount
	Issuing IPMI chassis OFF command to device @ 10.0.1.223: Unsuccessful. Trying again.
	Successful
	Issuing IPMI chassis ON command to device @ 10.0.1.223: Successful
	Setting ICN 10.0.1.223 to pxeboot: Successful
4	Waiting 360 seconds (max) for 1 NFS mount requests.
	...................:1:
	10.0.1.221
	key: 10.0.1.221 value: 24:6e:96:3c:16:d0
	10.0.1.221 [ 24:6e:96:3c:16:d0 ]
	Returned from watch_log after 101 seconds.
	10.0.1.221
	Mount requests received in 101 seconds.
5	Inserted logfile marker for ICN OS load/reboot
	Waiting 1800 (max) seconds for ICN OS load/reboot.
	10.0.1.221 24:6E:96:3C:16:D0
	Waiting 250 seconds to allow ICNs to finish booting.
	ICN OS load completed in 753 seconds.
6	ICNs booted with following temporary IP addresses:
	10.0.1.221 [ 24:6E:96:3C:16:D0 ]
	Entering ICN DMI validity check phase:
	vrevalidtest: Checking ICN configuration...
	Checking 10.0.1.221 [ 24:6E:96:3C:16:D0 ]
	VRF: 0
	BVER: 1.2.10
	SNAME: ICN Gen6
	DRIVES: 2
	PCIE: 1
	CMAN: General Electric Company
	BNAME: 086D43
	CVER: Not Specified
	CASS: 5931000-3rev4
	IB: 0
	We matched a VALID ICN configuration !
	Entering post_install phase:
	Entering icn_config phase
	ICN data file transfer:
	Completed for 10.0.1.221
	Configuring 10.0.1.221 [24:6E:96:3C:16:D0] as ICN1 and 10.0.1.100: Completed.
	Dell System...Getting cores per socket
	[All] cores are available
	Rebooting ICN1
	Entering host_config phase:
	Entering post_host_config phase:
	Waiting for the final reboot to finish........
	Final reboot is complete in 130 seconds...
	Waiting 30 seconds to allow ICNs to finish booting

If you get stuck in milestone 1, the VRE OS DVD is not good. Use a different VRE OS DVD.
If you get stuck in milestone 2, follow below steps for troubleshooting:
1. Login in by root/ operator and type flowing command:
  
  {sdc@blrmr194}[107] /export/home/insite/server/cgi-bin/getIcnVendor.pl
  1. If Dell|10.0.1.2xx (xx is random values) comes back, this means IPMI is successfully working on ICN. Implement step 2 to double confirm ICN BIOS is in communication.
  2. If error BMC not reachable Check BMC connections on the ICN and retry the diagnostics comes back, this means the IPMI commands are not working. Go to step 3 to check racadm connection.
2. SSH login IDRAC IP address for double confirm ICN BiOS is in connection.
  
  {sdc@GEHC}[XX] /export/home/insite/server/cgi-bin/racadm <BMC_IP> "get BIOS"
  
  If the feedback is same as the following, it means iDRAC and BIOS is normal:
  
  spawn /usr/bin/ssh -oStrictHostKeyChecking=no –
  
  oStrictHostKeyChecking=no root@icn1-bmc racadm get BIOS
  
  FIPS mode initialized
  
  root@icn1-bmc's password:
  
  BiosBootSettings
  
  IntegratedDevices
  
  MemSettings
  
  MiscSettings
  
  NetworkSettings
  
  OneTimeBoot
  
  ProcSettings
  
  PxeDev1Settings
  
  PxeDev2Settings
  
  PxeDev3Settings
  
  PxeDev4Settings
  
  SataSettings
  
  SerialCommSettings
  
  SlotDisablement
  
  SysInformation
  
  SysProfileSettings
  
  SysSecurity...
  
  If the feedback is not same as above, go to step 3 to check the racadm connection.
3. Check the racadm connection.
  1. Telnet iDRAC from host, type command firefox 10.0.1.2xx in xterm.
    Figure 4. Telnet iDRAC from Host
  2. Login IDRAC by root/calvin
    Figure 5. Login
  3. Click Launch.
    Figure 6. Launch
  4. Reboot ICN. Press power button in front panel till ICN off and then power on. Press F2 during ICN server boot to enter setup.
    Figure 7. Reboot ICN
  5. Select iDRAC Setting.
    Figure 8. iDRAC Setting
  6. Select” Network” and make sure the IP address is 10.0.1.2xx.
  7. Select “User Configuration and make sure the user name is root and set the password to calvin.
  8. Press ESC to exit and wait the system to boot.
  9. Go back to MR console and try to config the VRE again.
4. If you get stuck in milestone 3. Enter the Bios and check the PXE setting
  1. Telnet iDRAC from host
  2. Login iDRAC by root/calvin
  3. Click Launch.
  4. Restart ICN and press F2 to log in BIOS.
  5. Click system BIOS.
    Figure 9. System BIOS
  6. Click Boot Settings.
    Figure 10. Boot Settings
  7. Click BIOS Boot Settings.
    Figure 11. BIOS Boot Settings
  8. Make sure Hard Drive Sequence is C. if it is A or B, click Boot Sequence to change.
  9. Go back to System Setup and click Device Settings.
    Figure 12. Device Settings
  10. Click Integrated NIC 1 Port 1.
    Figure 13. Integrated NIC 1 Port 1
  11. Click NIC Configuration.
    Figure 14. NIC Configuration
  12. Make sure Legacy Boot Protocol is set to PXE.
    Figure 15. Legacy Boot Protocol
5. If you get stuck in milestone 4. Check the ICN-Host Ethernet connection.
6. If you get stuck in milestone 5 or 6. Take a screenshot of the iDRAC remote monitor window and ask the engineering team for support.
Note:
If your action still cannot identify the problem, collect the following files from host and ask the engineering team for help:
1. On host, open a terminal.
2. Type ssh sdc@vre. Answer yes if asked for unknown host.
  
  If this step returns an error, it means ICN is offline. In this case, skip step 3 to 5
3. Enter adw2.0 as the password.
4. Type cp ~sdc/ads.log* /usr/g/service/log.
5. Type "exit to exit to host.
6. Send the following files to the engineering team:
  1. /usr/g/service/log/ads.log and /usr/g/service/log/ads.log.X (X means 1-9) collect the log from other servers
  2. /usr/g/service/log/vretpsreset.log
  3. /usr/g/service/log/vreBmcUpCheck.log
  4. /usr/g/service/log/vrecontrol.log
  5. /usr/local/bin/install/log/VRE_Install.log
  6. /usr/local/bin/install/log/VRE_Install_Verbose.log
  7. /usr/local/bin/install/log/icn1_install.log
  8. /usr/g/service/log/vrediags.log
  9. /usr/local/bin/install/log/VRE_Install_Errors.log (if it exists)

Topic ID: id_SL16404753-1365992

Reference

Topic ID: id_SL16404754-1365992

6.1 Dolphin Link LED

Table 3. Dolphin Link LED State and Meaning
Dark	Yellow	Green	Green and Blinking
Power off	Power on and link down	Power on and link up	Power on and data transmitted

Note:

Take a SPRsnap before starting any troubleshooting. Also, if the VRE configuration has failed, take a picture of the ICN Remote console/monitor to capture the errors shown.

Topic ID: id_SL16404757-1365992

6.2 How to Get an IP Address

ETH1 and ETH2 are used on the ICN, they are backup for each other, so they have the same IP address 10.0.1.100. ICN iDRAC has a random IP address 10.0.1.2xx, xx is a random value assigned when doing the VRE configure.

Note: Gen7 and Gen7-DL ICNs do not use ETH2, only one Ethernet cable is required.

You can check the iDRAC address:

Using the ICN front panel: View->iDRAC IP->IPv4->IP->10.0.1.2xx
Figure 18. iDrac Address
Running arp-a. Address will be feedback with the name inc1-bmc.
Figure 19.