Topic ID: id_18718178
Version: 1.0
Date: Sep 20, 2018 2:05:29 PM

DARC Node Commands and Troubleshooting

1 Overview

This document provides manual commands for DARC Node troubleshooting. Not every available command is listed. The first step is to verify communication between the Host and the DARC Node. After verifying that the communication exists to the DARC Node, DIP diagnostics may be performed. Memory size is important and may be checked: however, exact memory values expected are not discussed as this memory size value may change over time and with different types of DARC Nodes offered. It is currently (July, 2006) around 2 GBytes for Main memory.

Application software can be started to verify that Image Generation is running on each IG Node present and configured in the Operator Console. The DARC Node contains the software and hardware that controls the IG Nodes. The Image Generation check assists by determining if the IG Node(s) is up (as we determine up and ready for recon) and under DARC Control. Refer to the System Diagrams to understand connections between Nodes. Refer to the Command List to determine which commands can be performed.

note:

It is important to understand that the DARC Node may not boot up properly due to a Disk Array issue.

note:

For All-In-One Console, There is no DARC node presence. CDIP card and scan data disks are mounted in Host computer.

For AIO console, it’s not necessary to run “rsh darc” for AIO console. All the commands ran on DARC node still can be ran on Host computer, but they are no use since DARC is not presence. Details please see below command list.

1.1 Video Cable

A DARC / IG Node Troubleshooting Video cable is helpful when troubleshooting the Operator Console. Utilize this cable whenever attempting to troubleshoot the DARC and IG Nodes. The cable should not be connected at either end when the system is operating normally, since the video may take over the monitor and cause Customer confusion. Additionally, connecting only the Node end of the cable will result in EMC violations. Sometimes the DARC Node requires a keyboard connection (such as pressing Enter to view DARC output). It is permitted to use the Operator Console keyboard when troubleshooting the DARC Node.

1.2 Repower Process

The IG Nodes require a manual re-power process so each IG Node present and configured (via LFC or a reconfig) can be initialized by the DARC Node. The Application software should be down during IG Node initialization. The DARC must successfully remote shell from the Host (rsh darc). LAN LED activity should be verified on each IG Node to ensure that an Ethernet communication between the DARC and IG Node is present. Then the front panel re-power sequence is performed for any IG Nodes that have been swapped (physical Node swap, or one end of the Ethernet cable between the Nodes), installed, or are having issues. The user should wait for the IG Nodes to successfully boot up. The user should then verify that each IG Node can be communicated with via the DARC Node (rsh ig#). If the remote shell to each IG Node is successful, then the user may decide to verify lhinv Main memory and perform the rac_menu –a diagnostic tool.

1.3 Application Software

Application software may be started and Image Generation should be verified as running. Verify that the prompt returned for each rsh ig# is as expected. If the prompt is not correct it is usually a software issue with the DARC Node, and a DARC software reload is the first step to correct the issue. If image_generation is not running on a specific IG Node, physically swap one end of the Ethernet cable and re-power the IG Nodes with Application software down. If the issue follows the IG Node, then a replacement IG Node may be ordered. Always verify the part number on the replacement hardware upon receipt of a FRU. The labels on shipping boxes can be incorrect (a PAR looks just like a IG Node but does not contain a rac board).

1.4 Host Software

The Host software cannot be loaded (OS and Apps) after the DARC software. If the Host software load was started, the DARC software must follow. Failure to perform procedures in the specified order may result in startup issues. In this case, the ssh command from Host to DARC will fail and Application software will not launch properly. Additionally, if files on the Host for the DARC were not loaded properly, these files will be corrupted when they transfer to the DARC during the software load. If the -reload command fails to correct the problem, perform a LFC.

1.5 Document Updating

Updating documents in a timely manner is sometimes impossible. Always verify that all service packs are loaded after any DARC software reload. The setdate command is performed during an LFC twice. Verify that the Host and DARC date and time are synced up.

2 Command List

The following is a list of manual input commands.

[root@hostname] ifconfig	Allows the user to determine what the Host understands in relation to the Ethernet Address / port for the DARC Node. When a DARC Node is configured /active in the system, look for RUNNING and verify that the address is correct. The connection is always port C on the Host. Even if you see RUNNING, there still could be a swapped connection (like DARC cable connected to Gantry and Gantry connected to DARC). Also, if the Host Apps is not loaded, there will only be the lO present.
[root@hostname] ethtool –p eth# [root@darc] ethtool –p eth# [root@ig#] ethtool –p eth#	The ethtool command allows the user to visually verify that the NIC is working. After the command is entered, the user will visually verify that one LED at the NIC is blinking much faster. To stop the process, press Ctrl-C. This can be performed on the host, DARC, or IG Node.
[root@hostname] ipmitool -I lan -H darc -A NONE chassis power status	The ipmitool may be available. This command allows the user to determine what the Host understands in relation to the power state for the DARC Node.
[root@hostname] service darcpower start	Restarts the DARC Node power. This is useful when the Host is up but the DARC was re-powered and does not seem to come up and ping in a timely manner. note: Not applicable for AIO console.
[root@hostname] service darcpower stop	Turns off the DARC Node power. note: Not applicable for AIO console.
[root@darc] halt	Halts the DARC Node so power can be turned off without losing files on the DARC or hardware. note: Not applicable for AIO console.
[root@hostname] service cliservice start	Restarts or verifies that the cliservice dpcproxy server is running on the Host to allow for Serial Over LAN (SOL). Returning message will be either: “The dpcproxy is running,” or “dpcproxy cliservice has been restarted.” note: Not applicable for AIO console.
telnet localhost 623 Server: darc Username: <Enter> Password: <Enter> Login successful dpccli> exit	Verifies dpcproxy server running / SOL connection. note: Not applicable for AIO console.
ping darc	Press Ctrl-C to stop the ping process. note: Not applicable for AIO console.
rsh darc	OS and Apps software must be loaded on the DARC Node. note: Not applicable for AIO console.
ssh darc	OS and Apps software must be loaded on the Host first, then the DARC Node for this to work. See additional ssh commands. note: Not applicable for AIO console.
[root@hostname] rm -rf /usr/g/ctuser/.ssh/known_hosts [root@hostname] config_sshd [root@darc] service sshd start [root@darc] chkconfig sshd on	Additional ssh related commands.
cd /var/log more messages	A status log on the DARC Node can assist in determining issues (used in conjunction with the log on the Host).
lhinv	Lists devices, memory.
lhinv \| grep Main	Lists memory size (2048).
cat /proc/mdstat
telnet localhost 623 Server: darc Username: <Enter> Password: <Enter> Login successful dpccli> help	Shows the Help menu for commands / options. note: Not applicable for AIO console.
telnet localhost 623 Server: darc Username: <Enter> Password: <Enter> Login successful dpccli> reset ok console	Verifies dpcproxy server running / SOL connection. Resets the DARC Node. note: Not applicable for AIO console.
telnet localhost 623 Server: darc Username: <Enter> Password: <Enter> Login successful dpccli> console	Verifies dpcproxy server running / SOL connection. Lists the DARC Node information and current OS and kernel information. note: Not applicable for AIO console.
telnet localhost 623 Server: darc Username: <Enter> Password: <Enter> Login successful dpccli> sensors -v	Verifies dpcproxy server running / SOL connection. Lists if power is on or off, as well as last or current voltage/temperature/fan information for the DARC Node. note: Not applicable for AIO console.
[root@darc] lsmod \| grep dip	Checks the existence of a DIP in the DARC Node.
[ctuser@darc] DipDiag –x 1,2,3,4,5,6	When this test is performed, ignore the H16 reference “creating H16 dip interface.” Because tests 7 and 8 require the installation of a loopback cable, they are not being performed.
[root@darc] dmidecode \| grep –i version [root@darc] dmidecode \| grep Ver	Shows DARC Node BIOS version as loaded by the vendor. note: Not applicable for AIO console.
uname -a	Displays kernel name, hostname, kernel release and version, machine hardware name, and hardware platform.
free -m	Does not adequately display DARC Node total memory information, but does display used memory.
cat /proc/meminfo	Displays detailed memory information.
df –h	Displays partition information.
cat /proc/devices	Lists all devices detected by the system.
[root@darc] dmidecode \| less	Shows the Motherboard version.
`{ctuser@host}` ipmitool -I lan -H darc -P '' sel	Basic comm. over SOL, and log status.
`{ctuser@host}` ipmitool -H darc -P '' chassis power status	Check chassis power behavior at boot.
`{ctuser@host}` ipmitool -H darc -P '' fru	Lists board components.
`{ctuser@host}` ipmitool -H darc -P '' sdr type temp	Component temperature checks.
`{ctuser@host}` ipmitool -H darc -P '' sel list last 15	Lists all board sensors. Many details. Look for OKs.
`{ctuser@host}` ipmitool -H darc -P '' sdr	Simplified board sensors. Look for OKs. This has replaced sensors –v.
[root@ig#] ps –leaf \| grep –v grep \| grep image_generation	With APPS Up ONLY, provides a clue that recon is up and running on the specific IG Node. (Do not perform this command on the DARC Node – it is not updated properly there.)

3 DIP Diagnostic Test

To verify the DARC Node DIP Board Diagnostics pass, as {ctuser@darc} type:

DipDiag –x 1,2,3,4,5,6

note:

Because tests 7 and 8 require the installation of a loopback cable, they are not being performed.

note:

When this test is performed, ignore the H16 reference “creating H16 dip interface.”

note:

For AIO console, there is no DARC presence. This command can be ran from host computer. The functionality is the same.

12:05:40 opening LINUX device /dev/dip
creating H16 dip interface ***Ignore H16 reference here ***
DIP PN : 2326523 B ***Specific DIP Board and Revision identified here ***
test 1 Passed X-ray Enable Relay
test 2 Passed Test FEC Bit
test 3 Passed Test EDDR Bit
test 4 Passed Test Magic Number Register
fill mem
write to dip took 4.714 sec
verifying memory
read from dip took 19.469 sec
test 5 Passed Memory Test
test 6 Passed Interrupt Test
1 loops 0 failures