ExaCC: Patch Virtual Machine (VM) Cluster OS Image
Overview: Oracle is responsible for performing patches and updates to EXACC infrastructure every quarter (there is no patch in Dec). That includes: ... Read More
ODA Server Patching Failed while Patching Oracle’s Integrated Lights Out Manager
We're glad you're enjoying our content.
Enter your name and email to unlock content for free.
We won't spam you - ever.
Introduction
Recently, I was working on patching ODA X6-2L from 19.6 to 19.10. Server patching failed when trying to upgrade the ilom/bios with an error.
DCS-10001:Internal error encountered: Failed update Ilom using USB.
[root@dbserv]# odacli update-server -v 19.10.0.0.0 -sko
{
"jobId" : "04d92fb2-aaad-4b84-8e96-b0be10e04b1d",
"status" : "Created",
"message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
"reports" : [ ],
"createTimestamp" : "August 30, 2021 11:05:02 AM EDT",
"resourceList" : [ ],
"description" : "Server Patching",
"updatedTime" : "August 30, 2021 11:05:02 AM EDT"
[root@dbserv]# odacli describe-job -i "04d92fb2-aaad-4b84-8e96-b0be10e04b1d"
Job details
----------------------------------------------------------------
ID: 04d92fb2-aaad-4b84-8e96-b0be10e04b1d
Description: Server Patching
Status: Failure
Created: August 30, 2021 11:05:02 AM EDT
Message: DCS-10001:Internal error encountered: Failed update Ilom using USB.
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Server patching August 30, 2021 11:05:09 AM EDT August 30, 2021 11:24:04 AM EDT Failure
Server patching August 30, 2021 11:05:09 AM EDT August 30, 2021 11:24:04 AM EDT Failure
Patch location validation August 30, 2021 11:05:11 AM EDT August 30, 2021 11:05:11 AM EDT Success
dcs-controller upgrade August 30, 2021 11:05:11 AM EDT August 30, 2021 11:05:11 AM EDT Success
Creating repositories using yum August 30, 2021 11:05:11 AM EDT August 30, 2021 11:05:14 AM EDT Success
Updating YumPluginVersionLock rpm August 30, 2021 11:05:14 AM EDT August 30, 2021 11:05:14 AM EDT Success
Applying OS Patches August 30, 2021 11:05:14 AM EDT August 30, 2021 11:18:11 AM EDT Success
Creating repositories using yum August 30, 2021 11:18:11 AM EDT August 30, 2021 11:18:12 AM EDT Success
Applying HMP Patches August 30, 2021 11:18:12 AM EDT August 30, 2021 11:18:31 AM EDT Success
Client root Set up August 30, 2021 11:18:31 AM EDT August 30, 2021 11:18:35 AM EDT Success
Client grid Set up August 30, 2021 11:18:35 AM EDT August 30, 2021 11:18:51 AM EDT Success
Patch location validation August 30, 2021 11:18:51 AM EDT August 30, 2021 11:18:51 AM EDT Success
oda-hw-mgmt upgrade August 30, 2021 11:18:51 AM EDT August 30, 2021 11:19:23 AM EDT Success
OSS Patching August 30, 2021 11:19:23 AM EDT August 30, 2021 11:19:24 AM EDT Success
Applying Firmware Disk Patches August 30, 2021 11:19:24 AM EDT August 30, 2021 11:19:43 AM EDT Success
Applying Firmware Controller Patches August 30, 2021 11:19:43 AM EDT August 30, 2021 11:19:56 AM EDT Success
Server patching August 30, 2021 11:19:56 AM EDT August 30, 2021 11:24:04 AM EDT Failure
Checking Ilom patch Version August 30, 2021 11:19:58 AM EDT August 30, 2021 11:20:00 AM EDT Success
Patch location validation August 30, 2021 11:20:00 AM EDT August 30, 2021 11:20:00 AM EDT Success
Save password in Wallet August 30, 2021 11:20:00 AM EDT August 30, 2021 11:20:00 AM EDT Success
Server patching August 30, 2021 11:20:00 AM EDT August 30, 2021 11:24:04 AM EDT Failure
Apply Ilom patch August 30, 2021 11:20:00 AM EDT August 30, 2021 11:24:04 AM EDT Failure
This is a known issue addressed by Oracle support in (Doc ID 2733383.1). It applies to the below Oracle Database Appliance Hardware:
Oracle Database Appliance X7-2 S/M Hardware - Version All Versions to All Versions [Release All Releases] Oracle Database Appliance X8-2S/M Hardware - Version All Versions to All Versions [Release All Releases] Oracle Database Appliance X7-2 HA Hardware - Version All Versions to All Versions [Release All Releases] Oracle Database Appliance X8-2HA Hardware - Version All Versions to All Versions [Release All Releases] Oracle Database Appliance X6-2 S/M/L Hardware - Version All Versions to All Versions [Release All Releases]
Cause
There is a firewall in between the Oracle ILOM (Intergrated Lights Out Manager) network and the host network that is blocking port 623 (IPMI over UDP).
Recommendation
I highly recommend confirming the following two checks before you start server patching. You may add the below checks to your ODA pre-checks.
1- There is NO firewall in between the ILOM network and the host network that is blocking port 623.
You may add the below command to your ODA pre-checks to check if there is a network problem.
--- Run the command from ODA server. The command will prompt to enter ILOM root password.
ipmitool -I lanplus -H
-U root chassis status
Example:
[root@dbserv]# ipmitool -I lanplus -H dbserv-ilo.local -U root chassis status Password: Error: Unable to establish IPMI v2 / RMCP+ session >>> Above error message indicates there is a firewall in between the ILOM network and the host network that is blocking port 623. [root@dbserv]# ipmitool -I lanplus -H dbserv-ilo.local -U root chassis status Password: System Power : on Power Overload : false Power Interlock : inactive Main Power Fault : false Power Control Fault : false Power Restore Policy : always-off Last Power Event : Chassis Intrusion : inactive Front-Panel Lockout : inactive Drive Fault : false Cooling/Fan Fault : false >>> There is NO a firewall in between the ILOM network and the host network that is blocking port 623.
2- Make sure ILOM ipmi setting (/SP/services/ipmi/ipmi v2_0_sessions) is enabled.
[root@dbserv]# ssh dbserv-ilo.local -> show /SP/services/ipmi /SP/services/ipmi Targets: Properties: servicestate = enabled tls_sessions = enabled v1_5_sessions = disabled v2_0_sessions = enabled <<< this should be enabled --- If v2_0_sessions shows disabled, then you can enable it using -> set /SP/services/ipmi v2_0_sessions=enabled Set 'v2_0_sessions' to 'enabled'
More from this Author
ExaCC: Adding SSH Keys to Exadata Cloud @ Customer
Overview: You can connect to the virtual machines in an Exadata Database Service on the Cloud @ Customer system by using a Secure Shell (SSH) ... Read More