Upgrade day was here I packed my lucky rabbit foot and headed into the office. Please note the screen shots below were taken during the last upgraded I posted about as I forgot to take any this time, the process was almost identical but obviously the version numbers you are going to see will differ from those in the screenshots. About an hour before the upgrade was due to begin I completed the following pre-upgrade steps and checks.
- Check CPU and port usage is below 50%. statcpu -iter 1, statport -iter 1
- Suspend tasks. setsched -suspend_all, check tasks are suspended as expected showsched
- Check for any DO activity. showtask –active, for any tasks that are active canceltask taskID
- Stop system reporter by visiting the machine its installed on and stopping the Windows service
- Check for any connected users who may be making changes to the system showuserconn
- Check the connectivity of hosts before the upgrade. showhost –pathsum, I took a screenshot of this so I could verify connectivity was as before after the reboot of the first node
- Verify health is OK to do upgradecheckhealth –svc
- Check the system is ready for upgrade checkupgrade
- Plus I suspended all backups so the system was as quiet as possible
Next it was time to hand over to HP. The high level steps and expected timing was as follows:
Updating New codes on the service Processor – 60 Minutes (non-intrusive, can be performed in advance)
Performing the pre-upgrade checks – 30 Minutes (non-intrusive)
Node Upgrade to the new InForm OS – (15 Minutes per node) + 5 pause time = 40 minutes
Performing Post Upgrade Check and patch installations – 30 Minutes (non-intrusive)
Drive cage and Drive Firmware update – 110 Minutes for 7 cages (will be run as a background task & monitored till completion (non-intrusive)
Updating the Service Processor
The HP engineer first downloaded the update for the Service Processor and Inform OS. Next he disabled alerting in the Service Processor and chose to run the Service Processor update ISO. This stage was completed quite quickly and next he moved onto loading the Inform OS to the Service Processor.
Next was the health checks, again we moved quite quickly through this as I had ran most of them myself before the upgrade. In addition to the checks I ran above he also ran the following commands:
showsys –d, showversion –a –b, showpd –failed –degraded, shownet, showalert, shownode, showcage, showbattery, showport –d
The Inform OS update had already been loaded to the Service Processor so the next stage was to stage the new code to the controllers. This was achieved by connecting through SSH to the Service Processor and running a bunch of commands to transfer the files. When the upgrade was kicked off I took a handful of screenshots to show roughly what happens.
Frist the upgrade goes through some pre-upgrade checks
Next the staged software appears to be transferred so it is ready to be actively installed
Next node 0 reboots and picks up the new code
There is then a pause between reboots of the nodes during which HP will allow you to check all looks OK. I checked our alerting software, checked all VM’s were still online and ran a showhost –pathsum to check that all paths and accessibility to nodes was OK. Before the last node reboots HP are able to roll the upgrade back in an online manner, once the last node has been upgraded this must be done offline. All looked good in my case so I let the upgrade continue.
Post upgrade checks
Once both nodes were upgraded the HP engineer then ran the following checks shownode, showversion a – b, and then re-enabled scheduled tasks with setsched -resume_all
Cage and Drive Firmware Upgrade
Next it was time to upgrade the firmware of the cages, this was kicked off with the command starttask upgradecage –a. To check the task was running the following commands were run: showtask –active and then we were able to drill down for more details by running showtask –d taskID. Progress was also monitored by running showcage. In the screenshot below you can see that about half the cages were done at this stage with half on 320f and half on 320c (you can see this in the RevA and RevB column).
Once the cage firmware upgrade is completed it’s time to upgrade the firmware of the disks. Showpd –failed –degraded, those disks that require a firmware upgrade will show as degraded. To kick off the disk firmware upgrade run admithw. Progress can again be monitored though showtask and re-running Showpd –failed –degraded. To do all the disks and cages in our 7 cage system took about 1.5 hours.
Admithw appears to recreate the default CPG’s. I don’t like these to be there in case someone accidentally adds a VV to them so I did a showcpg to double check they contained no VV’s and then removed them with removecpg
I then ended the remote session with the HP engineer and next set about updating all the attached Windows 2008 and 2012 Servers to host persona 15. Setting the host persona to 15 presents LUN’s in the manner which Windows expects and the good news is this means the application of KB2849097 is no longer part of the upgrade process. To set each Windows host to persona 15 I ran the following command:
sethost –persona 15 servername
Once this was done I set about the following final tasks:
- Kicked backups off again
- Restarted system reporter service on system reporter service
- Checked for new alerts showalert –n
- Check the hosts path showhost –pathsum
- Ran a checkhealth
- Checked all VM’s were online without issues
- Checked our monitoring software
- Updated software – CLI and management console. This was again downloaded from HP’s FTP site and was a simple case of just clicking next through the install wizard.
That’s was it successfully onto 3.1.3, interestingly once you are on 3.1.3 you can perform your own upgrades without HP having to install the software for you. This new process is explained in this excellent post by Bart Heungens.
If you missed the first part of the series catch it here.