Good times, upgrade completed successfully at the end of last week! We were planning to go to 3.1.2 MU3 but when HP got in touch with me on the day of the upgrade they advised 3.1.2 MU5 was available for our system, the requirements and pre-upgrade checks would be identical. MU5 also contains all previous patches and so there would be no need to run the individual patches we were planning. I asked what happened to MU4 as a number seemed to have been skipped in the sequence and found out that this version had been released specifically for an individual customer.
HP were due to perform the actual upgrade. About an hour before the upgrade was due to begin I completed the following pre-upgrade steps and checks.
- Check CPU and port usage is below 50%. statcpu -iter 1, statport -iter 1
- Suspend tasks. setsched -suspend_all, check tasks are suspended as expected showsched
- Check for any DO activity. showtask –active, for any tasks that are active canceltask taskID
- Stop system reporter by visiting the machine its installed on and stopping the Windows service
- Check for any connected users who may be making changes to the system showuserconn
- Check the connectivity of hosts before the upgrade. showhost –pathsum, I took a screenshot of this so I could verify connectivity was as before after the reboot of the first node
- Verify health is OK to do upgradecheckhealth –svc
- Check the system is ready for upgrade checkupgrade
- Plus I suspended all backups so the system was as quiet as possible
Next it was time to hand over to HP. The high level steps and expected timing was as follows:
Updating New codes on the service Processor – 60 Minutes (non-intrusive, can be performed in advance, VIRTUAL ROOM)
Performing the pre-upgrade checks – 30 Minutes (non-intrusive)
Node Upgrade to the new InForm OS – (15 Minutes per node) + 5 pause time = 40 minutes
Performing Post Upgrade Check and patch installations – 30 Minutes (non-intrusive)
Drive cage and Drive Firmware update – 110 Minutes for 7 cages (will be run as a background task & monitored till completion (non-intrusive)
Updating the Service Processor
I connected with the HP representative, who was very helpful via a virtual room. From HP’s FTP site he downloaded the update for the Service Processor and Inform OS. Next he disabled alerting in the Service Processor and chose to run the Service Processor update ISO. This stage was completed quite quickly and next he moved onto loading the Inform OS to the Service Processor.
Health Checks
Next was the health checks, again we moved quite quickly through this as I had ran most of them myself before the upgrade. In addition to the checks I ran above he also ran the following commands:
showsys –d, showversion –a –b, showpd –failed –degraded, shownet, showalert, shownode, showcage, showbattery, showport –d
Node Upgrade
The Inform OS update had already been loaded to the Service Processor so the next stage was to stage the new code to the controllers. This was achieved by connecting through SSH to the Service Processor and running a bunch of commands to transfer the files. When the upgrade was kicked off I took a handful of screenshots to show roughly what happens.
Frist the upgrade goes through some pre-upgrade checks
Next the staged software appears to be transferred so it is ready to be actively installed
Next node 0 reboots and picks up the new code
There is then a pause between reboots of the nodes during which HP will allow you to check all looks OK. I checked our alerting software, checked all VM’s were still online and ran a showhost –pathsum to check that all paths and accessibility to nodes was OK. Before the last node reboots HP are able to roll the upgrade back in an online manner, once the last node has been upgraded this must be done offline. All looked good in my case so I let the upgrade continue.
Post upgrade checks
Once both nodes were upgraded the HP engineer then ran the following checks Shownode, Showversion a – b, and then re-enabled scheduled tasks with Setsched_resume_all
Cage and Drive Firmware Upgrade
Next it was time to upgrade the firmware of the cages, this was kicked off with the command starttask upgradecage –a. To check the task was running the following commands were run: showtask –active and then we were able to drill down for more details by running showtask –d taskID. Progress was also monitored by running showcage. In the screenshot below you can see that about half the cages were done at this stage with half on 320f and half on 320c (you can see this in the RevA and RevB column).
Once the cage firmware upgrade is completed it’s time to upgrade the firmware of the disks. Showpd –failed –degraded, those disks that require a firmware upgrade will show as degraded. To kick off the disk firmware upgrade run admithw. Progress can again be monitored though showtask and re-running Showpd –failed –degraded. To do all the disks and cages in our 7 cage system took about 1.5 hours.
Admithw appears to recreate the default CPG’s. I don’t like these to be there in case someone accidentally adds a VV to them so I did a showcpg to double check they contained no VV’s and then removed them with removecpg
I then ended the remote session with the HP engineer and took the following final steps
- Kicked backups off again
- Restarted system reporter service on system reporter service
- Checked for new alerts showalert –n
- Check the hosts path showhost –pathsum
- Ran a checkhealth
- Checked all VM’s were online without issues
- Checked our monitoring software
- Updated software – CLI and management console. This was again downloaded from HP’s FTP site and was a simple case of just clicking next through the install wizard.
That was it all done with zero down time or issues. For my first 3Par upgrade I was very pleased with how it had all run.
Catch parts one and two in this 3 Par upgrade series if you previously missed them
Published by