3 Par OS Upgrade – Part 3 Upgrade Checklist

As I went through the HP 3Par 3.1.3 Inform OS upgrade process I created a checklist of the key steps for quick reference. I have shared it below, but please double check this list against the official HP documentation to ensure all points relevant to your environment and version are covered. Any comments or additions you would like to see on the checklist get in touch through Twitter

A) Pre-Upgrade Checks

1. Hosts

  • Check compatibility of components. You can complete this step yourself by using SPOCK or send off a completed host worksheet to HP who will verify the compatibility for you.
  • Apply Windows multipath patches :
    • KB275704 needs to be applied to Windows Server 2008, and 2008 R2
    • KB2821052 needs to be applied to Windows Server 2012 and Windows Server 2008 R2
  • Check MPIO is set to round robin on Windows 2008 servers using the native MPIO driver. The process using psexec is covered in this post under point 2.

2. SAN

  • Load must be below 50% at the time of upgrade. Run statcpu and statport and check utilisation is below 50%
  • Check current version is on upgrade path. showversion
  • Check the system is healthy. checkhealth
  • There are some version specific checks when upgrading from 2.3.1 or 3.1.1 to 3.1.2 check HP 3par OS 3.1.X Upgrade Pre-planning Recommendations for more information

B) Day of upgrade

1. Pre-Upgrade

  • Load must be below 50% at the time of upgrade. Run statcpu and statport and check utilisation is below 50%
  • If possible suspended tasks that would cause a heavy load eg backups
  • Stop AO, DO, and RC tasks before the upgrade. Plus stop anything else that interacts with the system such as scripts
  • Stop system reporter. Stop windows service – 3Par System Reporter Sampler Service
  • Showtask – and check for any DO activity
  • Stop any software that directly interact with the SAN e.g. Recovery Manager, scripts etc.
  • Stop RC
  • Check no one is logged in showuserconn
  • Checkhealth again. checkhealth -svc detail performs a full health check on the system
  • checkupgrade to verify if the system is ready to undergo an online upgrade.
  • Check current connectivity status of the hosts showhost -pathsum
  • All upgrades until after 3.1.3 are performed by HP, so next HP should install the update for you.

2. Post Upgrade

  • Check version is as expected showversion –a -b
  • Verify connectivity of hosts showhost –pathsum, plus check monitoring software attached hosts etc. for any potential issues
  • Restart suspended tasks setsched -resume_all . Check schedules running again showsched
  • Change Windows 2008 and 2012 to host persona 15 sethost –persona 15 servername
  • Remove default CPG. removecpg
  • Restart system reporter windows service (3Par System Reporter Sampler Service) plus any other software you stopped such as Recovery Manager
  • Resume backups and any other systems you suspended to reduce the load on the SAN
  • Check for new CLI and management console from FTP supplied by HP or software depot.  Check compatibility in SPOCK
  • Plus if you use any other 3Par software such as Recovery manager VSS provider etc check in the release notes for the latest version and then download via supplied FTP link from HP or from the software depot

 

Catch the full series of posts in which I ran through the upgrade process in detail: Part 1 

Part 2

3ParDude

 

3Par OS Upgrade 3.1.3 – Part 2 Upgrade Day

Upgrade day was here I packed my lucky rabbit foot and headed into the office. Please note the screen shots below were taken during the last upgraded I posted about as I forgot to take any this time, the process was almost identical but obviously the version numbers you are going to see will differ from those in the screenshots. About an hour before the upgrade was due to begin I completed the following pre-upgrade steps and checks.

Pre-upgrade

  • Check CPU and port usage is below 50%. statcpu -iter 1, statport -iter 1
  • Suspend tasks. setsched -suspend_all, check tasks are suspended as expected showsched
  • Check for any DO activity. showtask –active, for any tasks that are active canceltask taskID
  • Stop system reporter by visiting the machine its installed on and stopping the Windows service

  • Check for any connected users who may be making changes to the system showuserconn
  • Check the connectivity of hosts before the upgrade. showhost –pathsum, I took a screenshot of this so I could verify connectivity was as before after the reboot of the first node
  • Verify health is OK to do upgradecheckhealth –svc
  • Check the system is ready for upgrade checkupgrade
  • Plus I suspended all backups so the system was as quiet as possible

Timings

Next it was time to hand over to HP. The high level steps and expected timing was as follows:

Updating New codes on the service Processor – 60 Minutes (non-intrusive, can be performed in advance)

Performing the pre-upgrade checks – 30 Minutes (non-intrusive)

Node Upgrade to the new InForm OS – (15 Minutes per node) + 5 pause time =  40  minutes

Performing Post Upgrade Check and patch installations – 30 Minutes (non-intrusive)

Drive cage and Drive Firmware update – 110 Minutes for 7 cages (will be run as a background task & monitored till completion (non-intrusive)

Updating the Service Processor

The HP engineer first downloaded the update for the Service Processor and Inform OS. Next he disabled alerting in the Service Processor and chose to run the Service Processor update ISO. This stage was completed quite quickly and next he moved onto loading the Inform OS to the Service Processor.

Health Checks

Next was the health checks, again we moved quite quickly through this as I had ran most of them myself before the upgrade. In addition to the checks I ran above he also ran the following commands:

showsys –d, showversion –a –b, showpd –failed –degraded, shownet, showalert, shownode, showcage, showbattery, showport –d

Node Upgrade

The Inform OS update had already been loaded to the Service Processor so the next stage was to stage the new code to the controllers. This was achieved by connecting through SSH to the Service Processor and running a bunch of commands to transfer the files. When the upgrade was kicked off I took a handful of screenshots to show roughly what happens.

Frist the upgrade goes through some pre-upgrade checks

Next the staged software appears to be transferred so it is ready to be actively installed

Next node 0 reboots and picks up the new code

There is then a pause between reboots of the nodes during which HP will allow you to check all looks OK. I checked our alerting software, checked all VM’s were still online and ran a showhost –pathsum to check that all paths and accessibility to nodes was OK. Before the last node reboots HP are able to roll the upgrade back in an online manner, once the last node has been upgraded this must be done offline. All looked good in my case so I let the upgrade continue.

Post upgrade checks

Once both nodes were upgraded the HP engineer then ran the following checks shownode, showversion a – b, and then re-enabled scheduled tasks with setsched -resume_all

Cage and Drive Firmware Upgrade

Next it was time to upgrade the firmware of the cages, this was kicked off with the command starttask upgradecage –a. To check the task was running the following commands were run: showtask –active and then we were able to drill down for more details by running showtask –d taskID. Progress was also monitored by running showcage. In the screenshot below you can see that about half the cages were done at this stage with half on 320f and half on 320c (you can see this in the RevA and RevB column).

Once the cage firmware upgrade is completed it’s time to upgrade the firmware of the disks. Showpd –failed –degraded, those disks that require a firmware upgrade will show as degraded. To kick off the disk firmware upgrade run admithw. Progress can again be monitored though showtask and re-running Showpd –failed –degraded. To do all the disks and cages in our 7 cage system took about 1.5 hours.

Admithw appears to recreate the default CPG’s. I don’t like these to be there in case someone accidentally adds a VV to them so I did a showcpg to double check they contained no VV’s and then removed them with removecpg

Final Tasks

I then ended the remote session with the HP engineer and next set about updating all the attached Windows 2008 and 2012 Servers to host persona 15. Setting the host persona to 15 presents LUN’s in the manner which Windows expects and the good news is this means the application of KB2849097 is no longer part of the upgrade process. To set each Windows host to persona 15 I ran the following command:

sethost –persona 15 servername

Once this was done I set about the following final tasks:

  • Kicked backups off again
  • Restarted system reporter service on system reporter service
  • Checked for new alerts showalert –n
  • Check the hosts path showhost –pathsum
  • Ran a checkhealth
  • Checked all VM’s were online without issues
  • Checked our monitoring software
  • Updated software – CLI and management console. This was again downloaded from HP’s FTP site and was a simple case of just clicking next through the install wizard.

 

That’s was it successfully onto 3.1.3, interestingly once you are on 3.1.3 you can perform your own upgrades without HP having to install the software for you. This new process is explained in this excellent post by Bart Heungens.

Further Reading:

HP 3PAR OS Upgrade Pre-planning Guide

HP 3Par 3.1.3 Release Notes

HP 3PAR Windows Server 2012 and Windows Server 2008 Implementation Guide

 

If you missed the first part of the series catch it here.

 

3ParDude

3Par OS Upgrade 3.1.3 – Part 1: Pre-Upgrade Planning

I have been updating one of our 3Par systems to 3Par OS 3.1.3 MU1. This short series of posts will cover the process to successfully upgrade a 3Par system to 3.1.3

There are a number of pre-upgrade tasks that need completing before an upgrade can begin these are listed below under two categories host based tasks and SAN based tasks.

 A) Host Tasks

1 Check compatibility of components – This is to ensure that you are running a tested configuration of components that have been proven to work together by HP. There are 2 ways to go about this. Firstly you can use SPOCK.  This site contains all the compatibility information you will need to complete your own checks. Or you can complete a host worksheet and return it to HP who will then verify the compatibility of all your components and firmware versions. The components you need to check are fairly standard to any SAN upgrade – Server OS, multipath software, HBA’s and fabric switch firmware versions. I was lucky I didn’t need to update any firmware versions in order to proceed.

2 Check load balancing is set to round robin – This is a requirement for any Windows servers running 2008 and using the native MS MPIO driver. I used a script to make this step quicker.

I used the Microsoft command line application mpclaim to view the multipath configuration. Specifically I ran mpclaim –s –d from the command line.

To run the command line on multiple servers remotely without having to logon I used psexec. You can download it from here . Here is an excellent article on how to use it: psexec guide

In this case I used it in the following way

A Choose the server you want to run the script from and create a folder on it called C:scripts. Copy psexec to this folder

B in C:scripts create a file called 3par_servers.txt. Populate this with a list of the servers you wish to check for multipath configuration

C Also in C:scripts create a batch fie called mpclaim.bat and enter the following command line into it mpclaim –s –d

D Finally, open a command line from the machine you wish to run the script on, change directory to C:scripts and then enter c:Scripts>psexec -c -f @C:scripts3par_servers.txt C:scriptsmpclaim.bat

E You should then see the window populate with the information you require. An example of the output is below:

\Server1

C:Windowssystem32>mpclaim -s -d

For more information about a particular disk, use ‘mpclaim -s -d #’ where # is the MPIO disk number.MPIO Disk   System Disk LB Policy   DSM Name

——————————————————————————-

MPIO Disk5   Disk 6       RR           Microsoft DSM

MPIO Disk4   Disk 5       RR           Microsoft DSM

MPIO Disk3   Disk 4       RR           Microsoft DSM

MPIO Disk2   Disk 3       RR           Microsoft DSM

MPIO Disk1   Disk 2       RR           Microsoft DSM

 

Check LB policy appears as RR for all volumes.

 

3 Windows MPIO Patches. Two patches needed applying to each Windows server running MS MPIO we had attached to the 3Par. The patches which are to ensure correct functioning of MS MPIO, in my case we pushed these out using WSUS.

  • KB275704 needs to be applied to Windows Server 2008, and 2008 R2
  • KB2821052 needs to be applied to Windows Server 2012 and Windows Server 2008 R2

B) SAN Tasks

The SAN tasks were much quicker than the host checks I just needed to check the following three things

 

1 Performance – The 3Par needs to be upgraded at a time of low utilisation, this is defined as the load on the CPU’s and ports being below 50% utilised.

  • Check CPU utilisation. statcpu -t -d 15 -iter 1 Must be less than 50%
  • Check port utilisation. statport -d 15 -iter 1 -ni

For each port, the average I/O, Kbytes per sec, needs to be less than 50%. The official documentation showed the following figures:

200,000 Kbytes per sec, for 4 Gb connections

400,000 Kbytes per sec, for 8 Gb connections ie less than 50%

 

2 Version Check – I just ran checkversion from the CLI and then checked in HP 3PAR Operating System Upgrade Pre-Planning Guide that the current version was on the upgrade path and if there were any special considerations .

 

3 Health Check – The system needs to be in a healthy state to allow an upgrade to go ahead so I ran a checkhealth from the command line and all came back healthy. If you do find any items flagged by the checkhealth you will need fix these items before proceeding with the upgrade.

That was all the pre upgrade tasks done so I went ahead booked in the upgrade with HP as only from releases post 3.1.3 onwards will the systems be customer upgradable.

 

Follow 3ParDude on Twitter here