3PAR Remote Copy Failover

3PAR storage Remote Copy Failover, Recovery & Restore using IMC

Today we have a highly detailed guest post covering the 3PAR remote copy failover and recovery process. If you fancy writing a guest post check out this post.

About the author

Written by Deepak Garg, a technocrat with over 10+ years of experience in storage administration and with technical expertise in managing the technical support functions. Expert on HPE 3PAR, HPE StoreOnce, HP EVA Storage, HP MSA/P2000, LHN and many other HPE products


Unidirectional 1-to-1 Configuration

A unidirectional 1-to-1 remote-copy configuration is composed of one remote-copy pair. Each storage system in the pair plays only one role: one system is the primary system, and one system is the backup system.

Unidirectional remote copy rellationship

Bidirectional 1-to-1 Configuration

A bidirectional 1-to-1 remote-copy configuration is composed of one remote-copy pair.Each storage system in the pair functions as both the primary and backup system and each system contains both primary and secondary volume groups.

Each system provides backup for the other, according to the direction of replication specified for each volume group. Below figure illustrates a bidirectional 1-to-1 remote-copy configuration.

System Information during Failover for 1-to-1 Configurations

Asynchronous Periodic Mode for 1-to-1 Configurations

When a failure occurs on systems with asynchronous periodic mode volume groups such that all links between the systems are broken, the following actions occur:

  • After 60 seconds, the system marks the sending links as down.
  • After another 200 seconds, the system marks the targets as failed

Recovering from Disaster for 1-to-1 Configurations

For the remote copy failover, recover and restore operation there are defined steps involved which need to followed as below.

  1. Failover
  2. Recovery
  3. Restore


a) Stop Remote Copy Group    

Reverse the role of all volume groups on the system that is still in normal operation (the failover system). Here we will consider that system “System-A” is in failed state or having some planned maintenance activity. Below screen shot shows where we want to do the failover of replication group “DRTEST” from primary system System-A to System-B

During this first of all we have to stop the remote copy replication group “DRTEST” to do the testing.

From the 3PAR management console, click on remote copy from the left pane. After this click on the groups and verify the replication group listed on the right side as shown below in picture.

View 3PAr remote copy groups

We will be doing the failover of “DRTEST” replication group. So right click on “DRTEST” and click Stop Remote Copy Group(s) as shown below in picture:

Stopping the remote copy groups

Once clicked on Stop Remote Copy Group(s) option a pop-up window as shown below will open which will state that remote copy mirroring will be stopped. Click on OK to proceed.

Confirm groups are stoped

Once we stop Remote Copy Group, replication from source to target will be stopped.

b) Failover Remote Copy Group   

Once the remote copy group is in failed state then we will see that Group state is a stopped state. Now we have to right click on the failed remote copy group (in our case we are preparing example with remote copy group “DRTEST”) and click “Failover Remote Copy Group (s)..” option as shown below

Now one pop-up window will open click OK and after that one more pop-up will open and click yes here

Note:- After this failover will be executed for the selected group. The failover operation changes the role of secondary groups on the backup system (i.e. System-B) from “Secondary” to “Primary-Rev”. Any LUNs (VLUNs) associated with the volumes in the selected groups become writable by hosts connected to the backup system (backup system is System-B in our case). These VLUNs will be in writable permission from both side 3PAR so we must ensure that no write operation on VLUN from primary site.

As per picture below, click YES to continue and the failover operation will start.confirm fsailoverNow there should be Group state as “stopped” as per below picture. Replication status in the table will be as “Stopped” and Backup system (i.e. System-B) will show its role as “Primary-Rev” with DR state as “Failover”.

This completes all failover steps.


a) Description of Recovery Option    

Recover option will recover the failed system (in our case failed system is System-A). When both systems in the remote-copy pair are ready to resume normal operation, reverse the natural direction of data flow and resynchronize the systems.

At this time we have to right click on the replication group which was failed earlier (in our case it is “DRTEST”) and click on “Recover Remote Copy Group (s).. This will copy data / initiates reverse replication and synchronize the delta changes from the reversed volume groups on the failover system (System-B) to the corresponding volume groups on the recovered system (System-A).

Once executed the role of the remote copy group on the source system (System-A) becomes “Secondary-Rev”. Also, any LUN associated with the volumes in the selected groups become non-writable on the source system (System-A).

Note: – The recovery operation can be executed only on groups that have successfully completed the failover option.

b) Recover Remote Copy Group  

Below two pictures shows the process on how to put a replication group in recover state and then click YES in pop-up window. Right click on the replication group which was failover and click on the option “Recover Remote Copy Group(s)”

Recovering the remote copy groups

confirm recover operationOnce clicked on OK prompt, there will be one more pop-up, press yes now. Important to note that VLUNs will not be writable now on source system “System-A” and it will be in writable mode on backup system “System-B”.

Now, we should be able to see Source role for “DRTEST” on source system should be “Secondary-Rev” and “Primary-Rev” on the Backup system role as shown in below picture. Now the DR state will be in “Recover” statchecking status of recoveryc) Verification:

1 Issue the showrcopy command from the CLI on the failover system (System-B).Verify the following:

  • The Status of the target system (recovered system – “System-A”) is ready
  • The SyncStatus of all volumes in the Primary-Rev volume groups is Syncing.
  • The Status of all sending links is Up,

2 Issue the showrcopy command from the CLI on the recovered system (System-A) Verify the following:

  • The Status of the target system (System-B) is ready
  • The Status of all sending links is Up
  • The Role of the synchronizing volume groups is Secondary-Rev
  • The SyncStatus of all volumes in the Secondary-Rev volume groups is Syncing


a) Description 

Once the recovery is completed for the remote copy group then we must restore back the actual state of replication group. I.e. bring the replication group (in our case it is “DRTEST”) back to “System-A” as primary role and again replication from the primary system (System-A) to backup system (System-B).

b) Procedure to Implement

For doing this right click on the replication group and click “Restore Remote Copy Group(s)..”.This restore operation restores replication for the selected Remote Copy group to a pre-failover state after the recovery operation has been completed. Once the Restore operation is executed the role of the remote copy group on source system (System-A) will be “primary” and the Remote copy group on the backup system will be “Secondary”. Also, any LUNs associated with the volumes in the selected groups become writable by hosts connected to the source system (i.e. System-A) and become non-writable by hosts connected to the backup system (i.e. System-B)

And once we will click on the OK in the above pasted picture, then there will be one more pop-up to click Yes or No, we have to click “Yes” to complete the process.

Now the system should be able to see the status of restored replication group as it was in actual state. Below picture shows the actual status with System-A system as Primary and normal state and replication pointing from System-A to System-B.

3par replication status















2040 firmware version check

HPE MSA Upgrade

Today we have a guest post covering the HPE MSA firmware upgrade process in excellent detail. If you fancy writing a guest post check out this post.


Hi Guys,

First, let me introduce myself:

My name is Armin Kerl, I have been a HPE Master ASE for 20 years in Server & Storage starting at 1992 with Compaq Server Technology. I’m living with my Wife and Son in Germany and working for the last 15 Years for SWS Computersysteme AG (Member of ACP Group). In 2014, I got a nomination to the “HPE Partner Ambassador Program”. My primary focus is HPE Storage and Server as Senior Consultant. After Years of HP EVA Systems, today I am installing and managing around 50 3PAR Systems at 20 customer Sites. You can learn more about me by connecting on LinkedIn.

HPE MSA Upgrade Background

Now let me share some tips for updating your HPE MSA & P2000 Storage Systems. HPE has released a new Microsite for “HPE MSA Storage Firmware & Release Notes” http://www.hpe.com/storage/MSAFirmware Well done HPE, I wish we could of had this for the last 20 Years. From here you can find and download all the necessary firmware for the MSA.

Before starting the Update, remember, it is always a good Idea to have a Backup. I have had issues where multiple drives have a malfunctioned after the update and data was put at risk. In addition, you should have a support contract if the system production, best is 4h response Time. I have had experience of where one controller goes down during a firmware update.

All the Issues I have had were with P2000 systems, until now I have seen no firmware update problem with MSA204x or MSA205x Systems and have read that the newer MSA Storages HPE updates are improved.

As today, you have still search and download for every Drive Type the Firmware Version. If you scroll down the new MicroSite you can find the “HPE MSA Hard Disk Drives – Hard Drive Model Number Matrix”.

For the P2000 Systems HPE Support told, that is best practice to have no IOs (or minimum IOs) during the Update. Even if Online Update is supported. I have had many Issus by Updating and always shut down all Systems, if in Production.

For MSA204x and MSA205x, I decide not to Shut Down all Systems, but shutdown databases (Exchange, SQL, Oracle,…) and doing it to Low IO Times. So long, I have no trouble.

Now here is my Action Plan:

1 Prepare the System

Connect via putty:

1a Check if System is OK

#show version

#show network

#show system

System Information


System Name: P2k05

System Contact: xxxxxxxxx

System Location: xxxxxxxxxxxxxx

System Information: P2000 G3

Midplane Serial Number: xxxxxxxxxxxx

Vendor Name: HP StorageWorks

Product ID: P2000 G3 FC

Product Brand: MSA Storage

SCSI Vendor ID: HP

SCSI Product ID: P2000 G3 FC

Enclosure Count: 2

Health: OK

Note health shows as OK

1b Check if all VDisks are “Fault Tolerance Online” (FOTL) and have no Job:

Output from vdisk FTOL commandIf Disk Scrub (VRSC) active, then abort

#abort scrub vdisk BKRSDS1_RG2-0

Info: Scrub was aborted on vdisk BKRSDS1_RG2-0. (BKRSDS1_RG2-0)

Success: Command completed successfully. (2014-10-15 09:53:46)

Another example

vdisk is initalising

This Disk is Initializing:

If other jobs active or volumes are not FTOL than, cancel update and schedule new, if system is OK then continue

1c Check if disks are OK:

show disks output

Example:This disk is initializing:

1d     Check if unwritten Data is in Cache

# show unwritable-cache

Percent not written Cache in Controller A: 0

Percent not written Cache in Controller  B: 0

If there is:

# clear cache volume (Name)

1e Disable Partner Firmware Update

# set advanced-settings partner-firmware-upgrade off

1f you powered off hosts and check, there are no IOPS. Below we can see IOPs are still present, we would like to see this at zero

This is all the pre-upgrade steps

2 Firmware Update Controller

Download ” Online ROM Flash Component for Windows – HP P2000 / MSA 1040/2040 Storage Arrays”.


Always use the Windows EXE for Firmware Update; this is using special Script’s.

The Firmware Update Tool connects to the Controller over IP and takes aprox 30 min. per Controller.

Running the Windows online ROM flash component for MSA firmware update

3 After Update check if it is OK

After the Update login to the Web GUI go to Tools > Firmware Update. Check, if both Controllers have identical Firmware versions.Often on P2000 Systems with older firmware sometimes, parts of Firmware were not updated.

This is P2000 with c2 GUI:

Checking firmware versions, post upgrade

checking firmware version for the MSA controller

And here in the newer v3 GUI:

MSA2040 update firmware

2040 firmware version check

4 Firmware Update Disk Shelf

Download the ” Online ROM Flash Component for Windows – HP MSA 2040/P2000 Dual I/O LFF or SFF Drive Enclosure”.

This Process is only OFFLINE supported; meaning no IOPS. Each IO Module need around 15 minutes.

update expansion modules

5       Firmware Update Disks

Login to the Web GUI and go to Tools > Firmware Update

Look at the Disk Types, than download the „Firmware Flash Component for Windows – HP MSA 1040/2040, P2000 G3 and MSA2000 …..”, if there is some newer.The Disk Firmware Update Tool connects over the IP to the Controller and does an update of the disk.

If different Disk Types are installed, than repeat this for every type. Per Disk it will need about 5 Minutes

6  Check if Disk Update is OK

update disk drives firmware




Video demonstrating how to quickly edit the Windows host file

Edit the Windows 10 hosts file – IN 60 SECONDS

Editing the Windows hosts file has changed a little in Windows 10.  In this video I show you how to edit the hosts file in 60 seconds! This method gets over the issues when you can’t save the hosts file in windows 10.  This method will also work for Windows Server 2012 and Server 2016. Please like the video if you enjoy it.  The full list of steps is documented below.

I have detailed the steps below for your convenience.  Don’t forget to subscribe to the channel and visit the blog at https://www.d8taDude.com for more great tech content
1 Open file explorer and browse to the hosts file location C:\Windows\System32\drivers\etc\hosts
2 Copy the hosts file to your desktop
3 Open the hosts file on your desktop and add the entry you wish to in the format:
Hostname  IP Address
4 Save the updated hosts file
5 Copy the host file back to C:\Windows\System32\drivers\etc\hosts
6 Choose yes to overwrite file
7 Choose yes to the Windows 10 administrative control prompt

Don’t forget to subscribe to the YouTube channel