3PAR Remote Copy Failover

3PAR storage Remote Copy Failover, Recovery & Restore using IMC

Today we have a highly detailed guest post covering the 3PAR remote copy failover and recovery process. If you fancy writing a guest post and being in with the chance of winning a $100 gift voucher, check out this post.

About the author

Written by Deepak Garg, a technocrat with over 10+ years of experience in storage administration and with technical expertise in managing the technical support functions. Expert on HPE 3PAR, HPE StoreOnce, HP EVA Storage, HP MSA/P2000, LHN and many other HPE products


Unidirectional 1-to-1 Configuration

A unidirectional 1-to-1 remote-copy configuration is composed of one remote-copy pair. Each storage system in the pair plays only one role: one system is the primary system, and one system is the backup system.

Unidirectional remote copy rellationship

Bidirectional 1-to-1 Configuration

A bidirectional 1-to-1 remote-copy configuration is composed of one remote-copy pair.Each storage system in the pair functions as both the primary and backup system and each system contains both primary and secondary volume groups.

Each system provides backup for the other, according to the direction of replication specified for each volume group. Below figure illustrates a bidirectional 1-to-1 remote-copy configuration.

System Information during Failover for 1-to-1 Configurations

Asynchronous Periodic Mode for 1-to-1 Configurations

When a failure occurs on systems with asynchronous periodic mode volume groups such that all links between the systems are broken, the following actions occur:

  • After 60 seconds, the system marks the sending links as down.
  • After another 200 seconds, the system marks the targets as failed

Recovering from Disaster for 1-to-1 Configurations

For the remote copy failover, recover and restore operation there are defined steps involved which need to followed as below.

  1. Failover
  2. Recovery
  3. Restore


a) Stop Remote Copy Group    

Reverse the role of all volume groups on the system that is still in normal operation (the failover system). Here we will consider that system “System-A” is in failed state or having some planned maintenance activity. Below screen shot shows where we want to do the failover of replication group “DRTEST” from primary system System-A to System-B

During this first of all we have to stop the remote copy replication group “DRTEST” to do the testing.

From the 3PAR management console, click on remote copy from the left pane. After this click on the groups and verify the replication group listed on the right side as shown below in picture.

View 3PAr remote copy groups

We will be doing the failover of “DRTEST” replication group. So right click on “DRTEST” and click Stop Remote Copy Group(s) as shown below in picture:

Stopping the remote copy groups

Once clicked on Stop Remote Copy Group(s) option a pop-up window as shown below will open which will state that remote copy mirroring will be stopped. Click on OK to proceed.

Confirm groups are stoped

Once we stop Remote Copy Group, replication from source to target will be stopped.

b) Failover Remote Copy Group   

Once the remote copy group is in failed state then we will see that Group state is a stopped state. Now we have to right click on the failed remote copy group (in our case we are preparing example with remote copy group “DRTEST”) and click “Failover Remote Copy Group (s)..” option as shown below

Now one pop-up window will open click OK and after that one more pop-up will open and click yes here

Note:- After this failover will be executed for the selected group. The failover operation changes the role of secondary groups on the backup system (i.e. System-B) from “Secondary” to “Primary-Rev”. Any LUNs (VLUNs) associated with the volumes in the selected groups become writable by hosts connected to the backup system (backup system is System-B in our case). These VLUNs will be in writable permission from both side 3PAR so we must ensure that no write operation on VLUN from primary site.

As per picture below, click YES to continue and the failover operation will start.confirm fsailoverNow there should be Group state as “stopped” as per below picture. Replication status in the table will be as “Stopped” and Backup system (i.e. System-B) will show its role as “Primary-Rev” with DR state as “Failover”.

This completes all failover steps.


a) Description of Recovery Option    

Recover option will recover the failed system (in our case failed system is System-A). When both systems in the remote-copy pair are ready to resume normal operation, reverse the natural direction of data flow and resynchronize the systems.

At this time we have to right click on the replication group which was failed earlier (in our case it is “DRTEST”) and click on “Recover Remote Copy Group (s).. This will copy data / initiates reverse replication and synchronize the delta changes from the reversed volume groups on the failover system (System-B) to the corresponding volume groups on the recovered system (System-A).

Once executed the role of the remote copy group on the source system (System-A) becomes “Secondary-Rev”. Also, any LUN associated with the volumes in the selected groups become non-writable on the source system (System-A).

Note: – The recovery operation can be executed only on groups that have successfully completed the failover option.

b) Recover Remote Copy Group  

Below two pictures shows the process on how to put a replication group in recover state and then click YES in pop-up window. Right click on the replication group which was failover and click on the option “Recover Remote Copy Group(s)”

Recovering the remote copy groups

confirm recover operationOnce clicked on OK prompt, there will be one more pop-up, press yes now. Important to note that VLUNs will not be writable now on source system “System-A” and it will be in writable mode on backup system “System-B”.

Now, we should be able to see Source role for “DRTEST” on source system should be “Secondary-Rev” and “Primary-Rev” on the Backup system role as shown in below picture. Now the DR state will be in “Recover” statchecking status of recoveryc) Verification:

1 Issue the showrcopy command from the CLI on the failover system (System-B).Verify the following:

  • The Status of the target system (recovered system – “System-A”) is ready
  • The SyncStatus of all volumes in the Primary-Rev volume groups is Syncing.
  • The Status of all sending links is Up,

2 Issue the showrcopy command from the CLI on the recovered system (System-A) Verify the following:

  • The Status of the target system (System-B) is ready
  • The Status of all sending links is Up
  • The Role of the synchronizing volume groups is Secondary-Rev
  • The SyncStatus of all volumes in the Secondary-Rev volume groups is Syncing


a) Description 

Once the recovery is completed for the remote copy group then we must restore back the actual state of replication group. I.e. bring the replication group (in our case it is “DRTEST”) back to “System-A” as primary role and again replication from the primary system (System-A) to backup system (System-B).

b) Procedure to Implement

For doing this right click on the replication group and click “Restore Remote Copy Group(s)..”.This restore operation restores replication for the selected Remote Copy group to a pre-failover state after the recovery operation has been completed. Once the Restore operation is executed the role of the remote copy group on source system (System-A) will be “primary” and the Remote copy group on the backup system will be “Secondary”. Also, any LUNs associated with the volumes in the selected groups become writable by hosts connected to the source system (i.e. System-A) and become non-writable by hosts connected to the backup system (i.e. System-B)

And once we will click on the OK in the above pasted picture, then there will be one more pop-up to click Yes or No, we have to click “Yes” to complete the process.

Now the system should be able to see the status of restored replication group as it was in actual state. Below picture shows the actual status with System-A system as Primary and normal state and replication pointing from System-A to System-B.

3par replication status















3PAR 9450 Overview

The HPE 3PAR 9450 is the new all flash model in the 3PAR line up.  In this video I discuss the new model with 3PAR product manager Alberto.

Key Take Aways

  • The 9450 is an new all flash model in the 3PAR range
  • It is an addition to the existing product range, not a replacement
  • It is based on the existing GEN5 ASIC seen across the range
  • The 9450 effectively doubles the horsepower of the existing 8450 with twice the CPU’s and more than twice the cache
  • The form factor is based on the 20,000 series, each node is 2U like the 8000 series.  But unlike the 8000 the 9450 does not contain any data disks and fans are at the front of the unit

  • This is the rear view of the controller nodes

  • Drive cages are not daisy chained like in the 8000, but have individual connection

You can keep up to date with more videos by subscribing to the YouTube channel and following the blog via e-mail.

Further Reading

New 3PAR 9450 offers higher performance & scale for the mid-range Philip Sellers

HPE doubles AFA All-Flash performance with new 3PAR 9450 and 20000 R2 models Bart Heungens

9450 Walkthrough Calvin Zito


3PAR Mega News Bundle – Including Compression

3PAR Announcements

Today HPE announced a significant number of enhancements to the 3PAR product plus some changes in how the product is owned. The feature enhancements are enabled with the upgrade to 3PAR OS 3.3.1 which has been announced today. This really is quite some list, so hold onto your hat’s and let’s start with some data reduction enhancements which in combination HPE is calling Adaptive Data Reduction


3PAR has had dedupe for some time but has not had compression available until the release of 3PAR OS 3.3.1.  Compression is going be available on flash disks only and will be available to the GEN 5 systems i.e. the 8000 and 20,000. The aim of dedupe and now compression is to reduce the data size as much as possible on flash to make flash more affordable. Compression operations are performed inline i.e. before the data hits the SSD’s and are enabled at a per volume level. The best news is that compression is going to be licence free! HPE are expecting a 2:1 data reduction from using compression. I am going to do a deep dive on the compression and other data reduction technologies in the next few days, so watch out for that

Data Packing

3PAR writes to SSD’s in 16K pages, with compression of course you end up with odd sized pages.  These sub 16K sized pages are not optimal for writing to SSD and would incur a post process garbage collection to neaten things up and optimise their layout.  To eliminate this need for the additional garbage collection over head the odd sized pages are stitched together, just like your Nan makes a jumper, to form neat 16K pages. This process of taking odd sized compressed pages and packing them together is shown below.



Dedupe gets a code update in 3PAR OS 3.3.1. This new update changes the way dedupe operates writing to a private area first, as opposed to a shared area previously.  This change in operation aims to make the process more efficient, reducing garbage collection and ultimately making the system more scalable. TDVV (Thin Deduplicated Virtual Volumes) are depreciated, now you just create a standard volume and just change the attribute for the volume to turn on dedupe. Setting the dedupe attribute can be done from the CLI or SSMC.  Dedupe and compression can be combined together and HPE are expecting a median 4:1 data reduction, when the technologies are combined. The good news is that dedupe is again licence free and can be applied to all the GEN4 (7000 + 10,000) and GEN5 (8000 + 20,000) systems.


I really like the idea of VVOLs and have covered them in depth previously.  Their take up has been a little slower than I would have expected and one of the main reasons I have seen cited was the lack of replication support.  vSphere 6.5 introduced VVOL replication and 3PAR now supports this functionality once upgraded to 3PAR OS 3.3.1.  Don’t forget given the granular nature of VVOLs this will mean that replication can be controlled at a per VM level.

Peer Persistence 3 Data Centres (3DC)

Peer Persistence has to be one of my favourite 3PAR features, this enables the creation of a metro cluster or stretched cluster if you like across data centres.  This high availability setup allows storage to be switched between data centres, online with no disruption to hosts.  Peer Persistence 3DC allows a 3rd leg to be added to the replication topology.  This 3rd location does not become part of the metro cluster but allows a 3rd copy of the data to be replicated asynchronously for data availability.


Free Licences

3PAR must have more native software allowing data services than any other SAN now.  Deciding which ones you wanted and then having to pay for them was painful. Well now all licences within a single system are free! If you have more than one system and are using features that allow the systems to talk to each other e.g. Remote Copy, Peer Persistence, Peer Motion you will need to upgrade to the multi-site licence.  Again once you have the site licence this enables all multi system functionality.

Replication to StoreVirtual VSA (Peer Copy)

I have been wondering for a while if and when they were going to enable replication between the StoreVirtual and 3PAR, well it is now possible with the use of RMC in a feature called Peer Copy.  RMC is now included in the free licence bundle and in this circumstance acts as the data mover, replicating crash consistent snapshots between a 3PAR and Store Virtual.  Other RMC enhancements include replication to Azure, deployment in Hyper-V and SAP HANA support.


Online Import

The online import tool which already supported the EMC Clarrion, VNX and VMAX now adds support for the EMX DMX. Other vendors already supported include Hitachi and IBM

File Persona

File persona gets a bunch of feature additions including doubling scalability, file locks for governance, cross protocol file sharing and a file system checker.

Extended SSD Warranty

The existing warranty on 3PAR SSD’s is 5 years, now this has been extended to 7 years.  This covers media wear out and electronics.  The offer is open to all SSD’s in 8000 and 20,000 systems bought after June 2015.

There are even more enhancements than this but I don’t want this blog turning into a hardback edition so I will bring more news in the coming days on other enhancements and deep dives on what has been discussed today. To make sure that you don’t miss any updates, you can get e-mail updates, or follow via Face Book, LinkedIN and Twitter.

This is a mind blowing amount of information and if your head is spinning check out these nice videos in which Calvin Zito summarises these announcements in two videos Video 1, Video 2.