Master Standby with ArubaOS 8

Reading Time: 7 minutes

In the last posts about ArubaOS 8, I talked a lot about Virtual Mobility Master. This time, I will not include the VMM in my scenario. The reason is simple. Many customers deploy two controllers today. They simply do not need more. So the big question is how to build an environment with ArubaOS 8 and two controllers. The answer is Master Standby. Likely the same design as with ArubaOS 6.x.

Just to make it clear, you get the most features, the most benefits and the most outcome from a deployment with VMM. So this is just a plan B if you cannot use the benefits of an implementation with VMM.

Master Standby: Requirements

The Master Standby design requires two controllers, obviously. They run in the standalone mode. Keep this in mind during the setup of the controller. Only the active controller can terminate AP’s in this design. But you can use a variation of the fast failover feature.

The design looks like this, from a logical point of view:

Master Standby - Design
Master Standby – Design

The two controller run in standalone mode. I use the master redundancy configuration to synchronize the configuration, user data and of course the licenses. If the master fails, you can use the licenses on the standby for 30 days. Enough time to recover.

To create minimal downtimes during failover, I also create an HA Group. With the HA Group configuration, the AP’s creates a tunnel to each of the controllers. During failover, the AP fails over to the tunnel to the standby controller. This works without a reboot of the AP. Even the radios stay active.

Sounds interesting to you? Read further.

Master Standby: Master Redundancy

The first step to build the above design is to create the master redundancy. To start with master redundancy, I assume you have the two controllers running in standalone mode. Preferably fresh vom the factory or after a factory reset. You have configured them with the initial setup wizard to standalone controllers and they have rebooted.

Login to the one, which will be the active master. In the future, I name them master and standby. On the master, go to “Configuration–>Services–>Redundancy” and select the “Virtual Routing Table”.  Make sure, you are in the device hierarchy and not in the “Mobility Controller” hierarchy. Add a new “Virtual Router” with the “+” sign:

Master Standby - Create VRRP
Master Standby – Create VRRP

Use a meaningful “Description” to avoid confusion later on. I always use the VLAN ID for the “ID”. This assumes I use VLAN 201. This is my controller management VLAN. So both controllers have IP addresses in that VLAN and the controller IP is in this VLAN as well. Enter an “Authentication password” and specify the VRRP “IP address”.  I set the “Priority” of the master to 110, to make sure he is always the master. I also use pre-emption to make sure the master is master again after he is back. Set the “Admin state” to up and select the VLAN.

“Submit” the configuration and configure the same, except for the priority, on the standby. For priority use the value of 100, which is the default.

After some seconds, VRRP is up and running. You can check on the CLI:

(Master) [mynode] (config) #show vrrp


Virtual Router 201:
    Description Master VRRP
    Admin State UP, VR State MASTER
    IP Address 10.201.201.10, MAC Address 00:00:5e:00:01:c9, vlan 201
    Priority 110, Advertisement 1 sec, Preemption Enable Delay 0
    Auth type PASSWORD, Auth data: ********
    tracking is not enabled
(Standby) *[mynode] (config) #show vrrp


Virtual Router 201:
    Description 
    Admin State UP, VR State BACKUP
    IP Address 10.201.201.10, MAC Address 00:00:5e:00:01:c9, vlan 201
    Priority 100, Advertisement 1 sec, Preemption Enable Delay 0
    Auth type PASSWORD, Auth data: ********
    tracking is not enabled

The next step is to enable database synchronization. On the master, go to the “Mobility Controller” configuration hierarchy and go to “Configuration–>Services–>Redundancy” and select “Master Redundancy”:

Master Standby - Database Synchronization
Master Standby – Database Synchronization

“Submit” the changes and go down to the controller hierarchy again, but the same configuration page:

Master Standby - Create Master Redundancy
Master Standby – Create Master Redundancy

Add the “Master VRRP” information, this is the VRRP from above and the “IP address of the peer”. This is the IP of the standby. For this lab, I use an “IPSec Key” for “Authentication”. Certificates are possible as well. “Submit” the changes and redo the configuration on the standby, except for the IP address. Here you use the IP of the master. It could take some minutes to form the master redundancy. To check if it is up and running use the command below:

(Master) [mynode] #show database synchronize 

Last L2 synchronization time: Mon Nov 27 04:41:29 2017
Last L3 synchronization time: Secondary not synchronized since last reboot
To Master Switch at 10.201.201.12:  succeeded
To Secondary Master Switch at unknown IP address:  succeeded
WMS Database backup file size: 31093 bytes
Local User Database backup file size: 38393 bytes
Global AP Database backup file size: 12953 bytes
IAP Database backup file size: 3750 bytes
Airgroup Database backup file size: 3052 bytes
License Database backup file size: 5168 bytes
CPSec Database backup file size: 3224 bytes
L2 Synchronization took 2 second
L3 Synchronization took less than one second

16 L2 synchronization attempted
15 L2 synchronization have failed

0 L3 synchronization attempted
0 L3 synchronization have failed

L2 Periodic synchronization is enabled and runs every 1 minute

L3 Periodic synchronization is disabled

Synchronization doesn't include Captive Portal Custom data
(Standby) *[mynode] #show database synchronize 

Last L2 synchronization time: Mon Nov 27 04:41:22 2017
From Master Switch at 10.201.201.11:  succeeded
WMS Database backup file size: 31093 bytes
Local User Database backup file size: 38393 bytes
Global AP Database backup file size: 12953 bytes
IAP Database backup file size: 3750 bytes
Airgroup Database backup file size: 3052 bytes
License Database backup file size: 5168 bytes
CPSec Database backup file size: 3224 bytes
L2 Synchronization took 1 second

8 L2 synchronization attempted
6 L2 synchronization have failed

L2 Periodic synchronization is enabled and runs every 1 minute

Synchronization doesn't include Captive Portal Custom data

I have set the synchronization time to 1 minute. This is to make the waiting time shorter. In production environments, 10 minutes is ok as well.

You could now connect an AP. You would use the VRRP address for the connection between AP and controller. During a failover, the AP would connect to the standby after VRRP changes. But the AP would reboot. To avoid this, we use “HA Groups”.

Master Standby: HA Groups

The “HA Group” feature enables fast failover. To do this, the AP creates a tunnel to the two controllers. The first tunnel is the active one. If the controller for tunnel one fails, the AP uses tunnel two to the second controller. This makes the failover time very short and the best, the AP did not reboot during the failover.

To create an “HA Group”, login to the master controller. Stay in the “Mobility Controller” hierarchy and go to “Configuration–>Services–>Redundancy” and select “HA Groups”. Create a new “HA Group” with the “+” sign:

Master Standby - Create HA Group
Master Standby – Create HA Group

Add the two controllers to the “HA Group”. You can use the role “dual”. “Active” for the master and “Standby” for the standby works as well. Enable “Pre-emption” and “State synchronization”. Enter a “Pre-shared-key” and “Submit” the changes.

The last step is to join the controllers to the group. Stay on the same page as bevor and select “HA Member”:

Master Standby - HA Member
Master Standby – HA Member

Select the “HA Group” from the drop-down list “Member of HA group”. Now, the two controllers form an “HA Group”.

You also have to change the LMS IP in the AP group. You can do this in the AP Systems profile as well. I personally, prefer the group. In the same hierarchy level go to “Configuration–>AP Groups” and select the AP group you would like to change:

Master Standby - LMS IP
Master Standby – LMS IP

Select the “LMS” tab and enter the two IP addresses of the two controllers. The master the first one and the standby is the “Backup IP address”. Save the configuration.

All AP’s establish now two connections. One to each controller. To check this use the command below:

(Master) [mynode] #show ha ap table 

HA AP Table
-----------
AP       IP-Address      MAC-Address        AP-flags  HA-flags
--       ----------      -----------        --------  --------
HA-Test  10.201.201.204  94:b4:0f:cb:75:cc  LU        

Total Num APs::1
Active APs::1
Standby APs::0
AP Flags: R=RAP; S=Standby; s=Bridge Split VAP L=Licensed; M=Mesh, U=Up
HA Flags: S=Standby, C=Standby connected, L=LMS, F=Sent Failover Request to AP, H=AP flaged for Inter Controller Heartbeat
(Standby) *[mynode] (config-submode)#show ha ap table

HA AP Table
-----------
AP       IP-Address      MAC-Address        AP-flags  HA-flags
--       ----------      -----------        --------  --------
HA-Test  10.201.201.204  94:b4:0f:cb:75:cc  SLU       

Total Num APs::1
Active APs::0
Standby APs::1
AP Flags: R=RAP; S=Standby; s=Bridge Split VAP L=Licensed; M=Mesh, U=Up
HA Flags: S=Standby, C=Standby connected, L=LMS, F=Sent Failover Request to AP, H=AP flaged for Inter Controller Heartbeat

You see, that the AP is “Licensed” and “Up” in the master and in addition “Standby” on the standby controller. In case of a failure, the AP uses the standby connection to stay active. Because if the synchronization, all user sessions are available on the standby as well. No need for a complete new authentication.

Once the master is back online. All the master functions change back, including the AP termination.

The user might see a small interruption, I had only 8 pings failing in the worst case. Normally around 3-5.

Do you use this design of master standby or do you prefer a design with virtual mobility master? Tell me and leave a comment below.

If you find this post interesting, leave me a comment and share it with your friends. If you don’t like the post, leave me a comment and share it with your enemy. But whatever you do, leave me a comment, now.

48 thoughts on “Master Standby with ArubaOS 8”

    • hi Ashik,

      maybe I can describe it with more details.
      So first of all, if you are running a network with a Mobility Master (the blog post above is not talking about a mobility master at all, but I would like to mention it for completeness), the Mobility Master (MM) cannot terminate AP’s. you can only terminate AP’s on managed Devices (MD).

      If you do not have a MM, the Mobility Controllers can run in different modes:
      MCM or Mobility Controller Master, this is a normal Controller, running some kind of Mobility Master functionality, but with fewer features as a normal Mobility Master. I would never recommend this and please forget about this just after reading, it is here just for completeness. In this mode, the controller cannot terminate AP’s.

      Standalone, this is what the post above is talking about. In this mode, the controller runs without mobility master, like it was running in 6.x. In this mode, the controller can terminate AP’s and you can also configure the controller directly.

      Managed Device or MD, here the controller needs a MM and can terminate AP’s. But you can only configure the controller through the MM.

      Hope this gives a better understanding of the different modes and where you can terminate AP’s or not.

      BR
      Florian

      Reply
  1. Thank you for the blog post. I can confirm that this also worked fin ArubaOS 8.5 with only one difference. That being the breadcrumb trail for configuring “Master Redundancy” and “HA Groups” has been shortened from “Configuration–>Services–>Redundancy” to “Configuration->Redundancy”.

    Reply
    • Hi Brad,

      thanks for your feedback. Really appreciated. and also thanks for the update.
      It would consume more time as I have to keep the posts up to date with new software versions. But I hope, if you know what you are looking for, you should be able to find it even if the config context moved to a different place.

      BR
      Florian

      Reply
  2. Great article, thanks for the info. I have 2 controllers with 8.3 and need to do the same redundancy mentioned here. Just one quick question though, if I have some access points already configured on one controller with it’s ip address (like 10.201.201.11 – no redundancy setup yet), will they lose connection through the VRRP setup process? In other words, would I have to configure the new master VRRP IP address (10.201.201.10) on the access points or will they get updated from the controller on their own with the new VRRP controller ip address 10.201.201.10? I just want to make sure I don’t take our access points down, if it can be avoided. Thanks

    Reply
    • Hi Ike,

      thanks for your feedback.

      If the AP’s are configured to the physical address of the controller and you add a VRRP address, they will not reboot. You should use the VRRP address for the discovery only. This means, depending on the discovery process (DHCP,DNS or Layer2) they will learn the VRRP address during the next scheduled reboot via those discovery methods. During normal operations, they will not connect to the VRRP address, only for the first connection to the controller in order to learn LMS and Backup LMS from the controller.

      So simply adding VRRP to the game does not interrupt normal operations.

      BR
      Florian

      Reply
  3. Hi!
    Great post! Could you please specify the OS version of the controller you are using? Is this feature supported on all AOS 8.X, or just from specific versions?
    Thanks!

    Reply
    • Hi Daniel,
      Not sure which version I was using but this should work for every version in AOS 8.
      I would recommend to use the latest conservative release for production environments. Except you need newer APs which are only available for newer versions.
      BR
      Florian

      Reply
    • Hi Tahir,

      Active-Active Deployment with ArubaOS 8 is not possible anymore, without using a Mobility Master and at least 2 managed devices. But if you already have a Mobility Master I would always go for cluster setup like this:

      ArubaOS 8 Cluster

      BR
      Florian

      Reply
  4. Thanks for the article. A couple of questions. Will any configurations on the master be copied to the standby contorller? Also, will the standby controller share the ap licenses assigned from the master controller?

    Reply
    • Hi John,

      thanks for your comment. I will try to answer your questions to my best knowledge.

      First, all of the configurations from the /md tree will be synched between the two devices. All of the /mm tree as well, but only from the top level. Everything under /mm/device needs to be configured on each device individually. But this should be not that much.

      The licenses are synched the same way most off the rest off the databases like WMS or AP whitelist is synched, but the standby controller can only use the licenses when the primary goes down. This means, as long as the primary is up and running, the standby knows about the licenses but cannot use them.
      This is the reason, why this kind of setup can only support an active-standby scenario. For real redundancy, I would recommend a cluster setup.

      BR
      Florian

      Reply
  5. Great article. Just a few questions. I use MAC authentication using Internaldb as the authentication source. (The two controller run in standalone mode for Master-Redundancy). When Master is down, all AP switch to standby, but users cannot pass the authentication.(Authentication failed). And I typed “show local-userdb” on standby AC, there are all users and password info, but the Status is Inactive. On Master AC, show local-userdb status is Active. So if two standalone run in Master-Redundancy with MAC authentication. Anything special to pay attention to? How to troubleshoot this scenario?

    Reply
    • Hi Holly,

      thanks for your comment. Really appreciate your feedback.

      actually, I have never tried MAC Authentication with local DB. I have always used a central radius server for this kind of setup. But I would first check if the database synchronization works. If this is the case I have no clue at the moment why this should not work.

      I’m currently recreating my lab so I think with the start of February I should be able to test this myself.

      BR
      Florian

      Reply
    • Hi jAyR,

      I just tested it without issues. I created a guest manager and created a guest account and two mac address accounts via CLI. After database synchronization was running from master to backup the users were active on the salve as well:
      (VMC-2) [mm] (config) #show local-userdb

      User Summary
      ------------
      Name Password Role E-Mail Enabled Expiry Status Sponsor-Name Remote-IP Grantor-Name
      ---- -------- ---- ------ ------- ------ ------ ------------ --------- ------------
      guest123 ******** guest Yes 2/25/2020 2:31 Active 0.0.0.0 guestmanager
      ac7ba18b2fae ******** employee Yes Active 0.0.0.0 admin
      fcd5d92d9b87 ******** guest Yes Active 0.0.0.0 admin

      User Entries: 3
      (VMC-2) [mm] (config) #show vrrp

      Virtual Router 201:
      Description Master_VRRP
      Admin State UP, VR State BACKUP
      IP Address 10.201.201.100, MAC Address 00:00:5e:00:01:c9, vlan 201
      Priority 100, Advertisement 1 sec, Preemption Disable Delay 0
      Auth type PASSWORD, Auth data: ********
      tracking is not enabled

      I was running the latest “ArubaOS 8.6.0.2” version. So maybe a bug in your version.

      BR
      Florian

      Reply
  6. Thanks for the article, I found it useful in case of no MM.
    it will be kind when you make a new post to clarify how we deal with the migration from 6.x to 8.x when we also don’t have a MM. for example how to transfer our old license to the new OS version.

    Reply
    • Hi Ady,

      The migration from 6.x to 8.x is a hard cut. This means if you install the new 8.x image in your controller and reboot, the old configuration will be gone and the controller did a factory reset. So you need to configure everything from scratch. Normally a partner is in charge to plan and do the migration during a maintenance window or during low usage hours, during the night for example.
      If you prepare well, your downtime might be under 1 hour. but this is just a guess, without knowing your configuration and the environment. And that everything works as planned, and we all know, how often this happens 🙂
      In regards to the licenses, the controller will do a factory reset, but your licenses will remain on the controller. You only need to convert them, if you plan to use a mobility master. Which is my recommendation for the following reasons:
      1. You can use a cluster after the migration, which gives you by far the best user experience in regards to failover times
      2. You can prepare your upgrade by configuring the Mobility Master before you actually upgrade the controllers so that everything is already in place when you upgrade the controllers. This will bring you definitely under one-hour downtime. If you upgrade the standby first you can even be faster. But this should be planned by a consultant or with the help of your peers at Aruba.

      I hope this helps.

      BR
      Florian

      Reply
  7. Thanks a lot…
    I am doing the migration myself and we can’t afford a MM, so do you recommend using the migration tool for saving the config time or its not perfect?

    Best Regards,
    Ady

    Reply
    • Hi Ady,

      Personally, I cannot comment on the migration tool, as I do not have used it. I did the migration always by myself.
      Maybe you can start the migration with one controller (you will lose redundancy for the 6.x environment) first, move the AP’s to the new controller and afterward migrate the second one. This should give you the lowest downtime.

      BR
      Florian

      Reply
  8. Hi Florian,

    I have recently upgraded our office controller from 6.x to 8.6.0.2 and we have 7210 model. I have currently made up one 7210 controller with all configuration in place all the APs’ are terminated and in the production.
    Now i i need to add one more controller as backup. Currently all the AP’s are registered with IP 10.99.99.100
    now please tell me if i bring them in VRRP and HA Group will the AP reboot to get the new controller IP(when you do VRRP and HA Group, AP will use which IP )to register with controller example my 1st controller IP is 10.99.99.100 currently in production and second controller IP is 10.99.99.101 which is freshly configured and VRRP ip is 10.99.99.102). Basically i dont want to take one more downtime as i already took long downtime to bring controller in 8.x version.

    I have 3 Sites which campus AP mode configured with DHCP server local to each sites. And AP discovery mode will be ADP with i have configured controller IP address in DHCP server for option 42 and 60.

    Please let me know if u need further info.

    Reply
    • Hi shridhar,

      jsut to make sure we are talking about the same thing:
      10.99.99.100 Controller 1
      10.99.99.101 Controller 2
      10.99.99.102 VRRP

      first, you would use the VRRP IP only for initial AP discovery. So you would insert this IP into the DHCP server config. I assume you have 10.99.99.100 there at the moment.
      This would not make the AP’s reboot!
      Secondly, you would simply configure the 10.99.99.100 as LMS IP and 10.99.99.101 as Backup-LMS in the AP Systems Profile. As the AP’s are already connected to 10.99.99.100 they would just create a second tunnel to 10.99.99.101. They will not reboot.

      All the above assumes, that VRRP and HA is already configured between the two controllers.

      Many thanks,
      Florian

      Reply
  9. Hi FLorian,

    Many Thank you for your reply on this.

    Actually currently VRRP not configured yet i received another controller just this week and need to bring them to vrrp and HA.

    So before doing the changes i was going throgh your blog and take consult on this.

    Current situation

    10.99.99.100 is the controller as the single standalone 7210 controller.
    now i need to create VRRP and HA with new 7210 controller.

    I dont want to take any big risk of major changes just add the new controller to the VRRP and HA with no downtime or minimal downtime.

    Please help me here.

    Thank you so much.

    Reply
    • Hi Shridhar,

      First of all, you should contact your local partner or Aruba representative as this might need onsite consulting to understand your infrastructure. I can only give some general guidance.

      In general, if the already deployed controller (10.99.99.100) is already running in standalone mode you first need to create the master redundancy. This will include setting up VRRP. If this is running successfully, with successful database synchronization you can start configuring HA between the two controllers. Afterward, you simply add LMS (master controller) and backup LMS (backup controller) to the AP group config. This will make the AP to create a backup tunnel to the second controller. It should not make the AP reboot.

      you should use the VRRP IP for AP controller discovery. So you should put this either into the DHCP server as option 43 or into DNS. (only if you do L3 based controller discovery).

      Hope this helps.

      BR
      Florian

      Reply
  10. i am trying this setup on 8.6 , when i enable master-redundancy , the high availabilty tab just disappears from the GUI on the MM level , i can get it work from the CLI only

    did you encouter similar issue ? , or do i need to get the HA config first before i enable the redundancy ?

    Reply
    • Hi MH,

      I haven’t checked with 8.6 so far. So I cannot tell if this is as designed or a bug. But I will check during the next days.

      Many thanks,
      Florian

      Reply
  11. Thanks Florian , can you check on your current code if you have the tab “high availabilty ” on the mobility controller level ( the /mm level or the parent level) ,apparently once the the L2 redudancy is formed , this tab is gone from the GUI

    it is only available on the 7005-1 (the device i am using for this test ) .

    i am not sure if i can post a screen shot so it is clear.

    Reply
    • Hi MH,

      you are correct. I’m currently testing with 8.6.0.3 and after the Master-Standby is formed the Tab is gone, at least from the MM hierarchy. If you go own to the device level it is still there. For the controller, acting as the master, you can do this from the main GUI. The standby controller needs to be configured from the device itself. I will try to fully test this in the next days.

      BR
      Florian

      Reply
  12. thanks Florian for confirming this , i am testing with 8.6.0.2 the production release and it is the same.

    on the device level , the tab is also gone from the backup controller (instead the l3 redundancy tab is showing up)
    the only way i managed to get it to work is from the /mm of the master cli.

    or maybe config the HA before forming the l2 redundancy.

    Reply
  13. Hi Florian,

    Thank you very much for posting this detailed setup. We have a 7210 controller running 8.2 in stand alone mode with all the configs. Now, we want to add a second 7210 controller and have them setup as Master/Standby. If we add the second controller and setup VRRP, Master Redundancy, Database Sync, will the second controller get all the existing configs like AP groups, ap system profile, vap profile, user roles ect? Or do we have to configure those things manually on the second controller before setup HA? How about license? Do we need to install any license on the second controller?

    Thank you for your help.
    Hai

    Reply
    • Hi Hai,

      it depends on which level you configure everything. If you did as I described above, the second controller should get all the configs like AP groups and so on. In a Master/Standby scenario, only the master need licenses, as the standby will get those after failover.

      BR
      Florian

      Reply
  14. Hi Florian

    I am setting this up in a lab with 2x 7030 and running v8.6.0.4. I can confirm the HA tab is still gone in the MM hierarchy. Have you found out more about this? Is this feature being phased out by Aruba, or just a bug? I could try configuring this in CLI, but if this feature will become unsupported in future versions, the CLI config does not help me either.

    Kind Regards
    Ingo

    Reply
    • hi Ingo,

      Master/Standby with HA is no longer the recommended configuration. The recommendation is to use Master/Standby with VRRP. Use the VRRP address for LMS IP and do not use any Backup LMS. The failover times should be similar to the ones with HA in 8.x (ca. 10sec). I will write a new post about this in the coming weeks.

      BR
      Florian

      Reply
  15. Hi, I have a production instalation with two 7030 controllers in master/standby with VRRP, behind a mobility master. But now I wish to add two new 7030 controllers to replace the old ones, How do I add them to the existing cluster and them remove the old ones? thanks in advance

    Reply
    • Hi Daniel,

      thanks for your comment. To be honest I do not understand your setup. You are running a Mobility Master and you use VRRP for your MD’s managed by the Mobility Master? You would like to replace the two MD’s? Is this correct?

      If so, I would recommend going to a Cluster setup. There you do not need Master/Standby, as the Cluster will handle it. There you can add the new members and afterward remove the old the one one by one.

      Hope this helps.

      BR
      Florian

      Reply
    • Hi Kori,

      I’m not aware of any command. Taking down the port from a physical point of view is a good option. I would either remove the link physically or set the port in a downstate, but from the switch. This would simulate a link failure and VRRP should act accordingly.

      BR
      Florian

      Reply
  16. Hi Florian,

    First of all, thank you for this very detailed guide. I’ve used this and it helped me though my config.
    Just a quick question, we’ve already configured the HA but we’re using a Cisco Switch. The only problem that we have encountered is that the interfaces from MC1 (0/0/0) and MC2 (0/0/0) seems to have mac-flopping. But we have already established the VRRP for Master and Standby and it seems to be working well. Is it normal to have mac flapping for the connected interfaces? Does it have something to do with the database sync timer as well?

    Any inputs from you will be very helpful. Again, thank you very much!

    Reply
    • Hi Michael,

      thanks for your feedback.

      You should not see any mac flapping. you should see only the physical IP if Port 0/0/0 for each controller and the on one port the VRRP MAC as well. Nothing more. If there is a flapping, something is not working or working not correctly. Which mac is flapping?

      BR
      Florian

      Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: