Over the last 3 weeks since the Christmas and New Year Holidays, I have been upgrading all of our firewalls globally, many of them are an High Availability Pair. This means they are redundant and being redundant allows me to upgrade them individually while the site stays full up and functional.
The instructions for upgrading an HA pair are recommended because:
-
-
- It verifies HA functionality before starting the upgrade.
- It ensures the upgrade is successfully applied to the first device before starting the upgrade on the second.
- At any point in the procedure, if any issue arises, the upgrade can be seamlessly reverted without any expected downtime (unless you are having any dynamic routing protocols line OSPF/BGP).
- When finished, the final active/passive device state will be the same as it was before the upgrade with the fewest number of fail overs possible (2).
-
Before you Begin :
Take backup of the configuration as well as Tech Support from both HA Peers. Give proper names to each file, here is how:
Device > Setup > Operations > Save Named Configuration Snapshot
Device > Setup > Operations > Export Named configuration Snapshot
Device > Setup > Operations > Export Device State (If device managed from panorama
Device > Support > Generate Tech Support File, and then download it. (Might be required if any issues)
(Optional but recommended) Disable preemption on High Availability settings to avoid the possibility of unwanted failovers. Disabling preempt configuration change must be committed on both peers. Likewise, once completed, re-enabling must be committed on both peers.
To disable preempt, go to
Device > High Availability > Election Settings and uncheck Preemptive.
Then, perform a commit.
If upgrade is between major versions (4.1 -> 5.0 OR 5.0-> 6.0), it is advisable to disable TCP-Reject-Non-SYN, so that sessions can failover even when they are not in sync. : Do this on both Firewalls from the CLI:
# set deviceconfig setting session tcp-reject-non-syn no # commit
(Optional but recommended) Arrange for Out-of-Band access (Console access) to the firewall if possible. This is again to help recover from any unexpected situation where we are unable to login to the firewall. If you have a Terminal Server awesome, if not a simple Cell Phone tethered to a Laptop with RDP is also fine.
The Upgrade Process
Suspend Backup Device
From the CLI
> request high-availability state suspend
From the GUI
Go to Device > High Availability > Operational Commands > Suspend local device.
Install the new PAN-OS on the suspended device
Device > Software > Install
Reboot the device to complete the install.
When the upgraded device is rebooted, check the dashboard to check the version, wait for all the interfaces to come backup green.
If the device is still in suspended state make it functional again
From the CLI
> request high-availability state functional
From the GUI
Go to Device > High Availability > Operational Commands > Make Local Device Functional
Repeat steps on other firewall.
Suspend Primary Device
From CLI
> request high-availability state suspend
From the GUI
Go to Device > High Availability > Operational Commands > Suspend local device.
*The Backup Firewall will become Active – it does take 30-45 seconds so don’t panic
Install the new PAN-OS on the suspended device:
Device > Software > Install
Reboot the device to complete the install.
When the upgraded device is rebooted, check the dashboard to check the version, wait for all the interfaces to come backup green.
If the device is still in suspended state make it functional again
From the CLI
> request high-availability state functional
From the GUI
Go to Device > High Availability > Operational Commands > Make Local Device Functional
To Get Primary Back to Primary by suspending the backup (current active) firewall (The Original Backup Firewall)
From the GUI,
Go to Device > High Availability > Operational Commands > Suspend local device.
Once the Primary became active again, enable the suspended backup firewall
Enable TCP-Reject-Non-SYN, so that sessions can failover even when they are not in sync. : (Do this on both Firewalls)
# set deviceconfig setting session tcp-reject-non-syn yes # commit
Re-Enable preempt configuration change must be committed on both peers. To re-enable preempt, go to Device > High Availability > Election Settings and uncheck Preemptive. Then, perform a commit.
How to Downgrade
If an issue occurs on the new version and a downgrade is necessary:
To revert to the previous PAN-OS screen, run the following CLI command:
# debug swm revert
This causes the firewall to boot from the partition in use prior to the upgrade. Nothing will be uninstalled and no configuration change will be made.
However please be aware while running this command –
After rebooting from a SWM revert, the configuration active at the time before upgrade will be loaded with the activation of the previous partion. Any configuration changes made after upgrade will not be accounted for and will need to be manually recovered by loading the latest configuration version and committing the changes.
