Firewall Degraded - Connectivity & Services Disrupted
February 13, 2024 at 1:36 PM PDTApplication Servers
Employee Portal
Online6
Aeries
Network Monitoring
Panorama Management
Secure File Transfer
Internet
VPN
- 2024-02-26
- 10:37AM - Our vendor believes they know what new changes have caused the issues, but are still working on determining what is causing the bug.
- 2024-02-17
- 7:18AM - No outages since workaround applied.
- 2024-02-16
- 6:52AM - No outages seen so far.
- 2024-02-15
- 4:51PM - Firewall is back up and outage is resolved.
- 4:33PM - Instead of having back to back outages..we are going to apply the workaround now.
- 4:31PM - Outage due to card reboot.
- 1:49PM - Outage cleared.
- 1:45PM - Recovery taking longer than usual.
- 1:22PM - Outage due to card reboot.
- 9:12AM - Outage cleared.
- 9:05AM - Outage due to card reboot.
- 3:07AM - Outage cleared.
- 2:51AM - Outage due to card reboot.
- 2024-02-14
- 11:31PM - Case team is still actively investigating the issue.
- 9:27PM - Outage cleared.
- 9:11PM - Outage due to card reboot.
- 4:10PM - The issue has been assigned an internal case number with our vendor; further diagnosis is being done.
- 3:44PM - Support Engineer states issue likely software related; working with another team to further diagnose.
- 3:27PM - Outage cleared.
- 3:19PM - Outage due to card reboot.
- 3:08PM - Outage cleared.
- 3:01PM - Outage due to card reboot.
- 2:24PM - Engineer assigned to case with initial analysis.
- 2:06PM - Outage cleared.
- 1:59PM - Outage due to card reboot.
- 12:21PM - Outage cleared.
- 12:16PM - Outage due to card reboot.
- 8:44AM - Awaiting case update.
- 2024-02-13
- 11:54PM - Outage Cleared: Card online.
- 11:47PM - Outage: Card rebooted.
- 6:58PM - Outage Cleared: Cards back online.
- 6:42PM - Outage: Card rebooted.
- 4:03PM - Cards are back in service.
- 3:53PM - One set of cards rebooted
- 2:45PM - Our case has been transferred to Engineering from Support.
- 1:57PM - Cards are back in service.
- 1:40PM - Cards are not automatically restarting. We are manually restarting the cards.
- 1:36PM - Cards have rebooted causing an outage.
SCOE is currently working with our firewall vendor to resolve an issue with restarting line cards. Connectivity through SCOE is currently intermittent.