There was an outage that affected a large portion of all services in the Florida River Valley area, the worst portion of which was on the Sundance Hills network. The time frame of the outage was from approximately 8:35am to 4:40pm. While not a hard outage, the traffic being passed through our main gateway where we transition to fiber optic to the internet was being limited well below the normal usage for the network. This resulted in experiences ranging from no service to very slow service with extremely high latency. Internet browsing would be extremely slow and any higher bandwidth usage like video streaming and voice communications wouldn’t work at all.
The symptoms of the problem were not something we’ve ever seen before. Network admins from AlignTec and our upstream provider were all challenged by what they were seeing. Contrary to our original assessment, this incident turned out to be in no way related to any previous issues. The actual cause was rooted in the system that regulates individual service speeds. These systems are still relatively new and therefore this type of issue had never been experienced by us, and even the manufacturer of the system wasn’t fully aware of such. The system logs became overloaded and caused the system to go into a ‘safe’ mode to avert a complete lockup. The system reallocated resources to prevent full overload causing the restricted traffic flow. Thankfully, we had the foresight when installing this system last year to provide for a means of bypassing in the event of a failure. Bypassing the system immediately revealed that it was the cause, and that ended the restriction on traffic flow at 4:40pm. With the manufacturer’s input, we made configuration changes that will prevent such an occurrence in the future. Additionally, now that we know the symptoms of just such a failure, we will be much more adept at solving the problem more quickly.