At approximately 3:49pm central time on 8/1/2010 our upstream network provider experienced a complete network outage, causing all inbound and outbound traffic to our network to be rejected. This outage lasted until approximately 5:05pm central time, at which time it appears that network services have been fully restored.

We are waiting on a report from our provider as to the causes and resolution of the issue. We will update this blog post once we receive more information.

During this outage mail would have queued on the sender’s mail server and would not be lost. It may be delayed due to the sender’s retry timing policies.

We have issued a 5% service credit to all of our customers based on our uptime guarantee.

UPDATE: The following is the RFO (Reason for Outage) provided to us by our upstream provider:

A carefully crafted Denial of Service attack was launched at a customer and several of our routers, which caused the routers to repeatedly drop IGP (internal) and BGP (external) routing protocols. We identified the attack and found that it exploited a loophole in the control-plane policer packet matching logic which normally protects against these types of attacks, allowing it to slip past these filters and cause the amount of impact that it did.

As a work-around we have applied additional packet filters at multiple network edges to block this type of traffic, and are working with our router vendor to address the root cause of the issue, but we believe that this attack has been successfully filtered and that the network should be stable going forward.