At approximately 3:49pm central time on 8/1/2010 our upstream network provider experienced a complete network outage, causing all inbound and outbound traffic to our network to be rejected. This outage lasted until approximately 5:05pm central time, at which time it appears that network services have been fully restored.

We are waiting on a report from our provider as to the causes and resolution of the issue. We will update this blog post once we receive more information.

During this outage mail would have queued on the sender’s mail server and would not be lost. It may be delayed due to the sender’s retry timing policies.

We have issued a 5% service credit to all of our customers based on our uptime guarantee.

UPDATE: The following is the RFO (Reason for Outage) provided to us by our upstream provider:

A carefully crafted Denial of Service attack was launched at a customer and several of our routers, which caused the routers to repeatedly drop IGP (internal) and BGP (external) routing protocols. We identified the attack and found that it exploited a loophole in the control-plane policer packet matching logic which normally protects against these types of attacks, allowing it to slip past these filters and cause the amount of impact that it did.

As a work-around we have applied additional packet filters at multiple network edges to block this type of traffic, and are working with our router vendor to address the root cause of the issue, but we believe that this attack has been successfully filtered and that the network should be stable going forward.

Many of you have noticed over the past week or so that we have been seeing a higher than normal amount of spam slip through the filters and make it to users’ inboxes.  We have been in contact with MailFoundry and they have told us that there was an issue with their system that takes messages from the reported spam feed and injects them into the rule creation engine, thus updating the spam definitions.  This has caused reported spam to not get put into the engine definitions in a timely manner, leading to an increase in non-filtered spam over the last week or so.

Reportedly this is looking better today, but the MailFoundry developers are monitoring the situation to see if any additional adjustment is necessary.  For those who have reported this increase in spam to our support department, please monitor the situation over the next few days and let us know if the amount of spam slipping through begins to reduce.

UPDATE: The issues with elevated levels of spam slipping through the filters appears to have been resolved.  Those of you who have reported this issue, please let us know by updating your support ticket or posting a comment on this post if the issue seems to be resolved for you.

We are currently performing emergency database maintenance on the G1 gateway to repair a corrupted user database.  We estimate a total time for repair of 30-60 minutes.  During a portion of this time SMTP services on G1 will be disabled.  Sending servers will queue the mail until services are restored and will deliver at that time.

Starting at approximately 8:30pm central time on Thursday, January 21, 2010, mail delivery on MF2 was impacted by an extremely large block of large virus-laden email, causing our antivirus processing to slow delivery of mail. This issue has been rectified and delayed mail is now being delivered at normal speeds. We expect that all delayed mail will be delivered within the next 30 minutes or so.

We moved our DNS servers several weeks ago, but apparently one of the OpenDNS servers had the wrong information cached. This was rectified a couple of days ago, which is why we are seeing these problems today.  Now, there are several senders that still have bad DNS information cached.  If you are not receiving messages from some senders, be sure to notify them that they need to update their DNS cache.

Here is how the sender can find out if their cache is bad:

The sender’s system administrator needs to do a lookup on their mail server to see what IP address they’re getting for our gateway. Additionally, it would be helpful if they could get us the nameserver records they’re seeing for ijnet.net as well.

Examples:

On windows:

nslookup -type=a g1.ijnet.net
nslookup -type=ns ijnet.net

On linux:

dig g1.ijnet.net a
dig ijnet.net ns

Our correct IP’s are:

MF1 – 216.246.89.41

MF2 – 216.246.89.42

G1 – 216.246.89.40

G2 – 216.246.89.37

As always, if you have any questions, feel free to contact us at support@purity.net.

Thank you for choosing Purity Networks.

Our G1 gateway has completed database maintenance, and normal mail flow has resumed.