We’ve received reports from several customers that they’re having issues receiving email through the G1 gateway. Our initial investigation shows that G1 has been accepting mail but is possibly queuing mail before delivering to customer’s servers.

We’re doing further investigation at this time to determine a resolution to this issue and will update this post as we have more information. During our investigation there may be brief periods where G1 is unavailable due to reboots or other maintenance steps.  During any unavailability mail will queue at the sender’s server.

We apologize for any inconvenience this has caused.

Update at 7:38AM Central: We are running a database maintenance on G1 at this time to resolve issues with the internal databases. During this time G1 will not be available to receive mail. Mail will queue on the sender’s servers and be delivered after the maintenance has been completed. Unfortunately it is not possible to determine an ETA because database maintenance has a variable length. It is currently 25% complete. We will update this post when the status changes.

Update at 8:34AM Central: Database maintenance completed approximately 45 minutes ago and mail should be flowing normally at this time.  Any messages that were backlogged either have been delivered or will be delivered shortly. There is a large backup of quarantined mail waiting to be delivered to the Live Quarantine system which will take time to catch up.  Please contact support@purity.net if you are still seeing issues, but at this time mail appears to be flowing smoothly.  We will continue to monitor the situation.

This morning we are experiencing queuing of email on the G1 gateway. We are performing an emergency database maintenance cycle to attempt to mitigate the situation. The ETA for resolution at this time is unknown.

UPDATE: During this maintenance there may be periods of time where the G1 gateway is unavailable to accept connections. This will cause mail to queue on the sender’s server and be retried at a later time according to their settings.

UPDATE 11:40AM: Database maintenance is complete and mail appears to be flowing properly again. Please note that it may take some time for mail flow to return to normal as the gateway has a backlog of mail to process. We apologize for any inconvenience this might have caused!

Due to a previously unrecognized bug in our reporting system we regret to say that reporting data for domains on the MF2 gateway will be missing from 10-12-2010 through 10-20-2010. This issue has been corrected and will not be an issue going forward.

At approximately 3:49pm central time on 8/1/2010 our upstream network provider experienced a complete network outage, causing all inbound and outbound traffic to our network to be rejected. This outage lasted until approximately 5:05pm central time, at which time it appears that network services have been fully restored.

We are waiting on a report from our provider as to the causes and resolution of the issue. We will update this blog post once we receive more information.

During this outage mail would have queued on the sender’s mail server and would not be lost. It may be delayed due to the sender’s retry timing policies.

We have issued a 5% service credit to all of our customers based on our uptime guarantee.

UPDATE: The following is the RFO (Reason for Outage) provided to us by our upstream provider:

A carefully crafted Denial of Service attack was launched at a customer and several of our routers, which caused the routers to repeatedly drop IGP (internal) and BGP (external) routing protocols. We identified the attack and found that it exploited a loophole in the control-plane policer packet matching logic which normally protects against these types of attacks, allowing it to slip past these filters and cause the amount of impact that it did.

As a work-around we have applied additional packet filters at multiple network edges to block this type of traffic, and are working with our router vendor to address the root cause of the issue, but we believe that this attack has been successfully filtered and that the network should be stable going forward.

Mail is temporarily being deferred on G1 (g1.ijnet.net) due to the user list index being corrupted.  We are currently rebuilding the index and expect service to be restored shortly.

Mail being temporarily deferred means that the gateway is responding to SMTP requests, but rather than accepting messages is responding with a 451 error to the sending server, asking the server to try again shortly.  Once the user list is rebuilt and mail is being accepted again, senders will be free to retry to deliver mail that was deferred.  No mail will be lost from this issue.

UPDATE: Service has been restored as of 8:30am central US time.

Starting at approximately 8:30pm central time on Thursday, January 21, 2010, mail delivery on MF2 was impacted by an extremely large block of large virus-laden email, causing our antivirus processing to slow delivery of mail. This issue has been rectified and delayed mail is now being delivered at normal speeds. We expect that all delayed mail will be delivered within the next 30 minutes or so.