We are currently performing emergency database maintenance on the G1 gateway to repair a corrupted user database.  We estimate a total time for repair of 30-60 minutes.  During a portion of this time SMTP services on G1 will be disabled.  Sending servers will queue the mail until services are restored and will deliver at that time.

MF2 is currently undergoing emergency user database maintenance to help improve performance on the gateway.  During this time mail services are stopped and mail will queue on sending servers, to be delivered normally within 4 hours based upon retry settings at the sender.

Emergency maintenance is expected to take approximately 30 minutes.

UPDATE: Maintenance complete @ 0243

Mail is temporarily being deferred on G1 (g1.ijnet.net) due to the user list index being corrupted.  We are currently rebuilding the index and expect service to be restored shortly.

Mail being temporarily deferred means that the gateway is responding to SMTP requests, but rather than accepting messages is responding with a 451 error to the sending server, asking the server to try again shortly.  Once the user list is rebuilt and mail is being accepted again, senders will be free to retry to deliver mail that was deferred.  No mail will be lost from this issue.

UPDATE: Service has been restored as of 8:30am central US time.

Starting at approximately 8:30pm central time on Thursday, January 21, 2010, mail delivery on MF2 was impacted by an extremely large block of large virus-laden email, causing our antivirus processing to slow delivery of mail. This issue has been rectified and delayed mail is now being delivered at normal speeds. We expect that all delayed mail will be delivered within the next 30 minutes or so.

Due to a hard drive failure in the RAID array on MF1, we have replaced the drive and the array is now rebuilding. We are monitoring the rebuild and expect it to complete within the next 8-10 hours. There should be very little customer impact related to this emergency repair, though there may be brief periods where the gateway is unable to respond to SMTP requests. These outages should be brief and infrequent.

This morning at approximately 7am central time, our MF2 gateway began to experience issues. Although the gateway was up and responding to network requests, it was not responding to SMTP requests. After investigating the issue it appears that the gateway was hit with an extremely heavy dictionary attack, essentially to the point of a denial of service, sent to a domain that was misconfigured and was not rejecting unknown users.

This caused our gateway to accept all mail received for that domain and queue it up in an attempt to deliver it to the destination server. Due to the large volume of mail, it caused the gateway’s processor load to spike, causing the gateway to shut down SMTP services to protect itself.

Once we determined what the issue was, it took us some time to go through the outgoing queue and remove the offending messages from the queue manually to reduce the amount of work the gateway had to do so that processor levels could be reduced enough to enable SMTP again.

SMTP services were re-enabled at approximately 8:30am central time and all mail should be flowing normally. No mail will have been lost during this outage, though some mail may be delayed past the point where services were restored due to the retry settings on the sender’s mail server.

We apologize for this outage and will be discussing internally ways to prevent something like this from happening again. The most likely outcome of this will be that we will fully activate the automated catch-all checking and disabling, which is designed to disable domains that accept mail for all users like this. We’ve put off doing so several times in the past due to customers expressing concern about it, but I believe it’s time to go ahead and implement this for the sake of all of our users.

All customers with service on MF2 will be receiving a 5% service credit due to this outage.