We are currently performing emergency database maintenance on the G1 gateway to repair a corrupted user database.  We estimate a total time for repair of 30-60 minutes.  During a portion of this time SMTP services on G1 will be disabled.  Sending servers will queue the mail until services are restored and will deliver at that time.

Mail is temporarily being deferred on G1 (g1.ijnet.net) due to the user list index being corrupted.  We are currently rebuilding the index and expect service to be restored shortly.

Mail being temporarily deferred means that the gateway is responding to SMTP requests, but rather than accepting messages is responding with a 451 error to the sending server, asking the server to try again shortly.  Once the user list is rebuilt and mail is being accepted again, senders will be free to retry to deliver mail that was deferred.  No mail will be lost from this issue.

UPDATE: Service has been restored as of 8:30am central US time.

This morning at approximately 7am central time, our MF2 gateway began to experience issues. Although the gateway was up and responding to network requests, it was not responding to SMTP requests. After investigating the issue it appears that the gateway was hit with an extremely heavy dictionary attack, essentially to the point of a denial of service, sent to a domain that was misconfigured and was not rejecting unknown users.

This caused our gateway to accept all mail received for that domain and queue it up in an attempt to deliver it to the destination server. Due to the large volume of mail, it caused the gateway’s processor load to spike, causing the gateway to shut down SMTP services to protect itself.

Once we determined what the issue was, it took us some time to go through the outgoing queue and remove the offending messages from the queue manually to reduce the amount of work the gateway had to do so that processor levels could be reduced enough to enable SMTP again.

SMTP services were re-enabled at approximately 8:30am central time and all mail should be flowing normally. No mail will have been lost during this outage, though some mail may be delayed past the point where services were restored due to the retry settings on the sender’s mail server.

We apologize for this outage and will be discussing internally ways to prevent something like this from happening again. The most likely outcome of this will be that we will fully activate the automated catch-all checking and disabling, which is designed to disable domains that accept mail for all users like this. We’ve put off doing so several times in the past due to customers expressing concern about it, but I believe it’s time to go ahead and implement this for the sake of all of our users.

All customers with service on MF2 will be receiving a 5% service credit due to this outage.

Earlier today, between approximately 3:00pm and 5:00pm central time, we experienced issues with the DNS servers providing resolution for the easyantispam.com and protectedmail.net domains.  This outage impacted access to web resources, such as the main easyantispam.com website, the Account Manager and the Live Quarantine.  Mail traffic was not impacted, as the ijnet.net domain which our gateways use is listed in a separate set of DNS servers.

This DNS outage was due to a third party DNS provider’s network issues.  We’re currently exploring different options for DNS for our website infrastructure.

At this time, all DNS issues appear to have been resolved.

This morning at approximately 7:00am central time, we experienced a power failure in our cabinet at our datacenter, causing loss of connectivity to our servers.  This issue was caused by a severe increase in power consumption by some of our equipment, causing our power distribution unit to shut down due to more power being drawn than it could handle.

Connectivity was restored at approximately 8:15am and all gateways are functioning properly at this time.

We’re investigating the cause of the increased power usage and have an order in for a PDU that will handle larger power loads in case of situations like this.  We will need to take a short planned maintenance when that new PDU arrives in order to install it.

As some of you may remember, we experienced a similar issue in May.  The unit we experienced the issue with in May was replaced by our datacenter technicians with a new PDU of the same make and model that was in stock as a replacement part.  We had determined at the time that the PDU itself likely had a problem, which was why we replaced it.