Starting at approximately 8:30pm central time on Thursday, January 21, 2010, mail delivery on MF2 was impacted by an extremely large block of large virus-laden email, causing our antivirus processing to slow delivery of mail. This issue has been rectified and delayed mail is now being delivered at normal speeds. We expect that all delayed mail will be delivered within the next 30 minutes or so.

Due to a hard drive failure in the RAID array on MF1, we have replaced the drive and the array is now rebuilding. We are monitoring the rebuild and expect it to complete within the next 8-10 hours. There should be very little customer impact related to this emergency repair, though there may be brief periods where the gateway is unable to respond to SMTP requests. These outages should be brief and infrequent.

This morning at approximately 7am central time, our MF2 gateway began to experience issues. Although the gateway was up and responding to network requests, it was not responding to SMTP requests. After investigating the issue it appears that the gateway was hit with an extremely heavy dictionary attack, essentially to the point of a denial of service, sent to a domain that was misconfigured and was not rejecting unknown users.

This caused our gateway to accept all mail received for that domain and queue it up in an attempt to deliver it to the destination server. Due to the large volume of mail, it caused the gateway’s processor load to spike, causing the gateway to shut down SMTP services to protect itself.

Once we determined what the issue was, it took us some time to go through the outgoing queue and remove the offending messages from the queue manually to reduce the amount of work the gateway had to do so that processor levels could be reduced enough to enable SMTP again.

SMTP services were re-enabled at approximately 8:30am central time and all mail should be flowing normally. No mail will have been lost during this outage, though some mail may be delayed past the point where services were restored due to the retry settings on the sender’s mail server.

We apologize for this outage and will be discussing internally ways to prevent something like this from happening again. The most likely outcome of this will be that we will fully activate the automated catch-all checking and disabling, which is designed to disable domains that accept mail for all users like this. We’ve put off doing so several times in the past due to customers expressing concern about it, but I believe it’s time to go ahead and implement this for the sake of all of our users.

All customers with service on MF2 will be receiving a 5% service credit due to this outage.

First, my apologies for the issues that we’re having and the confusion surrounding it.  We’ve isolated the issue and have taken steps to resolve it.  In this blog post I’m going to explain what happened and what we did to fix the issue.

What Happened

A few weeks ago we decided to bring our DNS services in house after having them hosted by a third party for the last year.  We configured our zones in our new DNS servers and did testing against them.  Everything was working properly, so we switched our DNS servers with our domain registrar.  Our old DNS hosting was still up and running at this point.  Unfortunately, when our account period with our old DNS host expired they changed the IP addresses on our record to one of their internal IP addresses.  I believe it’s their way of disabling an account.  Normally this wouldn’t be an issue, as we had changed our DNS servers well before that to point to our own DNS servers.

Unfortunately, we had an issue with our DNS servers that we didn’t notice during the changeover.  The serial number on the records on our DNS server was lower than the ones on our old DNS hosts records.  This caused some DNS servers to continue to look to our old DNS provider for records, thinking that our new servers had out of date information.  When our old DNS provider changed the IP addresses to disable the account, some mail senders picked up that change.

What We Did To Fix It

We have updated the serial numbers on all of our domains to be newer than the serial number on our old provider’s DNS servers.  This will allow DNS servers to pick up the proper, current records from our DNS servers and see them as valid.  I’m not sure exactly how long this will take for DNS servers that have cached incorrect information, so if you are still seeing issues with senders having trouble getting the correct DNS information for our gateways you’ll need to have them force a DNS cache update to get the newest information.  Feel free to point people to this blog post for an explanation of what happened.

We will be issuing all customers a 10% credit to their account based on the services they have with us.  For example, if you have a DomainProtect account at $6.95 per month, you’ll receive a 70 cent credit.  If you have a ServerProtect account at $39.95 per month, you’ll receive a $4.00 credit.  You should see the credit on your account in the next couple of days.

Once again, I’d like to apologize for this oversight on our part and assure you that we’ve learned from this situation should we need to transition DNS services again.

Our G1 gateway has completed database maintenance, and normal mail flow has resumed.

Our G1 gateway began experiencing issues sometime around 8:00 am CST today.  We did not become aware of the issue until 10:30 this morning, and once we knew there was a problem we began taking the necessary steps to fix it.   Apparently there was a database error in the gateway that began causing all mail to be put into an unscanned queue and not processed.  The gateway is currently undergoing database maintenance, and should be back up and running within the hour.

We apologize for the inconvenience this has caused, and we thank you for choosing Purity Networks.