This morning at approximately 7am central time, our MF2 gateway began to experience issues. Although the gateway was up and responding to network requests, it was not responding to SMTP requests. After investigating the issue it appears that the gateway was hit with an extremely heavy dictionary attack, essentially to the point of a denial of service, sent to a domain that was misconfigured and was not rejecting unknown users.

This caused our gateway to accept all mail received for that domain and queue it up in an attempt to deliver it to the destination server. Due to the large volume of mail, it caused the gateway’s processor load to spike, causing the gateway to shut down SMTP services to protect itself.

Once we determined what the issue was, it took us some time to go through the outgoing queue and remove the offending messages from the queue manually to reduce the amount of work the gateway had to do so that processor levels could be reduced enough to enable SMTP again.

SMTP services were re-enabled at approximately 8:30am central time and all mail should be flowing normally. No mail will have been lost during this outage, though some mail may be delayed past the point where services were restored due to the retry settings on the sender’s mail server.

We apologize for this outage and will be discussing internally ways to prevent something like this from happening again. The most likely outcome of this will be that we will fully activate the automated catch-all checking and disabling, which is designed to disable domains that accept mail for all users like this. We’ve put off doing so several times in the past due to customers expressing concern about it, but I believe it’s time to go ahead and implement this for the sake of all of our users.

All customers with service on MF2 will be receiving a 5% service credit due to this outage.

First, my apologies for the issues that we’re having and the confusion surrounding it.  We’ve isolated the issue and have taken steps to resolve it.  In this blog post I’m going to explain what happened and what we did to fix the issue.

What Happened

A few weeks ago we decided to bring our DNS services in house after having them hosted by a third party for the last year.  We configured our zones in our new DNS servers and did testing against them.  Everything was working properly, so we switched our DNS servers with our domain registrar.  Our old DNS hosting was still up and running at this point.  Unfortunately, when our account period with our old DNS host expired they changed the IP addresses on our record to one of their internal IP addresses.  I believe it’s their way of disabling an account.  Normally this wouldn’t be an issue, as we had changed our DNS servers well before that to point to our own DNS servers.

Unfortunately, we had an issue with our DNS servers that we didn’t notice during the changeover.  The serial number on the records on our DNS server was lower than the ones on our old DNS hosts records.  This caused some DNS servers to continue to look to our old DNS provider for records, thinking that our new servers had out of date information.  When our old DNS provider changed the IP addresses to disable the account, some mail senders picked up that change.

What We Did To Fix It

We have updated the serial numbers on all of our domains to be newer than the serial number on our old provider’s DNS servers.  This will allow DNS servers to pick up the proper, current records from our DNS servers and see them as valid.  I’m not sure exactly how long this will take for DNS servers that have cached incorrect information, so if you are still seeing issues with senders having trouble getting the correct DNS information for our gateways you’ll need to have them force a DNS cache update to get the newest information.  Feel free to point people to this blog post for an explanation of what happened.

We will be issuing all customers a 10% credit to their account based on the services they have with us.  For example, if you have a DomainProtect account at $6.95 per month, you’ll receive a 70 cent credit.  If you have a ServerProtect account at $39.95 per month, you’ll receive a $4.00 credit.  You should see the credit on your account in the next couple of days.

Once again, I’d like to apologize for this oversight on our part and assure you that we’ve learned from this situation should we need to transition DNS services again.

We moved our DNS servers several weeks ago, but apparently one of the OpenDNS servers had the wrong information cached. This was rectified a couple of days ago, which is why we are seeing these problems today.  Now, there are several senders that still have bad DNS information cached.  If you are not receiving messages from some senders, be sure to notify them that they need to update their DNS cache.

Here is how the sender can find out if their cache is bad:

The sender’s system administrator needs to do a lookup on their mail server to see what IP address they’re getting for our gateway. Additionally, it would be helpful if they could get us the nameserver records they’re seeing for ijnet.net as well.

Examples:

On windows:

nslookup -type=a g1.ijnet.net
nslookup -type=ns ijnet.net

On linux:

dig g1.ijnet.net a
dig ijnet.net ns

Our correct IP’s are:

MF1 – 216.246.89.41

MF2 – 216.246.89.42

G1 – 216.246.89.40

G2 – 216.246.89.37

As always, if you have any questions, feel free to contact us at support@purity.net.

Thank you for choosing Purity Networks.

Our G1 gateway has completed database maintenance, and normal mail flow has resumed.

After some further research, we’re still pretty confident that any DNS issues that are being experienced are not on our end.  From everything we can see, our DNS servers are all responding properly and are returning the proper IP addresses for our gateways.

A recent DNS cache poisoning vulnerability was announced in the last few weeks, and there are reports of it being exploited in the wild.  US-CERT VU#800113 contains details of the vulnerability.  We do not know whether this may be impacting certain ISPs DNS servers, but it is a supposition based on the evidence we have available to us at this time.

To show that our DNS is working properly, we’ve used DNS Stuff to compile a report for each of our gateways showing that all the root DNS servers list our DNS servers as authoritative and that each of our DNS servers is returning the proper response for our gateway addresses.  Click the link for each gateway to see the report.  You can use this report when talking with an ISP about them resolving the issue.

MF1

MF2

G1

G2