We’re currently investigating an issue on the G1 gateway that is causing mail to not be delivered properly. It appears at this time that mail is queuing on the gateway and not being delivered to destination servers. We will be performing a database maintenance on the gateway to attempt to alleviate this issue.

We will update this post as soon as we have additional information.

Update: Database maintenance has completed and mail is flowing smoothly again. Please contact support@purity.net if you have any additional issues.

We’ve received reports from several customers that they’re having issues receiving email through the G1 gateway. Our initial investigation shows that G1 has been accepting mail but is possibly queuing mail before delivering to customer’s servers.

We’re doing further investigation at this time to determine a resolution to this issue and will update this post as we have more information. During our investigation there may be brief periods where G1 is unavailable due to reboots or other maintenance steps.  During any unavailability mail will queue at the sender’s server.

We apologize for any inconvenience this has caused.

Update at 7:38AM Central: We are running a database maintenance on G1 at this time to resolve issues with the internal databases. During this time G1 will not be available to receive mail. Mail will queue on the sender’s servers and be delivered after the maintenance has been completed. Unfortunately it is not possible to determine an ETA because database maintenance has a variable length. It is currently 25% complete. We will update this post when the status changes.

Update at 8:34AM Central: Database maintenance completed approximately 45 minutes ago and mail should be flowing normally at this time.  Any messages that were backlogged either have been delivered or will be delivered shortly. There is a large backup of quarantined mail waiting to be delivered to the Live Quarantine system which will take time to catch up.  Please contact support@purity.net if you are still seeing issues, but at this time mail appears to be flowing smoothly.  We will continue to monitor the situation.

This morning at approximately 7am central time, our MF2 gateway began to experience issues. Although the gateway was up and responding to network requests, it was not responding to SMTP requests. After investigating the issue it appears that the gateway was hit with an extremely heavy dictionary attack, essentially to the point of a denial of service, sent to a domain that was misconfigured and was not rejecting unknown users.

This caused our gateway to accept all mail received for that domain and queue it up in an attempt to deliver it to the destination server. Due to the large volume of mail, it caused the gateway’s processor load to spike, causing the gateway to shut down SMTP services to protect itself.

Once we determined what the issue was, it took us some time to go through the outgoing queue and remove the offending messages from the queue manually to reduce the amount of work the gateway had to do so that processor levels could be reduced enough to enable SMTP again.

SMTP services were re-enabled at approximately 8:30am central time and all mail should be flowing normally. No mail will have been lost during this outage, though some mail may be delayed past the point where services were restored due to the retry settings on the sender’s mail server.

We apologize for this outage and will be discussing internally ways to prevent something like this from happening again. The most likely outcome of this will be that we will fully activate the automated catch-all checking and disabling, which is designed to disable domains that accept mail for all users like this. We’ve put off doing so several times in the past due to customers expressing concern about it, but I believe it’s time to go ahead and implement this for the sake of all of our users.

All customers with service on MF2 will be receiving a 5% service credit due to this outage.

First, my apologies for the issues that we’re having and the confusion surrounding it.  We’ve isolated the issue and have taken steps to resolve it.  In this blog post I’m going to explain what happened and what we did to fix the issue.

What Happened

A few weeks ago we decided to bring our DNS services in house after having them hosted by a third party for the last year.  We configured our zones in our new DNS servers and did testing against them.  Everything was working properly, so we switched our DNS servers with our domain registrar.  Our old DNS hosting was still up and running at this point.  Unfortunately, when our account period with our old DNS host expired they changed the IP addresses on our record to one of their internal IP addresses.  I believe it’s their way of disabling an account.  Normally this wouldn’t be an issue, as we had changed our DNS servers well before that to point to our own DNS servers.

Unfortunately, we had an issue with our DNS servers that we didn’t notice during the changeover.  The serial number on the records on our DNS server was lower than the ones on our old DNS hosts records.  This caused some DNS servers to continue to look to our old DNS provider for records, thinking that our new servers had out of date information.  When our old DNS provider changed the IP addresses to disable the account, some mail senders picked up that change.

What We Did To Fix It

We have updated the serial numbers on all of our domains to be newer than the serial number on our old provider’s DNS servers.  This will allow DNS servers to pick up the proper, current records from our DNS servers and see them as valid.  I’m not sure exactly how long this will take for DNS servers that have cached incorrect information, so if you are still seeing issues with senders having trouble getting the correct DNS information for our gateways you’ll need to have them force a DNS cache update to get the newest information.  Feel free to point people to this blog post for an explanation of what happened.

We will be issuing all customers a 10% credit to their account based on the services they have with us.  For example, if you have a DomainProtect account at $6.95 per month, you’ll receive a 70 cent credit.  If you have a ServerProtect account at $39.95 per month, you’ll receive a $4.00 credit.  You should see the credit on your account in the next couple of days.

Once again, I’d like to apologize for this oversight on our part and assure you that we’ve learned from this situation should we need to transition DNS services again.

We moved our DNS servers several weeks ago, but apparently one of the OpenDNS servers had the wrong information cached. This was rectified a couple of days ago, which is why we are seeing these problems today.  Now, there are several senders that still have bad DNS information cached.  If you are not receiving messages from some senders, be sure to notify them that they need to update their DNS cache.

Here is how the sender can find out if their cache is bad:

The sender’s system administrator needs to do a lookup on their mail server to see what IP address they’re getting for our gateway. Additionally, it would be helpful if they could get us the nameserver records they’re seeing for ijnet.net as well.

Examples:

On windows:

nslookup -type=a g1.ijnet.net
nslookup -type=ns ijnet.net

On linux:

dig g1.ijnet.net a
dig ijnet.net ns

Our correct IP’s are:

MF1 – 216.246.89.41

MF2 – 216.246.89.42

G1 – 216.246.89.40

G2 – 216.246.89.37

As always, if you have any questions, feel free to contact us at support@purity.net.

Thank you for choosing Purity Networks.

Our G1 gateway has completed database maintenance, and normal mail flow has resumed.