Mail is temporarily being deferred on G1 (g1.ijnet.net) due to the user list index being corrupted.  We are currently rebuilding the index and expect service to be restored shortly.

Mail being temporarily deferred means that the gateway is responding to SMTP requests, but rather than accepting messages is responding with a 451 error to the sending server, asking the server to try again shortly.  Once the user list is rebuilt and mail is being accepted again, senders will be free to retry to deliver mail that was deferred.  No mail will be lost from this issue.

UPDATE: Service has been restored as of 8:30am central US time.

Starting at approximately 8:30pm central time on Thursday, January 21, 2010, mail delivery on MF2 was impacted by an extremely large block of large virus-laden email, causing our antivirus processing to slow delivery of mail. This issue has been rectified and delayed mail is now being delivered at normal speeds. We expect that all delayed mail will be delivered within the next 30 minutes or so.

First, my apologies for the issues that we’re having and the confusion surrounding it.  We’ve isolated the issue and have taken steps to resolve it.  In this blog post I’m going to explain what happened and what we did to fix the issue.

What Happened

A few weeks ago we decided to bring our DNS services in house after having them hosted by a third party for the last year.  We configured our zones in our new DNS servers and did testing against them.  Everything was working properly, so we switched our DNS servers with our domain registrar.  Our old DNS hosting was still up and running at this point.  Unfortunately, when our account period with our old DNS host expired they changed the IP addresses on our record to one of their internal IP addresses.  I believe it’s their way of disabling an account.  Normally this wouldn’t be an issue, as we had changed our DNS servers well before that to point to our own DNS servers.

Unfortunately, we had an issue with our DNS servers that we didn’t notice during the changeover.  The serial number on the records on our DNS server was lower than the ones on our old DNS hosts records.  This caused some DNS servers to continue to look to our old DNS provider for records, thinking that our new servers had out of date information.  When our old DNS provider changed the IP addresses to disable the account, some mail senders picked up that change.

What We Did To Fix It

We have updated the serial numbers on all of our domains to be newer than the serial number on our old provider’s DNS servers.  This will allow DNS servers to pick up the proper, current records from our DNS servers and see them as valid.  I’m not sure exactly how long this will take for DNS servers that have cached incorrect information, so if you are still seeing issues with senders having trouble getting the correct DNS information for our gateways you’ll need to have them force a DNS cache update to get the newest information.  Feel free to point people to this blog post for an explanation of what happened.

We will be issuing all customers a 10% credit to their account based on the services they have with us.  For example, if you have a DomainProtect account at $6.95 per month, you’ll receive a 70 cent credit.  If you have a ServerProtect account at $39.95 per month, you’ll receive a $4.00 credit.  You should see the credit on your account in the next couple of days.

Once again, I’d like to apologize for this oversight on our part and assure you that we’ve learned from this situation should we need to transition DNS services again.

Some of our customers may have received warning notices regarding possible suspension of their account due to unpaid invoices.   Please take a look at the notice and if the notice is for an invoice that is due in the next few days (i.e. not due yet) you can disregard it.  If it is for an invoice that is already past its due date, please make payment on the invoice to avoid service interruption.

The issue with the invalid notices has been corrected and should not happen again in the future.

Some customer have reported problems with emails to our trouble ticketing system bouncing since last night.  First, I apologize for any inconvenience this may have caused.

This issue has been fixed and emails to our ticketing system are working again.  Please re-send any emails that bounced so that we can assist you.

This issue was caused by an upgrade to our trouble ticketing system and a configuration error after the upgrade.

MailFoundry has pushed a new set of spam updates that corrects the problem that was causing the caching of mail and mail is once again flowing smoothly through the gateways.