First, my apologies for the issues that we’re having and the confusion surrounding it.  We’ve isolated the issue and have taken steps to resolve it.  In this blog post I’m going to explain what happened and what we did to fix the issue.

What Happened

A few weeks ago we decided to bring our DNS services in house after having them hosted by a third party for the last year.  We configured our zones in our new DNS servers and did testing against them.  Everything was working properly, so we switched our DNS servers with our domain registrar.  Our old DNS hosting was still up and running at this point.  Unfortunately, when our account period with our old DNS host expired they changed the IP addresses on our record to one of their internal IP addresses.  I believe it’s their way of disabling an account.  Normally this wouldn’t be an issue, as we had changed our DNS servers well before that to point to our own DNS servers.

Unfortunately, we had an issue with our DNS servers that we didn’t notice during the changeover.  The serial number on the records on our DNS server was lower than the ones on our old DNS hosts records.  This caused some DNS servers to continue to look to our old DNS provider for records, thinking that our new servers had out of date information.  When our old DNS provider changed the IP addresses to disable the account, some mail senders picked up that change.

What We Did To Fix It

We have updated the serial numbers on all of our domains to be newer than the serial number on our old provider’s DNS servers.  This will allow DNS servers to pick up the proper, current records from our DNS servers and see them as valid.  I’m not sure exactly how long this will take for DNS servers that have cached incorrect information, so if you are still seeing issues with senders having trouble getting the correct DNS information for our gateways you’ll need to have them force a DNS cache update to get the newest information.  Feel free to point people to this blog post for an explanation of what happened.

We will be issuing all customers a 10% credit to their account based on the services they have with us.  For example, if you have a DomainProtect account at $6.95 per month, you’ll receive a 70 cent credit.  If you have a ServerProtect account at $39.95 per month, you’ll receive a $4.00 credit.  You should see the credit on your account in the next couple of days.

Once again, I’d like to apologize for this oversight on our part and assure you that we’ve learned from this situation should we need to transition DNS services again.

Some of our customers may have received warning notices regarding possible suspension of their account due to unpaid invoices.   Please take a look at the notice and if the notice is for an invoice that is due in the next few days (i.e. not due yet) you can disregard it.  If it is for an invoice that is already past its due date, please make payment on the invoice to avoid service interruption.

The issue with the invalid notices has been corrected and should not happen again in the future.

Some customer have reported problems with emails to our trouble ticketing system bouncing since last night.  First, I apologize for any inconvenience this may have caused.

This issue has been fixed and emails to our ticketing system are working again.  Please re-send any emails that bounced so that we can assist you.

This issue was caused by an upgrade to our trouble ticketing system and a configuration error after the upgrade.

MailFoundry has pushed a new set of spam updates that corrects the problem that was causing the caching of mail and mail is once again flowing smoothly through the gateways.

Last night we experienced a complete infrastructure power outage due to a problem with a power distribution unit that delivers power to our cabinet at our datacenter.  Unfortunately, it took some time to diagnose exactly what particular piece of equipment was causing the problem and then to resolve the issue.

The problem has been resolved at this time, however, we will need to schedule a short downtime in the next week or so to upgrade a piece of power hardware to reduce the chances of an outage like this happening again in the future.

By way of apology for the extended downtime, we will be issuing $10 service credits for all ServerProtect customers and $2 service credits for all DomainProtect customers.  This credit will appear in your account in the next week or so.