At approximately 7:45pm CST on 4/5 an anomaly occurred on the MF1 gateway, causing all ServerProtect, DomainProtect and cloudprotect domains on the gateway to lose their “auto-discover” setting. This setting essentially tells the gateway to keep a dynamic list of users for each domain based upon the results of the destination server. This allows our customers to not have to input a list of valid accounts for each domain.

The loss of this setting caused the gateway to not be able to determine which users were valid and thus it rejected mail as “user unknown”. We have opened a ticket with MailFoundry, the provider of our antispam technology, to determine what might have caused the loss of this setting.

We have run a script to re-enable this setting for all domains on the gateway and mail services should be restored at this time.

We have issued a 15% refund for all ServerProtect, DomainProtect and cloudprotect customers on the MF1 gateway under our Uptime Guarantee.

We’d like to apologize to our affected customers! This is a very unusual situation that we’ve never seen before. If we are able to determine a root cause we will report that here.

At approximately 11:00am CDT we began experiencing periodic issues with the G1 gateway. In order to correct these issues we’ve begun a database maintenance, which will cause the gateway to be unable to accept connections for the duration of this maintenance.

During this maintenance sending servers will queue the mail and will attempt to redeliver once service is restored. There is no ETA for completion of the maintenance, as there are many variables that make it difficult to determine how long the maintenance will take.

We apologize for the inconvenience!

Update 1:51pm: Service has been restored on G1. All customers impacted have received a 5% service credit which will be applied towards your next bill.

At approximately 7:15pm central time we began experiencing periodic issues with the MF1 gateway. In order to correct these issues we’ve begun a database maintenance, which will cause the gateway to be unable to accept connections for the duration of this maintenance.

During this maintenance sending servers will queue the mail and will attempt to redeliver once service is restored. There is no ETA for completion of the maintenance, as there are many variables that make it difficult to determine how long the maintenance will take.

Update: Our apologies for missing the update on this! The maintenance took 45 minutes to complete and all services were restored after that period.

It appears that over the last 10 minutes our datacenter has experienced intermittent network availability issues. They are aware of the issue and working on a resolution. As we post this it appears that the network has recovered, but the possibility of additional issues may occur until we receive word of final resolution from our datacenter.

As the issues continue it is important to note that if a sending mail server is unable to reach our gateway it will queue any mail and retry again according to the configuration on that server. We will update this post as more information becomes available.

UPDATE:

At approximately 7:20 PM CST, datacenter network operations team detected a metro fiber cut which triggered an event affecting the network at the datacenter. The network has stabilized and the event has concluded, but network engineers are currently investigating the root cause of the instability after the fiber cut. While our metro fiber is redundant, we continue to work with our fiber vendor on the repair of the fiber cut and are monitoring situation closely.

Network instability lasted for approximately 15 minutes until approximately 7:35 PM CST.

Our support portal provider will be performing maintenance on their infrastructure from 6:00pm to 8:00pm Central Time tomorrow, 01/21/2012. During this time, emails sent to support@purity.net will be queued until the maintenance is complete. The support portal web site will be unavailable during this maintenance window.

For urgent support issues, please contact us via telephone at +1 815 261-4600. Non-urgent support issues can still be sent to support@purity.net but will be processed after the maintenance is complete.

Over the last 10 minutes we have been experiencing issues with the MF2 gateway leading to periodic connectivity issues. We are performing a database maintenance on the gateway in an attempt to clear up the instability. During this maintenance window MF2 will not accept mail. All mail sent to users on MF2 will queue at the sending server and will be delivered once the maintenance has completed and the sending server re-attempts to send the message.

Unfortunately there is no way to predict an ETA on database maintenance as we don’t know how long it will take to clean any problems it might find. We will update this blog post when MF2 service has been restored.

Our apologies for the inconvenience!

Update: At 10:17am central US time service for MF2 was restored. Mail queued at the sending server will be retried according to that sender’s retry delivery settings.