We are currently experiencing issues with mail queuing on the MF1 gateway and will be starting an emergency database maintenance in the next few minutes. Unfortunately, there is no way to determine an ETA for this process. During the database maintenance there will be periods where the gateway will be unable to accept mail and mail will queue on the senders’ mail servers. We will update this blog post when the maintenance is complete.

Update 2:11pm: Database maintenance completed at approximately 1:30pm central time, however the issue of mail queuing does not appear to be resolved. MailFoundry Support is now monitoring the situation and will update us as soon as a resolution is found. We are monitoring the mail queues and forcing delivery on them to keep the mail flowing properly.

Update 4:48pm: We’ve been monitoring the situation for over 2 hours now and it appears resolved. There was a series of messages that came into the queue with some strangely crafted headers that caused the queues to back up. MailFoundry is aware of the issue and is looking for a way to stop this from happening in the future.

Purity Networks phone support will be unavailable from 10/25 through 11/01. Our support department will still be available and will be responding to tickets opened via email to support@purity.net and through the Account Manager.

Due to a previously unrecognized bug in our reporting system we regret to say that reporting data for domains on the MF2 gateway will be missing from 10-12-2010 through 10-20-2010. This issue has been corrected and will not be an issue going forward.

Beginning at 3:30pm central time on September 21, 2010, Purity Networks phone support availability will be limited. Full support will resume at 9:00pm central time on September 22, 2010. During this time period we recommend that you contact support by opening a ticket via the Account Manager or by emailing support@purity.net.

At approximately 3:49pm central time on 8/1/2010 our upstream network provider experienced a complete network outage, causing all inbound and outbound traffic to our network to be rejected. This outage lasted until approximately 5:05pm central time, at which time it appears that network services have been fully restored.

We are waiting on a report from our provider as to the causes and resolution of the issue. We will update this blog post once we receive more information.

During this outage mail would have queued on the sender’s mail server and would not be lost. It may be delayed due to the sender’s retry timing policies.

We have issued a 5% service credit to all of our customers based on our uptime guarantee.

UPDATE: The following is the RFO (Reason for Outage) provided to us by our upstream provider:

A carefully crafted Denial of Service attack was launched at a customer and several of our routers, which caused the routers to repeatedly drop IGP (internal) and BGP (external) routing protocols. We identified the attack and found that it exploited a loophole in the control-plane policer packet matching logic which normally protects against these types of attacks, allowing it to slip past these filters and cause the amount of impact that it did.

As a work-around we have applied additional packet filters at multiple network edges to block this type of traffic, and are working with our router vendor to address the root cause of the issue, but we believe that this attack has been successfully filtered and that the network should be stable going forward.

The Purity Networks team had a great time at HostingCon 2010 in Austin, TX. It was great to see some of our customers and meet some new people in the hosted services industry too!

Congratulations to John Brancela of Infinitie Networks — he was the winner of the Xbox 360 250GB that we gave away at the show!

There were some great parties at the show, too! Thank you to all the vendors who hosted them.  The shot below is from the cPanel party on Tuesday night.

Beta Program Update

First, thank you to everyone who contacted us about beta testing our new scanning engine. We haven’t forgotten you! Due to travel for HostingCon, we’re a little bit behind schedule, but never fear! We’re still working hard on getting the new engine in place. We hope to be ready to go with the beta in the next couple of weeks. We have the new engine up and running (it works great, by the way!) and we’re just finishing the integration with the Account Manager and reporting.

Talk to you all soon!