Mail is temporarily being deferred on G1 (g1.ijnet.net) due to the user list index being corrupted.  We are currently rebuilding the index and expect service to be restored shortly.

Mail being temporarily deferred means that the gateway is responding to SMTP requests, but rather than accepting messages is responding with a 451 error to the sending server, asking the server to try again shortly.  Once the user list is rebuilt and mail is being accepted again, senders will be free to retry to deliver mail that was deferred.  No mail will be lost from this issue.

UPDATE: Service has been restored as of 8:30am central US time.

The staff here at Purity Networks has been working hard on some new features and functionality that we’ll be rolling out over the next few months. I wanted to take a few minutes and tell you about some of the things that are coming and approximate timeframes on when you’ll see them roll out:

New Integrated Live Quarantine

We will soon be integrating the Live Quarantine system into the Account Manager.  This will allow us to integrate the logins into one site, rather than having two separate sites as there are now.  One of the advantages of this is that the ability to theme the Account Manager is much more robust than the current Live Quarantine system.
Planned Release: February 2010

Additional Admin Accounts and Fine-grained Access Control

When the Live Quarantine is combined into the Account Manager, one of the major advantages is that we will also be rolling out the ability for our customers to give accounts access to administrative functions.  This access control will be very flexible and will allow the customer to give access to various areas of the Account Manager and even restrict access to specific domains.
Planned Release: February 2010

New Backend Technology

We will soon be replacing the backend scanning technology that we use with a completely new system.  The new technology we’ll be using is the same technology that will be going into our forthcoming antispam appliances.  Many of you know that we currently use technology from MailFoundry as our scanning engine.  We’ve used their products as the core of our service since we launched in 2005, and were extremely happy with their technology until recently.  Unfortunately, we feel that they have not been keeping up with changes necessary to effectively combat spam and other threats.

The core scanning technology we’ll be using in our new backend services and in our appliances is from Cloudmark, and we’ve been testing its effectiveness for several months now.   We’re happy to report we are extremely pleased with the performance and effectiveness and are beginning integration work now.
Planned Release: April 2010

We have some more things planned for this year as well, but I’ll leave those for a later message.  I think the things I posted above are enough to keep us busy for a little bit!

Please let us know if you have any questions or thoughts about our forthcoming changes.

Starting at approximately 8:30pm central time on Thursday, January 21, 2010, mail delivery on MF2 was impacted by an extremely large block of large virus-laden email, causing our antivirus processing to slow delivery of mail. This issue has been rectified and delayed mail is now being delivered at normal speeds. We expect that all delayed mail will be delivered within the next 30 minutes or so.

Due to a hard drive failure in the RAID array on MF1, we have replaced the drive and the array is now rebuilding. We are monitoring the rebuild and expect it to complete within the next 8-10 hours. There should be very little customer impact related to this emergency repair, though there may be brief periods where the gateway is unable to respond to SMTP requests. These outages should be brief and infrequent.

This morning at approximately 7am central time, our MF2 gateway began to experience issues. Although the gateway was up and responding to network requests, it was not responding to SMTP requests. After investigating the issue it appears that the gateway was hit with an extremely heavy dictionary attack, essentially to the point of a denial of service, sent to a domain that was misconfigured and was not rejecting unknown users.

This caused our gateway to accept all mail received for that domain and queue it up in an attempt to deliver it to the destination server. Due to the large volume of mail, it caused the gateway’s processor load to spike, causing the gateway to shut down SMTP services to protect itself.

Once we determined what the issue was, it took us some time to go through the outgoing queue and remove the offending messages from the queue manually to reduce the amount of work the gateway had to do so that processor levels could be reduced enough to enable SMTP again.

SMTP services were re-enabled at approximately 8:30am central time and all mail should be flowing normally. No mail will have been lost during this outage, though some mail may be delayed past the point where services were restored due to the retry settings on the sender’s mail server.

We apologize for this outage and will be discussing internally ways to prevent something like this from happening again. The most likely outcome of this will be that we will fully activate the automated catch-all checking and disabling, which is designed to disable domains that accept mail for all users like this. We’ve put off doing so several times in the past due to customers expressing concern about it, but I believe it’s time to go ahead and implement this for the sake of all of our users.

All customers with service on MF2 will be receiving a 5% service credit due to this outage.