Many of you have noticed over the past week or so that we have been seeing a higher than normal amount of spam slip through the filters and make it to users’ inboxes.  We have been in contact with MailFoundry and they have told us that there was an issue with their system that takes messages from the reported spam feed and injects them into the rule creation engine, thus updating the spam definitions.  This has caused reported spam to not get put into the engine definitions in a timely manner, leading to an increase in non-filtered spam over the last week or so.

Reportedly this is looking better today, but the MailFoundry developers are monitoring the situation to see if any additional adjustment is necessary.  For those who have reported this increase in spam to our support department, please monitor the situation over the next few days and let us know if the amount of spam slipping through begins to reduce.

UPDATE: The issues with elevated levels of spam slipping through the filters appears to have been resolved.  Those of you who have reported this issue, please let us know by updating your support ticket or posting a comment on this post if the issue seems to be resolved for you.

We are currently performing emergency database maintenance on the G1 gateway to repair a corrupted user database.  We estimate a total time for repair of 30-60 minutes.  During a portion of this time SMTP services on G1 will be disabled.  Sending servers will queue the mail until services are restored and will deliver at that time.

MF2 is currently undergoing emergency user database maintenance to help improve performance on the gateway.  During this time mail services are stopped and mail will queue on sending servers, to be delivered normally within 4 hours based upon retry settings at the sender.

Emergency maintenance is expected to take approximately 30 minutes.

UPDATE: Maintenance complete @ 0243

Mail is temporarily being deferred on G1 (g1.ijnet.net) due to the user list index being corrupted.  We are currently rebuilding the index and expect service to be restored shortly.

Mail being temporarily deferred means that the gateway is responding to SMTP requests, but rather than accepting messages is responding with a 451 error to the sending server, asking the server to try again shortly.  Once the user list is rebuilt and mail is being accepted again, senders will be free to retry to deliver mail that was deferred.  No mail will be lost from this issue.

UPDATE: Service has been restored as of 8:30am central US time.

Starting at approximately 8:30pm central time on Thursday, January 21, 2010, mail delivery on MF2 was impacted by an extremely large block of large virus-laden email, causing our antivirus processing to slow delivery of mail. This issue has been rectified and delayed mail is now being delivered at normal speeds. We expect that all delayed mail will be delivered within the next 30 minutes or so.

Due to a hard drive failure in the RAID array on MF1, we have replaced the drive and the array is now rebuilding. We are monitoring the rebuild and expect it to complete within the next 8-10 hours. There should be very little customer impact related to this emergency repair, though there may be brief periods where the gateway is unable to respond to SMTP requests. These outages should be brief and infrequent.