This morning we are experiencing queuing of email on the G1 gateway. We are performing an emergency database maintenance cycle to attempt to mitigate the situation. The ETA for resolution at this time is unknown.

UPDATE: During this maintenance there may be periods of time where the G1 gateway is unavailable to accept connections. This will cause mail to queue on the sender’s server and be retried at a later time according to their settings.

UPDATE 11:40AM: Database maintenance is complete and mail appears to be flowing properly again. Please note that it may take some time for mail flow to return to normal as the gateway has a backlog of mail to process. We apologize for any inconvenience this might have caused!

We are currently experiencing issues with mail queuing on the MF1 gateway and will be starting an emergency database maintenance in the next few minutes. Unfortunately, there is no way to determine an ETA for this process. During the database maintenance there will be periods where the gateway will be unable to accept mail and mail will queue on the senders’ mail servers. We will update this blog post when the maintenance is complete.

Update 2:11pm: Database maintenance completed at approximately 1:30pm central time, however the issue of mail queuing does not appear to be resolved. MailFoundry Support is now monitoring the situation and will update us as soon as a resolution is found. We are monitoring the mail queues and forcing delivery on them to keep the mail flowing properly.

Update 4:48pm: We’ve been monitoring the situation for over 2 hours now and it appears resolved. There was a series of messages that came into the queue with some strangely crafted headers that caused the queues to back up. MailFoundry is aware of the issue and is looking for a way to stop this from happening in the future.

At approximately 3:49pm central time on 8/1/2010 our upstream network provider experienced a complete network outage, causing all inbound and outbound traffic to our network to be rejected. This outage lasted until approximately 5:05pm central time, at which time it appears that network services have been fully restored.

We are waiting on a report from our provider as to the causes and resolution of the issue. We will update this blog post once we receive more information.

During this outage mail would have queued on the sender’s mail server and would not be lost. It may be delayed due to the sender’s retry timing policies.

We have issued a 5% service credit to all of our customers based on our uptime guarantee.

UPDATE: The following is the RFO (Reason for Outage) provided to us by our upstream provider:

A carefully crafted Denial of Service attack was launched at a customer and several of our routers, which caused the routers to repeatedly drop IGP (internal) and BGP (external) routing protocols. We identified the attack and found that it exploited a loophole in the control-plane policer packet matching logic which normally protects against these types of attacks, allowing it to slip past these filters and cause the amount of impact that it did.

As a work-around we have applied additional packet filters at multiple network edges to block this type of traffic, and are working with our router vendor to address the root cause of the issue, but we believe that this attack has been successfully filtered and that the network should be stable going forward.

Mailfoundry engineers have temporarily taken the G1 gateway offline as they investigate an issue causing some mail to get stuck in queue.  We expect services to be restored shortly but currently do not have an ETA.  At this time mail will queue on the sending server and will be delivered after G1 comes back online and the sender retries.

Update: G1 is back up and performing normally.

We have disabled the SORBS black list, as we’ve been getting a lot of reports of it being overly aggressive lately.  Those of you who have been having issues with SORBS blocking mail to your users should see that behavior stop immediately.

Many of you have noticed over the past week or so that we have been seeing a higher than normal amount of spam slip through the filters and make it to users’ inboxes.  We have been in contact with MailFoundry and they have told us that there was an issue with their system that takes messages from the reported spam feed and injects them into the rule creation engine, thus updating the spam definitions.  This has caused reported spam to not get put into the engine definitions in a timely manner, leading to an increase in non-filtered spam over the last week or so.

Reportedly this is looking better today, but the MailFoundry developers are monitoring the situation to see if any additional adjustment is necessary.  For those who have reported this increase in spam to our support department, please monitor the situation over the next few days and let us know if the amount of spam slipping through begins to reduce.

UPDATE: The issues with elevated levels of spam slipping through the filters appears to have been resolved.  Those of you who have reported this issue, please let us know by updating your support ticket or posting a comment on this post if the issue seems to be resolved for you.