Over the last 10 minutes we have been experiencing issues with the MF2 gateway leading to periodic connectivity issues. We are performing a database maintenance on the gateway in an attempt to clear up the instability. During this maintenance window MF2 will not accept mail. All mail sent to users on MF2 will queue at the sending server and will be delivered once the maintenance has completed and the sending server re-attempts to send the message.

Unfortunately there is no way to predict an ETA on database maintenance as we don’t know how long it will take to clean any problems it might find. We will update this blog post when MF2 service has been restored.

Our apologies for the inconvenience!

Update: At 10:17am central US time service for MF2 was restored. Mail queued at the sending server will be retried according to that sender’s retry delivery settings.

We are currently performing emergency database maintenance on the MF1 gateway to address an issue that was causing errors in mail delivery.

ETA: Unknown – database maintenance times can vary, though it should be complete within the next couple of hours

Mail Delivery Status: Mail will be queued on the sender’s server and will be delivered once mail services restart after the database maintenance

We will update this post as soon as the issue has been corrected.

Update 9:22PM Central: After further investigation it was determined that mail services on all three of our gateways was impacted for approximately 1 hour and 20 minutes this evening due to issues with the DNS servers that our gateways use to resolve IPs.  This issue caused our gateways to be unable to resolve IP addresses properly causing mail to not be able to be delivered properly.

We have since changed to different DNS servers for resolution, resolving the issue.  MF1 continues to work on its database maintenance but mail is being delivered as it works.  Customers on MF1 may notice a slight delay in delivery times as the database maintenance completes.

During the outage emails destined for customer servers will have queued on the sending servers and should be delivered as those servers retry at a later time.

Our apologies for this situation! Due to the length of the outage we will be crediting all customers 5% of their monthly service charges which will be applied against your next bill. Please contact us if you have any questions.

This morning we are experiencing queuing of email on the G1 gateway. We are performing an emergency database maintenance cycle to attempt to mitigate the situation. The ETA for resolution at this time is unknown.

UPDATE: During this maintenance there may be periods of time where the G1 gateway is unavailable to accept connections. This will cause mail to queue on the sender’s server and be retried at a later time according to their settings.

UPDATE 11:40AM: Database maintenance is complete and mail appears to be flowing properly again. Please note that it may take some time for mail flow to return to normal as the gateway has a backlog of mail to process. We apologize for any inconvenience this might have caused!

We are currently experiencing issues with mail queuing on the MF1 gateway and will be starting an emergency database maintenance in the next few minutes. Unfortunately, there is no way to determine an ETA for this process. During the database maintenance there will be periods where the gateway will be unable to accept mail and mail will queue on the senders’ mail servers. We will update this blog post when the maintenance is complete.

Update 2:11pm: Database maintenance completed at approximately 1:30pm central time, however the issue of mail queuing does not appear to be resolved. MailFoundry Support is now monitoring the situation and will update us as soon as a resolution is found. We are monitoring the mail queues and forcing delivery on them to keep the mail flowing properly.

Update 4:48pm: We’ve been monitoring the situation for over 2 hours now and it appears resolved. There was a series of messages that came into the queue with some strangely crafted headers that caused the queues to back up. MailFoundry is aware of the issue and is looking for a way to stop this from happening in the future.

Mailfoundry engineers have temporarily taken the G1 gateway offline as they investigate an issue causing some mail to get stuck in queue.  We expect services to be restored shortly but currently do not have an ETA.  At this time mail will queue on the sending server and will be delivered after G1 comes back online and the sender retries.

Update: G1 is back up and performing normally.

We are currently performing emergency database maintenance on the G1 gateway to repair a corrupted user database.  We estimate a total time for repair of 30-60 minutes.  During a portion of this time SMTP services on G1 will be disabled.  Sending servers will queue the mail until services are restored and will deliver at that time.