Following the discovery of an intermittent but serious network issue, Gandi teams have determined that a rolling maintenance will be necessary.
We regret that, while most of the affected systems will simply require migration and no interruption in service, some will almost certainly require restarting. We will endevor to make these interruptions as short as possible, and only perform them when absolutely necessary.
We are starting with dc0 (Paris) this week. We will proceed on to dc2 (Luxembourg) on Monday, June 9.
The issue is not detected on dc1 (Baltimore) at the moment but, if necessary, we will proceed to fix it there. We apologize for any inconvenience this may cause.
Alert 10:20 AM [Paris Time]:
An incident is currently impacting our email services.
Our team is working on solving this issue as soon as possible. Please accept our apologize for the temporary service interruption.
Edit 11:20 AM [Paris Time]:
The Mail Incident is now over and resolved.
Zone files updates may be delayed.
Additional information for impacted users 12:30 PM [Paris Time]:
We confirm that this incident will not have any impact on your e-mails, which will be delivered with a slight delay. No loss will be noticed.
A more detailed timeline is available
We experienced an outage on the Gandi mail platform this morning. The outage was due to human error, rather than any failure of equipment or software.
Here is the timeline (in UTC):
14:57 - A human error was made that introduced a problem for customers connecting to Gandi's email infrastructure.
15:00 - Notification of the outage posted to Gandi's web site
15:06 - Physical connectivity restored. One machine unresponsive.
15:23 - Full connectivity and machine function restored.
In summary, some 15,975 mailboxes were not accessible for checking received messages or sending emails from 14:57 to 15:23 UTC. No data was lost.
We apologize for any inconvenience this incident may have caused.
Some physical machines hosting IaaS VMs are unreachable due to an issue we are currently analyzing.
We are fixing the issues and restarting the VMs on other physical machines as soon as possible.
Thank you not to do any operation on your VM until the emergency maintenance is ongoing. and not finished.
At the end of the maintenance, if your server does not respond well, please contact our hosting support team by mail using the 'blocked server' option.
Sorry for the inconvenience this issue may cause to you.
UPDATE 12h20 : the situation is back to normal, we are still analyzing the metrics and the logs.
A maintenance will be performed on our hosting infrastructure at Baltimore (USA) and Bissen (Luxembourg).
The following services will be interrupted for a few minutes starting at 2014-04-29 05h00 UTC (2014-04-29 10pm PST):
New SFTP/GIT sessions
Reverse resolvers for customer IPs
Operations on servers and Simple Hosting
Please accept our apologies for the inconvenience.
An emergency maintenance will be done to improve our storage on Paris datacenter Wednesday 22 April 2014 between 3pm and 9pm PST (2014-04-22 22h00 and 2014-04-23 4h00 UTC).
There may be network interruptions lasting a few seconds per storage unit. The I/O will come back without any operation needed on your side. It is not necessary to reboot your virtual server.
Please accept our apologies for any inconvience caused by this maintenance. This post will be updated when maintenance has been completed.
This maintenance has been completed. Some additional issues were uncovered, and the entire maintenance took a little over 2 hours.
We do apologise for the inconvenience.
EDIT Tue Apr 29 23:30:00 CEST 2014 : The maintenance will be completed from 00:00 to 01:00 tonight for the storage units that have not been upgraded last time.
Our technical team is working on an issue impacting the emergency console on the Luxembourg datacenter.
It is currently unreachable.
Our technical team will reboot the machine as soon as possible.
Update: The issue has been resolved.
DNS resolution was interrupted for a few minutes on the hosting platform at our Paris datacenter.
While working on the machines which handle the DNS resolution for the Paris datacenter, some of them stopped responding at 11:04 AM (CEST).
Our technical team found and fixed the source of the issue at 11:24 AM.
We apologize for any inconvenience this problem may have caused to you.
An emergency maintenance is currently operation is underway for Sitemaker.
Due to this, the service will be unavailabel from 4:00 PM CET until 5:00 PM CET.
Please accept our apologies for the inconvienence.