Alert 10:20 AM [Paris Time]:
An incident is currently impacting our email services.
Our team is working on solving this issue as soon as possible. Please accept our apologize for the temporary service interruption.
Edit 11:20 AM [Paris Time]:
The Mail Incident is now over and resolved.
Zone files updates may be delayed.
Additional information for impacted users 12:30 PM [Paris Time]:
We confirm that this incident will not have any impact on your e-mails, which will be delivered with a slight delay. No loss will be noticed.
A more detailed timeline is available here.
We experienced an outage on the Gandi mail platform this morning. The outage was due to human error, rather than any failure of equipment or software.
Here is the timeline (in UTC):
14:57 - A human error was made that introduced a problem for customers connecting to Gandi's email infrastructure.
15:00 - Notification of the outage posted to Gandi's web site
15:06 - Physical connectivity restored. One machine unresponsive.
15:23 - Full connectivity and machine function restored.
In summary, some 15,975 mailboxes were not accessible for checking received messages or sending emails from 14:57 to 15:23 UTC. No data was lost.
We apologize for any inconvenience this incident may have caused.
Some physical machines hosting IaaS VMs are unreachable due to an issue we are currently analyzing.
We are fixing the issues and restarting the VMs on other physical machines as soon as possible.
Thank you not to do any operation on your VM until the emergency maintenance is ongoing. and not finished.
At the end of the maintenance, if your server does not respond well, please contact our hosting support team by mail using the 'blocked server' option.
Sorry for the inconvenience this issue may cause to you.
UPDATE 12h20 : the situation is back to normal, we are still analyzing the metrics and the logs.
A maintenance will be performed on our hosting infrastructure at Baltimore (USA) and Bissen (Luxembourg).
The following services will be interrupted for a few minutes starting at 2014-04-29 05h00 UTC (2014-04-29 10pm PST):
- New SFTP/GIT sessions
- Reverse resolvers for customer IPs
- Operations on servers and Simple Hosting
Please accept our apologies for the inconvenience.
An emergency maintenance will be done to improve our storage on Paris datacenter Wednesday 22 April 2014 between 3pm and 9pm PST (2014-04-22 22h00 and 2014-04-23 4h00 UTC).
There may be network interruptions lasting a few seconds per storage unit. The I/O will come back without any operation needed on your side. It is not necessary to reboot your virtual server.
Please accept our apologies for any inconvience caused by this maintenance. This post will be updated when maintenance has been completed.
This maintenance has been completed. Some additional issues were uncovered, and the entire maintenance took a little over 2 hours.
We do apologise for the inconvenience.
EDIT Tue Apr 29 23:30:00 CEST 2014 : The maintenance will be completed from 00:00 to 01:00 tonight for the storage units that have not been upgraded last time.
Our technical team is working on an issue impacting the emergency console on the Luxembourg datacenter.
It is currently unreachable.
Our technical team will reboot the machine as soon as possible.
Update: The issue has been resolved.
DNS resolution was interrupted for a few minutes on the hosting platform at our Paris datacenter.
While working on the machines which handle the DNS resolution for the Paris datacenter, some of them stopped responding at 11:04 AM (CEST).
Our technical team found and fixed the source of the issue at 11:24 AM.
We apologize for any inconvenience this problem may have caused to you.
An emergency maintenance is currently operation is underway for Sitemaker.
Due to this, the service will be unavailabel from 4:00 PM CET until 5:00 PM CET.
Please accept our apologies for the inconvienence.
A major vulnerability in the OpenSSL cryptographic software library has just been published (CVE-2014-0160). If you have a Gandi SSL certificate, please read this post carefully before taking action.
This flaw has existed for some time, and there is a possibility that X509/SSL private keys have been compromised undetectably.
This flaw is present in OpenSSL from version 1.0.1 up to and including 1.0.1f, referred to as the "heartbleed" bug (heartbleed.com).
If your servers are using an affected version of OpenSSL (If you aren't sure if your server is affected, you can try a tool like this one, not provided or tested by Gandi):
- If you are using our SSL certificates on our PaaS platform (Simple Hosting) or via our web accelerator, you should know we fixed this vulnerability as soon as we were informed of it, and we will try to give further details about how your private keys could have been exposed by our platform.
- If you are using our IaaS infrastructure (Gandi Cloud VPS), or that of another hosting provider, and your servers are using an affected version* of OpenSSL, you need to:
- Patch the openSSL version on any server you own and operate yourself by installing security updates provided in your package manager. (For example, on Debian, ensure you're using the official debian-security repository, then run `apt-get update` and `apt-get install openssl`, then restart all services that use SSL.)
- Generate new private keys and certificates to restore security of your services (see below if you're using a Gandi SSL certificate).
- If you are using our SSL certificates, either on our infrastructure (on our PaaS/Simple Hosting instances) or on external services, then it is recommended that you regenerate a CSR and private key. Note: Do not revoke the certificate! Replaced certificates will be automatically revoked (see update below). If you revoke the certificate yourself, you will not be able to replace it afterwards, and you'll instead have to buy a new one.
Additional technical information is available in this GandiKitchen blog post.
========= Update 17 April 2014 =========
If you regenerated a Gandi SSL certificate between 8 and 17 April:
Many customers have regenerated their SSL certificates as a result of the Heartbleed bug. Until today, old certificates which were replaced with new, regenerated ones were not automatically revoked. Due to popular demand, we promised (here and here) to revoke the old certificates.
We sent an email this morning to users who have regenerated a Gandi SSL certificate between 8 and 17 April to notify you that your old certificates will be revoked in 24 hours. Your old certificate will be revoked on the morning of 18 April, Paris time (as early as 1am PST). If you have regenerated a certificate but have not yet installed it on your infrastructure, now is the time to do so!
If you intend to regenerate a Gandi SSL certificate in the future:
We have implemented automatic revocation, which means that from today forward, regenerated certificates will be automatically revoked 48 hours after the replacement certificate has been issued.