Today (January 4th 2011), one of our routers went offline. This led to the partial and temporary loss of our network, impacting some of our services such as our website, SiteMaker, GandiBlogs, some email accounts, and all operations towards servers. Domain names did not encounter any unavailability, though some network paths to certain servers were unavailable.
The incident is currently being resolved, and services will progressively return to normal.
Please accept our apologies for the inconvenience.
UPDATE: Here is the technical explanation for yesterday's network incident:
Part of the Gandi France network is based on legacy topologies built over the past ten years, including multi-site spans for various VLANs and in some cases a relatively flat architecture. This part of the architecture relies, perhaps unwisely, on spanning-tree protocol to ensure a loop-free layer-2 topology in a bridged or switched network. Whilst we have have been performing various engineering works over the past 18 months to simplify the architecture, it takes a considerable amount of time to completely unbuild what has been built piece by piece over a period of ten years without significant outages of the Gandi services.
The incident yesterday was exacerbated by the legacy elements of the Gandi France network infrastructure and was caused by a fault in a downstream access switch cluster which created a layer-2 loop in the architecture. This in turn caused an unfortunate situation whereby the layer-2 topology of the legacy network was being constantly recalculated resulting in the spanning-tree protocol failing to converge, consuming 100% resources on the affected switches and thus preventing traffic flow. The offending switch cluster was isolated from the network, but we were also required to reload another switch in another datacentre to stop the "snowball" effect caused by the fault.
We have already scheduled for this quarter significant network engineering activities to finally unpick the remainder of the legacy topology and migrate to a fully hierarchical model limiting the layer-2 domains to locally contained subnets, and minimising the reliance upon such protocols as spanning-tree which was never designed to be used in such large scale designs in the first place. We will be communicating the dates and times of the maintenance windows over the coming weeks.
We apologise again for any inconvenience caused during this network incident yesterday.
We are currently experiencing an abnormally increased load on the incoming mail spools on the GandiMail service. As a result, new mail deliveries may be slower than usual. Our teams are investigating the source of this increased load and we will keep you updated as we have more information.
Update: 14:00 : The slow spool performance is related to an increased load on the antispam/antivirus filtering on the mail spools. Our teams are actively working to resolve the issue as soon as possible. Inbound mail is still being delivered, but of course at a slower rate.
Update: 17:50: Our teams have isolated the issue and have tweaked the processes on the antispam filters to further optimise performance. All mail in the spools have been delivered to the recipient mailboxes and the system is now running nominally.
We apologise for any inconvenience caused by the slower than normal delivery of mails today.
A problem with one of our storage units has led to a disruption in mail for several thousand email mailboxes. We are currently working to restore service as soon as possible.
Please accept our apologies for the inconvenience.
The technical team
14:24 GMT: We have identified that the problem was caused by data corruption. We are corrently working on repairing the file system.
16:16 GMT: The filesystem has been found to be severely corrupted. Our technical teams are investigating whether it is possible to recover any data on the affected filer. We will keep you updated with any further information/progress.
12:00 GMT (1 Dec): The work was completed to rebuild the affected filer last night and undelivered mails still in the queue will have been delivered overnight. Unfortunately the filesystem corruption was replicated on the resilient disk arrays which meant that we were sadly unable to recover the data for the affected mailboxes. These mailboxes have been brought back online, but unfortunately are empty for users accessing via webmail or IMAP. We have sent further communication to customers affected by this regrettable event. We have put measures in place to mitigate any repeat occurence, and will be further developing and hardening our mail infrastructure. On behalf of the entire Gandi team, please accept our sincerest apologies for this most regrettable 'first'.
Ubuntu distributions 10.04 and 10.10 are now available in 64-bits
We would also like to remind you that the price for additional RAM has been reduced to $5 (£3.16, or €4) excl. VAT) under A rates for 256MB, and the limit that your server can be configured for has been increased to 12GB.
We have lost contact with a rack for the hosting service -- our technical teams are on site to repair the issue. As soon as we know if the problem is related to the network or the servers, we will migrate your server or reconnect the rack to reestablish service.
Please do not execute any operations on your server (stop/reboot) in the meantime, we will restart your server if necessary.
We apologise for any inconvenience caused.
Update: 13:00 CET (12:00 UTC) : The servers are now reachable. The brief outage was due to a partial electrical failure which is being investigated. The outage only affected the network connectivity for the affected servers.
As a reminder, the .CAT extension is reserved for those that promote the language and culture of Catalonia. It is open to everyone, regardless of where they are located, however it must have a website in Catalan within 6 months of the domain's creation.
Starting today (November 16th, 2010), we are dropping our prices for creation and renewals of Pro SSL certificates:
* "1 address" Pro SSL certificates drop to $65 (£43, or €34) excl. VAT under A rates * "Multi-address (wildcard)" Pro SSL certificates drop to $340 (£237, or €265) excl. VAT under A rates
Please feel free to view our SSL comparison page, or our simulator, to help you choose the right certificate for you. As a reminder, with every domain creation or transfer at Gandi, you get the first year free on e standard SSL certificate.