| |
BOSTON DATA CENTER UPDATES
|
|
March 7 2007
|
According to the latest report from Savvis (dated March 05), Savvis have delivered on their promises, and have improved the internet connectivity in the datacenter significantly. Since the incident on January 2nd, they have taken the following actions:
- Two new, diverse-path OC-12 lines from Boston to New York were implemented.
- An additional OC-48 line will be implemented to provide further datacenter redundancy. Currently the needed infrastructure is being built.
- Replenished all on-site hardware spares with new tested hardware.
- A new test lab has been deployed, with the purpose of testing new spare hardware before being approved as spare.
We continue to monitor Savvis operations closely. As a whole, we are satisfied with the overall progress and development of the datacenter connectivity and redundancy issue. Also, we continue to investigate the alternative options for datacenters in Boston.
|
|
|
Jan 6 2007
|
SAVVIS datacenter incident (follow-up)
The Management of Savvis abided to take the following actions:
- One new Boston to New York OC-12 on diverse carrier facilities will remain available as a secondary path to the primary Chicago and New York OC-48s.
- A second OC-12 will be added across the same diverse carrier facilities to provide additional secondary bandwidth from Boston to New York.
- A new 2.5Gig carrier diverse facility from the Vendor B02 Data Center directly to New York will be added within 3 months to provide improved data center resiliency.
- On-site metro hardware spares will be replenished with new, tested hardware from the manufacturer.
Our official position is:
- we will continue our research for another collocation provider around Boston. It must meet certain requirements in terms of datacenter specifications, financial health and stability, and market positions of the company which runs it.
- after the new datacenter is tested and we start placing equipment in it, our customers will have the opportunity to choose the datacenter where their accounts will be placed.
- we will be monitoring closely all Savvis actions and procedures. If they do not fulfil the list of the actions which they abided to take, the plan for moving our equipment out of there will be commenced.
Although moving shared-hosting servers between datacenters is not an easy task, we have a technology for moving servers, already tested real-time.
In April 2006 we moved 12 shared-hosting servers from the datacenter of G4 due to the unsatisfactory quality of their service. During the move there was zero dataloss for our customers, no downtime for static websites and a 2 hour "Site is moving" message for dynamic websites. The move involved purchasing 12 new servers (for over $40000) and gradual synchronisation of the content of the old and the new server.
This technology involves larger costs than simply moving servers physically, however the uptime of our customers is a top priority and we will not spare resources when it comes to uptime.
|
|
|
Jan 5 2007
|
According to the information from Savvis, the sequence of events that lead to the outage is:
on 31/Dec a main backbone communication line experienced an outage due to a hardware failure. the repairs started on 01/January, but due to additional hardware problems they continued on 2/Jan as well. as the repairs were in progress, a second backbone communication line experienced an outage. This lead to disruption of communication for the whole datacenter.
When we started using this datacenter back in 2003, according to the datacenter specifications provided by its owner - Cable & Wireless at that time- it fully met our requirements, including in terms of connectivity and redundancy. When Savvis Corp acquired the datacenter in 2004, they confirmed that it will be developed as a high-end colocation and managed hosting facility.
Our Management, along with the upper Management of Savvis will revise all redundancy and failover procedures of the facility. If our requirements and the expectation of our customers are not met, we will consider moving equipment out of there.
|
|
|
Jan 5 2007
|
According to the information from Savvis, the sequence of events that lead to the outage is:
on 31/Dec a main backbone communication line experienced an outage due to a hardware failure. the repairs started on 01/January, but due to additional hardware problems they continued on 2/Jan as well. as the repairs were in progress, a second backbone communication line experienced an outage. This lead to disruption of communication for the whole datacenter.
When we started using this datacenter back in 2003, according to the datacenter specifications provided by its owner - Cable & Wireless at that time- it fully met our requirements, including in terms of connectivity and redundancy. When Savvis Corp acquired the datacenter in 2004, they confirmed that it will be developed as a high-end colocation and managed hosting facility.
Our Management, along with the upper Management of Savvis will revise all redundancy and failover procedures of the facility. If our requirements and the expectation of our customers are not met, we will consider moving equipment out of there. |
|
|
|
Jan 4 2007
|
SAVVIS outage on 2nd of January, 2007 (follow-up)
Savvis representatives have reported that the needed hardware and redundancy lines are completely restored. Since Jan 2 2007 16:00 EST, we have not noticed any connectivity problems. We expect a final incident report from Savvis network engineers within the next 24-48 hours. Since SAVVIS failed to maintain the redundancy levels they have advertised, we are also expecting a complete revision of their failover procedures. Improvement of the failover procedures is a matter which must be addressed in a report from their upper management. We would like to apologize to our customers for the occurrence and would also want them to know that the case will not be closed until we are convinced that similar problems will not happen again.
|
|
|
Jan 2 2007 16:08 EST
|
SAVVIS engineers just reported that the issue in the Boston datacenter has been fully resolved. In case you still experience troubles with your website, please do not hesitate to contact our support team here
|
|
|
Jan 2 2007 15:54 EST
|
Repairs are currently in progress. Intermittent loss of connectivity can expected while defective hardware is being replaced by the Boston network engineers. We expect to have the service fully restored soon.
|
|
|
Jan 2 2007 15:04 EST
|
Service has been partially restored in the Boston data center. Packet loss and slower access can still be experienced. The SAVVIS people are doing their best to have the issue fully resolved.
|
|
|
Jan 2 2007 14:07 EST
|
Both the main and backup/redundant lines providing connection for the Boston datacenter are down. SAVVIS are currently negotiating new backup lines with other upstream providers. Unfortunately, an estimated time of resolution is still unknown at this time.
|
|
|
Jan 2 2007 12:30 EST
|
The Boston data center is still isolated due to a network related issue. SAVVIS continues to work with the local Telco vendor and hardware vendor to restore connectivity. The SAVVIS Executive team is engaged in this customer impacting event. Unfortunately, an estimated time of resolution is still unknown at this time.
|
|
|
Jan 2 2007 10:14 EST
|
Currently, there are connectivity problems within the network of our US upstream provider - SAVVIS, and all servers in our US datacenter cannot be reached. Please be advised that the SAVVIS data center facility team has reported that the Boston data center is isolated due to a circuit related issue. At this time equipment is not reachable. SAVVIS is working with the local Telco vendor to restore connectivity. An estimated time of resolution is unknown at this time. An update for this incident will be published here;
|
|
|