Archive for December, 2005
Our colocation facility bounced the power again (apparently in order to deal with the failed UPS that caused our previous incident.) We are on route to the facility to bring servers back online now and that should be done shortly. We have been and will continue to work with Yahoo! to migrate to better facilities as fast as humanly possible.
(chronology moved inside the post)
Update (10:50 PM EST): We are at the facility and are bringing up machines now.
Update (11:14 PM EST): Bringing up slaves now. Should be done shortly.
Update (12:09 PM EST): All but one server up now. Bringing up inbox, search, etc.
Update (12:20 PM EST): The facility will be cutting power back over to the UPS at 1 AM. Supposedly this will not affect service, but neither was the test that brought everything down… cross your fingers.
Update (2:04 AM EST): Still waiting for the cutover back to the UPS. They’re planning to give us an hour warning, and have not done so yet. We’ll bring down the site beforehand so we can boot cleanly in case they drop power again.
Update (6:37 AM EST): They just cut over power to the UPS and bounced power again. We’re bringing stuff back online again, everything should be good in a few minutes.
Update (6:57 AM EST): Everything is back online.
December 27th, 2005
Due to the power outage earlier in the week, we are dealing with a number of continued hiccups. We’ve taken everything offline to properly rebuild and restore everything. I apologize and hope to have this resolved as soon as possible. Thank you for your continued patience.
(Chronology moved inside the post)
Update: Indexes are almost done rebuilding. When that is done, we can then copy the data to the various database servers and start the service back up. Almost done — hang in there.
Update: Still waiting for the last remaining index to build. No data has been lost — we just need to fix the tables so the databases can find things quickly. This appears to be largely due to a RAID failure after our power outage earlier in the week – one of the indexes became corrupted and crashed the master database; for some reason the slaves replicated bad data from the master and then ended up crashing infinitely.
Update: We’re done rebuilding databases and I am now pushing copies to the slave servers. When that is done I can bring the site up.
Update: We are back up! Now working on tag intersections, the search engine, and the inbox. Yahoo! has started helping out, so I hope to provide a higher level of service as soon as possible.
December 18th, 2005
Our datacenter has had full power outage, and we are now recovering from the aftermath. About half of the machines are still down and we are working on bringing them back as soon as possible. To keep things moving, I’ve turned off RSS and some other features so that posting will still work and so that you will still have access to your bookmarks.
5pm EST Update: Everything is back up. We apologize for the inconvenience. It looks like a number of slave servers are corrupted. We’re rebuilding them now and taken the site down so this goes as quickly as possible.
Update: Some servers are coming up. Should be going much faster from here on in.
PS — we are obviously not yet on the Yahoo! infrastructure, and this is exactly the kind of stuff they can do a lot better for us (and you.)
Update: Most stuff is up, finally. Tag intersections are currently not working, and search is not updating — a few servers would not come up. We’ll bring them up tomorrow when we return to the east coast.
December 14th, 2005
We’re proud to announce that del.icio.us has joined the Yahoo! family. Together we’ll continue to improve how people discover, remember and share on the Internet, with a big emphasis on the power of community. We’re excited to be working with the Yahoo! Search team – they definitely get social systems and their potential to change the web. (We’re also excited to be joining our fraternal twin Flickr!)
We want to thank everyone who has helped us along the way – our employees, our great investors and advisors, and especially our users. We still want to get your feedback, and we look forward to bringing you new features and more servers in the future.
I look forward to continuing my vision of social and community memory, and taking it to the next level with the del.icio.us community and Yahoo!
December 9th, 2005
