Sunday night we lost access to a couple of our secondary domains, after we started to move off hardware owned by Knology and onto hardware owned by Mediaphormedia, our software development company. This was related to The World Company’s sale of Sunflower Broadband late last year. The domains, which included HometownLawrence.com and our site admin, were inaccessible because of IP address conflicts. Last night while addressing those problems we had two issues that combined to screw up our news sites pretty badly. The first was with the [NFS mounts][1] our web nodes use to have access to our media and templates. The second was the loss of our [memcached][2] server. Trouble shooting the NFS problems initially hid the loss of our caching server. Without memcache, our sites were not able to keep up with the morning traffic and everything became unusably slow.
We are working on contingency plans to keep problems like this from hampering our ability to deliver your news the way they did this morning. Internally we are codifying and improving our emergency action plans. Our goal is to greatly improve our reaction times when faced with these kinds of problems.
Secondarily, we are making sure we’re better communicating to you our problems and where we stand. By the end of the week we’ll have designated places — externally hosted — for each of our sites so if we’re ever down for an extended period of time you’ll know where you can go to stay up to date. We’re also making sure that we have off-site locations to post and report news so that we’re not interrupting our coverage even if our primary sites are down.
[1]: http://en.wikipedia.org/wiki/Network_File_System_(protocol)
[2]: http://memcached.org/