Category Archives: Outages

Upcoming Service Outage

As part of the Tower Network datacenter move, we need to take all services down so that hardware can be physically moved to the new location, as well as to give our upstream ISP time to migrate the circuit over to the new location.

The first outage will be *TONIGHT*, June 10, starting at approximately 4PM, lasting 1-2 hours.

The second outage will be sometime either June 11 or June 12, time to be determined.  This outage could possibly last as long as half a day, but could be less.  This is when the actual, physical move will be taking place.

During these outages, *ALL* Tower Network services will be offline.  Our offsite “disaster recovery” host, siege, will still remain online to handle DNS queries and will queue incoming email, but all user-accessible services will be unavailable.

Resolved Outage: Network Outage

At approximately 10:30PM PDT, the DSL servicing Tower Network stopped passing traffic.  It is believed to be a problem with the upstream provider, and no ETR is available at this time.  Our offsite host, siege, should be providing minimal DNS and email capabilities until the circuit is back up (and by minimal email, we mean that it will accept and store incoming email.  It does not provide any ability for users to check their email).

And yes, we are aware that because of this outage, you won’t see this post until the DSL becomes live again, but at this point it’s the best we can do.

Update: The circuit came back online at almost exactly midnight.  The precise nature of the start/end times makes this smell like pre-scheduled maintenance, but this is purely conjecture.  At any rate, the circuit seems stable for the moment, so we’re going to call this “resolved.”

Weather Alert

The weather report for this area includes low temps and very high winds (20-30mph with gusts up to 65mph) over the weekend. The last time we had a storm like this there were many widespread power outages (some of them lasted many days). We were very lucky in that the longest power outage we endured was only a couple of hours.

However, if we are not as lucky as last time and the power goes out for a prolonged period, most Tower Network services will likely go offline, including this website, email services, etc., as we only have enough battery to last about 30 minutes (sorry, no generator yet…). The two exceptions to this are incoming email delivery and DNS queries, both of which are handled on our offsite emergency server, Siege (get it? Siege tower?), so unless St. Louis (where Siege lives) also has problems the same time we do, absolutely basic functions required to keep the domain alive will still be online.

We’ll try to keep everyone informed here if at all possible, but if you are mysteriously unable to connect to any of our services, you now probably know why.

Mail server migrated

Email services were successfully cut-over to the new server, and all users’ email has been migrated as well.  There would have been a period of 1-2 hours tonight from about 6:30-8:30 PM (PDT) where users would have been unable to log in, but all incoming email would have been successfully delivered to the users’ email accounts.

Any problems accessing email should be reported using the “Contact Us” tab found above.

All Services Down

(Editor’s Note: This is an import from the old outages site and has been backdated appropriately.)

Jan 7, 2008 10:00PM – Jan 8, 2008 4:45PM: A routine kernel update to the main VMware server London caused the machine to refuse to boot properly, preventing services from restarting. As of 16:45PM, all services are online. A review will be performed to ensure that an outage like this will not be as likely to occur, and if it does, prevent it from being so service affecting.

Email Delivery Down

(Editor’s Note: This is an import from the old outages site and has been backdated appropriately.)

Approx 10:00PM Aug 6, 2007 – 09:45AM Aug 7, 2007: An OS-patch update on the anti-spam/anti-virus scanning host caused Amavisd-new to not function properly. The root cause appears to be an inability for Amavisd-new to handle IPv6 correctly. A workaround is in place and mail is flowing once again.

Ongoing Known Webmail Timeout Issue

(Editor’s Note: This is an import from the old outages site and has been backdated appropriately.)

Patched/fixed Jan 07: There is a known issue where the webmail client will show a “timeout” error after only a few minutes of being idle (usually when the user is composing a new message). This issue is believed to be corrected in a patch from the vendor, which we are working on applying to the system. Until then, it is recommended that users compose their messages in a text editor and then paste them into the compose window, or use a full-featured IMAP client.

All Services Down

(Editor’s Note: This is an import from the old outages site and has been backdated appropriately.)

Dec 14, 2006 – 17:15PST Dec 17, 2006An extensive power outage in the Puget Sound region caused by a wind storm has taken out our network connection to the world. The DSL circuit is still up, but it would appear that Avvanta, our upstream provider, has dropped off the face of the earth. They are located in Bellevue, which is in the affected area of the power outage.

Update: Shortly after 17:05 today, Avvanta seems to have regained power and is routing our traffic again. All services appear to be back up.