Students trying to access online homework and e-mail yesterday
were in for a rude shock, as the failure of two power backup units
took down not only online coursework but much of the
University’s Web gateway including IFS space, e-mail and
Coursetools for most of the day.
The technological failure occurred shortly after noon and was
slowly resolved through the night, as University officials
struggled to identify and solve a problem which grew more complex
throughout the evening.
Kitty Bridges, associate vice president for IT Central Services,
said initially that only one backup unit was taken out, affecting
some but not all of the e-mail servers. After further
investigation, she reported that both units crashed because of a
power outage at the University’s Arbor Lakes complex, which
houses the headquarters for ITCS. The crash took out 23 servers,
including all e-mail and many of the University-administered
websites.
Early yesterday evening, she said an electrician was trying to
fix the problem. At that time, she estimated that all affected
services other than IFS network file space would be back
“soon.” She estimated that IFS space would probably not
be back until 9 p.m. last night, due to problems shutting it down
once the server problems were discovered. “This seems like a
straight-forward problem,” Bridges said.
However, she later said the problem was more complex. “It
turns out that the damage was more extensive than we had initially
anticipated,” she said at 9:30 p.m. “We have system
administrators at the data center bringing up the affected
services.”
Bridges said that as of 6 p.m. last night, the cause of the
outage had been determined and that IFS would be the last service
brought back up. However, she said at 9:30 p.m. that each e-mail
server had to be brought up individually and that only four had
been brought back at that time. She said the estimated time for
individual e-mail accounts to be brought back online would depend
on which server contained an individual’s e-mail account. She
said one of the administrators working on the problem estimated
that all e-mail would be back by midnight.
Bridges said the problem was unrelated to other e-mail outages,
such as one in late September which lasted a full day and
night.
A posting found on the Coursetools website specified that the
damage affected Kerberos Authentication, IMAP e-mail, e-mail
delivery, IFS, portions of umich.edu and USEnet News. Weblogin,
CourseTools, Wolverine Access, mail.umich.edu and login.itd were
also down all day.