At Monday's meeting of the Senate Advisory Committee on University Affairs, University officials named the reasons behind the three CTools outages between Apr. 18-22 and discussed the future of CTools. 


According to Laura Patterson, an associate vice president for Michigan Administrative Information Services, the principal cause behind all three CTools failures was “very high loads (of CTools activity) in very short windows of time.” 


John Williams, the executive producer of Digital Media Commons, said that at the time of the first outage, which lasted only a few minutes on Apr. 18, there were about 9,000 users on CTools, which is a typical end-of-term load.

However, Williams added that at the same time, e-mail notifications regarding course evaluations were sent out en masse to students at noon, and within two hours approximately 12,000 students began submitting evaluations, including some of the 9,000 whom were already using CTools.

By 2 p.m. there were approximately 12,000 to 21,000 students using CTools, which was an excessive load for the system. 


Williams said that if the notifications had been sent earlier, there would have been more time to allow for a more staggered response, but since notifications were sent at noon, CTools traffic became much heavier during a time of frequent use which subsequently caused the outage.

During fall semester, the e-mail notifications were sent at 6 a.m., which permitted enough time for users to gradually begin the course evaluations and to avoid a system overload. Last semester, the e-mail notifications were sent out later because they took a longer time to prepare than was expected, and were not readily available at an earlier time.

Williams said the second outage, which occurred on Apr. 20, was unexpected because the first outage lasted for only a few minutes.

“We thought the first outage was an anomaly,” Williams said.


Although the high loads of activity caused by the teaching evaluations were the easiest problem to identify, they were by no means the only activity that caused the second outage, Williams said.

Williams identified three factors that caused the second outage, including course evaluations, e-mail distribution and “presence" — the ability for students to view other students and faculty online in their courses on CTools. All of these factors contributed to a greater influx of users and a higher load.

Williams said that CTools was unprepared for the excessive loads in its pattern of use because the database’s load-testing program was modeled by the hour, and needed to be tested on a more frequent time scale to better anticipate periods of high usage.

“We really need to be modeling things on a minute by minute basis,” Williams said.

Williams added that upgrading CTools is difficult due to overlap operations with academic units. Information Technology has only one day during the summer to update the system, which is generally the day before the medical school starts its semester in July.

In addition, Williams said there is a lack of funds to support such an upgrade.

“Because of the economic climate, we haven’t seen those funds come through,” Williams said.

According to Patterson, one way that funds could be generated would be by merging information technology systems.

“Capturing redundancies in ITCS and Maize would increase investment in learning technology,” Patterson said, in reference to CTools.

Without much option to upgrade CTools, some SACUA members expressed concern over the lack of backup alternatives for CTools in the event of present and future outages. 


Williams explained that Blackboard, the only commercial vendor of education software, has pledged not to sue certain systems such as CTools, even though it has a patent that restricts many of its rival open-source companies.

In 2006, Blackboard successfully sued rival learning management developer Desire2Learn.