Black Friday: How It Played Out, An Eyewitness Account

by Alex Dault, NGSIS Program Assistant

“If we make it to 9:10 am, we’ll be okay.”

Miki Harmath utters these words at 8:54 AM, August 4th in the EASI war room. The room is lit up with projections of ACORN log-on stats and server usage. There is a box of untouched doughnuts and muffins on the table but no one has much of an appetite. All eyes are glued to the screens indicating the load balance of the four servers and the steadily increasing number of users attempting to access the system.

Laurel Williams, IT Analyst and Java Build Coordinator, nurses her coffee and murmurs:

“I told the barista that it was a big day for enrolment. He asked if I was a registrar. “No, I’m in IT” I said. He gave me this coffee for free.”

At 9:00 am, registration opens to the 4th year students. The number of active users starts to leap upwards, jumping up into the thousands in a matter of seconds.

This is Black Friday, the first Friday in August which is the “priority drop” enrolment period for all Faculty of Arts & Sciences (FAS) students. The day has become infamous among IT and Enrolment staff because for the past two years (2015, 2016) the massive volume of log-ins (33,000 anticipated but as many as 60,000 possible) to the ACORN and SWS site has overloaded servers and crashed the site.

A reddit thread from the r/UofT subreddit records the frustration of last year’s meltdown.

“RIP ACORN” posted user BenZion last year, “Couldn’t even make it to the login page”

“There is no hope.” wrote back user SoupDoge (also back in 2016)

Late in the afternoon in 2016, a representative from the ACORN project team replied to the chorus of acrimony with the following response:

We built ACORN to improve the student user experience. This takes into account not only features and interactions on the app itself but the entire experience of using it. Today we clearly failed big time. For those impacted we apologize sincerely. There’s not much to say that will change or help what happened this morning. The idea that this is an inevitable occurrence each year is not in line with our goals for ACORN or our values as a team. It’s obviously something we need to change and improve upon so this doesn’t happen again.

-ACORN Project Team

It was a contrite and heartfelt apology and sincere in that the ACORN and IT systems teams were determined not to allow this to happen next year.

* * *

“Well, it’s 9:11 so I think we’ll make it….” murmurs Miki Harmath, his knuckle tapping superstitiously on the wooden table. Up on the screen, the number of active users is dropping and the rush is coming to an end.

But the worst is yet to come. This was just the 4th Year registration. In less than an hour, the third years will log-on at 10:00 am, then the second years at 11:00 am.

“But the real tsunami wave comes at noon,” cautions Marilee Keogh, Manager for Technical Services.

“That’s when the first years come online.”

* * *

Over the course of the last six months, a number of initiatives have been undertaken to prepare for the Black Friday slam; some short term and some long term.

In the short term, the EASI/NGSIS team has developed Peak Load Mode ACORN (PLM ACORN) that disables some non-essential functionality (Notifications, Invoice, Financial Awards & Aid information and multiple degree invitations) to improve performance. This new lightweight landing web page presented to students on heavy registration days that has less overhead on the system resources than the regular ACORN dashboard and thus loads faster.

A specialized team within EASI and ITS has also improved load balancing on the ACORN application servers, ensuring the server performance is better shared amongst the multiple servers. Furthermore, they’ve streamlined the login process that speeds up the communications between the different systems involved in authenticating and authorizing a student when accessing ACORN and made a number of important technical changes that improve overall performance in ACORN’s backend systems.

ITS has also set its eyes on the bigger ‘backend’ issue, which is the Mainframe. The University is badly in need of new infrastructure to support the student database. Efforts to replace this back-end system are well underway but such a massive effort in a single year was not even a remote possibility.

* * *

It is 11:35 am. Fourth, Third and Second Year students have all been granted access to their enrolment cart. There are 1728 active users on the ACORN site and beginning to descend.

The EASI Black Friday Team in their War Room

Frank Boshoff, Data Architect and Client Services Manager, glances at an infographic indicating that 20% of users are registering for their courses on a mobile device.

“If I was a student, I wouldn’t be doing something so important on my mobile phone.”

“They’re totally different people than us, Frank.” answers Laurel Williams, taking a wistful sip of coffee.

“It’s a new age.”

Silence falls over the team as the clock hits 11:50 am.

The fact that there are still 1600 active users with ten minutes to go before the first years come online is making the team nervous. The Google Analytics page is showing that first year students are already starting to appear online by the hundreds and encounter the staggered enrolment page.

“They’re at the gates.” says Haroon Rafique, Technical Lead for SIS.

“What are they saying on social media?” asks Laurel Williams anxiously.

“It’s pretty quiet for the most part,” replies Mike Clark, EASI UX Manager.


“There’s a small thread going on the U of T subreddit comparing the load speed of SWS and ACORN.”

Up on the screens, it is obvious why SWS is running slightly faster than ACORN at the present moment. ACORN has 1700 live users compared to just 500 on the SWS. Power users accustomed to rapidly typing in course codes heavily favour SWS for its speed and familiarity.

At 11:58 a message comes in to Mike from a student asking if ACORN is down. The third year student says that they should have been able to access the service at 11:00. Mike needs more information to assist and writes a response to the student with a small flurry of questions.

“Something is going on with IDP,” says Andre Kalamandeen, an ACORN development team leader.

“Well, if they can just fix that in the next minute, that’d be great.” says Mike wryly as he shoots a second email out to registrars. His soldierly calm in the face of potential disaster is impressive to say the least.

It is now 12:00 pm.

The number of users online ticks up to 2,300 and keeps rising. A world map infographic shows that users are logging in from as far away as Beiijing and Siberia. If this were the olden days of lining up to enroll at the Registrar’s office, it would be akin to two thousand students coming in through the windows, doors, floorboards and ventilation pipes to simultaneously ask to be registered.

“They can’t get in. There’s a bottleneck at IDP.” says Andre Kalamandeen, a hint of anxiousness in his voice.

The pipes in the wall creak and groan sympathetically, the building itself manifesting the strain on the four servers.

The team watches helplessly as the number of users stuck on the IDP authentication page climbs into the thousands. The minutes tick by and there is a noticeable slowdown in service. The four servers are being hammered from all over the world.

This is the moment. All the preparations and tests and meetings have led to this moment. If ACORN is going to explode, it is going to happen right now.

But it doesn’t explode. And the number trapped on the IDP page start to drop. First year students are getting onto the site and registering for courses.

The service is working. There was a bit of slowdown, but it is still working.

* * *

A few hours later, the team has gone out for a celebratory lunch. The consensus is that there is still much more to be done but that huge strides have been made. Accolades have been pouring in from students and from the registrars’ offices.

“”I just wanted to say thank you to you and the team for managing a really complicated challenge today. Great job!”

“Congratulations! I did see the improvement on ACORN. This year I spent less time on loading in ACORN than last year. From 30 mins to 15 mins. Good job!”

“I really appreciate all the hard work your team put into preparations for today, and how communicative you’ve been with the registrarial community this morning. It’s made a big difference for our staff and our students”

“Good job guys, and thank you for the updates, they were very useful.”

Best of all, there is a rumour has they have even come up with a new name for the first Friday in August.

They’re calling it Light Grey Friday.

NGSIS releases pilot application ‘Timetable Builder’ at UTSC

by Alex Dault, NGSIS Program Assistant

“I mark the hours, every one,
Nor have I yet outrun the Sun.
My use and value, unto you,
Are gauged by what you have to do.”

These words are etched into the magical “time turner” featured in author J.K Rowling’s best selling children’s book Harry Potter and the Prisoner of Azkaban. The “time turner” allows character Hermione Granger to attend more classes in her third year at Hogwarts School of Witchcraft and Wizardry than time would normally allow.

This summer, NGSIS has given students at UTSC the next best thing to a “time turner”- only there’s no magic involved.

It’s called the Timetable Builder, an application that allows students to rapidly build an optimized course schedule. The tool builds a timetable algorithmically with the click of a button, based on the courses that students need (and want) to take. Students can then efficiently generate an optimal timetable according to their preferences, then explore custom variations as desired.

Timetable Builder co-creator Michael Clark, manager of the NGSIS User Experience (UX) team, says that the tool was inspired by students:

“Basically, [TTB] is for anyone who has previously found planning their course enrolment a cumbersome process. You wouldn’t believe the multitude of permutations that can result from the various class activities of all a students’ desired courses.”

Students who enrol in courses with multiple lecture, lab, tutorial or practical sections can have tens of thousands of potential options for their course timetable. It is a daunting task to build an ideal timetable especially for first and second year students. With the Timetable Builder, students are able to block off certain times from being booked in order to protect time in the day they would prefer to keep regularly reserved for practice, rehearsal, gym time, club meetings, power naps, etc.

It should be noted that Timetable Builder is currently just a pilot and available only at the University of Toronto Scarborough, where it has already made a favourable impression.

Anyone interested in learning more can watch the overview video of Timetable Builder and then try out the application at https://ttb.utoronto.ca.

Congratulations to the Timetable Builder team on this feat of technical wizardry.

Future of IT at UofT: Business Analysis

Technological demands are changing at the University of Toronto (and, if you’ve been paying attention, have been changing for some time). Staff are being asked to take enormously complex systems (SAP, ROSI) and make them appear simple to users. The ease with which students and staff can interact with U of T’s technology systems (SAP, ROSI) needs to mask not only the complexity of these systems, but also an intricate web of institutional relationships and business processes.

In order for to deliver this ‘vital simplicity’, strong functional partnerships exist must across the institution.

In other words: we, as staff, must prepare ourselves to work across traditional departmental and divisional lines.

EASI Director Cathy Eberts had the following advice for a group of staff who assembled last week for a Career Spotlight session:

“The future of IT at the University of Toronto will require master architects, negotiators, and technical staff who are able to work with an array of vendors and complex technical components and can articulate how this will all fit together.”

Cathy hinted that training in business analysis will be a vital part of this future. A business analyst is someone who can translate requirements for developers, think critically about organizational objectives and ultimately ensure the value of U of T’s investment in a project.

As part of a wider effort to spread these skills to the institution, EASI/NGSIS is offering a 2-day course in key techniques and competencies of Business Analysis, with both lectures and hands-on group work.  It is a practical course covering all aspects of the Business Analyst role – from requirements gathering to QA support.  The emphasis of the course is on learning practical tools and techniques that can immediately be put to use in project work and in interactions with stakeholders, whether it be local to the Division or in partnership with EASI/ITS initiatives.

The workshop, to be held at 215 Huron St, 6th Floor, ODLC on August 18 -19, 2016, filled almost immediately, but interested staff can still join the waiting list (if there is enough demand, perhaps we can schedule another course).  The cost of the workshop is $1000.

Applicants can register here:

http://www.odlc.utoronto.ca/hr/baworkshop

New NGSIS Website Set To Go Live

“Before enlightenment chop wood and carry water. After enlightenment, chop wood and carry water.” – Wu Li

If something isn’t necessary, you can probably live without it. And so to live simply is to rid your life of as many of the unnecessary and unessential things as you can, to make room for the essential. With the new NGSIS website, we’ve done just that.

The new site is set to go live on June 21st, 2016. The site aspires to be an easy to use resource for staff, students and stakeholders seeking to learn more about the status of the NGSIS program and its projects.

All in-progress projects with their completion percentages on our project list page.

Our blog will continue regular updates on program activities, a regular series of post contributors and invite guests to provide their take on NGSIS related matters. If you’re interested, subscribe to receive updates when a new post is up.

NGSIS shines at Excellence Through Innovation Program Awards

The Excellence Though Innovation Award program recognizes the contributions of administrative staff in advancing the University’s strategic objectives, encouraging administrative innovation and providing a platform for sharing best practices.

This program is an excellent way to “identify and celebrate improved practices within every area of our work” and to let the employees know that their contributions are important.

This year, two projects were selected which included NGSIS staff, the T Card Student Photo Project and the ACORN project, one of the flagship NGSIS projects.

The T Card Student Photo project enables the photo of a student stored in the T Card database to be reused in a suite of new validation applications, including our new exam invigilator and residence tools. NGSIS Architect Frank Boshoff and EASI staff Walter Ni and David Lock each played important roles in the project.

We’ve written about the ACORN project before. This was a massive undertaking and was the product of the collective efforts of NGSIS/EASI staff Hossein Aliabadi, Michael Clark, Miki Harmath, Mark Johnston, Andre Kalamandeen, Michael Wyers, David Lock, Mezba Mahtab, Michael O’Cleirigh, Haroon Rafique, Petru Sugar, Laurel Williams, Zoe Wong, and Terry Lago, amongst others.

Congratulations to everyone involved in these two projects!

NGSIS shines at Excellence Through Innovation Program (ETIA) Awards

The Excellence Though Innovation Award program recognizes the contributions of administrative staff in advancing the University’s strategic objectives, encouraging administrative innovation and providing a platform for sharing best practices.

This program is an excellent way to “identify and celebrate improved practices within every area of our work” and to let the employees know that their contributions are important.

This year, two projects were selected which included NGSIS staff, the T Card Student Photo Project and the ACORN project , one of the flagship NGSIS projects.

The T Card Student Photo project enables the photo of a student stored in the T Card database to be reused in a suite of new validation applications, including our new exam invigilator and residence tools.
NGSIS Architect Frank Boshoff and EASI staff Walter Ni and David Lock each played important roles in the project.

We’ve written about the ACORN project before . This was a massive undertaking and was the product of the collective efforts of NGSIS/EASI staff Hossein Aliabadi, Michael Clark, Miki Harmath, Mark Johnston, Andre Kalamandeen, Michael Wyers, David Lock, Mezba Mahtab, Michael O’Cleirigh, Haroon Rafique, Petru Sugar, Laurel Williams, Zoe Wong, and Terry Lago, amongst others.

Congratulations to everyone involved in these two projects!

Staff Profile: Andre Kalamandeen

As part of an ongoing series, we’ve reached out to contributing members of the NGSIS program and asked them to share some insights into their current involvement, career path, areas of professional expertise and personal interests that they bring to NGSIS projects.

Name:

Andre Kalamandeen

Role:

Intermediate Java Developer

Current Project:

ACORN. There’s still a lot of work to be done. We want to make it better, keep tweaking it. Then there’s ACORN Mobile, the staff-facing view…

Career Path:

I was born and raised in Guyana. I did my undergrad at the University of Guyana. After I finished a masters program in Computer Science at the University of Toronto, I saw a student job posting for a contract position doing mobile development. My first project at UTM was working on the card scanning system for shuttle bus running between UTM and the U of T. Since coming to EASI/NGSIS, I’ve worked on MyRes and the ACORN project.

Personal Highlight:

The ACORN project. I had never worked on a team that had so many functional roles! We had developers, functional analysis, UX, QA… Just having all these people work together; it was dramatic and it was interesting for me. It was great to be a part of that experience!

Off the clock:

I try to keep active which involves circuit training at the gym, swimming and cycling. I’m learning to play the piano, though I’m not very good at it. At times you’ll find me on battle.net playing StarCraft, but most times I’m working on mobile apps, or tinkering with new technology.

calmunderfire

Kalamandeen: Calm under fire. 

Changing of the Guard- Communicating ACORN to the University

In just a few short weeks, students at the University of Toronto will be introduced to ACORN, the Accessible Campus Online Resource Network. For students, ACORN will replace the existing ROSI Student Web Service (SWS). ACORN will provide a more convenient, personalized and guided experience for students using U of T’s online services.

Given the large-scale nature of this change, which will affect all three U of T campuses, we have worked very hard to get the word out to staff, faculty and students at the University. The last thing the team wants is for registrars or staff to feel surprised or worse, blindsided by the new service.

To achieve this, a wide array of activities have been undertaken by the team to reach out to the three campuses regarding ACORN. These activities have been both online and off; including staff training sessions, listserv e-mails, meetings with divisional representatives, a U of T conference presentation, a preview site and Q&A site for students, and an online resource kit for staff.

One of the most effective activities has been the hour-long ACORN staff training sessions, which saw a record turn out from staff joining us from a variety of perspectives including Registrar’s Offices, academic planning, academic advising, Dean’s offices, IT and Student Life staff. An extra four sessions had to be added on to accommodate the more than four hundred staff who came out to learn more about the ACORN service.

Communicating the news about the new service has been a careful balancing act. NGSIS is wary about “over-selling” ACORN, despite the enthusiasm we’ve heard to date from students and staff involved in the project and our own high hopes for its success, as there is always the risk that problems with the service will require that it be taken offline, and in anticipation of this remote possibility, the team has worked to strike a tone of ‘modest confidence.’

We are even offering students the option to opt-out for the time being and continue using ROSI-SWS.

We have been inspired by the interest and engagement of our students and staff community and very much look forward to our upcoming launch date.

Quick Numbers:

• 400+ staff attended training sessions

• 12 training sessions held

•35 student beta testing participants

• 100+ staff signed up for functional testing

 

NGSIS Impact Report: Business Intelligence

In the Fall of 2014, program director Cathy Eberts commissioned the development of a report examining the strategic impact of the NGSIS Program at the University of Toronto. The intention of the report was primarily to collect anecdotal and statistical feedback from key stakeholders representing the five strategic areas (student life, student accounts, registration & curriculum management, business intelligence and technical infrastructure) and create a “big picture” for the program sponsors.

The writer and designer of the NGSIS Impact Report was Andy Torr (Special Projects Officer, Strategic Initiatives). A gifted interviewer and writer, Andy was highly successful at connecting the divisional business case to the work being done by NGSIS to support staff, students and faculty. Andy worked in collaboration with Donald Boere (Vice-Principal, Innis College), who served as an editor on the report and joined Andy throughout the interview process.

The NGSIS Program is proud of what Andy & Donald were able to achieve with the report and believe it presents a strong case for the continuation of the NGSIS program in years to come.

Today, we offer a sneak peek of the report with an excerpt from the section on Business Intelligence.

You can view that article here.

An Awakening

Cover-2

I’ve just returned to NGSIS after a three month leave and feel a lot like Rip Van Winkel coming down from the mountains. So much has changed, so much has grown and developed in my absence.  Today, I got a demonstration of our new ACORN platform.

Wow.

Everything about this new tool, from the look & feel, to the functionality, to the sheer scope of information being unified on a single page is just breathtaking. I’ve worked here for several years and am well aware that many of these huge aspects of being a student- like registration, course planning, co-curriculars, paying tuition- are separate departments, led by different teams of people. The student of course sees none of this… and why should they need to? With this new ACORN platform, everything is there on one single, beautiful page.

As Piras and Mike were showing me how enrolment functioned in ACORN, my pupils started to dilate at the possibilities. This approach, which emphasized student planning and preparation, seems like it could be nothing short of revolutionary. The University will be able to use all of this information to plan for the popularity of courses well in advance of the registration crush.

My impression, upon awakening from a three month sleep, is that ACORN is going to make enormous waves at this institution when it is released!