1300 lose home directory access

A problem arising while moving user data between server disks last week resulted in the temporary outage of over 1300 student and staff profiles. The problem began on the morning of Monday 21st February, when staff and students found that their home directory was empty and that they had no profile settings. As a result, the default guest profile was used and the affected users were met with no desktop background or access to files on the Home Directory (H:/).

Heather Jones, Deputy President (Clubs & Societies), complained that she was “unable to open PDFs or Firefox and has called ICT every day since it happened”. Access to her files were restored on the 28th. Commenting on the problem, she said that “it was very problematic as a Sabbatical to have been in that position”.

ICT issued a statement regarding the issue, explaining that a routine shift of user data between disks on the weekend of the 18th/19th last month failed to complete. “Although the process is designed to cope with various failure, and has indeed done so without issue, this time a combination of circumstances conspired to defeat our process and caused this problem”, they explained.

Upon encountering the problem and receiving a vast number of calls, ICT established that 1300 users no longer had their files available. This represents less than 3% of the file directories ICT looks after. After indicating the problem in the daily bulletin to allow people with specific time deadlines to call for extensions to work, the action they took was to begin reloading the files from backup tape drives. ICT said “it was hoped that the 4 million files could be restored by the following day, but sod’s law was in full operation.” They were faced with a worst case scenario in which the maximum amount of incremental backups had to be used to restore the files, a situation in which full backups (rather than incremental) allow the quickest file reloading time.

User profiles are a collection of settings corresponding to the personalisation of a desktop. Since Imperial’s network is vast, profiles are stored on the servers and features such as desktop backgrounds and browser bookmarks are stored to allow users access from whichever machine they login from. When a user logs in to a machine, their profile is initially downloaded from the servers as part of the login process, and the settings are applied to restore the customised experience.

It is important to note that no data was lost and normal service resumed from mid-week to Friday the 25th. Paul Allatt, Head of IT Services, said that “for the immediate future, [they] are reviewing, fixing and re-testing the process” which resulted in the issue and “are in the process of adopting new disc technology facilitating regular snapshots and a much more rapid restore of files should the need arise.”

ICT apologise to all those impacted and inconvenienced by the disruption to their work and studies.