Change in Scratch Cleaning Policy - July 1st, 2016

Effective July 1, 2016, the 30-day file retention policy for networked scratch filesystems on both Helium and Neon will change. The current policy of purging files based on access time has proven technically unreliable, and the new policy is designed with feedback from the HPC policy committee to better meet the technical needs of the systems.

The new policy will be based on creation time, not access time. Under the new policy files will be deleted from 60 days after they are created.

The policy for /localscratch filesystems on compute nodes is not affected by this change: such local files are deleted 30 days after they are last accessed.

Please take this opportunity to review your data residing on the scratch filesystems and to move longer-term files to other storage. As a reminder these are scratch filesystems and are not meant to be long term storage. Home accounts and any storage purchased from Research Services are *not* affected by this cleaning policy.

  • The new policy will be implemented by a frequent cleaning process that deletes files created more than 60 days earlier.
  • Because the policy will take effect on July 1, it will purge files created up to May 1.
  • Due to the unreliable nature of the previous cleaning process, we expect a large number of files to be deleted during the initial July 1st run. Therefore a read-only snapshot of the scratch filesystems will be made available for 2 weeks after this initial run completes in case file retrieval is required.
  • Please find this snapshot here:
    /nfsscratch/.zfs/snapshot/pre_cleaning_2016-07-01/
    Inside, an individual user directory is therefore like this:
    /nfsscratch/.zfs/snapshot/pre_cleaning_2016-07-01/Users/<HawkID>
  • Neon will retain this snapshot until July 15, 2016.
  • Helium's initial cleaning process completed at a later date, and therefore Helium will retain this snapshot until July 22, 2016.