Shared Storage System (BASTO)

Critical Warning

AUTOMATIC FILE DELETION IS ENABLED on this storage system. Read this section carefully before storing any data.

Overview

Several "burros" (currently dalia, deimos, diva, divan, doctor, domino and dunedain) have access to a ~700 TB shared storage system hosted in a BSC-provided disk rack, accessible via the /storage/scratch mountpoint.

To prevent the storage from becoming full and to maintain equitable access for all IAC users, we have implemented an automatic file deletion policy.

Storage Health Levels

The system operates under three capacity thresholds:

  • Green: below 85% capacity (normal operation)

  • Yellow: between 85% and 90% capacity (cleanup recommended)

  • Red: above 90% capacity (automatic deletion triggered)

Yellow zone: Users will be notified and encouraged to delete unnecessary files to return to green status.

Red zone: AUTOMATIC DELETION activates and removes files until usage drops to 84%.

Note

From the command line you can check the usage of the system with the df command:

user@burro$ df -h /storage/
Filesystem                          Size  Used Avail Use% Mounted on
10.1.1.7@tcp:10.1.1.8@tcp:/storage  693T  204T  454T  31% /storage

Automatic Deletion Policy

When automatic deletion is triggered, files are selected for removal in the following priority order:

  1. Files not accessed in 1+ year:

    • First, files from the top five users by total storage consumption (in descending order)

    • Then, remaining files in order of oldest access time

  2. If still above 84%: Files not accessed in 6+ months:

    • Same prioritization: top five users first, then by access time

  3. If still above 84%: Files not accessed in 3+ months:

    • Same prioritization as above

  4. If still above 84%: Files accessed within the last 3 months:

    • Same prioritization (unlikely to be needed)

Rationale: This policy removes unused files while prioritizing those from the heaviest users, ensuring fair storage distribution across all IAC users.

Notifications

Status changes in the storage health levels trigger automatic notifications in the #computing/storage Zulip channel.

⚠️ If you store data in BASTO, you are STRONGLY ENCOURAGED to subscribe to this channel to receive deletion warnings.