<!-- Note: generated by tools/generate_agent_context.py from tools/agent_context/clusters/anvil.yml. Manual changes will be lost! -->
# Anvil Agent Context
This file is the assembled agent context for Purdue RCAC's **Anvil**
cluster, concatenated from the topic files in `/etc/agents.d/`. It is the
canonical guidance an agent should follow when operating on Anvil. Read it
before acting.
<!-- Source: unix.md -->
# Unix Environment (Anvil)
Anvil runs **Rocky Linux 8.10** with a standard GNU/bash userland. You are acting as a
specific user, over that user's existing SSH access — you have exactly their
privileges and nothing more.
## The login (front-end) nodes are shared
Connecting to `anvil.rcac.purdue.edu` places you on one of several shared front-end
nodes, used simultaneously by many people. They are for **editing, compiling,
submitting jobs, and light pre- and post-processing only**.
- **Do NOT run large, long, multi-threaded, parallel, or CPU-intensive work on a
login node.** Submit it to Slurm instead (see `slurm.md`). Heavy work on a front end
degrades the system for everyone and may be killed by administrators.
- **Do NOT start unbounded background processes or retry loops.** Clean up stray
processes, and watch for your own retry storms.
## No elevated privileges
- **Do NOT use `sudo` or attempt to gain root.** You cannot install system packages.
- Install software into user space, or preferably use the environment modules system
(see `lmod.md`).
## Ground yourself before acting
Prefer checking the real state of the system over assuming it. These commands are
read-only, cheap, and safe — run them eagerly before you plan or act:
- `myquota` — home and scratch usage and limits.
- `mybalance` — the allocation accounts you can charge and their balances in service units (SUs).
- `sfeatures` — node and GPU hardware features.
- `module avail` / `module list` — available and currently-loaded software.
- `uname -m` and `/etc/os-release` — the architecture and OS of the node you are on.
Compute nodes can differ from the login node.
Grounding your plan in real output prevents the most common failure mode: confidently
acting on an assumption that is wrong for this cluster.
## Anvil specifics
- Anvil usernames are the user's ACCESS username with an `x-` prefix (for example `x-jdoe`). Do not assume a Purdue career-account name.
<!-- Source: filesystems.md -->
# Filesystems (Anvil)
Anvil mounts several filesystems, each with a different purpose, technology, and
durability. Put data in the right place, and never assume a path is durable without
checking.
## Home — `/home/$USER` (`$HOME`)
- **Technology:** ZFS, with nightly snapshots kept 7 days, weekly for 3 weeks, and monthly for 2 months (recover with `flost`).
- **Use for:** source code, scripts, configuration, small files. It is
medium-performance and space-limited.
- **Do NOT** stage large datasets or run heavy parallel job I/O here.
## Scratch — `$SCRATCH`
- **Technology:** GPFS, a high-performance parallel filesystem.
- **Use for:** active job input/output and large working data. Point writable working
files here (`$SCRATCH`), not at home.
- **Not backed up, and purged after 30 days.** The purge runs continuously, with no grace period and no warning email, and it cannot be overridden. Do not try to circumvent it.
- **Do NOT** treat scratch as durable storage — move anything you want to keep to
durable project or archive storage.
## Project — `/anvil/projects` (`$PROJECT`)
- One project space per allocation, shared read/write by everyone on it. `$WORK` points to the same place.
- GPFS with snapshots; 5 TB and 1 million files by default (more on request).
- Use for shared group data and software installs, not for active job output.
- Removed 90 days after the allocation expires.
## Node-local — `/tmp`
- Ephemeral, per-node storage that exists only while your job runs on that node; not
backed up, not recoverable.
## Check before you write
- Run **`myquota`** to read home and scratch usage and limits before writing large
amounts of data. **Do NOT assume specific quota numbers** — they differ per user and
change over time; read them from `myquota`.
- Reference paths with environment variables (`$HOME`, `$SCRATCH`), not
hard-coded strings, since paths can change.
## Prohibitions
- **Do NOT** store secrets, credentials, or sensitive/regulated data in world-readable
paths.
- **Do NOT** perform bulk or recursive deletes (`rm -rf`) without confirming with the
user first.
## Anvil specifics
- **Do NOT** fill `$HOME`; it is limited to 25 GB. Keep conda or virtual environments and large files in `$PROJECT`.
- **Do NOT** look for Data Depot (`/depot`) or use `hsi`/`htar` on Anvil. Data Depot is not available, and Fortress is reachable only by SFTP or Globus.
<!-- Source: lmod.md -->
# Software Modules — Lmod (Anvil)
Software on Anvil is provided through **environment modules** (Lmod), not the
system package manager.
## Finding and loading software
- `module avail` — list available modules.
- `module spider <name>` — search for a package and see its versions and
prerequisites.
- `module load <name>/<version>` — load a specific version; `(D)` marks the default.
- `module list` — show what is currently loaded; `module purge` clears everything.
Load the same modules inside your job scripts that you used interactively, so batch
jobs run in the environment you tested.
## Recommended toolchain
The recommended compiler and MPI stack is **GCC 11.2.0 with OpenMPI**. Check
`module avail` for the exact versions available.
## Prohibitions
- **Do NOT** install software with `apt`, `yum`, `dnf`, or `sudo` — you do not have
root, and system package managers are not the mechanism here. Use modules, or install
into user space.
- **Do NOT invent module names or versions.** Verify a package and version exist with
`module spider` (or `module avail`) before claiming it is available or writing it
into a job script.
## Python
- Use the `conda` module (Miniforge) for Python, and create your own environments with
it rather than modifying a base installation.
- **Do NOT** use the `anaconda` module for new environments. `conda` replaces it.
- `uv` is a good choice for pip-style project environments; install it in user space.
## Anvil specifics
- The CPU module tree (`modtree/cpu`) is loaded at login. GPU software is visible only after `module load modtree/gpu`; load it before searching for GPU modules.
<!-- Source: slurm.md -->
# Slurm Scheduler (Anvil)
All real computation on Anvil runs as **Slurm** jobs. Never run compute on a login
node (see `unix.md`) — submit it here.
## Every Anvil job needs three things
A job request must specify **resources**, an **account**, and a **partition**. On
Anvil you MUST explicitly set **both the account (`-A`/`--account=`) and the
partition (`-p`/`--partition=`)**, and always a time limit (`--time`).
- Discover the accounts you can charge (and their balances) with **`mybalance`** — do not
guess an account name.
- Set a realistic `--time`; memory is allocated in proportion to the cores you request.
## Partitions
| Partition | For | Notes |
|-----------|-----|-------|
| `debug` | Testing and debugging (CPU) | up to 2 nodes / 256 cores; max 2 h; 1 running, 2 queued per user |
| `gpu-debug` | Testing and debugging (GPU) | 1 node, up to 2 GPUs; max 30 min; 1 running, 2 queued per user |
| `wholenode` | Multi-node CPU jobs (128-core nodes) | node-exclusive (charged 128 cores per node); up to 16 nodes; max 96 h |
| `wide` | Large multi-node CPU jobs | node-exclusive; up to 56 nodes / 7,168 cores; max 12 h; 5 running per user |
| `shared` | Single-node CPU jobs that share a node; the default partition | 1 node, up to 128 cores; max 96 h |
| `highmem` | Large-memory jobs (1 TB nodes) | 1 node; max 48 h; charged 4 SU per core |
| `gpu` | GPU jobs (4× NVIDIA A100 40 GB per node) | needs GPU credits; max 48 h; max 12 GPUs in use per user |
| `ai` | AI jobs (4× NVIDIA H100 80 GB per node) | needs AI credits; max 48 h; max 12 GPUs in use per user |
## Common commands
`sbatch` (submit a batch script), `squeue -u $USER` (your jobs), `scontrol show job
<id>` (job detail), `scontrol hold`/`release`, `scancel <id>` (cancel), `sinteractive`
(interactive job), `srun` (launch tasks).
RCAC helpers: `jobinfo <id>` (one view of a running or finished job, from `sacct`, `sstat`
and `squeue`), `jobscript <id>` (the job's submission script), `jobcmd <id>` (its
submission command line), `jobenv <id>` (its environment).
## Prohibitions
- **Do NOT** request a `standby`, `partner`, or owner queue or QOS. None exist on Anvil, and all partitions have equal priority.
- **Do NOT** omit `-A` or `-p`. Without them the job goes to `shared` and may be charged to an unintended account.
- **Do NOT** send small or single-core jobs to `wholenode` or `wide`. Those partitions are node-exclusive and charge all 128 cores per node; use `shared`.
- **Do NOT** use `highmem` unless the job needs more memory than a standard node. It is charged 4 SU per core.
- **Do NOT** charge `gpu` or `ai` jobs to a CPU-only account. CPU, GPU and AI credits are separate; check `mybalance` for an account with credits for that partition.
- **Do NOT** request `--mem-per-cpu=2G`. The default is about 1896 MB per core, and asking for more can allocate extra cores and raise the charge.
- **Do NOT** emit flags for other schedulers (PBS/Torque `#PBS`, LSF `#BSUB`) — this is
Slurm.
- **Do NOT** invent partition names; the valid partitions are listed above.
- **Do NOT** submit jobs without a `--time` limit, and **do NOT** `scancel` jobs you do
not own or run bulk cancellations without confirming with the user.
## Anvil specifics
- `standard` is the older name of `wholenode` (the same nodes, limits and charging); write `wholenode`. **Do NOT** submit to `profiling`; it is not for user jobs.
- Jobs are charged in service units (SUs) against the allocation account: on CPU nodes 1 SU is one core-hour, and on GPU nodes 1 SU is one GPU-hour. Find partition limits with `showpartitions`.
<!-- Source: policies.md -->
# Policies and Safety (Anvil)
You are acting as a specific user on shared, audited research infrastructure.
Everything you do is attributable to that user and charged to their allocation.
## Acceptable use
- Use of Anvil is bound by Purdue IT's Acceptable Use Policy and the ACCESS allocation
policies. Stay within the user's approved allocations and project scope.
## Data handling
- **Do NOT** place sensitive, regulated, or export-controlled data on the cluster
without prior approval, and **do NOT** store any sensitive data in world-readable
locations.
## Secrets and credentials
- **Do NOT** read, print, or transmit private keys, API tokens, or passwords, and
**do NOT** write them into files, prompts, or your own context. Treat anything under
`~/.ssh` and any `.env`-style file as off-limits unless the user explicitly directs
otherwise.
## Resource stewardship
- No heavy compute on login nodes; submit through Slurm with time limits (see
`slurm.md`). Watch allocation balances with `mybalance` — a runaway resubmission loop can
exhaust an allocation quickly.
## Destructive operations require confirmation
- **Confirm with the user before any destructive or irreversible action:** `rm -rf`,
mass file moves, permission changes on shared paths, or bulk `scancel`.
## Containers are not a sandbox here
- Anvil uses **Apptainer** (not Docker). RCAC's Apptainer configuration
**auto-mounts `/anvil`, `/home`, and `/tmp`** into containers, and those mounts
are writable — running inside a container does **not** protect the user's real files
from being edited or deleted. Do not assume container isolation.
## Getting help
- Verify AI-generated commands before running them. Report incorrect or harmful AI
output to the **ACCESS Help Desk** (https://support.access-ci.org/help-ticket).
## Anvil specifics
- `singularity` still works as a command and runs Apptainer.
- **Do NOT** run `mykeys` or print its output. It returns the user's S3 access and secret keys for Anvil Object Storage.