<!-- Note: generated by tools/generate_agent_context.py from tools/agent_context/clusters/scholar.yml. Manual changes will be lost! -->
# Scholar Agent Context
This file is the assembled agent context for Purdue RCAC's **Scholar**
cluster, concatenated from the topic files in `/etc/agents.d/`. It is the
canonical guidance an agent should follow when operating on Scholar. Read it
before acting.
<!-- Source: unix.md -->
# Unix Environment (Scholar)
Scholar runs **Rocky Linux 9 on the login and x86_64 compute nodes** with a standard GNU/bash userland. You are acting as a
specific user, over that user's existing SSH access — you have exactly their
privileges and nothing more.
## The login (front-end) nodes are shared
Connecting to `scholar.rcac.purdue.edu` places you on one of several shared front-end
nodes, used simultaneously by many people. They are for **editing, compiling,
submitting jobs, and light pre- and post-processing only**.
- **Do NOT run large, long, multi-threaded, parallel, or CPU-intensive work on a
login node.** Submit it to Slurm instead (see `slurm.md`). Heavy work on a front end
degrades the system for everyone and may be killed by administrators.
- **Do NOT start unbounded background processes or retry loops.** Clean up stray
processes, and watch for your own retry storms.
## No elevated privileges
- **Do NOT use `sudo` or attempt to gain root.** You cannot install system packages.
- Install software into user space, or preferably use the environment modules system
(see `lmod.md`).
## Ground yourself before acting
Prefer checking the real state of the system over assuming it. These commands are
read-only, cheap, and safe — run them eagerly before you plan or act:
- `myquota` — home and scratch usage and limits.
- `slist` — the accounts you can submit to.
- `sfeatures` — node and GPU hardware features.
- `module avail` / `module list` — available and currently-loaded software.
- `uname -m` and `/etc/os-release` — the architecture and OS of the node you are on.
Compute nodes can differ from the login node.
Grounding your plan in real output prevents the most common failure mode: confidently
acting on an assumption that is wrong for this cluster.
## Scholar specifics
- The `spark-batch` and `spark-interactive` nodes are the exception: they are aarch64 and run **Ubuntu 24.04**, not Rocky Linux. Check `/etc/os-release` and `uname -m` on the node you are on before building or running software.
<!-- Source: filesystems.md -->
# Filesystems (Scholar)
Scholar mounts several filesystems, each with a different purpose, technology, and
durability. Put data in the right place, and never assume a path is durable without
checking.
## Home — `/home/$USER` (`$HOME`)
- **Technology:** ZFS, with nightly snapshots kept for 7 days, weekly for 4 weeks, and monthly for 3 months (recoverable).
- **Use for:** source code, scripts, configuration, small files. It is
medium-performance and space-limited.
- **Do NOT** stage large datasets or run heavy parallel job I/O here.
## Scratch — `/scratch/scholar/$USER` (`$RCAC_SCRATCH`)
- **Technology:** ZFS, served by the same storage system as home (not a parallel filesystem).
- **Use for:** active job input/output and large working data. Point writable working
files here (`$RCAC_SCRATCH`), not at home. Find the path with `findscratch`.
- **Not backed up, and purged after 60 days of inactivity** (by last access and
content-modification time; touching metadata does not protect a file).
- **Do NOT** treat scratch as durable storage — move anything you want to keep to Data
Depot or Fortress.
## Node-local — `/tmp`
- Ephemeral, per-node storage that exists only while your job runs on that node; not
backed up, not recoverable.
## Long-term — Data Depot and Fortress
- **Data Depot** is group project space on **GPFS** (typically under `/depot`), for
active research data you need to keep.
- **Fortress** is the HPSS archive for long-term/cold storage; move data with `hsi`
and `htar`.
## Check before you write
- Run **`myquota`** to read home and scratch usage and limits before writing large
amounts of data. **Do NOT assume specific quota numbers** — they differ per user and
change over time; read them from `myquota`.
- Reference paths with environment variables (`$HOME`, `$RCAC_SCRATCH`), not
hard-coded strings, since paths can change.
## Prohibitions
- **Do NOT** store secrets, credentials, or sensitive/regulated data in world-readable
paths.
- **Do NOT** perform bulk or recursive deletes (`rm -rf`) without confirming with the
user first.
## Scholar specifics
- Scholar's home directory is separate from the user's home on other RCAC clusters and is not synchronized with it. Copy files between clusters deliberately.
<!-- Source: lmod.md -->
# Software Modules — Lmod (Scholar)
Software on Scholar is provided through **environment modules** (Lmod), not the
system package manager.
## Finding and loading software
- `module avail` — list available modules.
- `module spider <name>` — search for a package and see its versions and
prerequisites.
- `module load <name>/<version>` — load a specific version; `(D)` marks the default.
- `module list` — show what is currently loaded; `module purge` clears everything.
Load the same modules inside your job scripts that you used interactively, so batch
jobs run in the environment you tested.
## Recommended toolchain
Do not assume a default compiler or MPI version. Check `module avail` (and, where
provided, `module load rcac`) for the recommended stack, and verify with `module list`.
## Prohibitions
- **Do NOT** install software with `apt`, `yum`, `dnf`, or `sudo` — you do not have
root, and system package managers are not the mechanism here. Use modules, or install
into user space.
- **Do NOT invent module names or versions.** Verify a package and version exist with
`module spider` (or `module avail`) before claiming it is available or writing it
into a job script.
## Python
- Use the `conda` module (Miniforge) for Python, and create your own environments with
it rather than modifying a base installation.
- **Do NOT** use the `anaconda` module for new environments. `conda` replaces it.
- `uv` is a good choice for pip-style project environments; install it in user space.
## Scholar specifics
- The `spark-batch` and `spark-interactive` nodes are aarch64 (NVIDIA GB10) and run Ubuntu 24.04. They use their own module tree: run `module load modtree/spark` there and rebuild software for aarch64 (check with `uname -m`). **Do NOT** run x86_64 software or modules from the default tree on them.
<!-- Source: slurm.md -->
# Slurm Scheduler (Scholar)
All real computation on Scholar runs as **Slurm** jobs. Never run compute on a login
node (see `unix.md`) — submit it here.
## Every Scholar job needs four things
A job request must specify **resources**, an **account**, a **QOS**, and a
**partition**. On Scholar you MUST explicitly set **both the account
(`-A`/`--account=`) and the partition (`-p`/`--partition=`)**; also set the QOS
(`-q`/`--qos=`) and always a time limit (`--time`).
- Discover the accounts you can submit to with **`slist`** — do not
guess an account name.
- Set a realistic `--time`; memory is allocated in proportion to the cores you request.
## Partitions
| Partition | For | Notes |
|-----------|-----|-------|
| `cpu` | CPU jobs (`scholar-a`: 128 cores, 256 GB; `scholar-b`: 64 cores, 256 GB) | `-A scholar`; QOS `normal` (default, up to 4 h), `long` (up to 3 days) or `debug` (30 min) |
| `gpu` | NVIDIA GPU jobs (V100, A30, A30 MIG slices, A40 nodes) | `-A scholar`; request GPUs with `--gres=gpu:N` or `--gpus-per-node=N` |
| `spark-batch` | NVIDIA GB10 (DGX Spark) nodes with exclusive resources; aarch64, Ubuntu 24.04 | `-A scholar`; `module load modtree/spark` first; x86_64 binaries do not run |
| `spark-interactive` | NVIDIA GB10 (DGX Spark) nodes, shared, for interactive work; aarch64 | `-A scholar`; `module load modtree/spark` first; x86_64 binaries do not run |
## QOS
- **`normal`** — the default; jobs up to 4 hours.
- **`long`** — jobs longer than 4 hours, up to 3 days, at lower priority. Request it with **`-q long`**.
- **`debug`** — high priority for testing: one running job, up to 2 nodes, 30 minutes. Request it with **`-q debug`**.
## Common commands
`sbatch` (submit a batch script), `squeue -u $USER` (your jobs), `scontrol show job
<id>` (job detail), `scontrol hold`/`release`, `scancel <id>` (cancel), `sinteractive`
(interactive job), `srun` (launch tasks).
RCAC helpers: `jobinfo <id>` (one view of a running or finished job, from `sacct`, `sstat`
and `squeue`), `jobscript <id>` (the job's submission script), `jobcmd <id>` (its
submission command line), `jobenv <id>` (its environment).
## Prohibitions
- **Do NOT** request a `standby`, `preemptible`, or `training` QOS. Scholar has none; its QOS are `normal`, `long` and `debug`.
- **Do NOT** request more than 4 hours under the default `normal` QOS; add `-q long` for jobs up to 3 days.
- **Do NOT** invent an account name. The documented account is `scholar` (`-A scholar`); check `slist` for the accounts available to you.
- **Do NOT** submit to the `gpu` partition without requesting GPUs (`--gres=gpu:N` or `--gpus-per-node=N`).
- **Do NOT** assume `ai`, `smallgpu`, `highmem`, `a10`/`a30`/`a100-*` or `training` partitions. Those belong to other RCAC clusters; Scholar's partitions are `cpu`, `gpu`, `spark-batch` and `spark-interactive`.
- **Do NOT** emit flags for other schedulers (PBS/Torque `#PBS`, LSF `#BSUB`) — this is
Slurm.
- **Do NOT** invent partition or QOS names; the valid sets are listed above.
- **Do NOT** submit jobs without a `--time` limit, and **do NOT** `scancel` jobs you do
not own or run bulk cancellations without confirming with the user.
<!-- Source: policies.md -->
# Policies and Safety (Scholar)
You are acting as a specific user on shared, audited research infrastructure.
Everything you do is attributable to that user.
## Acceptable use
- Use of Scholar is bound by Purdue IT's Acceptable Use Policy and RCAC's resource
policies. Stay within the user's approved allocations and project scope.
## Data handling
- **Do NOT** place sensitive, regulated, or export-controlled data on the cluster
without prior approval, and **do NOT** store any sensitive data in world-readable
locations.
## Secrets and credentials
- **Do NOT** read, print, or transmit private keys, API tokens, or passwords, and
**do NOT** write them into files, prompts, or your own context. Treat anything under
`~/.ssh` and any `.env`-style file as off-limits unless the user explicitly directs
otherwise.
## Resource stewardship
- No heavy compute on login nodes; submit through Slurm with time limits (see
`slurm.md`). A runaway resubmission loop wastes shared resources; watch your jobs with
`squeue -u $USER`.
## Destructive operations require confirmation
- **Confirm with the user before any destructive or irreversible action:** `rm -rf`,
mass file moves, permission changes on shared paths, or bulk `scancel`.
## Containers are not a sandbox here
- Scholar uses **Apptainer** (not Docker). RCAC's Apptainer configuration
**auto-mounts `/home`, `/depot`, and `/scratch`** into containers, and those mounts
are writable — running inside a container does **not** protect the user's real files
from being edited or deleted. Do not assume container isolation.
## Getting help
- Verify AI-generated commands before running them. Report incorrect or harmful AI
output to **rcac-help@purdue.edu**.