AlephLM-0 E1 — execution environment
The exact stack the seed-1 arms (a1_anchored-s1, a2_dense-s1, a3_random-s1)
trained on, captured from the live container before it was released. Recorded
because several of this line's engineering laws are pinned to specific versions
and to the container's resource envelope — a replication on a drifted image
would change checkpointing and compile semantics silently.
Software
| component | version |
|---|---|
| Python | 3.11.10 (GCC 11.4.0) |
| PyTorch | 2.4.1+cu124 |
| CUDA (torch) | 12.4 |
| cuDNN | 9.1.0 (90100) |
| Triton | 3.0.0 |
| transformers | 5.14.1 |
| tokenizers | 0.22.2 |
| huggingface_hub | 1.26.0 |
| datasets | 5.0.1 |
| safetensors | 0.8.0 |
| numpy | 1.26.3 |
| tensorboard | 2.21.0 |
| torchvision / torchaudio | 0.19.1+cu124 / 2.4.1+cu124 |
| NVIDIA driver | 570.195.03 |
| OS / kernel | Ubuntu 22.04.5 LTS / 6.8.0-64-generic |
Full package set: pip_freeze.txt (179 packages) beside this file.
Why the torch pin matters. Three separate behaviors in this line are
torch-2.4-specific: torch.utils.checkpoint needs an explicit import (2.4.1
does not auto-import the submodule; 2.8+ does); compile-inside-checkpoint hits
2.4's RNG-fork/fake-tensor collision, so the compile path asserts
inner-checkpointing off; and 2.4's checkpoint recompute cannot re-enter dynamo,
a measured structural limit rather than a configuration choice.
Hardware and container envelope
| resource | value |
|---|---|
| GPUs | 2 × NVIDIA A40, 46,068 MiB each, PXB peer link |
| GPU UUIDs | GPU-b5d01167-717d-2198-1b7f-7ff14788d65a, GPU-0b319f75-ee74-46e3-9831-01dc3b8fe681 |
| Host CPU | Intel Xeon Gold 6342 @ 2.80 GHz, 96 vCPU visible |
| Host RAM | 540,644,167,680 B (~503 GiB) visible |
| Container memory quota | 110,999,998,464 B (103.38 GiB) |
| Container CPU quota | 17.85 cores (cfs quota 1,785,000 / period 100,000) |
| CPU affinity / NUMA | 0-23, 48-71 / node 0 |
| Container id | d2d7b1e1e4e4 |
| Storage | 200 GB container overlay; 400 GB network volume at the work root |
Why the quotas matter. The visible host figures (96 vCPU, ~503 GiB) are not what the container may use — the real envelope is 17.85 cores and 103.38 GiB. The streaming memmap target store exists precisely to fit that memory quota: an earlier build held the full 49 GB target buffer in RAM and, with transient spike, crossed the quota and took a silent cgroup SIGKILL at the final concatenate. Any rig sized against that law wants the exact byte value above, not a rounding.
Code revision per arm
The three seed-1 arms each fetched the training module at their own launch
instant, so they did not all execute the same revision. Launch times come from
the tensorboard filenames shipped with each run (events.out.tfevents.<epoch>.<container>.<pid>):
| arm | launch (UTC) | revision executed | target store implementation |
|---|---|---|---|
a1_anchored-s1 |
02:53:28 | abdd691 |
in-RAM concatenate |
a3_random-s1 |
03:36:35 | e21de50 |
streaming memmap cache |
a2_dense-s1 |
20:30:19 | ae93b6b |
streaming memmap cache (cache hit) |
e21de50 → ae93b6b changes 168 lines, but every hunk on a shared path is gated
behind the splat arm, which none of these three runs use — so a2 and a3 are
behavior-identical. abdd691 → e21de50 changes 86 lines located entirely in
the target store and not arm-gated: a1 therefore consumed its training
targets through a different store implementation than a2 and a3.
The store change alters how targets are held, not what they contain (the memmap byte arithmetic reconciles exactly to the source tensors: 31,905,616 × 768 × 2 bytes with zero slack), and a1-s1's capability lands within seed noise of a1-s0 (.6007 vs .6031, trained under yet another build). The recorded position is therefore that the arms are equivalent, stated as a checked fact with its one caveat named rather than as an unexamined assumption.
Console-log completeness caveat (a1_anchored-s1)
Two orchestrators launched a1 against the same fixed log path with truncating
redirection. The duplicate failed its VRAM probe and exited while the real run
continued writing at its own file offset, so the surviving console log carries a
490-byte hole and its first 7,195 bytes belong to the failed duplicate. Practical
consequences: the header line reporting a 14.3 GB allocator cap belongs to the
duplicate, not to the run that produced the shipped weights; and the real
run's own preamble (its allocator cap, VRAM-probe pass, config echo, resume
line) plus roughly the first 61% of its console output have no surviving
witness. The training record itself is unaffected — the tensorboard scalars on
the hub cover the full span, and the weights, metrics.json, and resume state
are complete. Fixed for future campaigns by giving every launch a distinct log
path and appending rather than truncating.
Captured 2026-08-07 by AbstractPhil before container release.