← AI Foundry lab dashboard › Worklog
Covers Sun 11 Oct 2026 (week 41), 16:40 PDT to 23:00 PDT: aifoundry1, aifoundry2 and aifoundry3 in the AI Foundry lab at Studio 45. Written Sun 11 Oct 2026, about 22:45 PDT, from the dashboard's own data before and after, each machine's logs and settings, and checks run on the three machines. Work on a machine someone was using waited until they were done.
The dashboard went from four warnings to "All clear". Three of the warnings were one problem: the cable that joins the three machines had no addresses on it, so each machine's network start-up timed out. The fourth, a "root login" on aifoundry3, was the dashboard misreading a lab user's own session. Along the way, the sweep repaired 41 of 42 damaged disk blocks on aifoundry1 and deployed a waiting fix to the live monitor. A fix for the card-use log's growth was written and reviewed, and needs one more round before it goes on the machines. aifoundry3's share waited for the lab user working there all evening: its wired address and the live-monitor fix went on on 13 October, and only its kernel reboot still waits for that user.
What the dashboard said, and what it says now
| Alert at 15:52 | Level | Now | What was done |
|---|---|---|---|
| aifoundry1, aifoundry2, aifoundry3: "systemd degraded: NetworkManager-wait-online failed" | warning ×3 | gone | Fixed addresses on the wired network (1) |
| aifoundry3: "1 terminal logged in as root, the shared login" | warning | gone | The dashboard's rule fixed (2) |
| aifoundry1: "pool rpool: 42 errors at the last scrub" | info | 1 error | 41 of 42 blocks repaired (3) |
| aifoundry1, aifoundry3: "card-use log read only from …" | info ×2 | stays | A fix written and reviewed, not yet installed (4) |
| aifoundry3: "reboot pending" | info | stays, for now | Waits until the lab user working there is done (6) |
| aifoundry1: "reboot pending" | info | stays | Left for an on-site visit (what wasn't done) |
| "card telemetry sampling is on" | info | stays | Correct, though the sample itself hasn't run since 2 October (below) |
| aifoundry3: "card used since the last check" | info | comes and goes | Nothing to do: it shows while a lab user is on the card, as again at 22:21 |
What was done
1. The wired network between the three machines
The three machines have been cabled together since 7 October, but nothing on that network hands out addresses. So the wired port on each one kept asking for an address, giving up after 45 seconds and starting again: about 30,000 log lines per machine per boot. And because those requests never finished, the machine's network start-up never finished in time either. The check that waits for it gave up after a minute at every boot and stayed failed, which is what the dashboard reported.
17:13 and 17:14: aifoundry1 and aifoundry2 got fixed private addresses on the wired port, with no route to the internet through it, so the internet and every remote login stay on Wi-Fi. The setting is saved where the machines keep their network configuration, so it should come back after a reboot. The two machines now reach each other over the cable in under a millisecond, against about 6 ms over Wi-Fi. The failed start-up check was cleared on both, and the address requests stopped. On aifoundry3, where a lab user was at work, only the failed check was cleared at 17:15, and the network was left alone. Its wired address comes with its kernel update, once that user is done, and that reboot will be the setting's first real test.
2. The "root login" that wasn't
The dashboard counted a terminal as a root login when the root account owned it. On aifoundry3 the two root-owned terminals belonged to a lab user's own session: logging in over Tailscale SSH with a command, rather than to a shell, leaves the terminal owned by root, with only that person's programs on it, or with nothing on it at all. 17:16: the dashboard now counts a root-owned terminal as a root login only when a program on it runs as root, and counts a terminal with only one person's programs on it as theirs (95c0fef). At 17:36 a real root shell, opened briefly on aifoundry2 between two of the dashboard's checks, was still counted as one.
3. aifoundry1's disk
aifoundry1 keeps everything, home directories included, on one disk with no second copy. Its monthly check, at 00:24 on Sunday, found 42 blocks whose contents no longer matched their checksums, in four files. The same 42 were there at the check of 13 September, and none at the one before, on 9 August. The disk reports itself healthy: no media errors, 18% of its rated wear. What fits best is unclean power loss: every restart in the machine's log since September ended in a crash, not a shutdown, as when the lab's power switch cut all three machines on 7 October.
17:20 to 17:33, 41 of the 42 blocks were dealt with:
- 38 were in a leftover file from a January compiler build, a stage that the finished compiler, installed elsewhere, no longer needs. The installed compiler was checked before and after. The file was deleted.
- 3 were in two small vocabulary files of a public model, in a lab user's copy of llama.cpp. Both were restored byte for byte from that copy's own history, and checked against the public original, with their owner, permissions and dates unchanged.
- Then the error record was cleared and the disk checked again (2 min 39 s).
One block is left: a 1.7 KB piece of a departed user's old git clone. It exists nowhere public and no branch uses it, so it can't be restored, and it was left as it was. Deleting it would lose nothing a branch uses and would clear the last error; that is the lab lead's call.
4. The card-use log
The lab's live monitor reads each card's temperature once a second, and the card-use logger wrote one line for every one of those reads: 86,400 lines a card a day, 98% of the log, 32 MB a day on aifoundry1. The dashboard reads only the newest 64 MB, so its seven-day card-use table reached back two days on aifoundry1 and four on aifoundry3, and marked the days it could not read "not logged". At that rate the log's size cap would also have started deleting aifoundry1's history around 18 October, after 15 days instead of 90. A fix was written and reviewed, but not installed. It has the logger write one line a minute for the monitor's reads instead of one a second, and the table say "not read" rather than "not logged" for days it could not read. Three reviewers found that, as written, each joined line would count its whole minute as time the card was held, so the lab's card-use command would show an idle card as busy most of the day. It needs that fixed and a second review before it goes on the three machines. Until then the dashboard's notes are accurate: the log really is read only that far back.
5. The live monitor
A fix written on 4 October had never been deployed. 17:38: it went live on aifoundry2. The monitor now learns who holds a card from the card-use logger, instead of running a privileged command every second, which put about three lines a second into the system's login log. It also checks the card's own lock just before each reading. On aifoundry2 those log lines went from about 120 a minute to 1, and the readings carried on. Mon 12 Oct, 08:01: it went live on aifoundry1 too, once the session the lab lead had announced there was over: its log lines went from about 110 a minute to 5, and both cards kept reading. Tue 13 Oct, 08:07: it went live on aifoundry3 too, so all three machines now run the fixed monitor.
6. aifoundry3's reboot waits for its user
A lab user has worked on aifoundry3 since 15:36, with a long-running agent in a terminal session, and was on the card again at 22:21. A reboot would end that session. So nothing that restarts or reconfigures aifoundry3 was done tonight: its kernel update (waiting since 10 October), its wired address and the live-monitor fix wait until that user is done. It is ready for them: the new kernel, its start-up files and the card driver built for it are in place, and the services that must come back after a reboot (the card's clock settings, the lab demo, the card-use logger) all start at boot. This page will say when it is done.
Mon 12 Oct, 08:00: still waiting. No release was posted overnight, and the user's agent session on aifoundry3 is still open, so nothing there has changed.
Tue 13 Oct, 08:07: aifoundry3 has its wired address and the live-monitor fix, neither of which touches anyone's session: it reaches aifoundry1 and aifoundry2 over the cable in about 0.5 ms, so all three machines are now on the wired network, and the failed start-up check is gone. The kernel update still waits: a reboot was scheduled for 08:09 and cancelled at 08:08, because the user's session was still open, and it will wait until that session ends.
Wed 14 Oct: the cable dropped overnight. All three wired ports lost their link at about 22:35 on 13 October and got it back at 13:56 on 14 October, within seven seconds of each other, so traffic between the machines went over Wi-Fi in between, and the lab dashboard did not say so. The saved addresses held through it. Each port has now lost its link twice since its machine last started, so the cable and its switch need a look on site.
What wasn't done, and why
- aifoundry1's kernel update (waiting since 9 October). A reboot re-trains the link to card 0, which is in its best state in weeks: 37 corrected link errors in 3.8 days, against about one a second for the twelve days after 18 September. The lab also has no remote console if a boot hangs. Better done on a visit, with card 0's pending reseat.
- The dashboard's own card sample hasn't run since 2 October. The page's temperatures come from the live monitor and are current. The half-hourly sample is the fallback, and its 2 October readings are still in the page's data. It stopped for a different reason on each machine:
- aifoundry1: after a power cut, a person confirms which card is which before sampling resumes, and nobody has since the 7 October boot.
- aifoundry2: its "card quiet" check counts the card's management traffic, which the live monitor itself makes twice a second, so it can never pass.
- aifoundry3: a lab user was active, which is the check working as meant.
- The kernel's own log covers about 50 minutes. It fills with security-audit lines from sandboxed programs, about 23 a minute, so the health check's per-card kernel-error counts cover only the last hour on all three machines.
- The live monitor's reader crashes about once in 24,000 reads, on exit, inside the card vendor's management library, after its reading is printed. It loses about a dozen readings a day and leaves a core file each time. It is a bug for upstream.
- The card-open tracer runs only on aifoundry1.
- The power cuts themselves. A UPS would remove the likeliest cause of the disk damage, on all three machines.
What isn't known
- Whether the wired-address setting comes back after a reboot. It is saved where the machines keep their network configuration, and no machine has rebooted since; aifoundry3's reboot will be the first test.
Sources
- The dashboard's data at 15:52 and after each change, and the collector on aifoundry2 (
update.sh nowat 17:16). - On each machine: NetworkManager's settings and journal,
systemctl, the terminals and the programs on them, the disk's check results and event log, the disk's own health counters, the card-use logger's files, the live monitor's history and crash records. - The lab repository: the dashboard's collector and probe, the card-use logger, the live monitor and their tests (the dashboard's 32 and 61 tests; the card-use logger's own suite in review).
#community-labon the lab's Discord, for who was using which machine.- Written by agents in one session, each change checked by a second agent that changed nothing.