AI Foundry lab

AI Foundry lab, 11 October 2026: a maintenance sweep of the lab dashboard

← AI Foundry lab dashboard › Worklog

Covers Sun 11 Oct 2026 (week 41), 16:40 PDT to 23:00 PDT: aifoundry1, aifoundry2 and aifoundry3 in the AI Foundry lab at Studio 45. Written Sun 11 Oct 2026, about 22:45 PDT, from the dashboard's own data before and after, each machine's logs and settings, and checks run on the three machines. Work on a machine someone was using waited until they were done.

The dashboard went from four warnings to "All clear". Three of the warnings were one problem: the cable that joins the three machines had no addresses on it, so each machine's network start-up timed out. The fourth, a "root login" on aifoundry3, was the dashboard misreading a lab user's own session. Along the way, the sweep repaired 41 of 42 damaged disk blocks on aifoundry1 and deployed a waiting fix to the live monitor. A fix for the card-use log's growth was written and reviewed, and needs one more round before it goes on the machines. aifoundry3's share waited for the lab user working there all evening: its wired address and the live-monitor fix went on on 13 October, and only its kernel reboot still waits for that user.

What the dashboard said, and what it says now

Alert at 15:52 Level Now What was done
aifoundry1, aifoundry2, aifoundry3: "systemd degraded: NetworkManager-wait-online failed" warning ×3 gone Fixed addresses on the wired network (1)
aifoundry3: "1 terminal logged in as root, the shared login" warning gone The dashboard's rule fixed (2)
aifoundry1: "pool rpool: 42 errors at the last scrub" info 1 error 41 of 42 blocks repaired (3)
aifoundry1, aifoundry3: "card-use log read only from …" info ×2 stays A fix written and reviewed, not yet installed (4)
aifoundry3: "reboot pending" info stays, for now Waits until the lab user working there is done (6)
aifoundry1: "reboot pending" info stays Left for an on-site visit (what wasn't done)
"card telemetry sampling is on" info stays Correct, though the sample itself hasn't run since 2 October (below)
aifoundry3: "card used since the last check" info comes and goes Nothing to do: it shows while a lab user is on the card, as again at 22:21

What was done

1. The wired network between the three machines

The three machines have been cabled together since 7 October, but nothing on that network hands out addresses. So the wired port on each one kept asking for an address, giving up after 45 seconds and starting again: about 30,000 log lines per machine per boot. And because those requests never finished, the machine's network start-up never finished in time either. The check that waits for it gave up after a minute at every boot and stayed failed, which is what the dashboard reported.

17:13 and 17:14: aifoundry1 and aifoundry2 got fixed private addresses on the wired port, with no route to the internet through it, so the internet and every remote login stay on Wi-Fi. The setting is saved where the machines keep their network configuration, so it should come back after a reboot. The two machines now reach each other over the cable in under a millisecond, against about 6 ms over Wi-Fi. The failed start-up check was cleared on both, and the address requests stopped. On aifoundry3, where a lab user was at work, only the failed check was cleared at 17:15, and the network was left alone. Its wired address comes with its kernel update, once that user is done, and that reboot will be the setting's first real test.

2. The "root login" that wasn't

The dashboard counted a terminal as a root login when the root account owned it. On aifoundry3 the two root-owned terminals belonged to a lab user's own session: logging in over Tailscale SSH with a command, rather than to a shell, leaves the terminal owned by root, with only that person's programs on it, or with nothing on it at all. 17:16: the dashboard now counts a root-owned terminal as a root login only when a program on it runs as root, and counts a terminal with only one person's programs on it as theirs (95c0fef). At 17:36 a real root shell, opened briefly on aifoundry2 between two of the dashboard's checks, was still counted as one.

3. aifoundry1's disk

aifoundry1 keeps everything, home directories included, on one disk with no second copy. Its monthly check, at 00:24 on Sunday, found 42 blocks whose contents no longer matched their checksums, in four files. The same 42 were there at the check of 13 September, and none at the one before, on 9 August. The disk reports itself healthy: no media errors, 18% of its rated wear. What fits best is unclean power loss: every restart in the machine's log since September ended in a crash, not a shutdown, as when the lab's power switch cut all three machines on 7 October.

17:20 to 17:33, 41 of the 42 blocks were dealt with:

One block is left: a 1.7 KB piece of a departed user's old git clone. It exists nowhere public and no branch uses it, so it can't be restored, and it was left as it was. Deleting it would lose nothing a branch uses and would clear the last error; that is the lab lead's call.

4. The card-use log

The lab's live monitor reads each card's temperature once a second, and the card-use logger wrote one line for every one of those reads: 86,400 lines a card a day, 98% of the log, 32 MB a day on aifoundry1. The dashboard reads only the newest 64 MB, so its seven-day card-use table reached back two days on aifoundry1 and four on aifoundry3, and marked the days it could not read "not logged". At that rate the log's size cap would also have started deleting aifoundry1's history around 18 October, after 15 days instead of 90. A fix was written and reviewed, but not installed. It has the logger write one line a minute for the monitor's reads instead of one a second, and the table say "not read" rather than "not logged" for days it could not read. Three reviewers found that, as written, each joined line would count its whole minute as time the card was held, so the lab's card-use command would show an idle card as busy most of the day. It needs that fixed and a second review before it goes on the three machines. Until then the dashboard's notes are accurate: the log really is read only that far back.

5. The live monitor

A fix written on 4 October had never been deployed. 17:38: it went live on aifoundry2. The monitor now learns who holds a card from the card-use logger, instead of running a privileged command every second, which put about three lines a second into the system's login log. It also checks the card's own lock just before each reading. On aifoundry2 those log lines went from about 120 a minute to 1, and the readings carried on. Mon 12 Oct, 08:01: it went live on aifoundry1 too, once the session the lab lead had announced there was over: its log lines went from about 110 a minute to 5, and both cards kept reading. Tue 13 Oct, 08:07: it went live on aifoundry3 too, so all three machines now run the fixed monitor.

6. aifoundry3's reboot waits for its user

A lab user has worked on aifoundry3 since 15:36, with a long-running agent in a terminal session, and was on the card again at 22:21. A reboot would end that session. So nothing that restarts or reconfigures aifoundry3 was done tonight: its kernel update (waiting since 10 October), its wired address and the live-monitor fix wait until that user is done. It is ready for them: the new kernel, its start-up files and the card driver built for it are in place, and the services that must come back after a reboot (the card's clock settings, the lab demo, the card-use logger) all start at boot. This page will say when it is done.

Mon 12 Oct, 08:00: still waiting. No release was posted overnight, and the user's agent session on aifoundry3 is still open, so nothing there has changed.

Tue 13 Oct, 08:07: aifoundry3 has its wired address and the live-monitor fix, neither of which touches anyone's session: it reaches aifoundry1 and aifoundry2 over the cable in about 0.5 ms, so all three machines are now on the wired network, and the failed start-up check is gone. The kernel update still waits: a reboot was scheduled for 08:09 and cancelled at 08:08, because the user's session was still open, and it will wait until that session ends.

Wed 14 Oct: the cable dropped overnight. All three wired ports lost their link at about 22:35 on 13 October and got it back at 13:56 on 14 October, within seven seconds of each other, so traffic between the machines went over Wi-Fi in between, and the lab dashboard did not say so. The saved addresses held through it. Each port has now lost its link twice since its machine last started, so the cable and its switch need a look on site.

What wasn't done, and why

What isn't known

Sources