The net that watches the screen

Start with what convolution is, because you've been using it for years: every blur, sharpen, and edge-detect filter in a photo editor is a tiny grid of numbers (a kernel, here 3×3) slid across the image, multiplying and summing at every stop. A convolutional layer is a bank of such filters with one upgrade: the numbers in the grid are learned, not chosen. Two properties follow. It's shift-invariant (move the ship left and its response moves left, automatically), which is exactly the right assumption to build into vision. And it's local: each output sees only a small patch, but stacking layers grows that patch: the receptive field, the amount of image that can influence one output. Each layer also runs many filters at once, producing channels: at every position, not one number but a short list of learned local descriptors ("edge here," "glow here," "bright-against-dark here"). Between layers sits a nonlinearity: ReLU, max(0, x), a hinge that lets evidence accumulate without canceling (negative evidence is dropped rather than subtracted, so features can only pile up). Without it the whole stack would collapse into one big fixed filter.

Stride is downsampling: a stride-2 layer computes its output at every second position, so the map shrinks by half, like a thumbnail. This trunk strides three times, so its deepest map lives at 1/8 resolution: each position there summarizes an 8×8 neighborhood (and, through the stacked filters, sees far wider). That pyramid (sharp edges at the bottom, coarse "what is this region" at the top) is the standard shape of a vision network; your own visual system opens with the same architecture. Hold onto the 1/8 number: chapters 04 and 05 are both about what that shrinking costs.

Do the parameter bookkeeping like chapter 02 of the pilot's notebook: a 3×3 filter bank from C_in channels to C_out costs 9·C_in·C_out + C_out numbers. The trunk's six layers (3→8→16→16→24→24→32) total 19,344; the decoder that paints answers back at full resolution adds ~8,500; about 28,000 all in. One more term you'll meet: batch normalization ("BN") re-centers and re-scales each channel during training to keep the arithmetic well-behaved. And because that correction is just a multiply and an add, at export it folds into the neighboring filter weights and vanishes (a scale and shift after a linear filter is just a different linear filter, so the two merge into one). The shipped net has no BN; it trained with it and absorbed it. Three more parts you'll see in the deep dive's parts list: a 1×1 conv mixes the channel list at a single position (no spatial reach, a per-pixel re-blend); an average-pool shrinks a map by averaging small blocks; a nearest-neighbor upsample grows one by copying each value into a 2×2 block. And its "no concat, no attention" just means none of the channel-stacking or big-model machinery. Every operation on the list is arithmetic at this level.

Finally the task itself: segmentation means per-pixel classification. The output isn't one label for the image but a full-resolution map of 5 scores per pixel (per-pixel logits; the argmax at each pixel paints the overlay). The same classifier machinery as ever, asked everywhere at once. You can now read the deep dive.

Unlike the pilot (reinforcement learning, no answer key), the eye is trained supervised: inputs paired with ground-truth targets, adjust the weights to shrink the error. Here the error is one number: across all 61,440 pixels, how far the five scores sit from piling their weight on the true class. Training walks all 28,000 weights downhill on that number, exactly as in the pilot's notebook (the field's word for it is the loss). Everything therefore hangs on where the targets come from and whether you can trust them. Here the renderer is the labeler: the pretty frame and the answer sheet are two drawings of the same underlying scene geometry, so the labels are pixel-perfect by construction. There is no label noise in this experiment at all, which is exactly what makes it a clean teaching instrument: every error you'll meet in chapters 04–05 is genuinely the model's.

The discipline vocabulary: the train / validation / test split exists because a model must be judged on data it never fit ("validation" steers design choices; "test" is touched last, once). Leakage is any path by which test information sneaks into training: the silent killer of machine-learning results, the equivalent of grading students on the exact questions they rehearsed. The defense here is structural: every scene is generated from a seed (a starting number that makes the randomness exactly repeatable), and the splits draw from disjoint seed blocks, non-overlapping by construction rather than by careful bookkeeping. Plus one whole recorded duel held out untouched, because random scenes can't test what correlated, duel-shaped sequences test.

The catch you should be asking about: synthetic data is too clean. A net trained on noiseless renders can learn to depend on that cleanliness. Chapter 05 of the pilot's notebook called this training against a lie. The counter is domain randomization, encountered there as theory and used here for real: every training frame gets randomly jittered brightness, color balance, and sensor-style grain, so the net has to learn objects, not exact pixel values. Train in every weather. A separate, deliberately harsher stress set probes robustness beyond the training jitter: the final exam is harder than the homework, on purpose.

Trap one is a boundary bug with a satisfying explanation. A filter sliding near the edge of a small, isolated tile runs out of image, so the missing pixels are invented as zeros (zero-padding), and every output near a border is computed partly against synthetic blackness. On a 32×32 tile, with a filter stack whose receptive field is wider than the tile, effectively every output is a border output: the net learned the statistics of padded crops. Run the same weights across a full frame and the padding vanishes (real neighbors everywhere), and the cues the net had leaned on simply never occur. It learned faces from passport photos and was then shown a crowd. The trained-on view and the deployed view were two different worlds that happened to share pixels.

Trap two is a sampling error, the phenomenon physicists know as aliasing: a signal narrower than your sampling grid slips between the samples. The coarse first-stage net (next chapter) makes one decision per 8×8 tile, and each tile was initially labeled by its center pixel's class. A laser is a 3×3-pixel spark, nine pixels: the chance one of them sits exactly on that sparse lattice of center pixels is about one in seven (9 of 64). So six of seven laser tiles were labeled "background" while visibly containing a laser, and the net did exactly what it was taught: ignore lasers. No amount of training fixes a labeling scheme that erases the class. The fix is correct sampling too: label each tile by the rarest class present anywhere in it (an "is it in there at all?" test, not a point sample), which guarantees every object lights at least one tile.

The meta-lesson generalizes far past machine learning: both traps produced excellent scores. Nothing inside the training loop can warn you that the benchmark measures the wrong thing: the 99.9% was real, on a test set built with the same flaw. The only defense is an end-to-end check under deployment conditions, which is why every stage of this project gates on full-frame, held-out evaluation before proceeding. A great score on the wrong test is the most expensive kind of success.

First, the metric, because every number here uses it: IoU (intersection-over-union) for a class is overlap ÷ union of the predicted and true pixel sets. 1.0 is perfect, and it punishes painting too much and painting too little symmetrically. Small objects make IoU brutal: misdraw an 18-pixel hull by a 1-pixel ring and you lose tens of points, because that ring is a large fraction of a small area. That's the whole plateau in one sentence: an 8× upscale can only place edges to within ±4 pixels, and ±4 on an 18-pixel object caps IoU near 0.6. The decoder wasn't undertrained. It was information-limited. Once downsampling threw the fine positions away, no amount of training could conjure them back: you can't unshred the page by studying the shreds harder.

The fix is the field's standard answer, worth knowing by name: skip connections. The sharp spatial detail isn't gone from the network: the early layers still run at full and half resolution; it's only missing from the deepest map. So the decoder, at each 2× upscale, adds in a lightly-transformed copy of the matching-resolution early layer: coarse meaning arriving from above, crisp location wired in from the side. This is the U-Net idea, the backbone of a decade of segmentation work (medical imaging especially). One engineering nuance from this lab: the skips are additive (plain elementwise addition, not the textbook's channel-stacking), chosen deliberately because an add stays trivial in 8-bit integer arithmetic (chapter 06) while stacking complicates it. Design for the cheap chip before you shrink to fit it.

Second theme of the chapter: class weighting, the knob for imbalanced data. Lasers are a few thousandths of the pixels, so an unweighted loss barely notices losing them; weighting laser errors up makes the net eager: high recall (it misses almost nothing) at lower precision (it paints lasers a bit generously, about 2× fat). That trade is one knob on one curve, and where to sit on it is a product decision, not a math one: in a live demo, a missed laser reads as blindness while a slightly fat laser reads as glow. The lab chose recall, on purpose, and wrote the reason down. Chapter 06's calibration decision rhymes with this one. (Two training names you'll see in the deep dive: AdamW is the pilot notebook's Adam optimizer with a standard tidy-up, and a cosine schedule just eases the step size down along a smooth curve as training finishes.)

Quantization is rounding with a plan. Every weight and every activation gets mapped from smooth floating-point numbers onto 256 integer levels (int8) by a scale factor: choose a full-scale F for each signal and code it as round(127·x/F). Choosing F is the entire game, and it's the same dilemma as setting recording levels for audio: set the level too low and quiet detail is lost in coarse steps; too high and the loud peaks clip. Weights are known exactly at export, so each output channel gets its own perfect scale (per-channel). Activations depend on the input, so their ranges are estimated empirically, by streaming calibration frames through the float net and recording the extremes each signal actually reaches. The arithmetic then runs int8 × int8 with the sums accumulated in int32 (a wide running total, so nothing overflows partway through), after which each sum is requantized: re-leveled back onto the 256-step scale, one precomputed multiplier per channel. Everything is symmetric around zero (no offset terms anywhere), which is what kept chapter 05's additive skips trivial: integer adds, one shared rescale.

The calibration decision deserves a slower telling. The textbook heuristic sets each activation's F at the 99.9th percentile of what calibration observed: "don't waste your 256 levels on freak outliers," which on photographs (sensor noise, sun glints) is usually right. On these synthetic frames it is precisely wrong, and the reason is the shape of the signal: the images are nearly binary (flat dark space, a few small bright objects), and the laser-detecting channels fire rarely and hard. Their top 0.1% of activations isn't a noise tail. It is the signal. Clip it and you amputate the laser detector. Max calibration (F = the true observed maximum, clip nothing) spends a little resolution everywhere to keep the spikes, and here that cost is absorbable. Laser accuracy under percentile clipping: fails the gate. Under max: identical to float. Know the shape of your signal before you trust anyone's heuristic about it.

Why is such a crude scheme sufficient (round the finished net, no retraining; the field calls it PTQ, post-training quantization)? Margin. A well-trained classifier's decisions sit far from its decision boundaries, and int8's worst-case rounding (~0.4%) nudges the scores far less than the typical winner's lead. When PTQ isn't enough (huge models, more aggressive formats), the field retrains with the rounding simulated inside the training loop (QAT, quantization-aware training). Rule of thumb worth exporting: try the cheap thing first and measure per class. Aggregate accuracy hides exactly the small, rare classes that clipping kills.

The bundle is a contract, and its enforcement mechanism is old and beautiful: golden vectors, recorded inputs with their exact recorded outputs. The reference implementation (a few hundred lines of plain array arithmetic) is the executable spec: not a description of the computation but the computation itself, short enough to read in one sitting. Anyone porting the eye to new hardware re-implements the arithmetic however they like, feeds it the same eight inputs, and compares against the goldens. Match, and the port is correct by the contract's definition. No argument about methodology required. It's the logic of a checksum, or of the old standard kilogram: correctness defined as agreement with a fixed reference artifact, not as anyone's opinion.

What ML adds is a subtlety about tolerances, worth internalizing. The float gate can't demand bit-for-bit equality: adding the same list of numbers in a different order legitimately changes the last decimal place, and different hardware sums in different orders. So the float spec says "within 1e-4, and all 61,440 pixel decisions must agree." The int8 gate is stricter than the float one: because chapter 06 defined the integer path entirely in integer arithmetic, there is no order-of-addition ambiguity. The correct output is one specific string of bytes, compared byte for byte. That determinism isn't luck; it was purchased back in chapter 05 (additive skips) before quantization ever ran: adds keep the whole path in whole numbers, and whole-number arithmetic has exactly one right answer.

The reproducibility claim is the other half. One seed drives scene generation, splits, initialization, and training order, and re-running the whole pipeline lands on the same shipped file, bit for bit, on the same machine and software stack: floating-point training is only bit-repeatable on the platform that ran it, which is why the bundle's gates, not its bits, are the cross-platform contract. That still makes the experiment falsifiable end to end (every number in this notebook can be regenerated, or caught wrong), which is rarer in machine learning than it has any right to be (nondeterministic training, undocumented data filtering, and "the checkpoint we happened to keep" are endemic). And note where the notebook deliberately stops: the bundle says what a port must compute, never how. No target, no toolchain, no vendor. Any student with a microcontroller, an FPGA, or a GPU shader can port the eye and prove their port against the same goldens. The two experiments on this site are the two halves of an autonomy stack: a policy that decides (the pilot) and perception that sees (the eye), each shipped as a folder of checkable arithmetic.