Notes on Skild S1 and NVIDIA's September data releases

Skild AI published its S1 robot foundation model on September 10, and NVIDIA published a companion post the same day describing the infrastructure underneath it. The headline claim is that S1 learns a task it has never seen from a single video of a person doing it. The same fortnight NVIDIA also put out two autonomous-vehicle items that matter for anyone producing training data: a technical post on re-rendering driving scenes for new sensor rigs with Omniverse NuRec, and a robotaxi round-up that gives the current numbers for its simulation stack. Below are the items that touch the synthetic data pipeline, and one ratio from Skild's write-up that we keep coming back to.

One video instead of 380 demonstrations

S1 takes a video demonstration as input and executes the task without updating its weights or post-training. Skild calls this in-context learning for robotics. The numbers Skild reports are for long-horizon tasks held out from pretraining: potting a plant, cooking pancakes, pour-over coffee, assembling a kit, each running up to 10 minutes and dozens of manipulation steps. With 100,000 hours of pretraining, S1 hits 66% step success on those unseen tasks from one video prompt. A language-prompted VLA trained on the same data reaches 9%, and Skild says it never completed a full unseen task once. On tasks that were in the pretraining set, S1 reaches 96%.

The figure most people will quote is the exchange rate. Skild puts a single in-context demonstration at roughly 380 post-training episodes for a conventional VLA. Collecting 380 long-horizon demos of four to ten minutes each takes 50 to 100 hours of teleoperation. The plant-potting example went from recording the demonstration to autonomous execution on hardware in 11 minutes, including five minutes of setup. The commercial side is already running: NVIDIA lists a Foxconn deployment where dual-arm S1 systems install busbars and fasten 16 screws on Blackwell assemblies, and Skild reports a $100 million annual run rate ten months after its first deployment, with more than 60 deployment partnerships.

Two caveats from the same write-up. The metric is per-step success, not task completion, and trial counts are not reported. And 66% from one video is not the ceiling: the same post shows a VLA reaching 86% on unseen tasks after 2,000 demonstrations. So the demonstration is cheap now, but the distance between 66% and a production success rate is still closed with data.

Where the 100,000 hours come from

The August post on this blog argued that once everyone downloads the same policy architecture and the same simulator, the training distribution is the differentiator. S1 is a concrete case. Skild describes its pretraining mix as a balance of four sources with different trade-offs: robot teleoperation, close to the hardware but slow to scale; UMI handheld-gripper data, moderate on every axis; egocentric human video, diverse and cheap but far from any specific robot; and simulation, scalable but narrow in diversity. Deployment data flows back in where customer agreements allow.

The NVIDIA post fills in the tooling. Cosmos world foundation models diversify the training data and convert video into structured descriptions; Cosmos Curator does the annotation and filtering. Isaac Sim provides the physically based environments for data generation and edge-case testing. Isaac Lab runs reinforcement learning on the Newton physics engine, whose GPU solvers for grip, contact, and pressure NVIDIA says it developed jointly with Skild. So the synthetic half of the pipeline is not a side dataset here. It is inside the pretraining loop, generating the variety that a small set of real demonstrations lacks.

Skild's post also says, almost in passing: "For every dollar we spend on collecting data, we spend three on quality control." That is from a team whose main claim this month is a 380x drop in collection cost. Nothing in either post suggests the quality-control side moved.

Driving: re-rendering scenes instead of recapturing them

NVIDIA's AV items from the same weeks are both about getting more training data out of a scene you already validated. The NuRec post (August 31) addresses a plain production problem: a perception stack tuned for one carline degrades when the cameras move, change lens model, or lose coverage on the next variant, and labelling a new dataset for every variant is too slow early in a vehicle program. NuRec reconstructs the original drive with 3D Gaussian splatting, re-renders it through the target rig, pinhole, fisheye, or f-theta, at new extrinsics and fields of view, and carries the trajectories, calibration, object tracks, and map annotations across so the labels adapt with the frames. A Harmonizer pass cleans temporal artifacts before training. The public dataset behind the workflow has more than 1,500 reconstructed scenes, about 20 seconds each from six cameras. NVIDIA reports relative gains in detection precision and recall across all object classes against a zero-shot baseline, without publishing the figures.

The robotaxi round-up (September 10) gives the scale NVIDIA is now quoting for the same pattern: NuRec reconstructs the corner cases a fleet has actually seen, Cosmos generates variations, and "thousands of real-world corner cases" become "millions of combinations" for closed-loop validation in AlpaSim. The one training number in the post is about annotation rather than volume. Adding reasoning data to the Alpamayo vision-language-action models cut minimum average displacement error from 2.08 to 1.18, which NVIDIA reports as a 43% improvement. The improvement came from what the scenes were annotated with, not from adding more of them.

One more item, on where the weights live. NVIDIA announced on September 3 that it will acquire Hugging Face for $12.93 billion. The Cosmos and GR00T weights, the NuRec dataset, and the LeRobot integration all ship through that hub. Jensen Huang's statement says it stays open to every model, framework, and cloud. We have no reason to doubt it. It does mean the world models, the simulator, and the hub the weights ship through now belong to the same company, which is worth knowing when you plan a pipeline around them.

The quality-control side

Taken together, the month lowered the cost of producing training data in three places: a task demonstration is now one video, a new sensor rig is a re-render of drives you already have, and a rare event is a reconstruction plus a Cosmos pass. None of those steps checks the result, and Skild's ratio gives a rough idea of how much work that check is.

Three specific gaps stay open. Coverage: one video shows one arrangement of one set of objects, and Skild's own robustness ladder, from same-placement L1 to L5 layouts that force the opposite arm, is where the language-prompted baseline degraded up to three times as much. Knowing which of those levels your deployment will meet, and generating and checking data for it, is still domain work. Physical fidelity: NuRec answers the geometry change of a new rig; friction, mass, contact, and sensor noise still have to be checked against the real line, which is why Newton needed co-development with Skild in the first place. Lineage: when the training signal for a task is a four-minute video, recording what it showed, who recorded it, and which deployment behavior traces back to it is straightforward to record and, in practice, often not recorded. The EU AI Act's Article 10 expects that record to exist.

NVIDIA and its partners keep lowering the cost of producing training data, and we build on that stack. The work of checking that a policy trained on the data holds up outside the demo room, comparing coverage against the deployment domain, grounding the physics, keeping the evidence, has not gotten smaller. By Skild's accounting it is three-quarters of the data budget. That is where datadoo works.