Skild AI Unveils S1: In-Context Learning for Long-Horizon Robot Manipulation

Key Takeaways

Hide

• Skild AI has unveiled S1, a robot foundation model driven by visual in-context learning rather than task-specific fine-tuning.
• When prompted with a single video demonstration, S1 executes previously unseen, multistep manipulation tasks spanning up to 10 minutes—such as potting plants and making pour-over coffee.
• In controlled scaling benchmarks on novel tasks, S1 achieved a 66% average step-success rate at 100k hours of pretraining, compared to 9% for a standard Vision-Language-Action (VLA) baseline.
• Internal evaluations indicate that a single in-context visual prompt provides the performance equivalent of roughly 380 post-training teleoperation demonstrations, though a VLA post-trained on 2,000 demonstrations still edges ahead.
• The release expands an increasingly competitive frontier of prompting-based physical AI, contrasting with Generalist AI's short-horizon GEN-1.5 and Dyna-2's video-generative world model scaling.

Skild AI has published results for S1, an embodied foundation model engineered to perform complex manipulation tasks purely through in-context prompting. By replacing text commands and heavy task-specific fine-tuning with a single video demonstration, the system executes multistep physical routines out of the box.

The announcement challenges the dominant paradigm in robot learning, which typically relies on collecting tens to hundreds of hours of teleoperation data to fine-tune specialist policies for every distinct task. Skild frames the current state of robotics as stranded in a "BERT era" of pretraining followed by heavy post-training, arguing that true foundation models must mirror the transition to large language model prompting: learning how to execute novel behaviors at inference time without modifying underlying weights.

The work extends a line the company began in locomotion. LocoFormer, released a year ago, was an in-context locomotion policy that adapted to changes in its own body and environment by accumulating live experience in its prompt, without gradient updates of any kind. S1 is Skild's attempt to carry that result into manipulation.

Video Demonstrations as Prompts

Rather than conditioning actions on natural language instructions, S1 takes a video demonstration as its prompt. Skild argues that language is fundamentally too coarse for dexterous manipulation, reaching for tasks nobody would attempt to convey in a sentence: folding a fitted sheet, whisking egg whites to stiff peaks, threading a sewing machine.

During pretraining, S1 is exposed to episodic data where tasks are defined solely through visual demonstrations from varying viewpoints, scenes, and embodiments. Because the demonstration may come from a different scene or body entirely, an inverse dynamics model is insufficient—the model is pushed to learn composable manipulation primitives and strategies for transferring demonstrated behavior across environments.

Skild describes the formulation in meta-learning terms: pretraining is the outer loop that teaches the policy how to learn from context, while at inference the demonstration drives the inner loop without touching any weights.

The policy does not simply mimic raw trajectories. Skild demonstrated that S1 can filter out human clumsiness—in one prompt, the demonstrator cracks an egg abruptly and makes a mess, and S1 performs the same step with a controlled motion. It treats the demonstration as a specification of the goal rather than a trajectory to reproduce.

Out-of-Distribution, Ten-Minute Horizons

The core differentiator claimed for S1 is its operating envelope. While recent one-shot systems like Generalist AI's GEN-1.5 have demonstrated rapid adaptation across short 3- to 12-second horizons, Skild showcased S1 on complex sequences lasting up to 10 minutes.

The company highlighted four entirely out-of-distribution, multistep tasks:

• Plant Potting: Digging into soil, seating a plant, and handling watering cans. In an ad-hoc deployment trial, autonomous execution began 11 minutes after the demonstration.
• Breakfast Preparation: Mixing batter and flipping pancakes in a skillet.
• Pour-Over Coffee: Seating paper filters into funnels, dosing grounds, and pouring hot water.
• Mechanical Kit Assembly: Including assembling a skateboard wheel.

Skild notes that several of the underlying primitives here were absent from pretraining entirely—flipping a pancake, digging into soil, pressing a filter into a funnel—making these more than a recombination of known skills.

According to Skild, S1 demonstrated organic error recovery and perturbation resistance without task-specific programming. Researchers slid objects away from the robot as it approached, swapped objects mid-task, and changed the lighting; none of these perturbations appeared in the prompt, and the policy completed the tasks regardless. When S1 fails a step, the company says, it typically re-attempts rather than blindly proceeding—behavior usually baked in through targeted data collection.

In-Context Learning Scaling Laws

Skild conducted controlled benchmark ablations comparing its in-context learning (ICL) architecture against conventional VLA pipelines across identical compute envelopes and pretraining corpora scaling from 1,000 to 100,000 hours.

One caveat on the numbers below: these are average per-step success rates, not end-to-end task completions. Skild used human intervention to recover from failures during rollouts so that every step in a sequence could be graded.

Language-conditioned VLAs held an advantage in low-data regimes for familiar tasks, which Skild attributes to task specification being compressed into a handful of language tokens while the in-context equivalent is spread across a larger and noisier context. The company reports that the ordering reverses on seen tasks as data scales, though it does not publish figures for that crossover.

Skild offers two explanations for the divergence. Visual prompts carry lower conditional entropy—a language instruction admits many valid executions, while a demonstration selects one mode. And in-context learners retrieve behavior from context rather than storing it in weights, which the company argues leaves fixed-capacity VLAs prone to underfitting as datasets grow.

Crucially, Skild calculated that a single in-context video demonstration matches the performance yield of roughly 380 post-training teleoperation examples on a held-out task, with the crossing point interpolated between measured data points. Post-training does eventually overtake it: the VLA reaches 86% at 2,000 demonstrations. Skild is careful to note the two approaches are not mutually exclusive, since ICL policies can themselves be post-trained on task-specific data.

Graceful Degradation—Up to a Point

Beyond aggregate success rates, Skild tested how the policy degrades as deployment conditions drift from training conditions, using a five-level ladder of increasing displacement. Levels two and three perturb object pose with increasing magnitude, level four substitutes objects of matched affordance, and level five forces half the actions onto the opposite arm.

Across this ladder the VLA degrades up to three times more than the ICL policy. Skild's reading is that language-conditioned policies tolerate small variations in object pose but fail once novel motions are required, whether from new objects or from placements demanding a different execution plan.

The company ran the same ladder on the gap between the deployment scene and the demonstration held in context. S1 tolerated mismatched object positioning and even substituted objects, degrading significantly only at level five—where the demonstration implies a substantially different execution plan and actions must switch to the opposite arm. That failure mode is the clearest published limit on how far a single prompt stretches.

The Data Tradeoff and Industry Context

To feed the model, Skild relies on a multi-source data engine, scaling teleoperation, egocentric video, and other sources in-house rather than betting on any one. The company frames the tradeoff explicitly: teleoperation sits closest to the deployment hardware and scales worst, egocentric video scales best with the largest domain gap, and nothing wins on all three axes of hardware proximity, diversity, and scalability.

The team notes a deliberate policy of spending $3 on filtering and quality control for every $1 spent on raw acquisition, arguing that noisy data actively harms the effectiveness of scaling. Every data point entering pretraining is screened for low-level precision, task coherence, and annotation fidelity.

The release marks a divergence in physical AI philosophies. While Dyna-2 relies on million-hour video co-training to build generative world models that "dream" future states, and teams like Generalist AI pursue fast few-shot fine-tuning steps, Skild is betting that non-parametric in-context adaptation can bypass fine-tuning entirely for real-world deployments.

Skild is also careful to bound its own claim. Short, in-distribution tasks of five to thirty seconds do not need in-context learning at all, the company argues, since frontier VLAs already handle those zero-shot from a language prompt. The case for ICL rests on the hard regime: long, unseen, dexterous tasks where a single mistake cascades.

Whether visual prompting alone can deliver the reliability industrial deployment demands remains an open question, but S1 suggests the frontier of zero-gradient robot flexibility is widening rapidly.