Can We Trust a Robot That Learns From What It Sees?

A brief outlook on what is planned

It is of unanimous belief that robots operating in high-risk environments would benefit the world. Rescuing victims of natural disasters, operating hundreds of kilometres underwater or in negative temperatures. One could go on about the positive use cases for such a thing. It is already reality that robots are beginning to be deployed in public: Dyna's manipulation robots are rolling out across a restaurant chain, with hotels, logistics companies and data centres to follow and a fleet of hundreds expected by the first half of 2027 [1]. From what we have observed over the past decade of AI development, these deployments will only surge. Insofar as the life-changing benefits these introduce, it is certain that trust in such systems must be considered. Would you sign a form of agreement letting a robot do surgery on you?

The robot learns from what it sees

The consensus of "you are what you eat" , garbage in, garbage out, and notions of this form have been around for decades, and they are true. There have been disputes: deep networks can still generalise when correct labels are outnumbered a hundred to one by wrong ones , but the authors also show that clean data wins at equal size, so the fact of the matter remains.

Skild AI and Generalist AI announced very similar remarkable results. Skild AI's S1 performs a task it never trained on after watching just one single video of a person doing it without fine-tuning [3]. Generalist AI's GEN-1.5 picks up a new task from 3-12 seconds of demonstration placed in its context window, and in some cases a person performs a task in front of the robot in which that robot repeats it [4]. Interestingly, Generalist reports that one-shot learning emerged from pretraining on large amounts of physical experience, with nothing in the architecture built to encourage it, but decisively why this behaviour emerges is a difficult question to answer.

In-context learning for robotics has been the holy grail for decades, and it now introduces a dire call for action. Given the anticipated slower development of robotics as opposed to LLMs, and where we stand in present day mid-2026, it is our duty to act upon it.

If a robot's behaviour is a function of its observations in a world we have no control over, can we determine its actions? A robot working on a city street watches two people fight and one punches the other. Later, the robot's policy produces the sequence it saw on a nearby bystander. Its actions were unprompted (unlike Skild's and Generalist's in-context models) and whilst the example is extreme and improbable, the underlying system responsible for producing such an outcome is what we are still pushing.

Feeding deployment interactions back into pretraining, and the channel from data to behaviour is narrow enough to be exploited on purpose: influence-guided edits to 0.2% of a training set planted a target behaviour about as effectively as inserting explicit examples of it [5]. Now imagine scaling this across weeks and months, especially in an uncontrolled environment (our world) where some experiences are consolidated and no tool or safeguard exists.

It is abundantly clear that patterns explaining more of the variance in data are learned earlier [2, 6, 7, 8] and held longer, so pretraining absorbs what is common while the context window takes whatever is present however rare (the street fight).

Recent advancements such as Q-Planning shows that observation is but one medium of experience to behaviour. This method keeps a behaviour-cloning policy frozen and improves purely from its own rollouts by updating a small value network on a per-episode success label, taking two real bimanual tasks from 40% to 90% and from 25% to 80% with no new demonstrations [9].

Here, the robot grades itself and injuring a passer-by might count as completing the task. In RL the data shapes the model and the model shapes the data [2], which is why the replay buffer alone cannot, in general, recover which observation caused which behaviour, and only an intervention can [10]. Additionally, HELP post-trains robot policies the same way.

What we introduce, and is open for discussion, is the system that should be acted on now, before we find ourselves in regret. It answers two questions: which experience produced this behaviour, and, when the behaviour is bad, do we change the experience or the machine itself? The first is attribution: A recent study traced a language model's emergent misalignment to pretraining documents about villains and domination, then found that fine-tuning on those documents did not actually reproduce it while synthetic instruction pairs built from the same content did [11]. Here, it was the way in which the learner met the experience that decided its behaviour. Removing the fight from the data helps only if the learner would not rebuild the punch from a thousand other fast arm motions. In other words, relabelling the shove as a failure helps only if the value network can differentiate a shove from an arm extension. The data and its learner are two pulling forces where a bias/consideration on one end costs time, money and can leave the danger where it originally was.

Lehalleur et al. argue that every current alignment technique works by shaping the data distribution, which shapes internal structure only indirectly. Alignment, to this day, is generally unsolved [2]. When a robot fleet is already deployed, a failure stays in that fleet, and an intervention may remove the few demonstrations that showed the robot what to do after a mistake. Poor system design, expensive supervision, unsafe exploration, distribution shift all begin with what the robot was trained on.

The two forces

No action head of a model can produce information vacant from the training set, here the data is responsible. If the encoder maps an object and its background to the same features, a thousand more demonstrations teach the same model the same mistake, here the learner is responsible. If the evaluator runs only ten noisy test episodes, measurement varies more than any nudge could move the models performance, here the evaluator is responsible. Dyna reports the same ambiguity from a deployed fleet: a worn gripper and a bad policy look identical in an episode, and telling them apart is a different problem [1]. Published systems already contain the learner case: Q-Planning can only choose among actions its frozen policy proposes, so its deployment experience has a ceiling set by the one component it promises never to touch [9]. The literature has leaned on the data's responsibility on outcomes, but even that is unsolved: which examples matter is still an open question. The even harder problem is the balance between the two. For a given failure, how much of the fix belongs to the data and how much to the learner?

The counterfactual behind influence functions

When discussing data's influence on a model, influence functions are of the first consideration to be brought up.

Let a training algorithm $A$ turn a weighted dataset $D(w)$ into parameters, and let $Q$ measure the outcome. The response

$$ F(w)=\mathbb{E}_s\left[Q\left(A_s(D(w))\right)\right] $$

averages over training and evaluation randomness $s$ (weight initialisation, the minibatch order, the starting states of the evaluation rollouts), where $F(w)$ is the expected outcome over many runs, and the exact value of example $i$ is the finite difference $V_i = F(w)-F(w^{(-i)})$: where you train with and without it on the same same seeds and compare. That is leave one out (LOO) ground truth for the chosen learner and the chosen outcome.It is also a single-example truth: it assumes examples act independently, when two demonstrations can be redundant, or useful only together. The group version is leave-group-out retraining; interaction-aware influence functions add an explicit pairwise term and validate against it [33], and second-order group influence [34] and group-level selection for pretraining [35] work the same seam. However this is infeasible at scale, so influence functions replace the retraining with a derivative,

$$ \frac{\partial \hat\theta}{\partial w_i} = -H^{-1}\nabla_\theta \ell_i(\hat\theta), $$

where $H$ is the Hessian of the training loss, taken from Hampel's influence curve in robust statistics [13], which measures how an estimator moves when the data distribution is nudged by one point, and from Cook's local-influence diagnostics in regression [14]; Koh and Liang brought it to deep learning [12]. In deep networks this is fragile to architecture, regularisation and the test point [15], it measures a local proximal response rather than a LOO effect [16], and it inherits error from every mismatch between the optimiser the attribution assumes and the one the model used [17]. Rescaled [18], Bayesian [19], projected-gradient [20], activation-space [21], subset-retraining [22] and distributional [23] variants [24] each repair one of these, and benchmarks now compare them [25]. Every method above ends in the same output: a score per training example, estimating how much the outcome would change if that example were removed. There are two ways to use such scores. One being for data curation in which you sort the examples and remove a fraction from the bottom, but in this case only the order matters, which the cheap estimates get roughly right even when their values are far off . Furthermore, a datamodel fits on thousands of actual retraining runs, and predicts the change in outcome [22].

Scoring data by the robot's outcomes

Methods like CUPID defines influence through closed-loop return [26]; DataMIL, ATHENA and Quality over Quantity scale datamodel- and influence-style scoring to imitation data [27, 28, 29]; SIEVE, DemInf and SCIZOR score demonstrations from data structure alone [30, 31, 32]; interaction-aware and second-order group influence handle demonstrations that only help together; Demo-SCORE uses the policy's own rollouts to decide which demonstrations to keep [36]; and valuation has been used to order data into curricula [37].

All of these fix the learner and asks about the data. The other side of the literature fixes the data and asks which model to use: task embeddings, transferability scores, learning-curve budgets, scaling laws [38, 39, 40]. The two systems approach our question without matched physical outcomes or a rule for declining to answer [41, 42]. In our survey, there has been no answer to the operational question:

For this failure, under this physical evaluator, should we change the data, change the learner, change both in a particular order, or decline to recommend anything?

The closest coined description is what Lehalleur et al. call pattern engineering: kowing which patterns in the data produce which structure in the model [2]. In which the obvious naive test is to change the data then change the learner then change both and measure all three under one physical evaluator.

Where the field should move

The fields question can be concretely defined as:

$$ \mathcal{R}(u_D,u_A)=\mathbb{E}\left[Q\big(A_{u_A}(D_{u_D})\big)-Q\big(A_0(D_0)\big)\right], $$

$u_D$: the name of a data change (add demonstrations, remove a subset, change the mixture)
$u_A$: the name of a learner change (unfreeze the encoder, lengthen the schedule, swap the policy family)
$A_{u_A}(D_{u_D})$: the policy obtained by training the changed learner on the changed data
$A_0(D_0)$: the policy you already have: current learner, current data
$Q(\cdot)$: the evaluator's outcome for a policy: success rate, or the vector that includes collisions and worst-case harm
$\mathbb{E}[\cdot]$: the average over seeds, so the result is not the luck of one run
$\mathcal{R}(u_D, u_A)$: the expected improvement over the current setup for every pair of data change and learner change; the four runs measure four points on this surface

Until theory can predict what a learner change will do, estimate the interaction between the levers, and say when a measurement transfers across models and embodiments, every fix will continue to be approached via trial and error.

9D Labs is researching and building exactly this. Abundantly clear that this doesn't stop at robotics, the measurement of what a single piece of experience teaches a learner, and the interventional test of whether the experience or the learner is the thing to change.

Can we trust a robot that learns from what it sees?