Three Streams, One Clock

Our glove captures touch, anatomical hand tracking and vision as one synchronised record. Not three recordings stitched together afterwards. One record, aligned in hardware at the moment of capture.

That matters because touch on its own, however clean, is an incomplete record of a manipulation. Knowing that a finger pressed something with a particular force at a particular instant does not tell you what was pressed, what state it was in, or what the hand was doing to arrive there.

Why one sensor was never enough

Take one manipulation, picking up a part that has shifted slightly out of position, and ask what each stream can tell you alone.

Touch tells you that contact happened, where on the surface, and how hard. It does not tell you what the hand was doing to cause it. A deliberate press and a hand decelerating into a surface can leave a similar mark.

Anatomical hand tracking tells you how the hand moved to create that contact: the approach, the rotation of the wrist, the deceleration just before the grip closed. It does not tell you what was there to be touched.

Vision tells you what was being acted on and in what state, which object, which face, whether it had already moved. It does not reliably tell you when contact began or how hard, and it loses the moment altogether once the hand covers it.

Each stream is ambiguous alone. The information is not in any one of them. It is in their agreement.

Captured on one clock

The part that matters is not that the three streams end up in the same file. Anyone can put three recordings in one folder. It is that they are aligned when they are captured, in hardware, rather than stitched together afterwards in software.

Consider what stitching afterwards involves. Three devices sample on three clocks. Each timestamps against its own idea of now. To combine them you pick a reference, resample the others onto it, and accept the residual error. The error is small. It is also concentrated in exactly the wrong place.

The moment that carries the information in a manipulation task is the moment of contact. It is short. It is where the pressure edge, the deceleration and the visual state of the object all have to agree in order to mean anything. Every millisecond of misalignment smears that agreement. Nothing looks obviously broken afterwards. The one instant you were trying to capture is simply no longer sharp.

Aligned at capture, that problem does not exist to be solved. The three streams share a clock, so the pressure edge, the motion that produced it and the frame showing what it landed on all sit at the same index. Alignment becomes a property of the capture hardware rather than a post-processing step, so it does not vary from one run to the next and does not degrade as a session gets longer.

This is the difference between data that was collected and data that can be trained on. A policy learning dexterous manipulation has to learn the relationship between what the hand did, what it touched and what happened next. If those three are offset by an unknown and varying amount, part of what it learns is the offset.

What a single record makes possible

An aligned record is not simply a tidier one. It changes what you can do with a session after it has been recorded.

1. Segmentation stops being manual. Contact onset is a reliable event boundary when the tactile edge is sharp and the other two streams agree with it. A long session divides itself into individual attempts.
2. The streams supervise each other. Vision identifies what was handled and in what state. Touch says exactly when it was touched and how hard. Hand tracking says how the hand arrived. Each narrows the ambiguity in the other two, but only if all three refer to the same instant.
3. Failures become legible. When a grasp slips, the useful question is what the hand was doing in the tens of milliseconds before it went wrong. That window only exists if all three streams are sharp at the same moment.

None of that needs a larger model or a cleverer loss function. It needs the record to be right at the point of capture.

The stack is already collecting

The SoftShell sensor stack solves this, and it is working today. We are successfully capturing touch, anatomical hand tracking and vision as one fused, fully aligned stream at a dedicated station at a partner site, on real tasks at full duty cycle.

The vision element was upgraded to a higher frame rate and resolution, so the visual half of the record is now as sharp as the tactile half. Recent prototype runs hit their targets. Logging and the data pipeline run end to end, so every session lands ready to train on instead of needing to be reconstructed.

That success is what we are scaling now. More stations, more hands, more hours, and the same three streams arriving aligned every time. A tactile material we trust unsupervised and a stack that timestamps itself were both preconditions, and both are in hand.

Vision and language models for robotics scaled because images and text were already lying around in quantity. Nothing equivalent exists for contact. So we are building it, and we are collecting it today.

If you are building humanoids, prosthetics, teleoperation rigs, or any system that learns contact-rich manipulation from real-world demonstration, we would love to hear from you.