InStep logo: a footprint that is also a pause bar, beside a play button
Fire TV · Vega OS

The video that waits for you.

How-to videos on Fire TV that pause at each key moment and continue only when your pose, hand shape, rhythm or hold is right — using the phone or laptop you already own as the camera.

▶ Watch the film Try the phone camera Source on GitHub
0.0 KB/s of video sent — measured 5 channels · 7 lessons 241 unit + 6 e2e tests Open source · MIT

Demo film, 4:04 — TV shots recorded on the Vega Virtual Device; learner simulated where labelled; narration by a local synthetic voice.

The problem

Every how-to video plays at the same speed — whether you keep up or not.

For people doing prescribed home exercises, parents learning to sign, or anyone practising CPR, a video that can't see you means practising it wrong, or giving up.

684,000

people die from falls each year — the second leading cause of unintentional injury death worldwide.

WHO · Falls fact sheet
76%

of Americans surveyed did not complete all of their assigned at-home physical therapy exercises.

Ohio State Wexner survey · Medical Economics
90%

of deaf children are born to hearing parents — who have to learn to sign.

Mitchell & Karchmer (2004)
73%

of out-of-hospital cardiac arrests happen in homes. Immediate CPR can double or triple survival.

American Heart Association
How it works

Your phone sees. Your TV waits.

No new hardware: the TV you have, and a phone or laptop propped up as an external camera. Pose tracking runs in the phone's browser; only body points travel to the TV.

1

Pick a lesson on the TV

Five channels, built for the 10-foot screen with the Fire TV remote. Each profile keeps its progress.

2

Pair your phone in seconds

Scan the QR code or type the 4-digit room code. The phone becomes the camera — frames stay on it.

3

The lesson waits for you

At each key moment the video pauses and continues only when you've got it right — counting reps, timing holds, and giving one short, kind tip when you're stuck.

TV pairing screen with a QR code and room code 0393
Pairing on the TV — recorded on the Vega Virtual Device.
Heel raises lesson on the TV: animated teacher, raise 5 of 5, live skeleton beside it and the step rail below
Heel raises: the teacher video, your live skeleton, and the steps you've passed.
Lessons

One engine. Five ways to practise.

Every GIF below is a real screen recording of the TV app (played 3× faster) with a simulated learner passing each step. Each teacher video says on screen what it is.

Heel raises on the TV: chair safety check, posture, five reps, final hold, summary
Fall Prevention · hero

Heel raises (with chair)

No balance work until a chair is beside you. Posture check, five counted heel raises, a held rise onto your toes, and an automatic sway stop.

Single-leg stand: chair check, then a 5 second hold on each foot
Fall Prevention

Single-leg stand (with chair)

A five-second hold on each foot with a countdown ring, right then left.

Fingerspelling A B C L Y with finger-level feedback, then the word C A B
Sign Language for Families · hero

Fingerspelling: A B C L Y

Hand shapes checked finger by finger with Delta Glow — no sound needed — then spell a word.

Illustrated handshapes, not yet reviewed by Deaf signers. Learn with Deaf teachers and community too.

The I-love-you handshape matched on the TV
Sign Language for Families

Your first sign: I love you

Thumb, index and little finger up — matched by finger curls.

Hand rehab: open and close your hand, rep counter rising to 20
Stroke Hand Rehab

Open and close your hand — 20 reps

Only full-range reps count, and the video slows to the learner's pace.

Use exercises prescribed or approved by your physiotherapist.

CPR rhythm: compressions counted with a BPM meter and an arms-straight check
CPR Practice

Hands-only CPR rhythm

Compressions at 100–120 a minute with straight arms, following a metronome.

Practice aid for rhythm and straight arms only. It does not measure compression depth and is not certification. Take a certified CPR course.

The Forest Walk: a child stretches like a tree, crouches like a mouse, balances like a flamingo
Kids' Active Storytime

The Forest Walk

Act out a story to move it forward — tree, mouse, flamingo and wave, with a star burst after each.

Frames from all seven narrated lesson videos
Teacher videos

Illustrated, narrated, honest

Flat illustrations drawn in code and timed to every gate, a real CPR clip (Pexels, credited), and a local synthetic voice that never talks over a pause. Together mode lets two people practise at once.

The phone

A camera, a coach, and a creator studio.

The companion is a web app — nothing to install. It runs MediaPipe in the browser, speaks tips, and lets a physiotherapist or teacher record a lesson once and publish it to every TV.

Join by room code
Join with the room code
Live camera with step feedback
Live feedback and hands-free control
Creator Studio lesson list
Creator Studio
Creator Studio step editor with suggested steps
Steps suggested from still moments
Privacy, measured

Only body points leave the phone. Never video.

  • Frames stay in the browser. Pose and hand landmarks are computed on the phone; only normalised numbers are sent.
  • The privacy pill counts every byte the page actually sends — video is measured at 0, not hard-coded.
  • Strict content security policy blocks third-party calls (including the vision SDK's telemetry), checked by an end-to-end test on every push.
  • Only the instructor's demo is ever recorded, after an explicit "Record demonstration" action with a visible indicator.
Video sent
0.0 KB/s

Measured on the phone, live, for every lesson.

Phone sheet: what leaves this phone, video 0.0 KB/s
Architecture

One gate engine — on the phone, the server and the TV.

A pure TypeScript engine and an open lesson format, .istep, shared by every part of the system. The TV pauses its W3C media player at each gate and resumes when the engine says you passed.

System diagram: learner's phone, InStep server on Render, Fire TV app, Creator Studio, shared packages, content, and AWS switched off
Full walk-through with file paths in docs/02-architecture.md.

.istep — captions for movement

Like a WebVTT file for a video, but each cue is a gate: what must be true before the video continues. Seven gate types: pose, hold, reps, rhythm, hand shape, sequence and safety checks. 

{
  "id": "raises", "t": 22,
  "type": "rep_count",
  "label": "Heel raises — five times",
  "metric": { "kind": "heel_lift" },
  "reps": 5, "rewindTo": 12,
  "then": "loop_segment"
}
Architecture poster in the Blue Hour brand with a one-gate step-by-step panel
Built with

Built for Vega OS, end to end.

Fire TV app

  • React Native for Vega (Vega SDK 0.24)
  • W3C media player, react-tv-space-navigation
  • react-native-svg skeleton overlays, tvOS-grade 10-foot UI

Phone & server

  • React + MediaPipe Tasks in the browser
  • Node 22, Fastify, WebSocket relay, SQLite — hosted on Render
  • Coach on Amazon Bedrock + Polly (built; switched off on the demo)

Open source

  • vega-gated-video-player — the format, engine and player for any Vega app
  • zod-validated protocol, JSON Schema
  • Vitest, Playwright, GitHub Actions
Honest by design

What InStep is — and isn't.

The camera is external. Your phone or laptop is the camera; the TV has no camera of its own.

A practice aid, not treatment. InStep has not been clinically tested and never diagnoses.

CPR is rhythm only. It does not measure compression depth and is not certification.

Sign language with Deaf teachers. The illustrated handshapes have not yet been reviewed by Deaf signers.

The coach is templated on the demo. Bedrock and Polly are integrated and tested, but switched off — no AWS credits are spent.

Every number is sourced. Statistics come only from the cited sources above.

Brand

Blue Hour.

Night-indigo for the living room, sky for the brand, sun for warmth — flat illustration without outlines.

InStep brand sheet: logo, colours, channels, type, controls and voice