A mobile app that watches an exercise through the camera, runs pose detection on-device, and validates form against motion benchmarks while the user is still moving.
Where it started
Form correction is only useful while someone is mid-repetition. A round trip to a server is too slow to be coaching, and streaming continuous camera footage off the device is both a bandwidth problem and a privacy one. Inference had to happen locally, on ordinary consumer phones.
On-device pose detection
Joint positions are detected from the live camera feed on the phone itself, so no video leaves the device.
Form validation
Detected joints are compared against motion benchmarks per exercise to judge whether a repetition was performed correctly.
Feedback during the rep
Correction arrives while the movement is happening, which is the only point at which it is useful.
Session tracking
Workouts, sets and history are recorded so progress is visible over time.
Analytics
Aggregate views over past sessions, served by the backend rather than computed on the handset.
Mid-range hardware
The capture and inference path is tuned for ordinary phones, not only flagships.
- Pose detection runs locally through MediaPipe against a React Native Vision Camera feed; inference never makes a network round trip.
- The device is responsible only for inference and capture; history and analytics live server-side.
- NestJS service for accounts, workout sessions and analytics APIs.
- React Native with TypeScript, with the camera and inference path treated as the performance-critical surface.
- Privacy follows from the architecture: footage is processed and discarded on the handset.
- 01
Ran pose detection on-device against the live camera feed, so no video leaves the phone and feedback is not gated on a network round trip.
- 02
Compared detected joint positions against motion benchmarks per exercise to judge whether a repetition was performed correctly.
- 03
Tuned the capture and inference path for sustained use on mid-range hardware rather than only on flagship devices.
- 04
Backed the app with a service handling accounts, workout sessions and progress analytics, keeping the device responsible only for inference.
- React Native
- TypeScript
- MediaPipe
- Vision Camera
- NestJS
- Node.js
Where it landed
Users get correction during the movement rather than a summary afterwards, and their camera footage never leaves the handset. The backend holds history and progress while the phone does the real-time work.
A short conversation with an engineer, not a sales qualification call. If we're the wrong people for it, we'll say so and point you somewhere better.