Personal project · December 2025 – present
A robot that listens, talks, and comes when called — with nothing leaving the room.
Speech recognition, a language model, text-to-speech, and person-tracking vision, all running on one Jetson Orin Nano bolted to a mecanum-wheeled chassis. No cloud, no internet connection, no ROS. Every decision was made against a hard budget of 8GB of memory shared between the CPU and GPU.
The goal
Build a robot my kids can talk to that works the way a device in the home should: it answers in about a second, it doesn’t send audio or video anywhere, and it keeps working when the Wi-Fi doesn’t. That rules out every hosted model. What’s left is the interesting engineering problem — fitting a full conversational and vision stack onto an edge board with the memory of a mid-range phone, and keeping it safe enough to drive around a living room.
What it does
- Listens and talks. Voice activity detection gates a local speech-to-text model, a local LLM answers with the last several turns of context, and a local neural voice speaks the reply. Conversation history expires after a few idle minutes so the next person doesn’t inherit someone else’s chat.
- Keeps people in view. A CSI camera on a pan-tilt mount tracks whole-person boxes from YOLO11n at 30 frames per second, so it holds a lock on someone in profile or walking away. When the servo runs out of travel, the base pivots to give it room.
- Comes when called. “Come here” turns the camera bearing into a chassis heading, drives to a standoff distance using sonar and box-height range estimates that hand off where the other fails, and then stops and says so. An approach that can’t advance ends out loud rather than stalling silently.
- Never relies on the brain for safety. Collision reflexes live in Arduino firmware with a serial watchdog: an obstacle inside 22cm halts forward motion no matter what the Jetson commands, and a dropped link stops the wheels.
Architecture
Four isolation domains, chosen so any one can crash and restart without taking the others down, and so each can be memory-capped on its own. A Python orchestrator coordinates subsystems over an internal message bus; inference runs in separate services over HTTP; vision is its own process; and the firmware owns the motors.
┌─ Jetson Orin Nano (8GB unified) ─────────────────────────────┐
│ orchestrator (Python, threads + MessageBus) │
│ audio_input / transcription / conversation / tts │
│ movement / approach / vision fusion │
│ │ HTTP │ HTTP │ stdin/stdout │
│ ┌────▼─────┐ ┌──────────▼──┐ ┌───────▼────────┐ │
│ │ llama.cpp│ │ faster- │ │ camera_tracker │ │
│ │ gemma 4 │ │ whisper │ │ (own venv, │ │
│ │ e2b GGUF │ │ (container) │ │ CSI + PCA9685)│ │
│ └──────────┘ └─────────────┘ └────────────────┘ │
│ │ /dev/arduino @115200 (JSON lines, watchdog) │
└─────────┼─────────────────────────────────────────────────────┘
┌────▼──────────────────────────────┐
│ Arduino Mega 2560 (firmware) │
│ motors, ultrasonic, IR, tracker, │
│ headlights + REFLEX SAFETY STOPS │
└───────────────────────────────────┘- Language model
- Gemma 4 E2B, quantized, served by llama.cpp in an NVIDIA Jetson container. Around 25 tokens per second with an 8k context window. Migrated off Ollama after measuring its serving layer added ~800ms to every first token on identical weights.
- Speech
- Silero VAD → faster-whisper → LLM → Piper text-to-speech. Kept decoupled on purpose: the model’s native audio input was evaluated and rejected because it cost a gigabyte of memory and removed the ability to swap or tune any single stage.
- Vision
- A fast reflex tier (YOLO11n compiled to a TensorRT engine, plus appearance re-identification) and a slow semantic tier (image prompts to the same LLM via its vision projector, enabled only when the memory budget allows).
- Messaging
- A small in-process message bus instead of ROS 2. At this scale DDS middleware and node overhead bought nothing three HTTP services didn’t already provide.
- Firmware
- Arduino Mega 2560 speaking JSON lines over USB serial. Owns motor ramping, ultrasonic and IR sensing, and the reflex stops. The IR veto is switchable by voice because the sensors saturate in sunlight; the ultrasonic veto is not, and a reset always restores both.
Fitting it in 8GB
On a unified-memory board every model competes with the OS, the camera buffers, and each other. The budget was measured rather than estimated: process by process, with the full stack running, under peak load. Some of what that turned up:
- Only the smallest Gemma 4 variant fits without swap; vLLM was rejected because it preallocates GPU memory and would starve the speech and vision services.
- Context length is nearly free thanks to sliding-window attention — going from 2k to 8k tokens cost about 40MB — so multi-turn history was cheap to add.
- A CUDA context on Tegra costs ~120MB, not the 500–800MB folklore suggests; the real cost of an extra vision process is the duplicated runtime libraries, so detection and re-identification share one process.
- Enabling image input in the LLM costs almost a full gigabyte and drags in an unused audio encoder, so it stays off by default and is a one-line switch when needed.
- Each inference container runs with an explicit memory cap so a leak kills one service, not the whole robot.
Hardware
- Compute
- NVIDIA Jetson Orin Nano 8GB developer kit, 25W mode, JetPack 6.2, NVMe boot
- Chassis
- OSOYOO mecanum-wheel car kit: four encoder motors, motor driver shield, Arduino Mega 2560
- Camera
- Arducam IMX219 8MP CSI camera on an I²C pan-tilt platform (PCA9685 servo driver)
- Sensing
- Ultrasonic rangefinder and forward IR obstacle sensors on the firmware side
- Audio
- USB omnidirectional conference microphone and a USB soundbar
- Power
- 12V LiFePO₄ battery feeding the Jetson directly and a buck converter for the servos; motors on their own 18650 pack
- Build
- Three stacked levels on brass standoffs and acrylic platforms; roughly $250 in parts
How it’s built
The codebase treats its documentation as load-bearing. A hardware source-of-truth file is backed by a read-only self-check script that verifies the mic, speaker, camera, servos, and Arduino against the configuration, so config drift fails loudly. Every behaviour rule — why the approach never reverses, why tracking never speaks, why a turn is one continuous motion rather than pulsed — is written down alongside the log session that motivated it. Architecture decisions carry their measurements, so the next change starts from numbers instead of guesses.