How the FamilyHub AI Assistant Works
How the FamilyHub AI Assistant Works
The AI assistant is the fastest way to interact with FamilyHub. Tap the microphone, say what you need in Dutch, and FamilyHub handles the rest — adding groceries, completing tasks, creating calendar events — all without touching your phone.
Here's how it actually works.
The Flow: Voice → Text → Intent → Action
Tap mic → speak → audio sent to backend
→ Whisper transcribes Dutch audio to text
→ Claude parses text into a structured Intent
→ IntentExecutor routes the Intent through the same use cases as REST
→ Real-time WebSocket push → all devices update instantly
Every step happens in under 5 seconds from the moment you release the mic button.
Step 1: Speech to Text
Audio is recorded in the browser using the native MediaRecorder API (webm/opus format) and sent to the backend as a multipart upload. The backend forwards it to a faster-whisper server (self-hosted) or the OpenAI Whisper API, running the large-v3 model specifically chosen for its strong Dutch accuracy. Silence is auto-detected after ~1.5 seconds.
Step 2: Natural Language → Intent
The transcribed text — along with the current state of your shopping lists, tasks, and calendar — is sent to Claude (Anthropic's API) or an OpenAI-compatible model. The AI responds with a structured JSON intent, not free text. This is key: the AI never directly modifies any data. It only decides what should happen.
The AI provider is configurable per family through Admin > Services.
Step 3: Intent Execution
The structured intent is handed off to the IntentExecutor, which routes it through the same application use cases that the REST API uses. Whether you tap a button or speak a command, the same code runs. Over 20 intent types are supported, covering shopping, tasks, calendar, and meta actions like undo.
Supported Commands
| Example command | What happens |
|---|---|
| "voeg melk, eieren en brood toe" | Adds 3 items to the Boodschappen list, auto-categorized |
| "voeg Billy kast toe aan de IKEA lijst" | Adds to the IKEA list specifically |
| "heb ik al boter op de lijst?" | Checks and responds yes/no |
| "kruis boter af op de lijst" | Checks off the item |
| "zet melk terug" | Unchecks an item |
| "planten zijn water gegeven" | Marks today's task instance as complete |
| "maak een taak aan om de goten schoon te maken volgende week" | Creates a one-off task with a due date |
| "snooz de vuilnis taak naar morgen" | Snoozes the task instance to tomorrow |
| "voeg tandarts toe volgende dinsdag 10:00" | Creates a calendar event |
| "verplaats de tandarts naar woensdag" | Updates an event |
| "wanneer is Louen's volgende voetbaltraining?" | Looks up and responds with date/time |
| "voeg een IKEA lijst toe" | Creates a new shopping list |
| "maak dat ongedaan" | Undoes the last AI action |
If the AI doesn't understand a command, it responds: "Ik begreep dat niet, kun je het anders formuleren?" — and nothing is modified.
Context-Aware Responses
Before every AI call, FamilyHub builds a snapshot of your current data: active shopping lists with their items, upcoming tasks, and calendar events for the next two weeks. The AI receives this context in every request, which is why it can answer questions like "heb ik al boter op de lijst?" without you specifying which list.
Text Mode
Prefer to type? The same AI bar accepts text input. Type a command in Dutch, hit enter, and the same pipeline runs — skipping only the speech-to-text step.
Privacy Note
Your text and audio are sent to the AI provider you configure (Anthropic or OpenAI) for intent parsing. The current state of your lists and tasks is included as context. No audio is stored by FamilyHub after transcription. If you self-host faster-whisper for speech-to-text, audio never leaves your network.