// IRIS Desktop / Architecture
IRIS treats AI authority as a security problem first. Below the glowing orb sits a deterministic governance stack where native code holds every ounce of power — and everything else, including the models, can only ask.
The core stack
The part you see and speak with: the Three.js particle orb, voice capture, the annotation overlay, Canvas window, macro engine, pattern detector, and the 18-page Control Center. Sixteen components and eleven hooks wired to a typed tool registry.
Sixty-five-plus native commands for window management, audio capture via cpal, workspace snapshots, system control — plus the entire governance plane: the Trusted Execution Host, durable job supervision, credential brokering, bounded perception, and GUI control sessions. Renderers are sandboxed under a strict production CSP.
Reasoning services with backend-native memory: Lethe stores your personal continuity, Mnemosyne holds shared cross-agent knowledge. Speech-to-text via Whisper, low-latency streaming TTS over Socket.IO, and dynamic tool loading that refreshes at runtime.
The governance principle
"Callers may request work. Only native code grants and exercises authority."// IRIS NEXT P0 AUTHORITY BOUNDARY // DELEGATION CAN NARROW AUTHORITY — IT CAN NEVER EXPAND IT //
Renderers, models, agents, adapters, or remote peers cannot create authority by supplying identity strings, approvals, or session IDs. Every identifier is created or validated in Rust. Delegation preserves-or-reduces capability, never grows it. This is what lets IRIS be powerful on your desktop without being a liability on it.
Every capability is exercised by native code under per-capability policy: allowlisted low-risk reads, single-use expiring approvals for opening URLs, path-canonicalized file reads with output bounds, time-boxed clipboard grants, and five-minute revocable input-control sessions bound to a specific window, process, and action class. Shell is quarantined outright.
A deterministic control plane — not an LLM — owns long-running local work. Jobs persist in WAL-journaled SQLite with append-only event history, worker leases, immutable authority profiles, and crash reconciliation that classifies ambiguous steps instead of blindly replaying side effects.
Secrets live in Windows Credential Manager under opaque locators — the durable database stores metadata only, never keys. Provider execution is pinned to exact endpoints with redirects disabled, atomic use-counting, and hash-only response receipts. Possessing a credential does not grant provider authority; provider authority does not supply a credential.
Job-scoped capture of windows, displays, or regions — with identity proven down to the process creation epoch. Raw pixels never touch disk or logs: they exist briefly in native memory, then die. Remote frame analysis requires both an active session and a job-bound credential grant, and visible screen text is always treated as untrusted data.
When IRIS must drive a desktop app durably, it does so through native Win32 targeting identified by executable path, PID, and process start time — defeating PID recycling. Every single action passes ten fail-closed revalidation checks, including live OS focus verification of the exact target.
A native fact-ingestion spine turns desktop events into governed work: events carry metadata only, triggers are typed matchers rather than scripts, and occurrence identity is hashed for exactly-once materialization into supervised jobs. Backpressure bounds everything.
The OBSERVE → REASON → PROPOSE → VALIDATE → ACT → VERIFY loop runs toward a user-approved goal inside an immutable work envelope. Hard budgets cap steps, observations, model calls, typing characters, and wall-clock time. The model can request help — it can never mint authority.
Voice pipeline
A full-duplex voice state machine — idle, listening, hearing, processing, speaking — engineered so IRIS never talks over you, and never hears herself.
"Hey IRIS" · enrolled owners skip it
Echo-cancelled mic or native cpal fallback
Speaker confidence & owner detection
Memory context + tool selection
Chunked, gapless, orb-synced
Post-speech cooldown keeps her from triggering on her own voice. If the WebSocket stream drops, playback falls back to HTTP TTS automatically. Audio levels drive the orb in real time.
Memory architecture
Desktop IRIS is wired directly into the backend memory stack — personal context leads, retrieved knowledge follows:
User-scoped memory built from your day-to-day interactions with desktop IRIS. What you did, how you phrased it, what mattered — carried forward so tomorrow starts with context.
Reusable, cross-agent knowledge. Summaries worth keeping beyond one user's flow get written here and become available across the wider agent ecosystem.
Pattern detection adds a temporal dimension: activity logging plus a deterministic sequence detector surfaces recurring workflows — three or more days of the same dance — and offers to turn them into one-word macros.
Perception systems
MediaPipe-powered vision runs on-device at ~12 FPS with debounced triggers — presence detection, attention tracking, and gesture recognition without shipping your camera feed anywhere unless you ask for analysis.
Face Detection
BlazeFace short-range — bounding box, confidence, attention direction
Hand Tracking
21-point skeleton → 14+ gestures at 3-frame debounce
Capture
1280×720 webcam frames, base64 to vision analysis on demand
Screen Grid
10×10 calibration grid powering annotations and click targeting
Tech stack
Desktop Runtime
Tauri v2 (Rust) + React 19 + TypeScript + Vite
3D Avatar
Three.js particle sphere with live audio visualization
Audio Engine
cpal native capture, RMS-level VAD, streaming TTS scheduling
Vision
MediaPipe Tasks — BlazeFace, HandLandmarker
Durable State
SQLite with WAL journaling, leases, append-only events
Speech AI
OpenAI Whisper STT · ElevenLabs / OpenAI TTS
Security
Windows Credential Manager, CSP sandboxing, DOMPurify
Testing
166 Rust unit tests + TS contract/gate suites, all green
Where it's going
The published roadmap pushes IRIS toward reliable multi-app operation: a unified observation layer, plan-act-verify execution with evidence-backed results, versioned skills, deep app integrations, and enterprise-ready administration — benchmarked against real task suites at every milestone. It is deliberately labeled vision, not shipped feature: the same honesty standard we hold the guardrails to.