// IRIS Desktop / Architecture

An operating layer,
engineered like one.

IRIS treats AI authority as a security problem first. Below the glowing orb sits a deterministic governance stack where native code holds every ounce of power — and everything else, including the models, can only ask.

The core stack

Three layers, top to bottom.

React Frontend — Presence Layer

React 19 · TypeScript · Three.js

The part you see and speak with: the Three.js particle orb, voice capture, the annotation overlay, Canvas window, macro engine, pattern detector, and the 18-page Control Center. Sixteen components and eleven hooks wired to a typed tool registry.

Three.js orb Framer Motion annotations MediaPipe vision Macro engine Sequence detector

Rust Core — Tauri Native Host

Tauri v2 · Rust · SQLite

Sixty-five-plus native commands for window management, audio capture via cpal, workspace snapshots, system control — plus the entire governance plane: the Trusted Execution Host, durable job supervision, credential brokering, bounded perception, and GUI control sessions. Renderers are sandboxed under a strict production CSP.

Trusted Execution Host Agent Supervisor Credential Broker Event Bus Native audio (cpal)

Intelligence Backend — Memory & Senses

Python · Whisper · Streaming TTS

Reasoning services with backend-native memory: Lethe stores your personal continuity, Mnemosyne holds shared cross-agent knowledge. Speech-to-text via Whisper, low-latency streaming TTS over Socket.IO, and dynamic tool loading that refreshes at runtime.

Lethe memory Mnemosyne memory Whisper STT Streaming TTS Dynamic tools API

The governance principle

One invariant rules them all.

"Callers may request work. Only native code grants and exercises authority."
// IRIS NEXT P0 AUTHORITY BOUNDARY // DELEGATION CAN NARROW AUTHORITY — IT CAN NEVER EXPAND IT //

Renderers, models, agents, adapters, or remote peers cannot create authority by supplying identity strings, approvals, or session IDs. Every identifier is created or validated in Rust. Delegation preserves-or-reduces capability, never grows it. This is what lets IRIS be powerful on your desktop without being a liability on it.

GOV / 01

Trusted Execution Host

Every capability is exercised by native code under per-capability policy: allowlisted low-risk reads, single-use expiring approvals for opening URLs, path-canonicalized file reads with output bounds, time-boxed clipboard grants, and five-minute revocable input-control sessions bound to a specific window, process, and action class. Shell is quarantined outright.

  • Authority derived from the Tauri window itself
  • Credential directories & symlinks denied
  • Strict CSP, DOMPurify-sanitized previews
  • Node audit vulnerabilities: zero
GOV / 02

Durable Agent Supervisor

A deterministic control plane — not an LLM — owns long-running local work. Jobs persist in WAL-journaled SQLite with append-only event history, worker leases, immutable authority profiles, and crash reconciliation that classifies ambiguous steps instead of blindly replaying side effects.

  • 11-state job lifecycle, centralized transitions
  • Terminal states can never resurrect
  • 9-class failure taxonomy; ambiguous suspends
  • 128 non-terminal jobs, 10,000 retained
GOV / 03

Credential Broker

Secrets live in Windows Credential Manager under opaque locators — the durable database stores metadata only, never keys. Provider execution is pinned to exact endpoints with redirects disabled, atomic use-counting, and hash-only response receipts. Possessing a credential does not grant provider authority; provider authority does not supply a credential.

  • No get-secret command exists
  • Pinned endpoints, redirects disabled
  • Atomic grant use-counts, child grants narrow-only
  • Secrets zeroized after use
GOV / 04

Governed Perception

Job-scoped capture of windows, displays, or regions — with identity proven down to the process creation epoch. Raw pixels never touch disk or logs: they exist briefly in native memory, then die. Remote frame analysis requires both an active session and a job-bound credential grant, and visible screen text is always treated as untrusted data.

  • Exactly four perception capabilities exist
  • Session TTL 300s · max 4 concurrent sessions
  • ≤20 frames/window, ≤3 frames/display
  • Observations recorded as hashes, not images
GOV / 05

Job-Scoped GUI Control

When IRIS must drive a desktop app durably, it does so through native Win32 targeting identified by executable path, PID, and process start time — defeating PID recycling. Every single action passes ten fail-closed revalidation checks, including live OS focus verification of the exact target.

  • No PowerShell, COM, or spawned processes in the path
  • Dangerous chords (Win+, Alt+F4, Ctrl+Alt+Del) denied
  • All GUI authority dies on restart
  • Typed text stored as length + SHA-256 only
GOV / 06

Event Source & Trigger Bus

A native fact-ingestion spine turns desktop events into governed work: events carry metadata only, triggers are typed matchers rather than scripts, and occurrence identity is hashed for exactly-once materialization into supervised jobs. Backpressure bounds everything.

  • Immutable envelopes; triggers reject edits to history
  • No model-generated or regex predicates
  • SHA-256 occurrence identity = exactly-once jobs
  • Bounded watchers: depth 8, 2,048 entries
GOV / 07

Governed Autonomous Worker

The OBSERVE → REASON → PROPOSE → VALIDATE → ACT → VERIFY loop runs toward a user-approved goal inside an immutable work envelope. Hard budgets cap steps, observations, model calls, typing characters, and wall-clock time. The model can request help — it can never mint authority.

  • Max 20 steps · 8 model calls · 600s wall clock
  • Budgets survive restarts; operators lower, never raise
  • Screen content = untrusted data, zero authority
  • Visual prompt-injection pilots: no shell, ever

Voice pipeline

From wake word to spoken reply.

A full-duplex voice state machine — idle, listening, hearing, processing, speaking — engineered so IRIS never talks over you, and never hears herself.

Wake Word

"Hey IRIS" · enrolled owners skip it

→

Capture + VAD

Echo-cancelled mic or native cpal fallback

→

Whisper STT

Speaker confidence & owner detection

→

Reasoning

Memory context + tool selection

→

Streaming TTS

Chunked, gapless, orb-synced

Post-speech cooldown keeps her from triggering on her own voice. If the WebSocket stream drops, playback falls back to HTTP TTS automatically. Audio levels drive the orb in real time.

Memory architecture

Personal first. Shared second.

Desktop IRIS is wired directly into the backend memory stack — personal context leads, retrieved knowledge follows:

Lethe — personal continuity

User-scoped memory built from your day-to-day interactions with desktop IRIS. What you did, how you phrased it, what mattered — carried forward so tomorrow starts with context.

Mnemosyne — shared memory

Reusable, cross-agent knowledge. Summaries worth keeping beyond one user's flow get written here and become available across the wider agent ecosystem.

Pattern detection adds a temporal dimension: activity logging plus a deterministic sequence detector surfaces recurring workflows — three or more days of the same dance — and offers to turn them into one-word macros.

Perception systems

Eyes that run locally.

MediaPipe-powered vision runs on-device at ~12 FPS with debounced triggers — presence detection, attention tracking, and gesture recognition without shipping your camera feed anywhere unless you ask for analysis.

Face Detection

BlazeFace short-range — bounding box, confidence, attention direction

Hand Tracking

21-point skeleton → 14+ gestures at 3-frame debounce

Capture

1280×720 webcam frames, base64 to vision analysis on demand

Screen Grid

10×10 calibration grid powering annotations and click targeting

Tech stack

Production-grade, end to end.

Desktop Runtime

Tauri v2 (Rust) + React 19 + TypeScript + Vite

3D Avatar

Three.js particle sphere with live audio visualization

Audio Engine

cpal native capture, RMS-level VAD, streaming TTS scheduling

Vision

MediaPipe Tasks — BlazeFace, HandLandmarker

Durable State

SQLite with WAL journaling, leases, append-only events

Speech AI

OpenAI Whisper STT · ElevenLabs / OpenAI TTS

Security

Windows Credential Manager, CSP sandboxing, DOMPurify

Testing

166 Rust unit tests + TS contract/gate suites, all green

Where it's going

The roadmap reads like mission control.

The published roadmap pushes IRIS toward reliable multi-app operation: a unified observation layer, plan-act-verify execution with evidence-backed results, versioned skills, deep app integrations, and enterprise-ready administration — benchmarked against real task suites at every milestone. It is deliberately labeled vision, not shipped feature: the same honesty standard we hold the guardrails to.

Read the code behind the guardrails.

The core of IRIS lives in the open. Audit it, fork it, build on it.

github.com/solsticeaistudio/iris-desktop-oss

ask iris
IRIS ONLINE
Drop files here Images, PDFs, text files