OpenAgent⌘
The Local macOS Body for Instinct

Say “Wake up, Jarvis.”
Then just talk.

OpenAgent gives Instinct a voice interface modeled on Tony Stark's Jarvis. Speak naturally from anywhere in the room. OpenAgent carries out your requests locally: shell execution, code diffs, native apps, your authenticated Chrome, and social accounts.

Quick Start (`./boot.py`)How It WorksExplore 55+ Tools →
Zero API Keys Needed
Offline Wake Word (Vosk)
Fail-Closed Chat Guard
Click sphere to simulate: "Wake up, Jarvis"
SESSION: CONTINUOUS_VOICE
Step 1/6
Jarvis

[ Offline Vosk Wake-Word Daemon Standing By ]

Paused
Engineered On Open Standards
macOS Darwin (14 & 15)Python 3.11+ via Astral uvBun 1.1+ TypeScriptwhisper.cpp & VoskFirecrawl Engine195 Passing Tests
Continuous Hands-Free Conversation

Talk while you make coffee. Jarvis handles the rest.

Traditional voice bots force you to wait for beeps or re-state wake words on every sentence. OpenAgent stays awake continuously. You speak, pause, and the request dispatches automatically. Context carries across turns in your verified chat.

CONTINUOUS_HANDS_FREE_CONVERSATION
[ Jarvis sleeping · Vosk offline wake daemon listening ]
You (making coffee)
"Wake up, Jarvis."
Jarvis
[ Audio chime · Jarvis awake and listening continuously ]
You
"Run the test suite on OpenAgent. If it's green, post a Threads update about what I shipped today."
Turn 4 of 14
01

Wake once, keep talking

Say "Wake up, Jarvis" and the session unlocks. Vosk monitors speech locally. Requests dispatch automatically after 2 seconds of natural silence.

02

Natural audio turn-taking

Instinct replies aloud through your Mac speakers. Built-in echo guards ensure Jarvis never hears or triggers its own voice feedback.

03

Hold F8 push-to-talk fallback

Working in an office or noisy café? Hold F8 to talk, release to send. Holding F8 during Jarvis's reply cuts it short instantly.

Closing the Desktop Gap

Instinct thinks. OpenAgent acts.

Instinct operates in the cloud. It can touch your accounts, but it cannot touch your laptop. OpenAgent closes the gap, giving cloud intelligence a local body with native macOS keyboard, mouse, terminal, and browser control.

CapabilityInstinct AloneInstinct + OpenAgent ⌘
Always-listening, hands-free voice sessions at your desk
Offline Vosk wake word + continuous turn-taking
Account-level tasks (email, calendar, GitHub, Notion)
Native Instinct cloud integrations
Proactive reminders and autonomous follow-ups
Reaches out proactively via iMessage/WhatsApp
Shell execution and code edits on your local machine
Sandboxed bash in zsh + atomic diff search/replace
Seeing your screen and controlling native macOS apps
Quartz window screenshots + non-stealing PID clicks
Driving your authenticated, everyday Chrome profile
Real Chrome CDP with cookies and session logins
Operating WhatsApp Desktop via Accessibility (AX)
Auto-calibrated native macOS Accessibility tree
Posting & engaging on Threads & Reddit from your browser
LocoAgent engine with anti-duplication ledger
Private, self-hosted web scraping and crawling
Dockerized local Firecrawl without cloud API leaks
Instinct thinks in the cloud. OpenAgent acts on your Mac.
Transport Architecture

How OpenAgent works without an API key

OpenAgent turns WhatsApp Desktop into a deterministic local bridge using macOS Accessibility APIs. No cloud proxies, no open inbound ports, and no API subscription costs.

STEP 01

Voice Ingestion

You speak after "Wake up, Jarvis". OpenAgent captures audio via SoX and attaches a voice note or local Whisper transcript to your Instinct chat.

STEP 02

JARVIS_CALL Envelope

Instinct replies with structured JSON payload wrapped in a base64 envelope, preventing WhatsApp markdown from stripping symbols like asterisks and tildes.

STEP 03

AX Observation

OpenAgent reads incoming messages through the macOS Accessibility tree without requiring webhooks or exposing local network ports.

STEP 04

Fail-Closed Verification

A security guard verifies the message header against your configured Instinct phone number (`BRIDGE_SAFE_MODE=true`). Unverified messages are discarded.

STEP 05

Dispatcher Routing

The call is parsed and routed to one of five local engines: developer harness, native Quartz/AX, browser CDP, Firecrawl, or LocoAgent.

STEP 06

Response & Voice Loop

Execution diffs, stdout, and window screenshots are staged and sent back to WhatsApp. Instinct plays voice responses out loud automatically.

Read the full technical Architecture & Sequence Specification →
Modular Architecture

Five engines. Over 55 local tools.

Every tool is engineered for safety, atomicity, and low latency. Click through the engines below to inspect capabilities and protocol envelopes.

BUILT ONBun + TypeScript + zsh

Developer Harness

Gives Instinct a full engineering environment on your Mac. Unlike naive execution, edits are validated through exact string matching with unified diffs to prevent catastrophic hallucinations.

Key Primitives
Sandboxed bash in zsh with timeout guards
Exact chunk search-and-replace with atomic verification
ripgrep multi-threaded regex search across entire repositories
Native AppleScript execution via osascript bridge
Explore all 8 Developer Harness tools in catalog
PROTOCOL TRANSPORTSAFE_MODE: ON
Payload Structure
JARVIS_CALL:eyJ0b29sIjoiZWRpdCIsImFyZ3MiOnsicGF0aCI6InNyYy9tYWluLnRzIiwib2xkX3N0cmluZyI6ImNvbnN0IFBPUlQgPSAzMDAwOyIsIm5ld19zdHJpbmciOiJjb25zdCBQT1JUID0gODA4MDsifX0=:END
Encapsulated inside WhatsApp markdown-safe envelope. Base64 encoding protects brackets and characters from messaging formatting corruption.
Zero-Touch Tooling

Self-bootstrapping autonomous scripts

OpenAgent includes autonomous Python scripts for every lifecycle task: environment setup, preflight system audits, WhatsApp UI calibration, and test execution.

$./boot.py --voice --send-mode audio
[boot] OpenAgent Autonomous Supervisor initializing...
[runtime] Auto-detected macOS Darwin (Apple Silicon arm64)
[uv] Environment .venv active, dependencies synchronized
[models] Checking speech intelligence: vosk-model-small-en, ggml-base.bin [OK]
[watchdog] Launching WhatsApp Desktop in background...
[bridge] Fail-closed chat verification: +16508702892 matched
[voice] Vosk wake-word listener standing by (say 'Wake up, Jarvis')...
[ready] OpenAgent running with exponential backoff supervisor PID: 48192
Security By Design

Your computer, guarded

Local agentic execution demands strict boundaries. OpenAgent is architected with defense-in-depth principles:

Offline Wake Word

Vosk runs completely on your Mac. No microphones stream to cloud providers while Jarvis is asleep. Audio records only after wake-word activation.

Fail-Closed Sender Auth

OpenAgent checks the active WhatsApp chat header before every tool invocation. Messages from unknown numbers or outside groups are rejected instantly.

Anti-Duplication Ledger

Social posts and interactions are permanently hashed and logged. The agent cannot accidentally spam forums, repost existing items, or enter infinite loops.

Questions & Answers

Frequently Asked Questions

Instinct (instinct.com) is an invite-only personal AI assistant that lives in your messaging apps (WhatsApp and iMessage). It handles account-level tasks, schedules, and cloud computing. OpenAgent provides the local macOS body for Instinct: an always-listening, hands-free bridge that executes local terminal commands, code diffs, native app actions, authenticated Chrome CDP tasks, and social media automation on your actual Mac.
OpenAgent ⌘ · MIT License

Give your AI a physical desktop.

Clone the repository, run `./install.py`, and say “Wake up, Jarvis.” Turn Instinct into your personal Tony Stark assistant today.

Get Started in 4 MinutesView on GitHub