GitHub - awlevin/typesafe-computer-use: Computer use for about $0.0002 a step: OCR the screen, classify the next action with TypeSafe, click. macOS. · GitHub
Skip to content
Navigation MenuSign inAppearance settingsPlatformAI CODE CREATIONGitHub CopilotWrite better code with AIGitHub Copilot appDirect agents from issue to mergeMCP RegistryIntegrate external toolsDEVELOPER WORKFLOWSActionsAutomate any workflowCodespacesInstant dev environmentsIssuesPlan and track workCode ReviewManage code changesCode QualityEnforce quality at mergeAPPLICATION SECURITYGitHub Advanced SecurityFind and fix vulnerabilitiesCode securitySecure your code as you buildSecret protectionStop leaks before they startEXPLOREWhy GitHubDocumentationBlogChangelogMarketplaceView all featuresSolutionsBY COMPANY SIZEEnterprisesSmall and medium teamsStartupsNonprofitsBY USE CASEApp ModernizationDevSecOpsDevOpsCI/CDView all use casesBY INDUSTRYHealthcareFinancial servicesManufacturingGovernmentView all industriesView all solutionsResourcesEXPLORE BY TOPICAISoftware DevelopmentDevOpsSecurityView all topicsEXPLORE BY TYPECustomer storiesEvents & webinarsEbooks & reportsBusiness insightsGitHub SkillsSUPPORT & SERVICESDocumentationCustomer supportCommunity forumTrust centerPartnersView all resourcesOpen SourceCOMMUNITYGitHub SponsorsFund open source developersPROGRAMSSecurity LabMaintainer CommunityGitHub StarsArchive ProgramREPOSITORIESTopicsTrendingCollectionsEnterpriseENTERPRISE SOLUTIONSEnterprise platformAI-powered developer platformAVAILABLE ADD-ONSGitHub Advanced SecurityEnterprise-grade security featuresCopilot for BusinessEnterprise-grade AI featuresPremium SupportEnterprise-grade 24/7 supportPricingSearch/Sign inSign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.
You signed out in another tab or window. Reload to refresh your session.
You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
awlevin
/
typesafe-computer-use
Public
Notifications
You must be signed in to change notification settings
Fork
16
Star
259
Code
Issues
4
Pull requests
1
Actions
Projects
Security and quality
0
Insights
Additional navigation options
Code
Issues
Pull requests
Actions
Projects
Security and quality
Insights
mainBranchesTagsGo to fileCodeOpen more actions menuLatest commit History26 Commits26 CommitsFolders and filesNameNameLast commit messageLast commit date.github/workflows.github/workflows docsdocs teststests typesafe_computer_usetypesafe_computer_use .env.example.env.example .gitignore.gitignore CONTRIBUTING.mdCONTRIBUTING.md LICENSELICENSE README.mdREADME.md pyproject.tomlpyproject.toml uv.lockuv.lock View all filesRepository files navigationREADMEContributingMIT licenseMore items
typesafe-computer-use drives a Mac toward a goal you type in plain English, for about a
fiftieth of a cent per step. It never sends a screenshot to a big model. Instead it
reads the screen deterministically, asks a small classifier which action comes next,
and only calls a writing model when a text field genuinely needs free text.
clicker "go to techcrunch and take me to the checkout page for the cheapest tickets to their next upcoming event" --act
Why
Frontier-model computer use is capable and expensive: every step ships a screenshot and
waits several seconds for a plan. Most steps do not need a plan. They need one choice
from a short list, made quickly and cheaply, with a confidence number you can gate on.
TypeSafe sells exactly that: a decision model that answers
a Choice over up to 255 options with a full probability distribution and a calibrated
confidence, in a few hundred milliseconds, with free output tokens. This project is a
computer-use loop built around it.
Measured on the same screenshot and goal, one decision each:
typesafe (jev)
Claude Opus 5, bare screenshot
multiplier
input tokens
4,882
4,785
same
cost per decision
$0.0002
$0.032
155x cheaper
cost per decision, realistic loop with history
$0.0002
$0.035 to $0.08
170x to 390x cheaper
cost per 12-step task
$0.003
$0.40 to $0.90
130x to 300x cheaper
model latency
0.13 to 0.38 s
5.2 s
14x to 40x faster
end-to-end step, with capture and OCR
about 1.5 s
about 5.5 s
3.7x faster
The honest caveat: the big model read the event dates off the pixels and compared them
unaided. The classifier needed the date parsing described below. Every piece of
reasoning the frontier model does for free has to be rebuilt here as deterministic state.
Install
macOS 14 or newer, Python 3.12 or newer, uv.
git clone https://github.com/awlevin/typesafe-computer-use
cd typesafe-computer-use
uv sync
cp .env.example .env # fill in the keys
variable
required
purpose
TYPESAFE_API_KEY
yes
every decision
ANTHROPIC_API_KEY
no
type_text, writer-proposed URLs, and the final answer
CLICKER_EMAIL
no
enables the type_email action
CLICKER_BROWSER
no
defaults to Google Chrome
CLICKER_WRITER_MODEL
no
defaults to claude-haiku-4-5
CLICKER_ANSWER_MODEL
no
reads the last screen for the final answer; defaults to claude-sonnet-5
Grant your terminal Screen Recording and Accessibility in System Settings >
Privacy & Security. Without the first, captures are wallpaper. Without the second,
synthetic clicks are silently dropped, and --act refuses to start.
Use
uv run clicker "open the Playground" # dry run: one step, prints what it would do
uv run clicker "open the Playground" --act # drives the machine, up to 100 steps
uv run clicker "log in" --act --steps 20 --delay 3 # longer and slower
uv run clicker-inspect "any goal" # 3-2-1, capture, open the annotated screen + payload
Clear the terminal first. It is on screen, so its text is OCR input.
Stopping a live run. Ctrl-C when the terminal has focus, or slam the mouse into the
top-left corner of the screen from any app. The loop also stops itself on done or
none, on confidence under --min-confidence (0.4), after two consecutive no-ops, or
at --steps.
The answer. When the loop stops itself, the writer reads the screen it stopped on
and prints the result: the information the goal asked for, or where things stand and
the next step when the screen does not hold it. A dry run that would have acted, and
an aborted run, print no answer.
How a step works
screencapture ─► Vision OCR ─► merge lines into blocks ─► drop lines echoing the goal
accessibility ─► actionable elements (role, label, frame), pruned to the display,
the labelled pressable ones it pruned kept as off-screen controls
│
└─► one numbered list of items, each carrying its source
│
accessibility ─► focused field (role, label, placeholder, value, frame)
AppleScript ─► frontmost app and pid, active tab URL
clock ─► local date and time
dates.py ─► "dated 2026-10-13 (in 27 days)" on any block containing a date,
"near a line dated ..." on its neighbours
│
▼
one TypeSafe request, three Choices, four with off-screen controls
┌────────────────────────────────────────────────────────────┐
│ kind : click_item | use_browser | type_text | scroll… │
│ item : which item (used only for click_item) │
│ site : which website (used only for use_browser) │
│ offscreen : which hidden control (only for press_offscreen)│
└────────────────────────────────────────────────────────────┘
│
▼
deterministic action ─► wait ─► next step
Items carry where they came from: ocr for a text block, ax for a control the app
declared, ax+ocr when both found the same thing. An ax item reads as
button 'Share' (top-right) in the criteria, so the classifier can tell a real control
from a line of text.
Splitting the decision into three questions keeps screen noise out of the action
choice. Every stall found while building this came from two options that meant the
same thing. Confidence measures concentration, so overlapping options always read as
doubt. Keep the action set mutually exclusive.
OCR cost
Vision is about two thirds of a step, and it charges by the amount of text rather than
the number of pixels, so the only real saving is reading less of the screen.
Crop. Each step reads the frontmost window with an 8 pt margin, plus the menu bar
strip over the same columns, clamped to the display. Text on the desktop and in
background windows is noise to the decision. Clipping the strip to the window's width is
what makes the crop pay on a full-height window. The cost: the clock and the menu extras
to the right of the window go unread. They stay clickable through the accessibility tree.
Reuse. The capture is compared with the previous one at 1/8 scale, in 256 px tiles.
Unchanged tiles keep the lines they produced last step. The changed tiles are clustered
into blobs, sides and corners counting as touching, and each blob becomes a rectangle
read on its own. Scattered change is the ordinary case, a clock digit plus one repaint,
and one rectangle around both would span the display. Each rectangle grows until no known
line straddles its edge, because a crop through a line returns the half it can see; ones
that meet after growing merge, and more than four merge by closest pair down to four.
Past 60% changed tiles, past 60% of the region in summed rectangle area, or on an app
switch or a window move, the whole region is read instead.
The timing line says how much was read, and in how many pieces: ocr 0.31s (22% of screen, 2 rects). A replay (--image) always reads the whole image and never reuses, so an offline
repro matches the original run.
Accessibility tree
OCR cannot see an icon. The accessibility tree can, so each step also walks the frontmost
process for labelled, on-screen controls. Coverage is uneven, measured on ten apps on one
Mac: Finder 100% of on-screen controls labelled, Chrome 88%, Slack 85%, Notion 68%,
Spotify 0 (its CEF shell exposes three window buttons and nothing else). Terminals expose
the grid as one text area. So AX is a bonus source, never a replacement.
Labels live in AXDescription for web and Electron, AXTitle for AppKit, and a short
AXValue otherwise. A decorative image takes the label of the control around it; a list
row takes it from a shallow AXStaticText.
Frames lie, so the walk prunes hard:
skip any subtree whose real frame misses the display (Notes reports rows 200 screens
down, Chrome parks scrolled-out nodes above the viewport)
skip any node under 4 pt wide or tall (Chromium clamps scrolled-out web nodes to slivers)
skip AXMenu subtrees, which are thousands of zero-sized items behind a closed menu
skip nameless AXGroup layout boxes, even pressable ones
stop at 4000 nodes or 0.6 s and say so
Walks measured here: Finder 152 controls in 0.08 s, Chrome 172 in 0.59 s. The assistive
handshake attributes (AXManualAccessibility, AXEnhancedUserInterface) are unsupported
on this macOS, so nothing relies on them.
Off-screen controls
AXPress does not need an element to be visible. Notes selects a row parked thousands of
points below the display, Chromium delivers a click to a link it clamped to a 1 px sliver
because the page is scrolled past it, and an auto-hidden Dock hands over all 37 of its
items from 5 pt below the bottom edge. So the same walk keeps the labelled, pressable nodes
it pruned, and offers them as a separate capped list rather than mixing them into the items:
nothing on the capture points at them, and a mouse click would land somewhere else entirely.
The list is deduplicated by role and label, drops any label the visible items already carry,
and stops at 120 controls, after which those subtrees are pruned as before, so the walk costs
what it always did. It is offered only when it is not empty, as a press_offscreen action
plus an offscreen question, and the step log counts it next to ax=. A refusal is the end
of it: there is no pixel to fall back on, so it reads as a no-op. What a walk finds depends
on the app, and the node and time caps bind first on a big tree: Notes and Chrome spend all
4000 nodes on what is already on screen and report nothing hidden.
Action space
key
does
click_item
press the element through the accessibility tree when the item came from it, so the press lands on the control rather than on whatever covers it; a mouse click at the center of the box otherwise, and as the fallback when the press is refused
press_offscreen
AXPress a labelled control the app exposes but does not show, chosen from the off-screen list; offered only when that list is not empty, and a refusal counts as a no-op since there is no pixel to fall back on
use_browser
go to the browser, showing the website the site answer names: none brings it forward on the page already open there, a SITES catalog key opens that URL through AppleScript open location, and other opens a URL the writer proposes
type_text
the writer composes the string; it is set on the focused element through the accessibility tree, with keystrokes as the fallback when the value does not read back, and a TypeSafe Noul then checks the field's value
type_email
fills in $CLICKER_EMAIL the same way; refused unless a text field is focused
press_enter, press_escape
keyboard
scroll_down, scroll_up
10 lines, after parking the cursor over the frontmost window
wait
screen still loading
done, none
stop
Where free text comes from
The classifier never generates text. The writer model runs in three places, each with a
small packet and a structured reply:
type_text receives the goal, recent actions, the focused field's label and
placeholder, and the OCR lines near the field. It returns {fill, text}. Credential
fields come back fill: false and nothing is typed. After typing, a Noul scores
whether the field now holds a sensible value. Under 0.5 the field is cleared.
use_browser with site: other receives the goal and returns {ok, url}.
Code rejects anything that is not a clean https URL with a hostname.
The answer, once, when the loop stops itself. It receives the goal, every action
taken, why the run stopped, the text of the last screen, and the capture itself,
because OCR misreads a letter here and there and drops layout. It returns
{achieved, answer}, and is told to take the answer from the screen alone. When an
action ran after the last capture, the screen is captured again first. This one
call uses CLICKER_ANSWER_MODEL, a stronger reader than the per-step writer.
Passwords are never typed. Rely on the browser's password manager or an SSO button
the OCR can read.
Run folder
Every run writes runs/<timestamp>/ so a stall can be replayed and fixed offline:
file
contents
run.log, run.json
everything printed; goal, outcome (done, nothing helps, low confidence, stalled, step limit, dry run, aborted, crashed), answer and goal_achieved, seconds, every action, config, and timing (mean and max seconds per phase, with steps_timed)
answer-raw.png
the capture the answer was read from, when an action made the last step's capture stale
step-NNN-raw.png
the capture
step-NNN.png
items numbered in blue, accessibility ones orange, the chosen one red, the focused field green
step-NNN-payload.txt
the exact state and criteria sent to TypeSafe, then every item with source, role, box, click point, confidence, then the off-screen controls
step-NNN-answers.json
every probability the classifier returned, the off-screen controls it was offered, plus timing for that step
Each step also logs what it cost, so a slow phase is obvious:
timing: capture 0.31s screenshot 0.28s app 0.01s window 0.02s field 0.01s url 0.01s ocr 0.31s (22% of screen) ax 0.06s decide 0.21s act 0.05s total 0.95s
capture covers the four round trips under it; act is left out when the step did not act.
Replay a saved capture as if it were live, without touching the screen:
uv run clicker "same goal" --image runs/<ts>/step-003-raw.png --app "Google Chrome" --url "https://example.com/"
Layout
typesafe_computer_use/
macos.py the only module that touches Quartz, AX, AppleScript (platform adapter)
including the bounded walk for actionable elements
perception.py capture, OCR, the read region and the changed-tile cache,
block merging, goal-echo filter, the accessibility item
source, and the merge of the two
dates.py date parsing and "in N days" hints
decide.py state, criteria, the three-Choice request, the Noul check
writer.py the writer model, structured replies, URL validation, the final answer
actions.py one handler per action, each returning a history line
runner.py the step loop, run folder, stop rules, the hand-off for the answer
report.py logging, annotated screenshots, payload dump
timing.py phase stopwatches, the timing line, run summary
cli.py `clicker` and `clicker-inspect`
tests/ pure logic: dates, merging, reading order, echo filter, config,
decisions, the tree walk against a fake tree
A Linux port replaces macos.py with xdotool and AT-SPI, and swaps Vision OCR for
PaddleOCR or RapidOCR. The tree walk itself takes its children, attributes, and actions
as callables, so only those three bindings change. Nothing else knows the platform.
Known limits
OCR only sees text, and the accessibility tree only covers apps that publish one.
In a terminal, a canvas, or Spotify, an icon-only button reaches neither source.
Two identical labels get only a coarse region hint and split the vote.
Only the main display is captured.
Using the machine during an --act run fights it for focus and the cursor.
The site catalog is small on purpose; the writer covers the rest.
Development
uv run ruff check . && uv run ruff format --check .
uv run pytest -q
CI runs the same on macOS. See CONTRIBUTING.md.
License
MIT
AboutComputer use for about $0.0002 a step: OCR the screen, classify the next action with TypeSafe, click. macOS.Topicsai-agentsautomationcomputer-usemacosocrtypesafeResourcesReadmeMIT licenseContributingContributingActivityStars259 starsWatchers1 watchingForks16 forksReport repositoryReleasesPackagesContributorsLanguages
Footer
© 2026 GitHub, Inc.
Footer navigation
Terms
Privacy
Security
Status
Community
Docs
Contact
Manage cookies
Do not share my personal information
You can’t perform that action at this time. |
The project typesafe-computer-use is an endeavor to automate computer interaction on macOS by translating plain English goals into executable actions with extremely low cost, aiming for approximately two hundredths of a cent per step. The core methodology involves performing actions by reading the screen using Optical Character Recognition, classifying the subsequent required action using a specialized decision model called TypeSafe, and then executing the click. This approach contrasts with methods relying on large frontier models that generate full plans, as this system focuses on making quick, cheap, and deterministic decisions based on immediate context.
The capability stems from recognizing that most steps in computer use do not require complex planning but rather a quick choice from a limited set of options, which TypeSafe is designed to handle by assessing choices over up to two hundred fifty-five options in a few hundred milliseconds, providing a full probability distribution and calibrated confidence scores, often with free output tokens. This cost-effective decision-making is demonstrated by comparing the costs of decision-making in this system versus using large language models for planning, showing significant cost reductions for individual decisions and entire multi-step tasks.
The system operates through a carefully constructed computer-use loop. Each step begins with screen capture and OCR, followed by merging lines and identifying actionable elements via an accessibility tree. The process then feeds this visual and structural information into a deterministic mechanism. The decision-making process involves segmenting the problem into three choices to minimize screen noise, ensuring that the action selection process is highly concentrated. The system determines the next deterministic action, which involves waiting before proceeding to the next state.
The mechanism for interpreting the screen involves complex interplay between visual data, accessibility information, and temporal reasoning. The process involves detailed steps like screen capture, OCR, merging lines into blocks, and utilizing the accessibility tree to identify labeled, pressable elements. The system must account for dynamic changes, using image comparison between steps to determine which regions have changed, potentially reading the entire region if significant changes are detected, thereby optimizing the cost of vision processing.
The system defines a rich action space for the decision-making process, including interface interactions such as click_item, press_offscreen, use_browser, and typing text. These actions are mediated by the accessibility tree, which guides the interaction by finding the specific element based on its role, label, and frame, allowing the system to select controls from either visible or hidden locations. For actions requiring text input, the system uses the writer model to compose text, and a subsequent check verifies the field's sensible value, preventing unvalidated inputs.
The process of generating free text is handled by specialized components. The classifier does not generate text directly; instead, the writer model interacts with the system in specialized packets to receive context, such as the recognized text, the focused field's state, and OCR results. This allows for context-aware text input, where the system fills fields only when sensible values can be derived, thereby avoiding the typing of sensitive information like passwords.
A robust logging system is integral to the project, with every execution saving a folder containing detailed state information, including run logs, raw image captures, payload dumps detailing the exact criteria sent to TypeSafe, probability outputs from the classifier, and timings for every phase, such as screen capture, OCR, and decision-making. This logging enables the ability to replay previous runs or offline debug the system by replaying saved captures exactly as they occurred.
The architecture is modular, with distinct Python modules handling specific responsibilities, including perception for capture and OCR, decide for state and criteria management, writer for generating text responses, and a runner for managing the step loop and logging. While the system is implemented for macOS, the underlying logic of the tree walk and causal relationships is designed to be portable, with plans existing for a Linux port that substitutes macOS-specific tools with equivalents like xdotool and PaddleOCR, ensuring platform adaptability.
Known limitations exist, primarily concerning the scope of information derivable from the input. The system is constrained by what OCR can perceive (text only) and the limitations of the accessibility tree when dealing with non-standard applications or elements that do not expose sufficient labeling. Issues arise when distinguishing between identical labels, which can lead to coarse region hints, and when dealing with elements that lack visible pixels, where the system defaults to a no-operation if no visual fallback exists. Furthermore, the system is optimized to operate primarily on the main display, and dynamic real-time usage by the machine during execution can compete for focus, affecting the process. |