All you need is accessibility

I’ve been building FynPDF, a PDF reader and editor for macOS, with its own PDF engine written from scratch.

I’m primarily a backend engineer who occasionally gets the urge to make things on the side, so naturally I decided to build a PDF application for a platform I have comparatively little experience with. Sensible decisions were clearly made.

The code started out entirely handwritten because I had an itch to scratch: I wanted to build my own PDF rendering engine, write an interpreter, and continue learning Apple’s graphics APIs after working with them for perceptual hashing in DedupX, my other project.

The engine is the easy part to test: Feed it a PDF, render some pages, compare the pixels against Poppler and PDFKit, and you’re done. The tests run headless and nobody has to look at a window.

The user-facing app - the viewer and editor - is different:

• A thumbnail can update while the page stays stale.
• A save can blank the screen for 200 ms and look fine afterward.
• A popover can open and never close.
• A click can land on the wrong thing because something moved underneath it.

None of that shows up in a unit test.

And I wanted AI agents to do more than write the code. I wanted an agent to actually exercise the application: open a PDF, scroll it, zoom it, annotate it, save it, reopen it, and tell me when something went wrong.

That sounds straightforward until you try to do it on the Mac you’re actually working on.

The interesting part of this project wasn’t that I decided accessibility was the answer. I didn’t. The requirements and constraints emerged one failed testing approach at a time.

Attempt 1: “Look at it” with screenshots and System Events

My first version was a skill file for the agent - a Markdown playbook describing how it should test the app. It would run FynPDF from Xcode, open a file through File > Open like a user, send shortcuts with AppleScript System Events, take a screenshot, read it, and repeat until the change was visible.

The basic rule was sound: if you didn’t see it, you didn’t finish. The mechanics were not.

The app jumped in front of whatever I was doing. If I clicked somewhere else mid-run, the agent’s ⌘W could land in my window. Background clicks through the screenshot tool often failed.

And to boot, screenshots are expensive. Every observation became a chunk of image data that the agent has to consume. An agent that needs to look after every interaction quickly turns a simple UI test into a token-burning exercise.

More importantly, screenshot-driven interaction is inherently slow. I don’t want an agent taking a screenshot, reasoning about it, clicking once, taking another screenshot, reasoning again, and doing that for every scroll or mouse movement.

A runaway render during one test run ate enough memory that I had to write a watchdog script just to kill it. Lesson learnt, have watchdog scripts for CPU/memory/disk/<whatever else> usage - it’ll save you from a system hangup.

I deleted the skill.

But this attempt gave me the first real requirement:

1. The test must not interfere with me.

Attempt 2: Separate macOS account + VNC

The obvious next idea was isolation. If the problem was that the test was fighting me for the desktop, why not give it another macOS user account? So, I set up a separate account and used VNC to interact with its desktop.

This fixed one problem immediately: my own desktop was no longer at risk. It didn’t fix the testing problem.

VNC gave me another remote desktop, but it was still fundamentally a screen-and-input interface. The agent still had to reason about pixels, and every observation was still expensive. Rapid scrolling, repeated interactions, and hit testing were particularly unpleasant. It also introduced another layer between the test and the application: the virtual display.

A test that cares about whether a particular control was hit should not have to ask: Which pixels currently represent that control on this remote desktop?

And I certainly didn’t want to run an agent in a loop where every action looked like:

• capture screen
• send image to model
• decide where to click
• click
• capture screen again
• send another image
• repeat

That is a terrible way to spend tokens. The separate account solved isolation, but not interaction.

So the second lesson was:

1. The agent should not have to continuously observe the screen just to perform deterministic actions.
2. I need the test to talk to the application, not continuously stare at its desktop.

I wanted something that could perform hundreds of interactions quickly and only involve the model when there was actually something interesting to inspect.

Attempt 3: XCUITest, the “proper” way

At this point I went for Apple’s proper UI testing framework: XCUITest. I did the groundwork properly: accessibility identifiers on every control (page-view-3, thumbnail-7, zoom-in), a launch argument to open a fixture PDF, and a regression suite covering every shipped viewer surface: scrolling, zooming, layouts, sidebar, and go-to-page.

It went wrong in 2 different ways for me.

It takes over the machine

• XCUITest drives the app through synthesized HID events, the same basic mechanism a person would use.
• It activates the app, owns the keyboard and pointer while it runs, and launches its own copy of FynPDF next to my Debug build. The 2 instances fought over windows and state. That wasn’t going to work on the machine I use for development.

It was brittle against my own UI

• The suite was written for a single-document window.
• And I had moved my app to workspace windows with AppKit tabs in the title bar, and those title-bar tabs turned out to be a bad accessibility target. I want easy conversion from a tab, window or pane. I like terminal multiplexer like layouts to read multiple docs side-by-side.
• Every test that assumed “the window” needed rework.
• This wasn’t necessarily a flaw in XCUITest. It was a reminder that a conventional UI test suite is tightly coupled to the UI architecture it was written against.

At this point the requirements had evolved again:

1. The test needs to be isolated from my desktop.
2. It needs to interact with the app without owning my keyboard and mouse.
3. It needs to perform rapid, deterministic interactions without a model looking at a screenshot after every step.

Attempt 4: A general computer-use agent

The next thing I tried was a computer-use MCP server that works in the background.

This was much closer.

It reads windows through the accessibility tree and posts events directly to a process instead of going through the global event stream.

Menus pressed through accessibility elements worked with the app in the background.

For the first time I could have an agent interact with FynPDF without simply handing it my desktop. But the rough edges were exactly where I cared about them.

• Pixel clicks were refused when anything covered that spot on screen. One afternoon a full-screen Dock overlay - Mission Control or Launchpad - blocked every click on the page. That meant no stamps, no notes, no text boxes, no signatures.
• Window snapshot cost a lot of tokens. An agent that snapshots after every step burns through its context window very quickly.
• “Background” wasn’t enforced. A typing helper set AXFocused on a text field before inserting text. macOS can activate an app when you focus one of its elements, so FynPDF jumped in front of me mid-sentence.
• My app was guilty too. The Debug build called NSApp.activate 0.3 seconds after launch, so every supposedly “background” launch brought the app to the foreground anyway.

And there was a deeper problem: A general-purpose computer-use agent is designed to behave like a person. That’s useful when I want it to explore an unfamiliar application; it is not what I want for a test runner.

For a test I already know I want to:

• scroll rapidly
• hit-test a series of controls
• place several annotations
• move something repeatedly
• save
• capture frames when needed
• compare states
• reopen the document
• check parity against other PDF viewers

I don’t want a language model involved in the control loop for any of that. The model should describe the test and inspect evidence.The runner should execute the boring, deterministic parts at machine speed.

The generic tool had finally given me the right primitive - accessibility - but it still had too much freedom and too much overhead.

So I stopped looking for a generic computer-use solution.

What finally worked: accessibility, but as a test API

The breakthrough was a throwaway Swift script that used nothing but the macOS accessibility API, AXUIElement, to drive a Debug build that was never allowed to come to the front. It pressed menu items, ran actions on the page, set text field values, clicked sidebar thumbnails, and captured the window. On the first day it found 2 real bugs that no unit test had caught: a popover that never dismissed, and a text box that was too small for its own text. That script became fynpdf-ax, a small Swift command-line tool of about 900 lines that lives in the repo next to the app. The important distinction is that I wasn’t building another computer-use agent, I was building a test API for my application, exposed through the accessibility tree. And accessibility turned out to be exactly the right layer because it’s semantic and doesn’t require focus.

• Press, don’t click: AXUIElementPerformAction(element, kAXPressAction) presses a button without a pointer, without a keystroke, and without the window being in front. Nothing can cover it because it never goes through screen coordinates.
• Set values, don’t type:AXUIElementSetAttributeValue(field, kAXValueAttribute, "Quarterly notes") fills a field without focusing it. No focus means no activation.
• Find elements by identity, not position: I look for the identifier first (page-view-1), then the role (role:AXTextArea), then the label. A thumbnail and a page can both be labelled “Page 1”; only one of them is page-view-1.
• Menus are a tree: Testing options like Annotate/Stamp etc. walk the AXMenuBar by title and presses the item. It works in the background, with no global shortcut to intercept.

This is where the pieces finally clicked.

The test can drive semantically and observe visually, without making the model sit in the middle of every interaction. Every failure from the earlier attempts became a rule that fynpdf-ax enforces rather than a sentence in a prompt.

• There is no focus or global input API
• The tool never sets AXFocused, never sends keystrokes, and never moves the pointer. There simply isn’t an API in the tool that would do those things.
• If I want an action to be testable, I expose it through accessibility.

So do VoiceOver and Switch Control, because these are real accessibility features that people who can’t use a mouse need anyway. Making the app testable made it more accessible, and it goes the other-way too.

Flows: UI tests an agent can write

A test in fynpdf-ax is a JSON file called a flow: a fixture PDF and a list of steps. Here’s a real one, checked into the repo, for a bug where a stamp’s thickness did nothing and every save flashed the window:

{
  "about": "A stamp's Thickness reaches its frame, and ⌘S changes nothing on screen.",
  "fixture": "Testdata/viewer/pages-12.pdf",
  "steps": [
    { "save": true, "steady": 0.05, "region": [0, 0.08, 1, 1] },
    { "menu": "Annotate/Stamp" },
    { "action": "page-view-1", "name": "Place Annotation" },
    { "capture": "stamp-default" },
    { "action": "page-view-1", "name": "Show Properties" },
    { "set": "annotation-properties-thickness", "value": "8" },
    { "press": "annotation-properties-apply" },
    { "capture": "stamp-thick" },
    // differ means this step visibly changed the page
    { "differ": ["stamp-default", "stamp-thick"], "min": 0.05 },
    { "save": true, "steady": 0.05, "region": [0, 0.08, 1, 1] },
    { "capture": "after-save" },
    // same means the save or reopen didn't
    { "same": ["stamp-thick", "after-save"], "max": 0.3 },
    { "parity": "1", "max": 0.5 },
    { "relaunch": true },
    { "capture": "reopened" },
    { "same": ["after-save", "reopened"], "max": 0.8 }
  ]
}

The agent doesn’t need to operate the application one screenshot at a time. It writes the flow, starts the runner, and lets the runner execute the whole thing. That distinction matters. The agent might spend a few seconds reading the accessibility tree and deciding what flow to write. The actual test can then perform hundreds of actions, rapid scrolling, repeated hit tests, and frame captures without consuming another model token for every interaction.

This is also where coding agents like Claude and Codex fit into the workflow. I’m building FynPDF solo, but I’m not writing every line by hand anymore. Claude and Codex are heavily involved in implementation and testing. I use them extensively before review to look for security issues and other problems in the code. The important part is that the agents aren’t the authority on whether the UI test passed. The runner is, and I am. The agent gets structured results and visual evidence from the runner, rather than being asked to hallucinate whether an action probably worked.

Where accessibility isn’t enough

The title oversells it a little. Accessibility solved driving: getting an agent to use a real app, in the background, without fighting me. It doesn’t solve seeing. Every useful visual check still ends in pixels: a window capture, a frame recording, or a render from another viewer. And some bugs only show up with the view hosted in-process, where a test can read the composited tile layer directly. For example, I had an edit after a save that reached the thumbnail but never reached the page. I keep a small suite of those tests too.

Accessibility also isn’t a substitute for genuine interaction testing. A real drag, freehand ink stroke, or signature still exercises code paths that an accessibility action doesn’t necessarily cover. But that’s okay.

• The point isn’t that accessibility can replace every form of UI testing.
• The point is that it gives me a semantic control surface that doesn’t require ownership of the desktop. Once I had that, I could build the rest of the testing system around it.

The actual lesson

I started this project wanting an AI agent that could use FynPDF like a person. That turned out to be the wrong abstraction. A person needs a mouse and a screen. A test runner needs neither.

What I actually wanted was:

• a semantic way to address controls
• deterministic actions that don’t require focus
• rapid interaction without a model in the loop
• visual evidence when something changes
• isolation from my own desktop
• and assertions that encode the failure modes I actually care about.

macOS accessibility gave me the control surface for all of that. The funny part is that making FynPDF easier for an AI agent to test also made it easier for assistive technologies to use. The test API and the accessibility API ended up being the same API.

So if you’re building a native app and want an AI agent to test it, my advice is short:

1. Give every control an accessibility identifier. It’s cheap, and it’s the foundation.
2. Add accessibility actions for anything that currently needs a mouse. Your VoiceOver and Switch Control users benefit too.
3. Keep the flows. A repro that runs once is a demo. A repro that’s checked into the repository is a test.