Tapilot is an open-source Android automation project with a specific constraint: the user should be able to describe and run a task on the phone, without a computer connected.
Here is how the phone-side workflow fits together, and where its limits are.
A task starts in Android
The native chat interface accepts a text task. Voice is another input path once a speech provider is configured. The embedded runtime runs the agent and keeps task context on the phone.
Midscene interprets screenshots, locates visible targets and guides actions. Android’s AccessibilityService provides the device capabilities, including screenshots, gestures, navigation and text input. The next screenshot provides evidence of what changed.
The distinction is between visual targeting and device execution. Screenshot-based targeting does not mean Android system interfaces disappear from the implementation.
The model is still remote
The phone hosts the agent, but inference uses the configured model provider. Screenshots and task context are sent to that service. Voice recognition and playback use separately configured providers.
This lets the phone run tasks without a connected computer. It does not make the workflow offline or remove the need to configure a model.
A report makes an attempt inspectable
Midscene’s HTML report contains screenshots, plans, actions and errors. The demo shows a phone receiving a spoken instruction and opening WeChat, followed by a report view.
The video is edited. It demonstrates the workflow, rather than establishing a success rate. Protected screens, custom-rendered inputs and vendor background restrictions can still interrupt execution.
A computer can be an optional controller
Tapilot also includes a Skill and Codex plugin that connect to a paired, unlocked phone over Wi-Fi. In default steps mode, the host agent chooses actions, the CLI executes them, and MCP supplies the embedded preview panel.
That is a separate entry point. Users do not need it to run the phone-side agent.
Trying it
You need an ARM64 Android 11+ phone, accessibility permission and a supported vision model. Check the current installation instructions and the difference between the downloadable build and the latest source before starting.
Demo and setup: https://tapilot.lifeos.md
Source: https://github.com/tapilot-ai/tapilot
For a useful bug report, include the device, model and a small reproducible task. Check screenshots and logs for private information before sharing them.
Top comments (0)