Productivity
Screen Guide
Type a goal and see which button to click next.
Get for Mac.dmg · 956 KB · or download .zip
- Category
- Productivity
- Requires
- macOS 13+
- Size
- 956 KB
- Price
- Free
Preview
About Screen Guide
Type a goal and see which button to click next.
What it does
- Screen aware. Captures the front window with ScreenCaptureKit and reads its text and interface elements.
- Local AI. Asks an Ollama or LM Studio model which element to use and draws a ring around it.
- Step by step. Press Next step to continue towards your goal. Screenshots never leave your Mac.
Notes
macOS asks for Screen Recording and Accessibility permission, and you need a local model server. Ad-hoc signed apps can lose these permissions when rebuilt. This demo was compiled and launched; the guidance itself has not been tested end to end.
Information
- Developer
- SiliconDevKit (generated by AI from a prompt)
- Category
- Productivity
- Compatibility
- macOS 13 or later, Apple silicon. Apple silicon Macs only (M1 or later), not Intel
- Size
- 956 KB (.dmg)
- Version
- 1.0
- Price
- Free
- Signing
- Ad-hoc signed, not notarized
The prompt behind it
This app was generated from the prompt below. Paste it into SiliconDevKit to build your own version, then change it however you like.
Build a macOS app called Screen Guide that shows you where to click. The user types a goal such as "export this as PDF" and presses Return. The app captures the frontmost window with ScreenCaptureKit (ask for Screen Recording permission, with a first-run explainer and a button that opens System Settings), reads the text on it with the Vision framework, and also reads the interface elements of the frontmost app with the Accessibility API (AXUIElement: buttons, menu items, text fields, their titles and frames, needs the Accessibility permission, explain it first). It then asks a local language model which element to use next: Ollama at http://localhost:11434/api/chat (model picked from /api/tags, and when a vision model such as llama3.2-vision or llava is chosen, send the screenshot as a base64 image too), or LM Studio's OpenAI-compatible server at http://localhost:1234/v1. The reply names an element from the list and gives a one-line explanation. Draw a pulsing ring around that element on a transparent click-through full-screen overlay window, and show the explanation in a small card next to it. Add Next step, which captures again and continues towards the goal, a history of steps for the current goal, and a Stop button. A global hotkey (Control+Option+G, registered with Carbon RegisterEventHotKey) opens the goal box from any app. If no model server is running, still list the detected elements and let the user pick one to highlight, and say what to install to get suggestions. Screenshots are processed in memory only and are never saved or sent anywhere except to the model server the user chose on this Mac. No Dock icon.
Opening it for the first time
- Open the .dmg and drag Screen Guide onto the Applications folder.
- Open it from Applications. macOS may say it can’t check the app for malicious software. Click Done.
- Open System Settings → Privacy & Security and click Open Anyway. This is needed once. Why this happens. To share an app without the warning, see how to notarize it.
