from vision_engine import read_screen_text, click_on_text
# Read text from screen
text_lines = read_screen_text()
# Click on detected text
success = click_on_text("Submit Button")
Voice Commands
"Jarvis, what is on screen?"
"Jarvis, click on [element]"
"Jarvis, read the text"
"Jarvis, find the button"
Performance
Detection Speed: 100-500ms
Accuracy: 90%+ for UI elements
Maximum Screen Resolution: 4K
Dependencies
OpenCV
Tesseract-OCR
PyAutoGUI
Pillow
numpy
Author
Aly-Joseph
Version
1.0.0
Last Updated
2026-01-31
1---2name: vision-automation3description: Vision Automation Skill4---5# Vision Automation Skill67## Overview8The Vision Automation skill provides screen reading, object detection, and automated clicking capabilities using computer vision and OCR.910## Slug11`vision-automation`1213## Features14- Screen text recognition (OCR)15- UI element detection16- Automated clicking on detected elements17- Visual search and locate18- Screenshot capture and analysis1920## Implementation21- **Modules**: `vision_engine.py`, `vision_ocr.py`, `vision_click.py`22- **Primary Libraries**: `OpenCV`, `Tesseract-OCR`, `PyAutoGUI`23- **Detection Methods**: Template matching, feature detection2425## Configuration26```python27from vision_engine import read_screen_text, click_on_text2829# Read text from screen30text_lines = read_screen_text()3132# Click on detected text33success = click_on_text("Submit Button")34```3536## Voice Commands37- "Jarvis, what is on screen?"38- "Jarvis, click on [element]"39- "Jarvis, read the text"40- "Jarvis, find the button"4142## Performance43- Detection Speed: 100-500ms44- Accuracy: 90%+ for UI elements45- Maximum Screen Resolution: 4K4647## Dependencies48- OpenCV49- Tesseract-OCR50- PyAutoGUI51- Pillow52- numpy5354## Author55Aly-Joseph5657## Version581.0.05960## Last Updated612026-01-31
Run npx skillmds@latest add aly-joseph/vision-automation in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.
Vision Automation Skill It is listed under Coding & Dev Tools on SkillMD.
This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.
This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.
Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.
Aly-Joseph (@aly-joseph) published this skill. Their other Agent Skills are listed on their SkillMD profile.
Aly-Joseph