GitHub Tool Lets Text-Only Models Use Vision in Codex

Project Overview

Anionex released

on GitHub. The repository supplies a proxy script that intercepts calls to Codex's built-in view_image tool. When a text-only model such as DeepSeek attempts the call, the proxy returns a description instead of an error. The code targets Codex sessions connected to DeepSeek and has been tested in actual agent runs. It also bundles optional CLI tools for image question answering, OCR, and visual grounding.

How the Proxy Works

The main file codex-vision-proxy.py sits between the agent and Codex's native image handler. When view_image is invoked, the script forwards the image to a multimodal model, caches the result, and injects the description back into the conversation. Multi-image requests are processed concurrently, and cross-turn caching prevents repeated calls for the same image within a session. No extra MCP servers or custom skills are required for the basic proxy path.

Installation follows the steps in AGENT_INSTALL.md. Users copy .env.example, set the multimodal endpoint, and run the proxy script before starting Codex. The repository states that the setup has been verified with DeepSeek V4 for both single-turn screenshot analysis and multi-step GUI debugging.

Optional Vision Toolkit

Beyond the proxy, the repository includes a set of CLIs under skills/vision-tools. The glance command supports image Q&A and OCR. The ground command performs visual grounding, allowing the model to output coordinates for UI elements. These tools can be installed separately for agents outside Codex. Tests in the repo show DeepSeek locating chess pieces on screen and issuing click coordinates without manual annotation.

Trade-offs appear in latency. Each view_image call now routes through an external multimodal model, adding network time. Description quality depends on the chosen vision model; generic captions may omit fine details needed for precise code edits. The project does not claim to match native multimodal performance inside the same model family.

Practical Limits

The proxy only handles the view_image surface. Other multimodal behaviors, such as direct image generation or real-time camera input, remain unsupported. Cache invalidation relies on conversation turns rather than image content hashes, so edited screenshots may return stale descriptions until the turn count advances. Users must supply their own multimodal API keys and manage rate limits separately from Codex quotas.

FAQs

Does the proxy require changes to the Codex client? No. It runs as a standalone script that the agent launches before connecting to Codex.

Can the toolkit be used without the proxy? Yes. The glance and ground CLIs install independently and expose commands that any text agent can call.

What happens on image description failure? The proxy returns the error message from the multimodal backend instead of crashing the Codex session.

---

📖 Related articles

• Agentic Coding: Una Trappola per lo Sviluppo Software?
• File agents.md: utili per gli agenti di coding?
• Phantom su GitHub: L'AI co-worker auto-evolvente e sicuro