agents-io/PokeClaw

Layered Agent Stack

Core Abstractions

• UiElement Tokens: Compressed representation of AccessibilityNodeInfo trees mapped to Gemma's vocabulary space using custom BPE encoding
• ToolCall Schema: Structured JSON defining action (tap/swipe/type), target (element hash), and params with coordinate bounds
• SessionManager: Ephemeral context window management with LRU eviction for UI state history and KV-cache persistence

Architectural Tradeoffs

The reliance on AccessibilityService introduces 50-120ms latency versus native input injection but ensures compatibility with non-rooted devices. The 4-bit quantized Gemma 4 sacrifices <1% accuracy for 60% memory reduction, enabling operation on 8GB RAM devices. Single-threaded inference serialization prevents race conditions in UI state but limits parallel tool execution.

PokeClaw pioneers the deployment of Gemma 4 as a fully autonomous on-device phone agent, leveraging LiteRT GPU delegates to achieve real-time UI understanding without cloud inference or API keys.

Technical Breakthroughs

1. Quantized Multimodal UI Parsing: Implements group-query attention (GQA) optimization from the Gemma 4 architecture to process screen bitmaps and accessibility trees simultaneously within a 4-bit quantized context window (8k tokens), reducing memory bandwidth by 40%.
2. Hierarchical Accessibility Tree Tokenization: Compresses Android's AccessibilityNodeInfo hierarchy using a custom BPE tokenizer trained on 50K+ UI layouts, reducing average context length by 73% compared to raw XML serialization while preserving semantic element relationships.
3. LiteRT NPU Delegation: Utilizes the GpuDelegate and NnApiDelegate with asymmetric quantization (per-channel for weights, per-tensor for activations) to achieve 12-15 tokens/second on Snapdragon 8 Gen 3 Hexagon NPU versus 3-4 tokens/sec on CPU.
4. Deterministic Tool-Calling via Constrained Decoding: Implements grammar-based sampling using LiteRT's external delegate to force valid JSON schema output for UI actions, eliminating hallucinated coordinates through finite-state machine validation during token generation.
5. Ephemeral Privacy Architecture: Zero-persistence design where screen captures and inference tensors are stored in android.ashmem shared memory and wiped post-action via Arrays.fill() and System.gc() hints, preventing forensic data recovery.

Implementation Snippet

val interpreter = Interpreter(
    modelFile, 
    Interpreter.Options().apply {
        addDelegate(GpuDelegate())
        numThreads = 4
        useXNNPACK = true
        setCancellable(true)
    }
)
// Constrained decoding for tool calls
val grammar = ToolCallGrammarBuilder()
    .addAction("tap", boundsParam())
    .addAction("swipe", vectorParam())
    .build()
interpreter.setExternalContext(grammar)

Benchmark Metrics

Scalability Constraints

• Context Bottleneck: Gemma 4's 8K context limit restricts historical action memory to ~5-7 previous screens, limiting multi-step task complexity
• Thermal Throttling: Sustained inference triggers CPU downclocking after 3-4 minutes of continuous operation, degrading token generation speed by 35%
• Accessibility API Bounds: Cannot interact with secure windows (banking apps, VPN dialogs) due to Android's FLAG_SECURE restrictions, requiring fallback to manual intervention

Optimization Techniques

Employs speculative decoding via LiteRT's experimental GPU backend to predict UI element indices, reducing token generation steps by 30%. KV-cache persistence across turns maintained in MappedByteBuffer to avoid recomputation of attention weights for static UI elements.

Competitive Landscape

Production Adoption Profiles

1. Enterprise MDM: Deployed in financial services for automated compliance testing on managed devices where cloud screen sharing violates SOC2 requirements
2. Accessibility Services: Used by motor-impairment users for voice-controlled phone navigation without internet connectivity in rural deployments
3. Privacy-First Consumers: Adopted by security researchers and journalists requiring air-gapped device automation for sensitive source communication
4. QA Automation: Mobile dev teams using PokeClaw for offline UI testing in Faraday cage environments where cloud connectivity is prohibited
5. Edge AI Research: Academic labs studying autonomous agent behavior without cloud inference bias or API rate limiting

Integration & Migration

Migrating from cloud-based agents (e.g., OpenAI's Operator) requires replacing REST API calls with local BroadcastReceiver intents. Integration with existing Android automation stacks possible via Intent delegation to PokeClawService. The project exposes an AIDL interface (IPokeClawController) for third-party apps to request autonomous actions without direct AccessibilityService access.