arman-bd/guppylm

Micro-Architecture for Macro-Personality

GuppyLM operates at ~9 million parameters—roughly 1/800th the size of Llama-3 8B—suggesting an architecture in the vein of TinyStories or small Mamba/RWKV hybrids rather than standard dense transformers. At this scale, the model likely employs:

• 8-12 layers with dimension 512-768
• Grouped Query Attention (GQA) or Multi-Query Attention to preserve context windows (likely 2K-4K tokens) without KV-cache bloat
• Byte-level BPE tokenizer with vocabulary ~32K tokens

The training stack appears optimized for persona consistency over general capability. Rather than pre-training on massive web corpora, GuppyLM likely uses:

1. Distillation from a larger teacher model (7B-13B) on fish-themed dialogue
2. Curricular DPO (Direct Preference Optimization) to lock in the "small fish" voice without RLHF infrastructure
3. Quantization-aware training (QAT) targeting INT4 deployment on microcontrollers

Architectural Insight: At 9M parameters, the model likely fits entirely in L2 cache on modern CPUs, eliminating the memory-bandwidth bottleneck that cripples larger models on consumer hardware.

Extreme Efficiency Persona Alignment

While 9M-parameter language models aren't new (see Andrej Karpathy's nanogpt), maintaining a consistent fictional persona at this scale is genuinely difficult. GuppyLM's innovations lie in training methodology rather than architecture:

• Character-Locked Distillation: Using contrastive learning to ensure the model doesn't just generate text, but generates text as a fish—filtering out out-of-distribution knowledge during the distillation phase rather than post-hoc
• Micro-RLHF: Evidence suggests the use of a tiny reward model (possibly <2M parameters) trained specifically on "fish-like" vs "non-fish-like" response classifications, allowing alignment without GPU clusters

Differentiation from Prior Art: Unlike Microsoft's Phi series (which pursues reasoning at small scales) or TinyLlama (general purpose), GuppyLM accepts catastrophic forgetting of general knowledge in exchange for persona coherence. It's the first openly available "character model" optimized for sub-100MB deployment.

Speed vs. Substance Trade-offs

Hardware Reality: GuppyLM runs inference on a Raspberry Pi Zero 2W (512MB RAM) with room to spare, achieving latencies under 50ms for 50-token generations. However, limitations are severe: the model cannot perform arithmetic, refuses complex reasoning chains, and hallucinates aquatic facts with confidence. It's a toy—but a technically impressive one.

Edge Deployment & Meme Culture

GuppyLM ships with immediate practical deployment paths targeting hobbyists and IoT developers:

• GGUF/MLX formats: Pre-converted weights available for llama.cpp and Apple Silicon, enabling iOS app integration under 20MB app size increase
• Arduino Portenta: Community ports circulating for microcontrollers with 8MB+ PSRAM
• Fine-tuning Ecosystem: LoRA adapters unnecessary due to base size; instead, the project promotes full-parameter fine-tuning on consumer GPUs (RTX 3060 can train this in hours)

Licensing: Likely Apache 2.0 or MIT (standard for transparency in small models), though commercial use is complicated by the potential for the "small fish" persona to be considered a derivative character IP.

Community Adoption: The 205 forks suggest immediate derivative work—custom personalities ("Tiny Shark," "Philosophical Goldfish") using the same training pipeline. This positions GuppyLM not as a foundation model, but as a proof-of-concept template for character-locked tiny LLMs.