Pull down to go back
Production LLM Systematically Violates Tool Schema Constraints to Invent UI Features; Observed Over ~2,400 Messages [D]

Production LLM Systematically Violates Tool Schema Constraints to Invent UI Features; Observed Over ~2,400 Messages [D]

生產環境大型語言模型系統性違反工具架構限制以創造UI功能;觀察超過2,400則訊息 [D]

Writeup of an emergent behavior I observed in production. Posting here for methodological critique and pointers to related work. Context: a conversational AI system (single-tool tool schema with 5 enumerated action types, each with explicit description). Observed across ~2,400 messages, the model uses the enum correctly most of the time. When it deviates, the deviation is the point of interest. Key observations: The action types are repurposed consistently across unrelated conversations: in