My Local AI Invented 162 Python Packages. I Reran Everything From Scratch, and 161 Came Back.
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine I...
First seen by Oriel on .
AI summary
Phantom Packages studies reproducible hallucinated Python package names from local coding models and provides an offline detector plus installation guard that blocks ghost packages before they reach pip.
- Three Ollama models produced 212 nonexistent packages among 1,410 install suggestions, with rates from 8.1% to 20.7% and 30.0% on obscure tasks.
- Rerunning the study reproduced 1,183 of 1,199 outputs and 161 of 162 ghost names, indicating that many hallucinations are predictable rather than random.
- TabPFN-3.5 uses package-name and generation features without registry lookups, achieving AUROC 0.975 on unseen prompts and 0.948 on unseen names.
- The Claude Code hook blocked all 212 ghost installs, wrongly stopped 13 of 1,165 real ones, and added 1.7 seconds for real installs.
- The project targets Python and PyPI workflows with phantom check, a Claude Code hook, and a GitHub Action; gradient boosting performed better at precision at 90% recall.
#python #llm agents #pypi #tabpfn
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Reactions
Total public reactions reported by the source; distinct from points and likes.
- DEV Community: 59, latest observation ; +43 over observed period since
Comments
Comment count reported by the source.
- DEV Community: 36, latest observation ; +32 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| DEV Community | 4955502 | https://dev.to/soumyadeepdey/my-local-ai-invented-162-python-packages-i-reran-everything-from-scratch-and-161-came-back-4j73 Read discussion |