Oriel — AI & technology worth noticing, every day.

My Local AI Invented 162 Python Packages. I Reran Everything From Scratch, and 161 Came Back.

Post · https://dev.to/soumyadeepdey/my-local-ai-invented-162-python-packages-i-reran-everything-from-scratch-and-161-came-back-4j73

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine I...

First seen by Oriel on .

AI summary

Phantom Packages studies reproducible hallucinated Python package names from local coding models and provides an offline detector plus installation guard that blocks ghost packages before they reach pip.

  • Three Ollama models produced 212 nonexistent packages among 1,410 install suggestions, with rates from 8.1% to 20.7% and 30.0% on obscure tasks.
  • Rerunning the study reproduced 1,183 of 1,199 outputs and 161 of 162 ghost names, indicating that many hallucinations are predictable rather than random.
  • TabPFN-3.5 uses package-name and generation features without registry lookups, achieving AUROC 0.975 on unseen prompts and 0.948 on unseen names.
  • The Claude Code hook blocked all 212 ghost installs, wrongly stopped 13 of 1,165 real ones, and added 1.7 seconds for real installs.
  • The project targets Python and PyPI workflows with phantom check, a Claude Code hook, and a GitHub Action; gradient boosting performed better at precision at 90% recall.

#python #llm agents #pypi #tabpfn

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Reactions

Total public reactions reported by the source; distinct from points and likes.

  • DEV Community: 59, latest observation ; +43 over observed period since

Comments

Comment count reported by the source.

  • DEV Community: 36, latest observation ; +32 over observed period since

Aliases

SourceExternal IDURL
DEV Community4955502https://dev.to/soumyadeepdey/my-local-ai-invented-162-python-packages-i-reran-everything-from-scratch-and-161-came-back-4j73 Read discussion