4 commits

Author	SHA1	Message	Date
George Hotz	ec00cefa5b	llm is the only app (#15779) ... * tinygrad/llm is the only app * upd pyproject * claude refs * scoping * min diff	2026-04-17 10:44:48 +08:00
George Hotz	b9eb5b5d49	clean up the LLM tokenizer (#12653) ... * clean up the LLM tokenizer * simple tokenizer is actually simple * ugh write good code	2025-10-14 14:22:01 +08:00
leopf	4f0ee4e982	BPE tokenizer (#11415) ... * BPE works * refactor tok * oops * basic tests * fix eval * smaller diff * fix error * proper vocab decoding * use regex for splitting * escape ucatrange * full compat --------- Co-authored-by: George Hotz <72895+geohot@users.noreply.github.com>	2025-08-04 09:52:38 -07:00
leopf	557ca7d757	testing SimpleTokenizer against OASST1 (#11214)	2025-07-14 17:09:31 -07:00