457 commits

Author	SHA1	Message	Date
chenyu	33b635d23a	Tensor.train -> TRAINING [PR] (#16705) ... * Tensor.train -> TRAINING [PR] * doc	2026-06-22 15:13:22 -04:00
wozeparrot	fe9b19b12d	llama: more mp mem fixes (#16701) ... * llama: more mp mem fixes * clean: unused * fix: batch	2026-06-22 10:54:35 -04:00
wozeparrot	70cac72781	llama: realize weight init (#16623)	2026-06-15 23:00:19 -07:00
qazal	f998b9930a	fp8 gemm inv_scale in epilogue (#16625) ... * fuse scale * remove python inv_scale * more inv_scale removal * more cleanups * cleaner * diff polish * work * rename * simpler * simpler * compute * c * Revert "c" This reverts commit 8941fec7ca. * Revert "compute" This reverts commit 9db573a6d3. * Revert "simpler" This reverts commit 910ad33f87. * Revert "simpler" This reverts commit bf75d235a1. * s_g * update types * less diff noise * remove	2026-06-15 18:44:41 +09:00
qazal	4d34590b7d	llama: less E kernels (#16517)	2026-06-12 19:49:25 +09:00
wozeparrot	e770805d21	llama: mxfp8 (#16574)	2026-06-11 22:15:24 -07:00
wozeparrot	f11f63007d	llama: immediate scaling on flag (#16494)	2026-06-04 10:30:00 -07:00
chenyu	0faaf6df26	remove kwargs from arange and linspace [PR] (#16505) ... it used to have requires_grad and device, now both are removed	2026-06-04 10:32:37 -04:00
qazal	3b1a5f9770	llama: a_bT and aT_b bf16 gemms (#16487) ... * hk_bf16_gemm * enable in 8b * cleanups * rename to USE_HK_BF16_GEMM * work * work * work * work * change the gemms * work * work * set as default * work * change	2026-06-04 23:30:21 +09:00
wozeparrot	fd13080636	deviceless const skip axis check (#16496)	2026-06-03 19:13:20 -07:00
wozeparrot	7dcfd144b6	llama: columnwise fp8 scaling (#16480)	2026-06-02 18:55:45 -07:00
wozeparrot	aab9a5a8a3	llama: allow specifying layer count (#16464)	2026-06-01 15:36:04 -07:00
wozeparrot	6795c2d5c9	llama: zero grad this way (#16445)	2026-05-29 20:25:21 -07:00
wozeparrot	c23652e486	llama: minimize peak init mem (#16440)	2026-05-29 18:00:37 -07:00
wozeparrot	6787de9f52	llama: fix mp (#16434)	2026-05-29 11:21:43 -07:00
wozeparrot	f86966af56	llama: optim amax margin (#16425)	2026-05-28 20:18:11 -07:00
wozeparrot	36c8ff70c1	llama: use old scale for dequant in optim (#16417)	2026-05-28 15:21:19 -07:00
George Hotz	edca5df25a	flip offset and shape in pad and shrink (#16414) ... * flip offset and shape in pad and shrink * dumb test	2026-05-28 11:58:19 -07:00
wozeparrot	dac3743d75	llama: delayed scaling in optim (#16407)	2026-05-27 15:40:03 -07:00
George Hotz	8ee3a37524	shrink/pad use (new_shape, offset) (#16405) ... * shrink uses offset and shape * pad does too * fix	2026-05-27 15:13:08 -07:00
wozeparrot	3a7a6da7d5	llama: fakedata uses real vocab size (#16389)	2026-05-26 18:58:55 -07:00
qazal	b73d2d17b9	viz/cli: add --interval (#16363) ... * interval support * add test_interval * llama uses interval	2026-05-26 03:35:06 +09:00
wozeparrot	68d2102fd2	llama: offload master weights (#16355)	2026-05-25 08:48:13 -07:00
chenyu	31424cda71	Tensor.requires_grad -> is_param (#16325) ... for optimizer	2026-05-21 19:39:57 -04:00
wozeparrot	0c41317a59	llama: update 405b scripts (#16309)	2026-05-21 14:03:34 -07:00
wozeparrot	fb718a5e9d	llama: realize amax (#16308)	2026-05-21 14:00:48 -07:00
wozeparrot	afc5bfa183	llama: remove fused grad accum (#16301)	2026-05-21 09:38:40 -07:00
qazal	e33e058d34	set SPLIT_W13=0 for 8b DP by default (#16302)	2026-05-21 22:09:10 +09:00
Christopher Milan	172f9493e1	move is_dtype_supported to renderer (#16226)	2026-05-20 21:19:37 -04:00
wozeparrot	825f30bf18	llama: apply_grad saves memory (#16275)	2026-05-20 13:14:06 -07:00
wozeparrot	361553c0a8	llama: match flat_llama with model_train (#16269)	2026-05-19 17:25:56 -07:00
wozeparrot	a3d59faef6	llama: don't save weight (#16252)	2026-05-18 17:05:45 -07:00
qazal	18b102f355	llama: also use 7.1 comgr, update startup_walltime.sh (#16253)	2026-05-19 08:59:02 +09:00
qazal	98b8a2b407	llama: use hipcc 7.1 version (#16250)	2026-05-19 08:09:57 +09:00
chenyu	dcee90aa3f	remove requires_grad use in extra/examples (#16238) ... except the ones fed into optimizer	2026-05-16 18:40:26 -04:00
qazal	ebcb7b7cc0	fp8 gemm tests with scale args (#16231) ... * update atol * update fp8 path * more work * update profile.sh	2026-05-16 20:47:58 +09:00
wozeparrot	159694347e	llama: fix running flat_llama (#16224)	2026-05-15 20:16:48 -07:00
chenyu	07a172dbbb	remove noop requires_grad_ calls (#16213)	2026-05-15 13:31:10 -04:00
chenyu	409bb0c9ad	requires_grad cannot be None (#16212) ... final goal is to remove requires_grad, first change the default to True, and don't allow None	2026-05-15 02:01:04 -04:00
wozeparrot	b4d267dfd4	llama: only save when small (#16208)	2026-05-14 17:46:29 -07:00
wozeparrot	88ac2ac1fd	llama: cleanups (#16189)	2026-05-13 17:08:06 -07:00
wozeparrot	e97f2c1114	llama: only gemm + fa custom kernel (#16180) ... * llama: tie store to grad directly * llama: set mp flags * llama: non fused grad fp8 quantize path	2026-05-12 21:03:49 -07:00
wozeparrot	e9359d9e7d	more llama mp fixes (#16151) ... * llama: SPLIT_W13 * llama: fix with no fused kernels * llama: cast to bf16 on non asm_gemm patH * llama: new mp flags	2026-05-11 21:29:23 -07:00
wozeparrot	026688f03f	llama: move to correct dir (#16118)	2026-05-08 19:42:16 -07:00
qazal	a9a87ad8fd	viz/cli: less flags (#16076) ... * viz/cli: merge -s and -i flags * only -t * merge parser * fix	2026-05-08 00:22:40 +09:00
wozeparrot	730fa66bf3	llama speed 6 (#16071)	2026-05-06 20:51:03 -07:00
wozeparrot	ab6218bc92	llama mp fixes (#16050)	2026-05-05 15:35:32 -07:00
wozeparrot	528d35e306	llama speed 4 (#15993)	2026-04-30 17:14:41 -07:00
wozeparrot	0080489abe	llama: use env vars (#15978)	2026-04-29 12:37:15 -07:00
wozeparrot	ef09071073	llama: speed 2 (#15960)	2026-04-28 20:44:37 -07:00