mirrors/tinygrad
2
0
Fork 0
mirror of https://github.com/tinygrad/tinygrad.git synced 2026-06-24 02:14:17 +00:00
Code Issues Projects Releases Packages Wiki Activity
12,211 commits 280 branches 15 tags 325 MiB
Commit graph

340 commits

Author SHA1 Message Date
wozeparrot
4b5d3bda1f
 		llama3: data seed (#14681) 2026-02-11 19:04:40 -08:00
wozeparrot
a60220bed9
 		llama3: move dl to numpy & jit more (#14677) 
Co-authored-by: George Hotz <72895+geohot@users.noreply.github.com>
 2026-02-10 18:16:40 -08:00
wozeparrot
69574542ab
 		fix: use correct fa implementation in eval (#14651) 2026-02-09 18:20:44 -08:00
qazal
50d3f6cea5
 		EVAL_BS=0 in llama profile (#14643) 2026-02-10 00:49:43 +09:00
nimlgen
e087c58ae0
 		print tables in llama/profile.sh (#14639) 2026-02-09 12:32:54 +03:00
qazal
b7e3fbe07e
 		llama: add VIZ=-1 to dev_run (#14583) 
* llama: add VIZ=-1 to dev_run

* readme

* cleaner

* add profile.sh script

* better grouping of options

* add other row

* readme edits

* work
 2026-02-06 22:59:22 +09:00
chenyu
d57d24c7d4
 		Buffer.as_buffer -> Buffer.as_memoryview [pr] (#14535) 
it casts to memoryview. also inline the as_typed_buffer checks to Tensor._data
 2026-02-04 11:31:11 -05:00
George Hotz
dd2de4f838
 		rename all DEFINE_GLOBAL to PARAM (#14511) 2026-02-03 15:09:38 +08:00
wozeparrot
bbcd3d67a3
 		fa: faster (#14453) 2026-02-02 21:34:17 -08:00
qazal
616e9c1483
 		CDNA assembly gemm in tensor.py with flag (#14310) 
* work

* work

* the assembly

* remove the old one

* remove ws bufs, assert splitk

* notes cleanup

* work

* gemm args

* gemm in mixins would be nice

* add gemm gradient

* print counters

* the realize is for DEBUG=2 aesthetics

* dedup

* rewrite to python dsl, no list copies

* leave that

* add B, M, N, K to gemm name

* it's M0 not NULL

* fp16 support

* test cleanup + more gemms

* work from viz

* more work

* gemm batch_size

* xccg path work

* tiny comments on the label naming

* s_waitcnt
 2026-01-31 22:34:14 +09:00
George Hotz
c9a3ddb341
 		benchmark llama walltime script (#14454) 
* benchmark llama walltime script

* adj layers
 2026-01-31 10:21:54 +08:00
George Hotz
f5346d6a1a
 		fix USE_ATOMICS for non float dtypes and make it the default (#14444) 
* embedded multistep test

* complex test

* with jit

* fix dtypes and reenable USE_ATOMICS

* that test didn't catch anything
 2026-01-31 09:44:16 +08:00
George Hotz
ee2c78709d mlperf/llama: disable USE_ATOMICS for now 2026-01-31 00:42:08 +08:00
George Hotz
838cd078bc
 		use atomics for embedding backward (#14400) 
* embedding is slow

* failing

* float is fine

* null

* it fails

* simplify embedding with broadcasting

* ATOMIC_ADD incoming

* min change

* simpler test

* better test

* fix test

* real test

* simpler

* cleanups

* types and names

* _zero_kernel

* grad multi

* hack

* none

* multi unshard

* more for call

* don't tag in call

* good

* call_multi

* call_multi wow claude is useless

* embedding backward mutli test

* test passes

* fix as_param

* shape_to_shape_arg

* add clip

* before cast

* fix spec=2, use atomics
 2026-01-30 18:10:59 +08:00
George Hotz
793afbd473
 		simplify nn.Embedding, support AFTER in CUSTOM_KERNEL (#14419) 2026-01-29 17:22:13 +08:00
wozeparrot
4845e42135
 		llama3 gradacc fixes (#14414) 2026-01-28 19:12:39 -08:00
nimlgen
aec1ae0de1
 		llama: set manual_seed (#14409) 2026-01-28 14:40:00 -08:00
George Hotz
0c6b3f50aa
 		add marker to llama training (#14401) 2026-01-28 22:44:28 +08:00
qazal
5bffa17f82
 		llama train: better NULL=1 EMULATE=AMD_CDNA4 dev experience (#14395) 
* beam opens devices

* switch to hip renderer

* amd: true?

* llvm true is for test_autogen
 2026-01-28 17:31:22 +09:00
wozeparrot
e496547720
 		llama3 gradacc (#14291) 2026-01-27 19:48:10 -08:00
chenyu
db010a31be
 		IGNORE_OOB -> CHECK_OOB [pr] (#14374) 
flip the meaning
 2026-01-27 12:20:59 -05:00
wozeparrot
a987a4abc3
 		feat: llama8b dev_beam.sh (#14358) 2026-01-26 14:51:23 -08:00
nimlgen
e152f1b0f5
 		llama: use ALL2ALL (#14353) 2026-01-26 22:01:53 +03:00
George Hotz
11ce1e847d llama train: null device support 2026-01-26 08:53:05 +08:00
wozeparrot
963c59ebdb
 		fix: pull fixes from gradacc branch (#14296) 2026-01-22 23:07:54 -08:00
wozeparrot
c1d14ea832
 		llama8b train fixes (#14264) 2026-01-20 20:34:47 -08:00
wozeparrot
ba90e1b52e
 		feat: script to run llama8b training (#14239) 2026-01-20 12:44:06 -08:00
wozeparrot
a879b54234
 		tk: fa jit fix (#14170) 2026-01-16 16:38:45 -08:00
b1tg
0fbc551622
 		train bert with fp8 (#13874) 
* fp8 train

* clean

* lint

* test fix from #13439

* skip first/last layer

* rm __init__, restore unroll <=32 check

* tests

* clean test, remove unused

* multi-gpu test, clean quantize_to_fp8

* remove bert contiguous

* run script

* test: better check

* run script search

* add seed in bert data shuffle

* move script to mi350x folder

---------

Co-authored-by: chenyu <chenyu@fastmail.com>
 2026-01-09 09:21:59 -05:00
b1tg
241f0402b4
 		add seed in bert data shuffle (#14054) 2026-01-07 10:02:05 -05:00
chenyu
da1cb6a9ec
 		update llama dataloader (#13825) 
separate creating dataset from itererating over the dataset to not create eval data for each eval
 2025-12-24 17:42:08 -05:00
chenyu
903753c60c
 		llama wandb logging (#13822) 2025-12-24 10:24:59 -05:00
chenyu
27d899ce97
 		TRAIN=0 to only eval llama (#13804) 2025-12-22 11:55:46 -05:00
chenyu
39d962106f
 		update llama logging (#13803) 
```
REWRITE_STACK_LIMIT=1000000 SMALL=1 BASEDIR=/raid/datasets/c4-8b SAMPLES=1000 BS=8 DP=8 DEFAULT_FLOAT=bfloat16 OPTIM_DTYPE=bfloat16 LLAMA3_SIZE=8B SEQLEN=1024 PYTHONPATH=. MODEL=llama3 python3 examples/mlperf/model_train.py

    1 93.44 s run, 11.8750 loss, 0.000000000001 LR, 642.43 GB used,  19644.30 GFLOPS
    2 101.78 s run, 11.8750 loss, 0.000000000001 LR, 1454.57 GB used,  17039.35 GFLOPS
    3 7.34 s run, 11.8750 loss, 0.000000000002 LR, 1454.57 GB used, 236258.78 GFLOPS
    4 4.32 s run, 11.8750 loss, 0.000000000002 LR, 1454.57 GB used, 401488.40 GFLOPS
    5 4.36 s run, 11.9375 loss, 0.000000000003 LR, 1454.57 GB used, 398116.13 GFLOPS
    6 4.32 s run, 11.8750 loss, 0.000000000003 LR, 1454.57 GB used, 401878.60 GFLOPS
    7 4.34 s run, 11.8750 loss, 0.000000000004 LR, 1454.57 GB used, 399822.57 GFLOPS
    8 4.35 s run, 11.8750 loss, 0.000000000004 LR, 1454.57 GB used, 398512.24 GFLOPS
    9 4.36 s run, 11.8750 loss, 0.000000000005 LR, 1454.57 GB used, 397832.61 GFLOPS
   10 4.40 s run, 11.8750 loss, 0.000000000005 LR, 1454.57 GB used, 394520.83 GFLOPS
```
 2025-12-22 11:28:29 -05:00
George Hotz
45c459848d
 		remove more stale stuff (#13765) 
* remove more stale stuff

* remove disassemblers/adreno

* stale
 2025-12-19 17:14:56 -04:00
chenyu
7cd7593c5d
 		add script to train bert on mi350x (#13743) 
adapted from mi300 config
 2025-12-17 16:54:04 -05:00
chenyu
e428fbfab6
 		verify dtype of llama model params (#13719) 2025-12-16 12:32:02 -05:00
chenyu
6cad622f59
 		don't FREE_INTERMEDIATE in bert (#13684) 
hangs green hcq consistently after an hour of training
 2025-12-14 14:27:42 -05:00
chenyu
fcaed1e1dd
 		don't use empty in bert fake data (#13661) 
somehow jit does not count empty as input
 2025-12-12 15:59:50 -05:00
chenyu
01e9ad0d52
 		clean up bert next_data (#13650) 
train iter was designed to never stop for both real and fake data
 2025-12-11 22:56:28 -05:00
chenyu
5034c6fb37
 		reenable FREE_INTERMEDIATE for bert (#13639) 
* reenable FREE_INTERMEDIATE for bert

* comment
 2025-12-10 12:08:09 -05:00
chenyu
016a59cafa
 		remove contiguous and use where in EmbeddingBert (#13632) 2025-12-09 15:49:21 -05:00
chenyu
2471b49e45
 		minor bert / llama change from grad acc branch (#13622) 
* minor bert / llama change from grad acc branch

* revert those
 2025-12-08 16:04:14 -05:00
chenyu
b981b6f89e
 		remove old llama grad_acc (#13611) 
* remove old llama grad_acc

* GRADIENT_ACC_STEPS=1
 2025-12-07 13:03:47 -05:00
chenyu
4562f217e1
 		more bert updates (#13597) 
prep split jit
also lower BS to 72
 2025-12-06 08:32:43 -05:00
chenyu
cb4c6324ef
 		revert bert grad accumulation (#13596) 
prep for the new split jit style
 2025-12-05 17:30:08 -05:00
chenyu
74db65cf72
 		update mlperf bert LOGMLPERF (#13065) 2025-11-02 15:26:37 -05:00
chenyu
70dd297a05
 		BS=96 for bert (#12675) 
96 trains fine now
 2025-10-14 09:07:43 -04:00
chenyu
77b5e6774e
 		fix bert training config (#12647) 
FREE_INTERMEDIATE=0 REWRITE_STACK_LIMIT=500000
 2025-10-13 15:03:47 -04:00
chenyu
0f776c6e46
 		examples/mlperf/training_submission_v6.0 (#12644) 
copied from v5.1
 2025-10-13 09:58:25 -04:00