mirrors/tinygrad
2
0
Fork 0
mirror of https://github.com/tinygrad/tinygrad.git synced 2026-06-24 02:14:17 +00:00
Code Issues Projects Releases Packages Wiki Activity
11,429 commits 280 branches 15 tags 325 MiB
Commit graph

1,227 commits

Author SHA1 Message Date
hooved
5d9035f5a6
 		Eval for Stable Diffusion mlperf (#12316) 
* add diff

* rerun ci

* refactor beam workaround, add test

* fix conflict

* linting
 2025-10-02 02:35:38 -04:00
hooved
0f804c9a83
 		Stable Diffusion model init for mlperf (#12314) 
* include clip pr diff

* updated unet and sd init

* dehardcode default device

* revert beam hang workaround

---------

Co-authored-by: chenyu <chenyu@fastmail.com>
 2025-10-02 02:28:41 -04:00
hooved
969a1b35ca
 		LR scheduler for Stable Diffusion mlperf training (#12201) 
* add lr scheduler for stable diffusion training

* add lr scheduler test

* rerun ci

* rerun CI

* use np for testing

* move test to CI path

* remove unneeded copy
 2025-09-30 21:21:08 -04:00
George Hotz
7129419500
 		fix cifar training in RANGEIFY (#12355) 
* fix cifar training in RANGEIFY

* even more wino fuse

* bugfix

* test to show issue
 2025-09-30 15:59:19 +08:00
wozeparrot
2a0caa09c2
 		push copy to disk (#12348) 2025-09-29 21:55:05 -07:00
chenyu
3a480b858f
 		use more getitem in gpt2 (#12343) 2025-09-29 23:08:03 -04:00
hooved
c2689c505e
 		Clip model updates for Stable Diffusion mlperf training (#12313) 
* stable diffusion mlperf clip changes

* add clip tests

* set gelu as attribute

* add more tests

* factor out GPUS

* rerun CI

* add imports to if blocks

* remove unneeded axis

* add clip tests to CI

* move clip tests

* add deps, disable max buf size
 2025-09-29 21:50:14 -04:00
George Hotz
baf3b60cfb
 		fix gpt2 on rangeify (#12335) 2025-09-29 19:16:44 +08:00
George Hotz
b899392f30
 		fix llm app with rangeify (#12334) 
* fix llm app with rangeify

* add gpt2 contiguous also
 2025-09-29 18:42:44 +08:00
chenyu
84d2d047ea
 		Tensor.pad_to and Tensor.shrink_to (#12210) 
most of the time i want this instead of spelling out the args

also add more input validation to shrink
 2025-09-16 12:24:55 -04:00
hooved
3a9db08b49
 		download data and ckpts for sd train/eval (#12170) 2025-09-15 00:31:45 -04:00
Steven Shi
25b1bc8eff
 		added top k sampling to examples/mamba (#12061) 2025-09-14 15:27:34 -04:00
chenyu
d2316ba91a
 		don't validate output in sdxl with fakeweights (#12160) 
NULL backend passed validation before because both desired and actual went through NULL backend
 2025-09-13 21:47:51 -04:00
chenyu
0e266f376c
 		ops_gpu -> ops_cl (#12103) 2025-09-10 15:15:48 -04:00
chenyu
0599e86186
 		replace hardcoded GPU in llama debug msg (#12102) 2025-09-10 13:56:40 -04:00
Sieds Lykles
5b73076e48
 		assert benchmark times (#12042) 
* assert jitted times in openpilot

* better error

* better error

* add ASSERT_MIN_STEP_TIME to more models

* t is step_times

* update benchmark times

* update times
 2025-09-09 23:40:02 +02:00
wozeparrot
d16cc6c012
 		feat: resume ckpt (#11970) 2025-09-02 15:47:48 -07:00
wozeparrot
7c21271a5f
 		feat: end_lr envvar (#11953) 2025-09-01 14:53:07 -07:00
wozeparrot
7e68045fb2
 		feat: small llama3 training (#11829) 2025-08-31 13:41:47 -07:00
NoahKusaba
0838021753
 		remove np from beautiful_cifar (#10988) 
* remove np from beautiful_cifar

* remove np from cifar

* rename variable and rename tensor.arrange to just tensor.randperm

---------

Co-authored-by: chenyu <chenyu@fastmail.com>
 2025-08-29 19:34:16 -04:00
chenyu
e39b25cd36
 		upcast float exp to at least float32 (#11758) 
* upcast float exp to at least float32

* unlucky seed
 2025-08-22 20:16:34 -04:00
wozeparrot
b979162c5d
 		llama3 eval train (#11706) 2025-08-20 19:56:35 -04:00
chenyu
dbd3b67657
 		clamp GRAD_CLIP_NORM in llama (#11761) 2025-08-20 19:55:50 -04:00
George Hotz
9366a23eb0
 		test backward in test_tiny (#11697) 
* test backward in test_tiny

* empty
 2025-08-16 20:29:39 -07:00
chenyu
e9d0027591
 		llama MP realize weight after shard (#11672) 
* llama MP realize weight after shard

prevents memory spike on device 0

* empty weight for FAKEDATA
 2025-08-14 16:17:46 -04:00
kevvz
e2873a3a41
 		[bounty] Muon optim (#11414) 
* newton schulz

* add muon + move newton schulz to tensor

* compact newton schulz

* better tests

* cleanup

* add comments for muon

* cleanup

* add export with tests

* match muon optim with test optim

* cleanup

* unsed import

* correct comment

* whitespace

* move export

* muon test fix

* match reference impl + tests

* remove export by moving muon device

* add credit

* cleanup

* remove print

* spacing

* spacing

* comma

* cleanup

* removal

* fix tests + optim momentum

* consistent is not/ not

* more consistency

* fix test

* cleanup

* fix the nones

* remove comment

* cast

* comment

* comment

* muon teeny test

* muon flag beautiful mnist

* set steps

* steps as hyperparam

* match default test steps

* name

* large cleanup

* dont care about steps

* nesterov false default

* match each other impl

* steps

* switch nest

* swap defaults

* update docstring

* add no nesterov test

* ban fuse_optim

* prints

* classical momentum

* alternative condition

* recon

* pre + post wd

* false default

* detach

* signature changes

* context

* swap order

* big cleanup

* 0 step instead

* parity

* remove fuse

* remove fused

* better paper

* assert message

* correct shape check + eps

* multidim

* add eps

* cleanup

* correct assert message

* lint

* better tests

* naming

* ns_steps,ns_params

* update docstring

* docstring

* match sgd and muon together

* sandwich

* add back fused

* parity

---------

Co-authored-by: George Hotz <72895+geohot@users.noreply.github.com>
 2025-08-13 14:27:55 -04:00
Sardor
ca7a641442
 		fix bugs at examples/yolov3.py (#11614) 
* Update load_weight. Give valid model url

* Fix bug in iou function
 2025-08-11 21:14:47 -04:00
chenyu
630edcffd8
 		remove .float calls in olmoe (#11610) 
still matches torch
 2025-08-10 20:33:22 -04:00
chenyu
ef17af85c6
 		remove .float call in llama logit (#11598) 
* remove .float call in llama logit

* bfloat item
 2025-08-10 00:02:18 -04:00
chenyu
7338ffead0
 		small beautiful_mnist update (#11596) 
gather is fast now. there's a conv/bw kernel that only gets fast with BEAM, but whole thing runs < 5 seconds now regardless
 2025-08-09 19:51:14 -04:00
chenyu
45baec1aab
 		model parallel llama (#11588) 
MP=8 GRADIENT_ACC_STEPS=3 BS=1 DEFAULT_FLOAT=bfloat16 OPTIM_DTYPE=bfloat16 LLAMA3_SIZE=70B SEQLEN=512 PYTHONPATH=. MODEL=llama3 python3 examples/mlperf/model_train.py
 2025-08-09 16:54:27 -04:00
chenyu
702e38dc19
 		remove FUSE_ARANGE_UINT (#11567) 
also add IGNORE_OOB=1 to bert runs. lowered BS on tinybox to 90 since 96 oom during eval without reset
 2025-08-07 16:49:06 -04:00
wozeparrot
7ae4335127
 		feat: generate blend index (#11566) 2025-08-07 14:20:28 -04:00
George Hotz
21570545d3
 		move view pushing to codegen, try 2 (#11534) 
* move view pushing to codegen, try 2

* fix up some linearizer tests

* fix test search

* fix test schedule

* delete that test

* fix test arange

* fix a few tests

* update tests

* push views

* ebs cleanup

* fix local/reg

* test and lint

* fix more tests

* test cleanups

* skipped that one
 2025-08-06 15:58:38 -07:00
wozeparrot
2d5bdc939d
 		faster llama3 dataloader (#11540) 2025-08-06 18:25:57 -04:00
chenyu
f7965f85aa
 		Revert "feat: faster index building (#11462)" (#11478) 
This reverts commit 3a4deb08d2.
 2025-08-02 12:50:48 -04:00
wozeparrot
3a4deb08d2
 		feat: faster index building (#11462) 
* feat: faster index building

* feat: correct training samples
 2025-08-02 11:50:18 -04:00
chenyu
9e8e6b45ab
 		grad acc train llama (#11467) 
* grad acc train llama

* log step time
 2025-08-01 15:54:50 -04:00
chenyu
7ad7329257
 		data parallel train llama (#11466) 2025-08-01 12:13:51 -04:00
George Hotz
8ff03806e8
 		add llama layers (#11460) 
* add llama layers

* add contig bw for speed
 2025-07-31 16:28:04 -07:00
wozeparrot
6252f7770e
 		feat: fake data (#11447) 2025-07-30 17:18:20 -07:00
chenyu
e300451f3a
 		update llama3 (#11446) 
`LR=1e-4 TRAIN_ON_VAL=1 DEFAULT_FLOAT=bfloat16 FUSE_ARANGE=1 JITBEAM=2 OPTIM_DTYPE=bfloat16 LLAMA3_SIZE=1B WARMUP_STEPS=36 DECAY_STEPS=360 SEQLEN=512 PYTHONPATH=. AMD=1 AMD_LLVM=0 MODEL=llama3 python3 examples/mlperf/model_train.py` trained to 7
 2025-07-30 19:34:21 -04:00
wozeparrot
5fb975351a
 		feat: flag for training on val (#11441) 2025-07-30 14:29:45 -07:00
wozeparrot
825b6a2505
 		feat: llama3 dataloader (#11340) 2025-07-30 13:27:55 -07:00
George Hotz
842184a1ab
 		rename kernelize to schedule, try 2 (#11305) 2025-07-21 11:18:36 -07:00
nimlgen
cc3c1e4c14
 		hcq: move cpu to hcq (#11262) 
* hcq: move cpu to hcq

* import time

* upd

* fix

* windows support

* hm

* cleaner

* fix timer

* fix timing

* std is ns

* skip profiler

* mypy

* cleaner

* cleanups

* after merge

* default is back
 2025-07-21 15:10:38 +03:00
chenyu
85ddd72038
 		simpler grouptop in hcopt (#11219) 
* simpler grouptop in hcopt

keep the only perf relevant conditions and the rest is handled by try except

* update openpilot read image count
 2025-07-13 16:06:09 -04:00
chenyu
a0438012af
 		remove Kernel.get_program [pr] (#11203) 2025-07-12 20:50:29 -04:00
geohotstan
5ce278b245
 		OnnxRunner file as input (#10789) 
* file path as input and have parse be in OnnxRunner.__init__

* modelproto_to_onnxrunner -> modelproto_to_runner

* whoops, fix import

* oh flakiness again, is it because it's getting gc-ed?

* small changes

* CI flaky so just move compile4 fix in

* copy typing of onnx_load

* actually can just import onnx_load instead of onnx.load

* fix external_benchmark_openpilot

* fix onnx_runner test to use onnx_helper

* rerun CI

* try run_modelproto

* spam CI a few times

* revert run_modelproto since that's flaky also

* no external onnx_load usage except onnx.py

* cursor tab complete is evil. Snuck a darn sorted in. But does order change result? Why?

* model_benchmark 193s -> 80s, add OnnxRunner.to()...

* minimize diff and clean up

* device can be None, weird but eh

---------

Co-authored-by: chenyu <chenyu@fastmail.com>
 2025-07-12 14:27:46 -04:00
chenyu
b072be0e2d
 		hotfix whisper main script (#11184) 2025-07-11 12:34:00 -04:00