mirrors/tinygrad
2
0
Fork 0
mirror of https://github.com/tinygrad/tinygrad.git synced 2026-06-24 02:14:17 +00:00
Code Issues Projects Releases Packages Wiki Activity
13,471 commits 280 branches 15 tags 325 MiB
Commit graph

5,694 commits

Author SHA1 Message Date
JaSpa99
d3d58a37e5
 		Bert: use Tensor.scaled_dot_product_attention (#1528) 
* use scaled attn from Tensor

* add a test for bert

* linter

* no more tokenizer

* without loading weights

* remove prints

* tribute to linter lords

* smaller input and less runs

* small bert
 2023-08-12 08:46:04 -07:00
wozeparrot
29d5801387
 		distributed collectives (#1519) 
* feat: world

* feat: tests

* feat: no more backwards

* feat: recv into

* feat: whoops

* feat: test in ci

* feat: some debug logging

* feat: workflow naming

* feat: need to set pythonpath

* feat: just send to same device

* feat: allreduce

* feat: test

* feat: need contiguous

* feat: test in ci

* feat: exit with correct code

* feat: don't need that

* feat: opencl wait_for just doesn't work

* feat: synchronize on out

* feat: try?

* feat: try again?

* feat: add extra realizes

* feat: print

* feat: seed

* feat: tol

* feat: test ones and zeros

* feat: remove print

* feat: are you just flaky

* feat: seperate scatter and gather?

* feat: just try synchronizing

* feat: remove print again

* feat: bring back difference

* feat: no sync

* feat: revert that

* feat: back to wait_for

* fix: typo
 2023-08-11 10:22:07 -07:00
George Hotz
38fe84d92b
 		cleanup mlops (#1521) 
* cleanup mlops

* that line belongs there
 2023-08-10 19:53:28 -07:00
wozeparrot
7e7c9001e9
 		distributed world (#1481) 
* feat: world

* feat: tests

* feat: no more backwards

* feat: recv into

* feat: whoops

* feat: test in ci

* feat: some debug logging

* feat: workflow naming

* feat: need to set pythonpath

* feat: just send to same device
 2023-08-10 10:00:51 -07:00
geohotstan
07b79f210f
 		llvmir support for bool <-> float casting (#1492) 2023-08-09 13:12:52 -04:00
Jacky Lee
ef5f648e2f
 		Tensor.scaled_dot_product_attention to match torch, used in LLaMA, and tested (#1502) 
* Implement scaled_dot_product_attention and test

* Support attn_mask

* Support is_causal too

* Use in llama

* Don't forget to reshape

* Set requires_grad=False for causal

* Remove staticmethod

* Remove extra spaces
 2023-08-08 23:27:13 -07:00
nimlgen
dabfd7569a
 		use allclose instead of equals in test_jit (#1504) 
Closes #1503
 2023-08-08 22:22:17 -07:00
Yixiang Gao
7c2ea85bb0
 		Raise memory limit for CIFAR test (#1499) 2023-08-08 19:40:56 -04:00
Yixiang Gao
6480a1a180
 		CIFAR 94.03% (#1340) 
* add disk_tensor

* fix jit

* new baseline before whitening

* whitening through torch

* whiting done currently at 91.65%

* 91.99%

* clean up mixup and 92.3%

* clean up 92.30%

* 92.49% before searching for new hyper-parameters

* fix CI

* fix white space

* add whitening init in test

* refactor, update hyperpara, 92.72%

* converting whiting to tinygrad operation

* update CI kernels count for CIFAR

* add pad reflect

* add random crop 92.53%

* update hyperpara 93%

* 93.15% on docker container, need to refactor the assignment for hyper param

* print out weights and bias to be separated

* bias/non-bias params separated

* fix whitespace

* clean up

* refactor hyper-param with dict

* refactor lr schedular params

* fix whitespace

* fix cross entropy loss

* fix whitespace

* move opt hyp to hyp dict

* minor fixup

* adjust model, loss scaling

* 92.74% while using half of compute as before

* update hyp for cutmix

* random shuffle during batches

* clean up

* updating the model

* update ConvGroup

* disable gradients for batchnorm layer weights

* whitespace

* 93.92%

* clean up

* finally 94%git add .!

* rewrite whitening to remove dependency on torch

* whitespace

* remove dependency on torch, 93.91%

* back to 94.03%

* clean up

* update test_real_world
 2023-08-08 15:13:24 -07:00
George Hotz
d24f936501
 		just cmplt (#1493) 
* just cmplt

* fix maximum

* don't save, there's no backward

* ugh, no slot either

* eq is a scam
 2023-08-08 13:58:10 -07:00
nimlgen
932dad1a2b
 		fix cast bool->float in llvmir (#1480) 
Closes #1479
 2023-08-07 21:30:51 -07:00
nimlgen
046fd7437a
 		use fake buffer for external_test_speed_llama.py (#1478) 2023-08-07 22:05:44 -04:00
Diogo
d7d1011f1e
 		Add WEBGPU tests to CI (#1463) 
* webgpu tests

* assert device is webgpu

* missed env set

* exclude failing ci tests

* ignore test file

* changed acc for adam test
 2023-08-06 10:32:01 -07:00
George Hotz
486a9dbfd9
 		speed v torch (#1464) 
* speed v torch

* always print

* change print

* torch speed tee

* all exposed
 2023-08-06 09:32:33 -07:00
George Hotz
2ab282bfec
 		run on update_benchmark too (#1460) 
* run on update_benchmark too

* amd inference test

* name it better

* add 10 CIFAR training steps
 2023-08-06 08:58:37 -07:00
George Hotz
d67e248d9b
 		simple bitcast 2 (#1445) 
* simple bitcast 2

* bc 2

* empty

* Revert "empty"

This reverts commit d8ee083655.
 2023-08-06 00:30:50 -07:00
George Hotz
bf21aec81f
 		do benchmarking (#1451) 
* do benchmarking

* system

* artifact

* go

* name artifact
 2023-08-05 23:35:01 -07:00
nimlgen
1ba8ae62a1
 		Match Torch speed for sum reduction (#1387) 
Co-authored-by: Alexander Edwards <alex@alexedw.com>
 2023-08-05 22:27:33 -07:00
George Hotz
7fa730b506 external model benchmark test 2023-08-05 22:10:48 -07:00
George Hotz
7b8d06c9f1
 		test uops (#1444) 
* test uops

* tests should pass

* improve uops

* precision
 2023-08-05 12:35:56 -07:00
George Hotz
84c430355e
 		fix backends for new style (#1443) 
* fix backends for new style

* fix method cache

* fix fakeless

* llvm blacklist

* fix kernel optimizer
 2023-08-05 11:07:04 -07:00
George Hotz
bd7f4b1249
 		move renamer to linearizer (#1442) 
* move renamer to linearizer

* uops converter

* Delete test_uops.py
 2023-08-05 08:53:25 -07:00
nimlgen
669b406ec6
 		correct children count with lazycache (#1429) 2023-08-05 00:30:16 -07:00
Felix
97a6029cf7
 		Corrected a few misspelled words (#1435) 2023-08-04 16:51:08 -07:00
Francesco Castelli
579f4615a0
 		Add assert for wrong matmul/dot shapes (#1438) 2023-08-04 18:16:56 -04:00
Umut Zengin
52db7d7435
 		inf, -inf support for pad (#1436) 2023-08-04 15:05:25 -04:00
Alex Telon
7325bc914f
 		fix: Context (#1430) 
* Fixed issue in Context

* Cleaned up fix

Now that DEBUG.value = 3 always works we can do so in __new__ as well.
 2023-08-04 10:53:48 -04:00
wozeparrot
801bed4f66
 		Add ops_shm (#1413) 
* feat: add ops_shm

* clean: extra newline

* feat: add test

* feat: ci doesn't like that

* feat: ci still doesn't like that

* feat: skip big test on ci

* feat: testing

* feat: big

* feat: testing again

* feat: reskip test
 2023-08-03 17:40:52 -07:00
chenyu
34f348643b
 		Support constant expand to symbolic shape (#1411) 2023-08-02 21:21:22 -07:00
chenyu
6572ca6835
 		support symbolic expand (#1407) 2023-08-02 20:03:46 -04:00
chenyu
18d0a93f09
 		LazyBuffer.get_variable_buffers() (#1391) 
* LazyBudder.get_variable_buffers()

* remove left_only, add ProdNode

* no vars for OpNode.b

* do not change symbolic vars, remove ProdNode
 2023-08-02 09:01:35 -07:00
Umut Zengin
8889821547
 		Const pad support to pad2d and slice (#1392) 
* slice to pad2d migrate

* Gain line

* Mypy happy

* Mypy happy

* Revert

* whitespace
 2023-08-02 08:58:52 -07:00
Alex Telon
b66361843a
 		Timing and Context can now be used as decorators (#1385) 
* Context and Timing can now be used as decorators

* Using Timing decorator in quickstart.md

The time formating is better and is a useful tool to learn.

Old: Time: 3.5260659999912605
New: Time: 3526.14 ms

* Updated env_vars documentation for Context

* Added test for Context decorator

* Put new import on same line as others
 2023-08-01 17:16:10 -07:00
Diogo
4dc8595069
 		simple exporting models (#1344) 
* unified exporting

* json exporting

* ignore more

* simplified buffer export

* added dtypes

* added assert

* swift example

* fix tests

* linter

* remove whitespace

* fixed tests

* remove swift example

* remove unintended changes

* allow callable models to be used

* whitespace

* more readable json export

* name change

* whitespace

* whitespace
 2023-08-01 09:35:48 -07:00
Diogo
ba5e3818a0
 		Limit dims based on max size (#1390) 
* working

* whitespace

* changed defaults to None

* linter

* last linter error
 2023-07-31 19:18:19 -07:00
chenyu
b2fde9ec36
 		reshape to register variable value (#1386) 
* reshape to register variable value

* better error message
 2023-07-31 17:10:02 -07:00
Umut Zengin
0de5f20970
 		Re-open constant pad support to Tensor.pad (#1388) 
* Added const padding support to .pad

* Linter
 2023-07-31 17:08:57 -07:00
Alex Telon
2d10e0340e
 		Refactored ContextVars (#1331) 2023-07-31 15:44:46 -04:00
chenyu
f5ef445cb6
 		trim space (#1381) 2023-07-31 10:37:57 -07:00
JaSpa99
5ab12059da
 		rng hlops: add normal and kaiming_normal (#1378) 
* add normal and kaiming_normal

* make sure its float

* add tests
 2023-07-31 10:37:02 -07:00
George Hotz
37fa7e96fb
 		Revert "update editorconfig, enforce via CI (#1343)" (#1380) 
This reverts commit da2efecbe2.
 2023-07-31 10:35:50 -07:00
Pavol Rusnak
da2efecbe2
 		update editorconfig, enforce via CI (#1343) 
* update editorconfig to set unix-style newlines and trim whitespace

* add editorconfig github action to the CI

* fix whitespace
 2023-07-30 18:44:30 -07:00
S-Lykles
c2b82ea8ac
 		fix to_shape_strides (#1374) 
* add tests for expr_node and expr_idxs

* simplify condition and add missing optimization
 2023-07-30 18:42:46 -07:00
chenyu
1fdf560fb1
 		simplify get_contraction (#1373) 2023-07-30 18:35:22 -07:00
S-Lykles
a32c677601
 		Fix off by one error in View.expr_node (#1363) 
* Fix off_by_one error in View.expr_node

* Add test for expr_node

* Remove whitespace before :

* test no arguments and properly test idx=None
 2023-07-29 08:10:37 -07:00
Karan Handa
e0a69bdbe6
 		Fix argfix and add tests (#1365) 
* Remove unreachable code

* Fixed argfix

* Add empty check and tests

* Removed redundant tests"
 2023-07-28 09:09:49 -07:00
wozeparrot
32d1afa4b5
 		feat: correct case when base is 0 (#1360) 2023-07-27 13:53:38 -04:00
wozeparrot
c22e77abfd
 		Match torch on fractional negative base pow (#1352) 
* feat: match torch on fractional negative base pow

* feat: tests for trunc
 2023-07-26 19:14:54 -07:00
Umut Zengin
d4ebadf2da
 		Small Tensor.cat optimization and reformating (#1347) 2023-07-26 18:01:12 -04:00
geohotstan
4056f97187
 		Gather (#1329) 2023-07-25 15:05:41 -04:00