670 commits

Author	SHA1	Message	Date
George Hotz	f4e49a7c1a	resnet 50 opt: correct loop + LARS (#4449) ... * correct loop + LARS * ops	2024-05-06 08:01:26 -07:00
George Hotz	fc995d4446	add backward to handcode_resnet50_opt	2024-05-06 06:42:26 -07:00
wozeparrot	603d3a351b	feat: allow keeping multiple cookies (#4440)	2024-05-05 19:26:48 -07:00
Francis Lam	709410071c	mlperf/resnet: updated BEAM params to increase performance (#4443)	2024-05-05 21:49:46 -04:00
chenyu	3b30756cbb	update mlperf submission system (#4435) ... more required fields.	2024-05-05 13:19:07 -04:00
David Hou	c0a048c044	batchnorm d(var)/d(mean) = 0 (#4430) ... * d(var)/d(mean) = 0 * drop the number in test_schedule!	2024-05-05 00:25:45 -04:00
qazal	fa17dcaf07	Fix llm.c/export.py (#4423) ... * fix headers * add CI * add stdio * merge clang tests * revert llm.c * revert ci * Revert "revert llm.c" This reverts commit 5fd17e3c8b.	2024-05-04 19:37:10 +03:00
George Hotz	cb7289f9c9	remove clang program header (#4422) ... * remove clang program header * proper max * bools are numbers * fix compile enet	2024-05-04 08:38:01 -07:00
chenyu	473ecb978a	remove SPLIT_REDUCEOP=1 from resnet scripts (#4404) ... SPLIT_REDUCEOP=1 is default	2024-05-03 12:36:23 -04:00
David Hou	b767d59684	resnet trainer: keep old cookie around until next step has been queued (#4401) ... * keep old cookie around until next step has been queued (-10ms 6gpu) * also for eval * drop cookie before data_get? * Revert "drop cookie before data_get?" This reverts commit b01e6aa2b2. * Revert "Revert "drop cookie before data_get?"" This reverts commit 23464e73d4.	2024-05-03 12:15:21 -04:00
chenyu	2c3b7f8e70	pad resnet training data with training data mean (#4369) ... update model_train resnet to pad training	2024-05-02 20:26:15 -04:00
Francis Lam	3cf8291f2f	mlperf/resnet: update beam params to increase time and quality (#4396) ... * mlperf/resnet: update beam params to increase time and quality * revert upcast 8 in search space and add rocm setup function * refactor to independent setup.sh script	2024-05-02 20:14:46 -04:00
chenyu	ab01a9433d	resnet eval 4n+3 if epoch < 33 (#4391) ... the rule is as thoroughly as 4n+k and we can stop the clock as soon as eval hits target. this can save 24 evals or 12 minutes	2024-05-02 16:52:07 -04:00
chenyu	7492e5d3e7	resnet correct log name for red (#4390)	2024-05-02 10:58:55 -04:00
chenyu	bf31837e6d	resnet correct steps_in_val_epoch in logging (#4389) ... also added random seed from system in scripts	2024-05-02 10:51:36 -04:00
ym555	3113785604	Llama 3 Models (#4339) ... * Full Impl * fix test * Fix inference loop --------- Co-authored-by: George Hotz <72895+geohot@users.noreply.github.com>	2024-05-02 06:06:07 -07:00
chenyu	22376e53b7	resnet mlperf logging (#4361) ... * resnet mlperf logging * cropping too much?	2024-05-02 00:00:04 -04:00
chenyu	ad116dc5c6	fill in mlperf system description (#4381) ... it did not ask too many details. will put software versions later with tinygrad commit. ``` python3 -m mlperf_logging.system_desc_checker examples/mlperf/training_submission_v4.0/tinycorp/systems/tinybox_red.json training 4.0.0 INFO - System description checker passed for tinybox red ``` ``` python3 -m mlperf_logging.system_desc_checker examples/mlperf/training_submission_v4.0/tinycorp/systems/tinybox_green.json training 4. 0.0 INFO - System description checker passed for tinybox green ```	2024-05-01 16:47:45 -04:00
chenyu	9358b62073	rename resnet script to dev_beam.sh and dev_run.sh (#4379) ... final run_and_time needs to be one script for both. rename the old scripts	2024-05-01 14:41:35 -04:00
chenyu	6628e13a5f	pad resnet eval data in model_train (#4374) ... asserted if eval sample count is different from total eval file count.	2024-05-01 14:33:42 -04:00
chenyu	826cccd54d	fix mean underflow for half tensor (#4377) ... * fix mean underflow for half tensor divide only the reduce factor. added unit test and non-nan assertion in resnet training. also added a failed test cast for symbolic shape var * skip for python backend	2024-05-01 13:38:57 -04:00
George Hotz	b683d0f496	hotfix: 100% accuracy is wrong	2024-05-01 08:07:18 -07:00
chenyu	683b7c605a	pad first batch of imagenet dataloader and update eval (#4368) ... * pad first batch of imagenet dataloader and update eval * pad zero instead of empty for training	2024-05-01 00:21:52 -04:00
Francis Lam	16838eae08	mlperf/resnet: update tinybox_red parameters to new best values (#4364) ... about 27 minutes to setup and 345ms/110TF steps	2024-04-30 18:08:12 -04:00
Francis Lam	0d33c54d99	kernel: change PADTO check to allow up to 4x padding (#4354) ... * kernel: change PADTO check to allow up to 4x padding also optionally remove PADTO from the search action space with BEAM_PADTO=0. * fix test_linearizer test_tensor_cores_padded tests * update resnet runs to use SPLIT_REDUCEOP=1 * fix up search TC axis and amt checking * fix up the dimensions of the TC tests	2024-04-30 15:29:34 -04:00
Elias Wahl	babe87a8ae	BERT: Checkpoint loading tests (#4359) ... * Move checkpoint init to helpers. Add test * linters * Move the steps outside of the main train loop * Move data_get * data_get belongs to helpers	2024-04-30 14:43:41 -04:00
Elias Wahl	71ff68b445	dropout after eval step (#4351)	2024-04-29 15:47:21 -04:00
Elias Wahl	27613dd881	MLPerf BERT: Main training loop (#4288) ... * BERT language modeling head + trunc normal initializers * add train loop + helpers * shuffle in dataloaders + slight changes in main loop * beam change * Minor changes * random.shuffle * HParam update * Use deque for dataloader * wandb bert project name * half fixes * BENCHMARK + remove epoch * cast + print() --------- Co-authored-by: chenyu <chenyu@fastmail.com>	2024-04-29 14:35:27 -04:00
Francis Lata	bb849a57d1	[MLPerf] UNet3D dataloader (#4343) ... * add support for train/val datasets for kits19 * split dataset into train and val sets * add tests for kits19 dataloader * add MLPerf dataset tests to CI * update unet3d model_eval script * fix linting * add nibabel * fix how mock dataset gets created * update ref implementation with permalink and no edits * clean up test and update rand_flip implementation * cleanups	2024-04-28 22:34:18 -04:00
Arnav Mehta	f3de17912f	added the download if not present missing function (#4318)	2024-04-28 16:31:08 +08:00
chenyu	ec65aea32f	resnet stop the script once hit target (#4303) ... * resnet stop the script once hit target * comment	2024-04-25 23:54:56 -04:00
George Hotz	1e37c4a7a1	minor llm.c improvements	2024-04-26 11:15:31 +08:00
chenyu	f9a7badace	use LR=7 for resnet with BS=1536 (#4299) ... had 3 runs after lr float32, seems quite stable and converges at epoch 34 and 35	2024-04-25 15:23:10 -04:00
chenyu	c11bad766d	prepare mlperf submission (#4270) ... * prepare mlperf submission * 28min compile and 3h53m * red 30 minute compile and 56 TFLOPS	2024-04-24 13:19:31 -04:00
chenyu	c1fbacb182	resnet benchmarks use DEFAULT_FLOAT=HALF (#4285) ... also update LR default to scaled based on 1536 (the BS we are submitting)	2024-04-24 12:10:57 -04:00
George Hotz	ad28fdecb1	si.inputs+outputs -> bufs (#4279)	2024-04-24 15:12:34 +08:00
chenyu	8401de9922	resnet benchmark return early in eval (#4278) ... only do few eval steps to compile, and skip second epoch when doing beam + benchmark. save 2 minutes	2024-04-24 00:55:01 -04:00
chenyu	6637ecc5fe	use IGNORE_JIT_FIRST_BEAM to not BEAM in jit cnt=0 (#4269) ... we want to have different BEAM values for resnet train and eval. global JITBEAM cannot do this. added the flag to change beam behavior at cnt=0 (so it default behaves the same with or without TinyJit), and for cnt=1 it uses existing BEAM.value. Also updated the context var BEAM in resnet to be outside of TinyJit. saves about 3 minutes compile time	2024-04-23 18:59:43 -04:00
Elias Wahl	3a48773f1a	BERT dataloader (#4252) ... * add dataloader * comment	2024-04-23 13:44:49 -04:00
chenyu	37f8be6450	resnet print epoch ops and mem in benchmark (#4244) ... * resnet print epoch ops and mem in benchmark also added a flag to optionally disable reset jitted steps * real per epoch stats	2024-04-21 18:32:31 -04:00
chenyu	30fc1ad415	remove TODO: remove explicit dtypes after broadcast fix in stable_diffusion (#4241) ... this is done	2024-04-21 00:31:24 -04:00
chenyu	a1940ced77	remove the assign hack in whisper (#4240) ... no longer needed, the commented test case was removed too	2024-04-20 23:56:44 -04:00
chenyu	3f126c7664	fix examples vits / converstion.py (#4239) ... it was passing a const numpy array into Tensor.arange	2024-04-20 23:29:12 -04:00
George Hotz	cd88afc98b	datasets isn't a feature + filter docstrings (#4228) ... * datasets isn't a feature * filter docstrings in sz	2024-04-19 16:16:10 +04:00
George Hotz	d99b512084	llm.c timing (#4219) ... * add timing info * fix malloc * 8s with beam	2024-04-19 12:43:21 +04:00
George Hotz	39b60a25f0	more llm c work (#4207) ... * more llm c work * print nicely * fake load pretrained * select warmups * output c code	2024-04-18 22:20:44 +04:00
chenyu	f7416916df	update resnet hparams based on BS=1632 RCP (#4210) ... https://github.com/mlcommons/logging/blob/master/mlperf_logging/rcp_checker/training_4.0.0/rcps_resnet.json	2024-04-18 12:01:46 -04:00
George Hotz	fa57c3e7ce	continue llm.c (#4190) ... * continue llm.c * export more * progress on llm.c * simpler optim, names work	2024-04-18 10:57:54 +04:00
Francis Lata	3644077a42	[MLPerf][UNet3D] Add DICE loss + metrics (#4204) ... * add DICE loss and metrics * update dice to include reference implementation's link * remove unused imports * remove unnecessary test file and update pred + label for metrics and losses test * add tests to CI + add exclusion of mlperf_unet3d --------- Co-authored-by: chenyu <chenyu@fastmail.com>	2024-04-17 20:09:33 -04:00
chenyu	cd801a15f3	scipy.signal.gaussian -> scipy.signal.windows.gaussian (#4205) ... fixed unet3d model_eval, will add to CI after merging new dice loss	2024-04-17 19:15:37 -04:00