56 commits

Author	SHA1	Message	Date
Elias Wahl	73bddc44f6	Fix fake dataloader (#5326)	2024-07-08 09:07:44 -04:00
Elias Wahl	e267f3161d	Add MLLogger (#5125) ... * add MLPerf logger * eval steps * start with step 1 * compliance for 3.1.0 and 4.0.0 * more compliance * assert, comment and contiguous	2024-06-26 12:23:56 -04:00
Elias Wahl	f31ef11537	Better default hparams for large BS (#5030) ... * better default hparams for large BS * bf16 too * use tuple	2024-06-18 11:13:06 -04:00
Elias Wahl	7bfa9101c0	Float in scaled dot product attention (#4985) ... * Monkeypatch scaled-dot-product-attention * Use dot instead of matmul * new api * imports * least_upper_dtype	2024-06-18 08:16:41 -04:00
Elias Wahl	d2e3c391e8	Residual in MLM loss + Change default steps (#4935) ... * Residual in mlm loss * Reduce default steps to 160K * 24 * oops * comment	2024-06-12 16:09:18 -04:00
Elias Wahl	e576aca044	Disable dropout (#4837)	2024-06-04 18:57:26 -04:00
Elias Wahl	bb248a0dd1	Optional half matmul (#4835) ... * half linear * move weight cast back * oops * matmul dtype var * todo comment	2024-06-04 17:53:41 -04:00
Elias Wahl	04e237328b	Refactor to class style (#4804)	2024-06-04 14:08:31 -07:00
Elias Wahl	c4b0acf095	Global norm + small changes (#4749) ... * norm * no empty * default loss scaler in float	2024-05-27 18:35:27 -04:00
Elias Wahl	acc0039cfc	Resume fix + scheduler for non weight decay params (#4679) ... * move ckpt dir * fix resume. Add scheduler group	2024-05-21 19:38:13 -04:00
Elias Wahl	993091adfa	loss scaler + nan fixes (#4661)	2024-05-20 17:08:35 -04:00
chenyu	bed70b130c	mlperf bert getenv-able EVAL_STEP_FREQ (#4534)	2024-05-11 14:36:56 -04:00
chenyu	04a4980a51	touchup bert script (#4531) ... small adjustments, remove duplicated training setting and stop the script once target is hit	2024-05-11 13:02:02 -04:00
chenyu	b00b6b16f0	fix TRAIN_BEAM and Tensor.training for mlperf bert (#4525) ... also hard coded bert model config instead of looking up a file	2024-05-11 00:18:36 -04:00
chenyu	b399d98e41	fix resnet eval (#4507)	2024-05-10 00:49:00 -04:00
chenyu	0e8aa0e288	use fake data in beam searching resnet (#4504)	2024-05-09 23:43:50 -04:00
chenyu	047c7f3e5b	polish resnet mlperf logging (#4490) ... don't include save final check point time in run time, and some cosmetic order changes	2024-05-09 13:04:24 -04:00
chenyu	d78e159aa3	resnet logging move RUN_START to start of the script (#4488)	2024-05-09 12:32:32 -04:00
chenyu	1f6bf9d2f7	real diskcache_clear in model_train resnet (#4445) ... clear cache if INITMLPERF is set, or running run_and_time. dev_beam and dev_run do not clear cache	2024-05-08 19:06:09 -04:00
chenyu	1b4645bea6	hotfix resnet move init_start to start of the script (#4481)	2024-05-08 19:03:52 -04:00
chenyu	db7e15c46f	hotfix resnet only log epoch start with RUNMLPERF (#4477)	2024-05-08 15:14:41 -04:00
chenyu	062c6dd65d	mlperf logging, truncate dir in logs and log seed (#4475)	2024-05-08 12:54:02 -04:00
chenyu	b62a65b617	redo faster sparse_categorical_crossentropy (#4461) ... update LR and DECAY in resnet default that help convergence too	2024-05-08 11:21:43 -04:00
wozeparrot	603d3a351b	feat: allow keeping multiple cookies (#4440)	2024-05-05 19:26:48 -07:00
David Hou	b767d59684	resnet trainer: keep old cookie around until next step has been queued (#4401) ... * keep old cookie around until next step has been queued (-10ms 6gpu) * also for eval * drop cookie before data_get? * Revert "drop cookie before data_get?" This reverts commit b01e6aa2b2. * Revert "Revert "drop cookie before data_get?"" This reverts commit 23464e73d4.	2024-05-03 12:15:21 -04:00
chenyu	2c3b7f8e70	pad resnet training data with training data mean (#4369) ... update model_train resnet to pad training	2024-05-02 20:26:15 -04:00
chenyu	ab01a9433d	resnet eval 4n+3 if epoch < 33 (#4391) ... the rule is as thoroughly as 4n+k and we can stop the clock as soon as eval hits target. this can save 24 evals or 12 minutes	2024-05-02 16:52:07 -04:00
chenyu	bf31837e6d	resnet correct steps_in_val_epoch in logging (#4389) ... also added random seed from system in scripts	2024-05-02 10:51:36 -04:00
chenyu	22376e53b7	resnet mlperf logging (#4361) ... * resnet mlperf logging * cropping too much?	2024-05-02 00:00:04 -04:00
chenyu	6628e13a5f	pad resnet eval data in model_train (#4374) ... asserted if eval sample count is different from total eval file count.	2024-05-01 14:33:42 -04:00
chenyu	826cccd54d	fix mean underflow for half tensor (#4377) ... * fix mean underflow for half tensor divide only the reduce factor. added unit test and non-nan assertion in resnet training. also added a failed test cast for symbolic shape var * skip for python backend	2024-05-01 13:38:57 -04:00
Elias Wahl	babe87a8ae	BERT: Checkpoint loading tests (#4359) ... * Move checkpoint init to helpers. Add test * linters * Move the steps outside of the main train loop * Move data_get * data_get belongs to helpers	2024-04-30 14:43:41 -04:00
Elias Wahl	71ff68b445	dropout after eval step (#4351)	2024-04-29 15:47:21 -04:00
Elias Wahl	27613dd881	MLPerf BERT: Main training loop (#4288) ... * BERT language modeling head + trunc normal initializers * add train loop + helpers * shuffle in dataloaders + slight changes in main loop * beam change * Minor changes * random.shuffle * HParam update * Use deque for dataloader * wandb bert project name * half fixes * BENCHMARK + remove epoch * cast + print() --------- Co-authored-by: chenyu <chenyu@fastmail.com>	2024-04-29 14:35:27 -04:00
chenyu	ec65aea32f	resnet stop the script once hit target (#4303) ... * resnet stop the script once hit target * comment	2024-04-25 23:54:56 -04:00
chenyu	f9a7badace	use LR=7 for resnet with BS=1536 (#4299) ... had 3 runs after lr float32, seems quite stable and converges at epoch 34 and 35	2024-04-25 15:23:10 -04:00
chenyu	c1fbacb182	resnet benchmarks use DEFAULT_FLOAT=HALF (#4285) ... also update LR default to scaled based on 1536 (the BS we are submitting)	2024-04-24 12:10:57 -04:00
chenyu	8401de9922	resnet benchmark return early in eval (#4278) ... only do few eval steps to compile, and skip second epoch when doing beam + benchmark. save 2 minutes	2024-04-24 00:55:01 -04:00
chenyu	6637ecc5fe	use IGNORE_JIT_FIRST_BEAM to not BEAM in jit cnt=0 (#4269) ... we want to have different BEAM values for resnet train and eval. global JITBEAM cannot do this. added the flag to change beam behavior at cnt=0 (so it default behaves the same with or without TinyJit), and for cnt=1 it uses existing BEAM.value. Also updated the context var BEAM in resnet to be outside of TinyJit. saves about 3 minutes compile time	2024-04-23 18:59:43 -04:00
chenyu	37f8be6450	resnet print epoch ops and mem in benchmark (#4244) ... * resnet print epoch ops and mem in benchmark also added a flag to optionally disable reset jitted steps * real per epoch stats	2024-04-21 18:32:31 -04:00
chenyu	f7416916df	update resnet hparams based on BS=1632 RCP (#4210) ... https://github.com/mlcommons/logging/blob/master/mlperf_logging/rcp_checker/training_4.0.0/rcps_resnet.json	2024-04-18 12:01:46 -04:00
chenyu	d5b67c1ca3	log resnet TRAIN_BEAM / EVAL_BEAM (#4181) ... also run eval in benchmark mode if either one is positive	2024-04-15 19:29:08 -04:00
chenyu	6a2168e698	TRAIN_BEAM and EVAL_BEAM for resnet (#4177) ... working on measuring compile time	2024-04-15 14:57:21 -04:00
chenyu	e20d6f9221	correct resnet estimate time (#4169) ... 7.99 hours was rendered as 7h0m.	2024-04-14 02:21:46 -04:00
George Hotz	97c402d69e	use imagenet spawn (#4096)	2024-04-06 08:34:10 -07:00
George Hotz	fffd9b05f5	mock mnist data for imagenet trainer (#4095) ... * mock mnist data for imagenet * move print and test * needed to reshape	2024-04-06 08:08:40 -07:00
George Hotz	93824e59eb	support MOCKDATA=1 for resnet (#4090) ... * mockdata for resnet * fix eval, revert hsa	2024-04-05 17:19:18 -07:00
chenyu	ecf38f498e	beam search resnet eval too in BENCHMARK (#4000)	2024-03-29 21:07:23 -04:00
David Hou	4b95350c41	fp16 resnet (without expand backwards sum in float, doesn't work) (#3816) ... * fp16 resnet * cast running mean and var back to default float * extra cast * check symbolic no overflow * add linearizer failure * loss scaler after grad contig * oops * i think this works * don't loss scale fp32 * remove overflow test case * remove symbolic bounds check * loss scaler should be float * temporarily disable padto cuz bug shruggie * make running stats in batchnorm float32? * calculate lars stuff in fp32? * oops * remove most changes * move loss scaler out of optimizer * no more FP16 var * oops --------- Co-authored-by: chenyu <chenyu@fastmail.com>	2024-03-28 01:25:37 -04:00
wozeparrot	9a9cac58f9	add lars to nn (#3750) ... * feat: add lars * feat: don't remove this comment * clean: smaller diff * clean: shorter line * feat: remove mlperf lars, switch resnet * fix: fully remove mlperf lars * clean: comment * feat: contiguous * feat: no weight decay on skip params * feat: optimizergroup * feat: classic momentum * fix: pylint * clean: move comment * fix: correct algo * feat: lrschedulergroup * feat: skip list tests * feat: :| forgot that params are a thing * feat: remove skip_list params from main params * feat: set moment --------- Co-authored-by: chenyu <chenyu@fastmail.com>	2024-03-24 11:43:12 -04:00