mirrors/tinygrad
2
0
Fork 0
mirror of https://github.com/tinygrad/tinygrad.git synced 2026-06-24 02:14:17 +00:00
Code Issues Projects Releases Packages Wiki Activity

revert layernorm to have axis param

Browse source
This commit is contained in:
George Hotz 2022-09-26 10:11:38 -04:00
commit dec5334da9
1 changed files with 1 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

3
tinygrad/tensor.py
View file

@ -312,8 +312,7 @@ class Tensor:
def sequential(self, ll:List[Callable[[Tensor], Tensor]]): return functools.reduce(lambda x,f: f(x), ll, self)
def layernorm(self, eps=1e-5):
axis = range(1, len(self.shape))
def layernorm(self, axis=-1, eps=1e-5):
y = (self - self.mean(axis=axis, keepdim=True))
return y.div((y*y).mean(axis=axis, keepdim=True).add(eps).sqrt())