tinygrad/examples
Oleg Rybalko 5e87083783
Whisper + LLAMA + VITS (#2332)
* feat: working voice 2 text using whisper

* feat: added llama generation

* feat: vits init

* feat: more accurate voice conversion

* feat: support for tts and working pipeline for the first pass

* fix: linter checks

* refactored vits initialization and inference, added mmts-tts support

* fixed process sync and now we can have an infinite conversation

* reuse output stream to remove overhead of creating a new one each time

* added pre-prompt configuration with yaml files

* adjusted code to merge PR which changed whisper

* optimized whisper, now it's blazing fast and also reduced number of lines

* added better debug printing

* use jitted encode function for whisper, added timings and removed response delim to save speed on generating those tokens

* fixed hf convert and now it's working with tinyllama

* added tinyllama config

* refactored code and made it work with all llama models

* prettier order

* prettier order

* fixed suffix for tinyllama and refactored convert_from_hf

* added missing parameters

* fixed stream release and added missing params

* jitted dp and encoder

* jitted flow forward

* removed re-init of espeak on each call to save up time

* jitted generator forward for blazing fast tts

* added contextmanager for displaying a chat log

* removed whitespace for pylint

* updated code to support latest fetch func

* wait for llama eos token and pass params from cli to llama

* listen for not fixed amount of time

* refactored code a bit

* removed thresholding and now the output streams directly to whisper

* tokenize llama output for vits batch size to work and stream each sentence to a speaker

* changed speaker

* whisper is now printing on the same line

* don't trigger llama on whisper output in parens

* added tinyllama chat model

* adjusted code to work with tinyllama chat model

* removed unused cli arg

* autofetch tokenizer and tinyllama model. add 3 chat tokens to the tokenizer

* fixed issue with long sentences by chunking them

* support for multiline llama output

* prettified log output

* adjusted sentence length

* remove quote from response to avoid funny tts

* fixed prompts

* added missing parameter
2023-12-02 15:03:46 -08:00
..
conversation_data Whisper + LLAMA + VITS (#2332) 2023-12-02 15:03:46 -08:00
mlperf move things, clean up extra (#2292) 2023-11-13 20:18:40 -08:00
sovits_helpers Implementation of SoftVC VITS SVC model (#1371) 2023-08-13 19:43:23 -07:00
vgg7_helpers move to new cached fetch (#2493) 2023-11-28 17:36:55 -08:00
webgpu/stable_diffusion move device to device.py (#2466) 2023-11-27 11:34:37 -08:00
__init__.py failing llama test 2023-03-11 16:28:10 -08:00
beautiful_cartpole.py A beautiful MNIST training example (#2272) 2023-11-17 19:42:43 -08:00
beautiful_mnist.py a lil more beautiful_mnist 2023-11-17 19:53:06 -08:00
benchmark_train_efficientnet.py move things, clean up extra (#2292) 2023-11-13 20:18:40 -08:00
coder.py coder.py can write and run code (#2439) 2023-11-25 12:27:54 -08:00
compile_efficientnet.py add name support to fetch (#2407) 2023-11-23 14:16:17 -08:00
compile_tensorflow.py moved extras/jit.py -> tinygrad/jit.py (#599) 2023-02-25 08:32:33 -08:00
conversation.py Whisper + LLAMA + VITS (#2332) 2023-12-02 15:03:46 -08:00
efficientnet.py torch and numpy don't share ops anymore (#2412) 2023-11-23 16:58:10 -08:00
f16_w_uint32.py move device to device.py (#2466) 2023-11-27 11:34:37 -08:00
gpt2.py fp16 in gpt2 attention (#2491) 2023-11-28 19:27:03 -05:00
handcode_resnet50_opt.py fix handcode_resnet50_opt.py (#2558) 2023-12-01 20:51:21 -05:00
hlb_cifar10.py new style device (#2530) 2023-11-30 17:07:16 -08:00
index.html Enable Multi-Output Export (#2179) 2023-10-30 18:42:26 -07:00
llama.py Whisper + LLAMA + VITS (#2332) 2023-12-02 15:03:46 -08:00
mask_rcnn.py move things, clean up extra (#2292) 2023-11-13 20:18:40 -08:00
mnist_gan.py move state to nn/state (#1619) 2023-08-22 07:36:24 -07:00
serious_mnist.py move state to nn/state (#1619) 2023-08-22 07:36:24 -07:00
simple_conv_bn.py with Tensor.train() (#1935) 2023-09-28 18:02:31 -07:00
so_vits_svc.py use class Foo: instead of class Foo(): (#1797) 2023-09-06 12:20:25 -07:00
stable_diffusion.py move device to device.py (#2466) 2023-11-27 11:34:37 -08:00
train_efficientnet.py move things, clean up extra (#2292) 2023-11-13 20:18:40 -08:00
train_resnet.py move things, clean up extra (#2292) 2023-11-13 20:18:40 -08:00
transformer.py move things, clean up extra (#2292) 2023-11-13 20:18:40 -08:00
vgg7.py waifu2x vgg7: testcase, auto-RGBA->RGB, function to grab pretrained models, training "fix" (#2117) 2023-10-19 22:07:15 -07:00
vit.py move to new cached fetch (#2493) 2023-11-28 17:36:55 -08:00
vits.py Whisper + LLAMA + VITS (#2332) 2023-12-02 15:03:46 -08:00
whisper.py Update Whisper to use fetch helper (#2401) 2023-11-23 12:59:59 -08:00
yolov3.py move to new cached fetch (#2493) 2023-11-28 17:36:55 -08:00
yolov8-onnx.py [ready] Replacing os with pathlib (#1708) 2023-08-30 10:41:08 -07:00
yolov8.py move to new cached fetch (#2493) 2023-11-28 17:36:55 -08:00