Ikko Eltociear Ashimine 955c317cab

chore: update tokenizer.go (#4571) ...

PreTokenziers -> PreTokenizers

2024-05-22 00:25:23 -07:00

..

sentencepiece

…

convert_test.go

add test

2024-05-21 11:28:22 -07:00

convert.go

…

gemma.go

fix conversion for f16 or f32 inputs

2024-05-21 11:28:22 -07:00

llama.go

fix conversion for f16 or f32 inputs

2024-05-21 11:28:22 -07:00

mistral.go

fix conversion for f16 or f32 inputs

2024-05-21 11:28:22 -07:00

mixtral.go

fix conversion for f16 or f32 inputs

2024-05-21 11:28:22 -07:00

safetensors.go

simplify safetensors reading

2024-05-21 11:28:22 -07:00

sentencepiece_model.proto

…

tokenizer.go

chore: update tokenizer.go (#4571)

2024-05-22 00:25:23 -07:00

torch.go

fix conversion for f16 or f32 inputs

2024-05-21 11:28:22 -07:00