5 Commits

Author	SHA1	Message	Date
Blake Mizerany	95af97b9f3	server: try github.com/minio/sha256-simd ... This is an experimental change to see if sha256-simd is faster than the standard library's sha256 implementation. It is not yet clear if this will be a net win, but it is worth trying.	2024-05-31 00:51:20 -07:00
Ikko Eltociear Ashimine	955c317cab	chore: update tokenizer.go (#4571) ... PreTokenziers -> PreTokenizers	2024-05-22 00:25:23 -07:00
Michael Yang	bbbd9f20f3	cleanup	2024-05-20 16:13:57 -07:00
Michael Yang	547132e820	bpe pretokenizer	2024-05-20 16:13:57 -07:00
Patrick Devine	2d315ba9a9	add missing file	2024-05-20 16:13:57 -07:00