Jesse Gross
5d34320b7c
runner.go: Fix off by one in batch size check
...
When adding tokens to a batch, the index is zero based but is
checked against being greater than the max batch size. This results
in an out-of-bounds access when the final token is added.
2024-09-03 21:15:14 -04:00
Jesse Gross
0c2f95f3de
runner: Initialize numPredict
...
numPredict is used to enforce a limit on the number of tokens to
generate. Is it passed in from Ollama but it is never stored to
be checked.
2024-09-03 21:15:14 -04:00
Daniel Hiltgen
47b0e81219
fix dolphin-mistral
2024-09-03 21:15:14 -04:00
Daniel Hiltgen
751009a5d7
Runtime selection of new or old runners
...
This adjusts the new runners to comingle with existing runners so we can use an
env var to toggle the new runners on.
2024-09-03 21:15:14 -04:00
Daniel Hiltgen
8527028bf4
Implement timings response in Go server
...
This implements the fields necessary for `run --verbose`
to generate timing information.
2024-09-03 21:15:14 -04:00
Daniel Hiltgen
e0241118d0
Get embeddings working
...
Truncation doesn't pass, but the other embeddings tests pass
2024-09-03 21:15:14 -04:00
Daniel Hiltgen
f97ee8c506
Fix parallel requests
2024-09-03 21:15:13 -04:00
jmorganca
1da6c40f4f
lint
2024-09-03 21:15:13 -04:00
jmorganca
de634b7fd7
fix issues with runner
2024-09-03 21:15:13 -04:00
jmorganca
795753be7e
move sync script back in for now
2024-09-03 21:15:13 -04:00
jmorganca
8f79a2e86a
cleanup stop code
2024-09-03 21:15:13 -04:00
jmorganca
7d0a452938
num predict
2024-09-03 21:15:13 -04:00
jmorganca
43efc893d7
basic progress
2024-09-03 21:15:13 -04:00
jmorganca
20afaae020
add more runner params
2024-09-03 21:15:13 -04:00
jmorganca
72f3fe4b94
truncate stop properly
2024-09-03 21:15:13 -04:00
jmorganca
a379d68aa9
wip stop tokens
2024-09-03 21:15:13 -04:00
jmorganca
b2ef3bf490
embeddings
2024-09-03 21:15:12 -04:00
jmorganca
ce15ed6d69
remove dependency on llm
2024-09-03 21:15:12 -04:00
jmorganca
c0b94376b2
grammar
2024-09-03 21:15:12 -04:00
jmorganca
72be8e27c4
sampling
2024-09-03 21:15:12 -04:00
jmorganca
d12db0568e
better example module, add port
2024-09-03 21:15:12 -04:00
jmorganca
ec17359a68
wip
2024-09-03 21:15:12 -04:00
jmorganca
fbc8572859
add llava to runner
2024-09-03 21:15:12 -04:00
jmorganca
a8f91d3cc1
add llava
2024-09-03 21:15:12 -04:00
jmorganca
9fe48978a8
move runner package down
2024-09-03 21:15:12 -04:00
jmorganca
01ccbc07fe
replace static build in llm
2024-09-03 21:15:12 -04:00
jmorganca
6129f30479
wip...
2024-09-03 21:15:12 -04:00
jmorganca
eb1aa97961
rename server to runner
2024-09-03 21:15:12 -04:00
