Jeffrey Morgan b24e8d17b2

Increase minimum CUDA memory allocation overhead and fix minimum overhead for multi-gpu (#1896) ...

* increase minimum cuda overhead and fix minimum overhead for multi-gpu

* fix multi gpu overhead

* limit overhead to 10% of all gpus

* better wording

* allocate fixed amount before layers

* fixed only includes graph alloc

2024-01-10 19:08:51 -05:00

..

ext_server

Update submodule to 6efb8eb30e7025b168f3fda3ff83b9b386428ad6 (#1885)

2024-01-10 16:48:38 -05:00

generate

clean up cmake build directory when cross compiling macOS builds

2024-01-09 17:13:56 -05:00

llama.cpp @ 328b83de23

revert submodule back to 328b83de23b33240e28f4e74900d1d06726f5eb1

2024-01-10 18:42:39 -05:00

dynamic_shim.c

Switch windows build to fully dynamic

2024-01-02 15:36:16 -08:00

dynamic_shim.h

Refactor how we augment llama.cpp

2024-01-02 15:35:55 -08:00

ext_server_common.go

Offload layers to GPU based on new model size estimates (#1850)

2024-01-08 16:42:00 -05:00

ext_server_default.go

Offload layers to GPU based on new model size estimates (#1850)

2024-01-08 16:42:00 -05:00

ext_server_windows.go

fix windows build

2024-01-08 20:04:01 -05:00

ggml.go

Offload layers to GPU based on new model size estimates (#1850)

2024-01-08 16:42:00 -05:00

gguf.go

Offload layers to GPU based on new model size estimates (#1850)

2024-01-08 16:42:00 -05:00

llama.go

Offload layers to GPU based on new model size estimates (#1850)

2024-01-08 16:42:00 -05:00

llm.go

Increase minimum CUDA memory allocation overhead and fix minimum overhead for multi-gpu (#1896)

2024-01-10 19:08:51 -05:00

shim_darwin.go

Offload layers to GPU based on new model size estimates (#1850)

2024-01-08 16:42:00 -05:00

shim_ext_server_linux.go

Code shuffle to clean up the llm dir

2024-01-04 12:12:05 -08:00

shim_ext_server_windows.go

Code shuffle to clean up the llm dir

2024-01-04 12:12:05 -08:00

shim_ext_server.go

Offload layers to GPU based on new model size estimates (#1850)

2024-01-08 16:42:00 -05:00

utils.go

partial decode ggml bin for more info

2023-08-10 09:23:10 -07:00