53b600921e
It is not safe to hold a mutex only while we are waiting for the condition variable to signal that a new sequence has been added. It's possible that a sequence could be added in the middle of batch processing. For example, if a new sequence is added while Decode() is running, it will get picked up for sampling, despite not having been added to the original batch. This change holds a mutex for the majority of the time when active processing is happening, releasing it only for a brief period each time around the loop. Depending on the workload and the scheduler is may result in unfairness between different requests. However, this was not actually observed in testing. This addresses the correctness issue - better performance and fairness can be achieved with additional improvements in the future.
llama
Note: this package is not used in Ollama yet. For now, see the
llmpackage.
This package integrates the llama.cpp library as a Go package and makes it easy to build it with tags for different CPU and GPU processors.
Supported:
- CPU
- avx, avx2
- macOS Metal
- Windows CUDA
- Windows ROCm
- Linux CUDA
- Linux ROCm
- Llava
Extra build steps are required for CUDA and ROCm on Windows since nvcc and hipcc both require using msvc as the host compiler. For these shared libraries are created:
ggml_cuda.dllon Windows orggml_cuda.soon Linuxggml_hipblas.dllon Windows orggml_hipblas.soon Linux
Note: it's important that memory is allocated and freed by the same compiler (e.g. entirely by code compiled with msvc or mingw). Issues from this should be rare, but there are some places where pointers are returned by the CUDA or HIP runtimes and freed elsewhere, causing a a crash. In a future change the same runtime should be used in both cases to avoid crashes.
Building
go build .
AVX
go build -tags avx .
AVX2
# go doesn't recognize `-mfma` as a valid compiler flag
# see https://github.com/golang/go/issues/17895
go env -w "CGO_CFLAGS_ALLOW=-mfma|-mf16c"
go env -w "CGO_CXXFLAGS_ALLOW=-mfma|-mf16c"
go build -tags=avx,avx2 .
Linux
CUDA
Install the CUDA toolkit v11.3.1:
make ggml_cuda.so
go build -tags avx,cuda .
ROCm
Install the CUDA toolkit v11.3.1:
make ggml_hipblas.so
go build -tags avx,rocm .
Windows
Download w64devkit for a simple MinGW development environment.
CUDA
Install the CUDA toolkit v11.3.1 then build the cuda code:
make ggml_cuda.dll
go build -tags avx,cuda .
ROCm
Install ROCm 5.7.1.
make ggml_hipblas.dll
go build -tags avx,rocm .
Building runners
# build all runners for this platform
make -j
Syncing with llama.cpp
To update this package to the latest llama.cpp code, use the sync.sh script:
./sync.sh ../../llama.cpp