diff options
Diffstat (limited to 'misc/llama-cpp/pkg-message')
| -rw-r--r-- | misc/llama-cpp/pkg-message | 9 |
1 files changed, 9 insertions, 0 deletions
diff --git a/misc/llama-cpp/pkg-message b/misc/llama-cpp/pkg-message index 157a4db6ea78..15534c8fd534 100644 --- a/misc/llama-cpp/pkg-message +++ b/misc/llama-cpp/pkg-message @@ -22,6 +22,15 @@ and navigate to http://localhost:8080: > llama_server_model=/path/to/models/llama-2-7b-chat.Q4_K_M.gguf > llama_server_args="--device Vulkan0 -ngl 27" +In order to use the multi-model feature do not use llama_server_model. +Instead add the argument "--models-preset /path/to/models.ini" +Add pre-downloaded models into models.ini, for example: +[Qwen3.5-35B-A3B-Uncensored] +model = /path/to/Qwen3.5-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf + +You can switch to the CPU-only operation by choosing the port option +VULKAN=OFF in misc/ggml (not in llama-cpp). + EOM } ] |