Preparations
To start, you need to compile llama.cpp. You can follow their README for instructions.
The server is compiled alongside other targets by default.
Once you have the server running, we can continue. We will use PHP Resonance framework.
Troubleshooting
Obtaining Open-Source LLM
I recommend starting either with llama2 or Mistral. You need to download the pretrained weights and convert them into GGUF format before they can be used with llama.cpp.
Starting Server Without a GPU
llama.cpp supports CPU-only setups, so you don't have to do any additional configuration. It will be slow, but you will still have tokens generated.
Running With a Low VRAM Memory
You can try quantization if you don't have enough VRAM on your GPU to run a specific model. That lowers the response quality and the memory the model needs to use. Llama.cpp has a utility to quantize models:
$ ./quantize ./models/7B/ggml-model-f16.gguf ./models/7B/ggml-model-q4_0.gguf q4_0
10GB of VRAM is enough to run most quantized models.
Starting llama.cpp Server
While writing this tutorial, I had a server started with a command:
$ ./server
--model ~/llama-2-7b-chat/ggml-model-q4_0.gguf
--n-gpu-layers 200000
--ctx-size 2048
--parallel 8
--cont-batching
--mlock
--port 8081
cont-batching
parameter is essential, because it enables continuous batching, which is an optimization technique that allows parallel request.
Without it, even with multiple parallel
slots, the server could answer to only one request at a time. cont-batching
allows the server to respond to multiple completion requests in parallel.
Configuring Resonance
All you need to do is add a configuration section that specifies the llama.cpp server location:
[llamacpp]
host = 127.0.0.1
port = 8081
Testing
Resonance has built-in commands that connect to llama.cpp and issue requests.
You can send a sample prompt through llamacpp:completion
:
$ php ./bin/resonance.php llamacpp:completion "How to write a 'Hello, world' in PHP?"
To write a "Hello, world" in PHP, you can use the following code:
<?php
echo "Hello, world!";
?>
This will produce a simple "Hello, world!" message when executed.
Programmatic Use
In your class, you need to use Dependency Injection to inject LlamaCppClient
:
<?php
namespace App;
use Distantmagic\Resonance\LlamaCppClient;
use Distantmagic\Resonance\LlamaCppCompletionRequest;
#[Singleton]
class LlamaCppGenerate
{
public function __construct(protected LlamaCppClient $llamaCppClient)
{
}
public function doSomething(): void
{
$request = new LlamaCppCompletionRequest('How to make a cat happy?');
$completion = $this->llamaCppClient->generateCompletion($request);
// each token is a chunk of text, usually few-several letters returned
// from the model you are using
foreach ($completion as $token) {
swoole_error_log(SWOOLE_LOG_DEBUG, (string) $token);
if ($token->isLast) {
// ...do something else
}
}
}
}
Summary
In this tutorial, we went through how to start llama.cpp server and connect to it with Resonance.
If you like Resonance, check us out on GitHub and give us a star! :)
Top comments (4)
Thanks for sharing your concerns and tips. I am building a framework around it to mitigate those resources issues, so hopefully I will be able to mitigate most of them.
My understanding is you can also tweak it by adjusting slot sizes to not ever exceed the maximum context, so in general I’m more optimistic. :D I will remember your advice, however.
Looks very promising, congrats on the integration!