This is the companion blog for this video
.
In this short post I will show you how to get started with Quarkus, Langchain4j and Ollama to set up your local dev environment to developer GenAI applications. Ready ? Let's go !
Installing Ollama
Ollama is one of those magic those that makes developers happy, basically you install it and then you can run LLMs locally, it has also a embedded web server so that you make requests to it, just like you would do with OpenAI.
Head over to Ollamaand follow the installation instructions.
Open a terminal and type :
ollama list
It will list you the LLMs that are installed locally on your computer. You can easily add a new LLM by doing for instance :
ollama pull mistral:latest
Check the model page to see what is available, it's that easy !
Creating the Quarkus project
Your Quarkus app will need for now only 2 extensions : quarkus-resteasy-reactive and quarkus-langchain4j-ollama , you can also use this link and click on the big blue button "Generate your application"
Once unzipped, open your project in your favorite IDE.
First steps with Langchain4j and Quarkus
We need to define in the properties which LLM we will be using, open your application.properties and add this line :
quarkus.langchain4j.ollama.chat-model.model-id=llama2:latest
Now let's create our first AI Service, create a new interface called MyAIService.java :
package org.sebi;
import dev.langchain4j.service.UserMessage;
import io.quarki
verse.langchain4j.RegisterAiService;
@RegisterAiService
public interface MyAIService {
public String chat(@UserMessage String message);
}
The important thing to notice here is the @RegisterAiService annotation, this indicates to Quarkus that when calling the method chat it will be using the LLM that we defined in the properties.
Run your app : mvn quarkus:dev , this will start your application in dev mode.
Once it's running, press d in the terminal to open the devUI :
You should see the langchain4j widget containing a link called chat, you can click on it and it will bring you to an interface where can prompt your LLM !
At any point, you can change the model in the properties, you can also enable logging for the outgoing requests to see what is exactly sent to the LLM :
quarkus.langchain4j.ollama.chat-model.model-id=llama2:latest
quarkus.langchain4j.ollama.log-requests=true
That's pretty cool, isn't it ?
Exposing our AI Service as a rest endpoint
The DevUI is useful but now we want to make it possible to access our LLM from a rest endpoint, we already have the resteasy-reactive extension and you should also have already a resource class called GreetingResource , change the content with :
package org.sebi;
import jakarta.inject.Inject;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;
@Path("/chat")
public class GreetingResource {
@Inject
MyAIService myAIService;
@POST
@Produces(MediaType.TEXT_PLAIN)
@Consumes(MediaType.TEXT_PLAIN)
public String hello(String body) {
return myAIService.chat(body);
}
}
Here we simply inject our AI Service and call it in our POST method.
Try it out with this curl command :
curl -X POST -H 'Content-Type: text/plain' -d 'Tell me a joke' http://localhost:8080/chat
I got this one :
Sure, here's one:
Why don't scientists trust atoms?
Because they make up everything!
Adding a system prompt
Let's fine tune our prompt by providing a system message. Usually you can use this to define how the chat bot must behave. Go to your AI Service and add this annotation :
@SystemMessage("You are a funny bot and always reply with a poem!")
public String chat(@UserMessage String message);
Try your curl command again and it should reply with a joke in the form of a poem !
You can change as you want the system prompt to experiment, change the model, all of this without having to stop&start your app because you are running in dev mode !
That all for today, next time we will explore how to add memory to our bot !
Top comments (1)
Thanks for the post. I am getting the error below on Windows. What could be going wrong?
`2024-03-08 12:20:23,461 INFO io.quarkus Installed features: [awt, cdi, langchain4j, langchain4j-ollama, poi, qute, rest-client-reactive, rest-client-reactive-jackson, resteasy-reactive, smallrye-context-propagation, vertx]
2024-03-08 12:20:44,197 INFO io.qua.lan.oll.OllamaRestApi$Ollam... Request:
2024-03-08 12:20:46,283 ERROR io.qua.ver.htt.run.QuarkusErrorHan... HTTP Request to /chat failed, error id: 8893484c-d8ac-463c-9e4f-c29831a5c82e-1: org.jboss.resteasy.reactive.ClientWebApplicationException: Received: 'Internal Server Error, status code 500' when invoking: Rest Client method: 'io.quarkiverse.langchain4j.ollama.OllamaRestApi#generate'
at org.jboss.resteasy.reactive.client.impl.RestClientRequestContext.unwrapException(RestClientRequestContext.java:194)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.handleException(AbstractResteasyReactiveContext.java:324)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:175)
at org.jboss.resteasy.reactive.client.impl.RestClientRequestContext$1.lambda$execute$0(RestClientRequestContext.java:313)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:276)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:258)
at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:56)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: jakarta.ws.rs.WebApplicationException: Internal Server Error, status code 500
at io.quarkus.rest.client.reactive.runtime.DefaultMicroprofileRestClientExceptionMapper.toThrowable(DefaultMicroprofileRestClientExceptionMapper.java:19)
at io.quarkus.rest.client.reactive.runtime.MicroProfileRestClientResponseFilter.filter(MicroProfileRestClientResponseFilter.java:52)
at org.jboss.resteasy.reactive.client.handlers.ClientResponseFilterRestHandler.handle(ClientResponseFilterRestHandler.java:21)
at org.jboss.resteasy.reactive.client.handlers.ClientResponseFilterRestHandler.handle(ClientResponseFilterRestHandler.java:10)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.invokeHandler(AbstractResteasyReactiveContext.java:231)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
... 12 more
`