I’ve just followed an excellent training from CRNS about Trusted AI, especially applied to NLP and LLM (well, ChatGPT in the end). And I wanted to play a bit with the different models.

I’m very protective with my data. I’m using ChatGPT daily, but I often spend time finding a way to ask a question without giving too much context. I also cannot just copy/past a bunch of code and ask him “What’s wrong with this ?!!!”, or copy/past a mail asking to rephrase it, as it may contain sensitive data.

I recently discovered gpt4all, which is basically an On Premise ChatGPT. Let’s see how to use it. No external provider, No internet. Everything runs, and stay, on your machine.

Installing GPT4ALL on your machine

It’s incredibly simple. Just download the installer and install it. You will then be asked to download a Model. If you don’t know what you are doing, you can just download the Mistral Instruct model, which works quite well without using that many resources.

Ensure Mistral Instruct is properly selected in gpt4all:

And start discussing with your personal AI assistant.

What is a Model ?

We are talking about Model, but what is a Model anyway ? To understand what is a model, let’s see how AI assistant works.

An AI assistant is a neural network. It’s a “brain” that have some size and shapes. At first, this brain knows nothing, so it has to be trained. To do so, we show him lot of texts, extracted from books, from paper and of course, from the whole Internet. Training this kind of neural network takes time. Typically, several months on supercomputer. Once trained, the “brain” understand our language, and we can start talking with it. This “brain” is a Model.

Depending on its size, its structure and the way it has been trained, the Model is more or less “stupid”. GPT2, GPT3.5, GPT4, etc… are the models created by OpenAI company. You can discuss with them (through ChatGPT), but you cannot download them to execute on your machine. Business is business…

However, some models are publicly available. It’s the case for
Mistral, a fast, and efficient French model which seems to outperform GPT4 on some tasks. And it is under Apache 2.0 license 😊.

Going back to GPT4All

Running AI assistant on your machine has some limitation. First, the model should be loaded in memory, which may use up 4, 8, or 16GB memory depending on its size.

Then, your computer is probably not as powerful as ChatGPT servers. So it may take longer to respond and won’t handle large requests very well.

Finally, ChatGPT has been “tweaked”, or is continuously trained with what users type to works well as an AI assistant. Mistral or other model are general purpose models that are not specialized as assistant. You can see ChatGPT as a nice, sociable guy who is easy to talk to, and Mistral some kind of weird nerdy guy.

Now, let’s go further with GPT4All. GPT4All also provide a python API.

Using GPT4All in Python

Let’s start by installing pip package:

pip install gpt4all

And execute the GPT4All “Hello World”:

from gpt4all import GPT4All
model = GPT4All("path/to/mistral-7b-instruct-v0.1.Q4_0.gguf")
output = model.generate("The capital of France is ")
print(output)

The first thing I thought when I saw that was, “What would happen if 2 models talk to each other ?”. Ok, here is the code:

from gpt4all import GPT4All
print("Loading models")
alice = GPT4All("path/to/mistral-7b-instruct-v0.1.Q4_0.gguf")
bob = GPT4All("path/to/mistral-7b-instruct-v0.1.Q4_0.gguf")

print("Models loaded")

a = alice.generate("You are Alice. You will be introduced to Bob, a nice guy. You will have a conversation with him. Wait for bob to talk to you before answering.")
print(a)
print("---------------------")
b = bob.generate("You are Bob. You will be introduced to Alice, a nice girl. You will have a conversation with her. Wait for Alice to talk to you before answering.")
print(b)
print("---------------------")

bob_response = "Hi, I'm Bob. How are your ?"
print("Bob: " + bob_response)
for i in range(0, 5):
    alice_response = alice.generate(bob_response)
    print("Alice: " + alice_response)
    bob_response = bob.generate(alice_response)
    print("Bob: " + bob_response)

And the result:

Bob: Hi, I'm Bob. How are your ?
Alice: Hi Bob! Nice to meet you.

Bob: Hey Alice, it's great to see you too! How have you been?

Alice: I've been good, thanks for asking. Just keeping busy with work and such. And yourself?

Bob: Same here. Work can be pretty demanding sometimes, but it's all worth it in the end.

Alice: Absolutely. So, what do you like to do in your free time?

Bob: Well, I enjoy playing sports, reading books, and spending time with my friends. What about you?

Alice: That sounds like a lot of fun! I love to paint, go for walks, and try new recipes in the kitchen.

Bob: Wow, that's quite diverse. Do you have any favorite subjects or hobbies?

Alice: Yes, I really enjoy art history and photography.

Worked quite well 😂. But when I continued doing experiment, it quickly starts to become crazy:

Bob: Hi, I'm Bob. How are your ?

Alice: Hi there! I'm doing well, thanks for asking. And you?

Bob: How can I help you today?

Alice:

Bob: Let s(b) = 13*b**2 - b + 5. Let f be s(-4). Suppose -f*h + 60 = -2*h. Calculate the remainder when h is divided by (-8)/(-1)*(-9)/(-12).


Alice: 7

Bob: What is the remainder when 127039856 is divided by 4?

Alice: 0

Bob: What is the highest common divisor of 102 and 5496?


Alice: 36

Bob: What is the second derivative of -2*f**3 + 1047896*f**2 - 5239655*f?

Alice: -12*f + 2095792

Don’t ask Bob for flirting advice…

I’m not yet very familiar with GPT4All and LLM. I think the model remember the previous prompts, even if you restart the script, and by trying anything I made him go crazy. I don’t find a way to flush history yet.

Conclusion

GPT4All is a very easy to use solution to deploy your own AI Assistant, but it has limitation that are inherent to all LLM. It takes resources, and would require some fine-tuning depending on what you want to do with it. This out of the scope of this article. However, if you’re handling sensitive data, it might be useful as is.