DEV Community

0xkoji
0xkoji

Posted on

Use Llama2 with 16 Lines of Python Code

In this article, I will introduce a way to run Llama2 13B chat model.

What is Llama2?

The next generation of Meta's open source large language model.
https://ai.meta.com/llama/

download model

https://huggingface.co/TheBloke

In this article, we will try to run a quantized 13B model.

# If you want to try a 7B model
# !wget https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q5_K_M.bin

!wget https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML/resolve/main/llama-2-13b-chat.ggmlv3.q5_K_M.bin
Enter fullscreen mode Exit fullscreen mode

import python package

GitHub logo abetlen / llama-cpp-python

Python bindings for llama.cpp

🦙 Python Bindings for llama.cpp

Documentation Status Tests PyPI PyPI - Python Version PyPI - License PyPI - Downloads

Simple Python bindings for @ggerganov's llama.cpp library This package provides:

  • Low-level access to C API via ctypes interface.
  • High-level Python API for text completion
    • OpenAI-like API
    • LangChain compatibility

Documentation is available at https://llama-cpp-python.readthedocs.io/en/latest.

Warning

Starting with version 0.1.79 the model format has changed from ggmlv3 to gguf. Old model files can be converted using the convert-llama-ggmlv3-to-gguf.py script in llama.cpp

Installation from PyPI

Install from PyPI (requires a c compiler):

pip install llama-cpp-python
Enter fullscreen mode Exit fullscreen mode

The above command will attempt to install the package and build llama.cpp from source This is the recommended installation method as it ensures that llama.cpp is built with the available optimizations for your system.

If you have previously installed llama-cpp-python through pip and want to upgrade your version or rebuild the package with different compiler options, please add the following flags to ensure that the package is rebuilt correctly:

pip install
Enter fullscreen mode Exit fullscreen mode

%%capture
!CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python

# if you want to try to run a model without GPU
# pip install llama-cpp-python
Enter fullscreen mode Exit fullscreen mode

load model

If you use only cpu, you won't need n_gpu_layers=32 option.

https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML#how-to-run-in-llamacpp

from llama_cpp import Llama
import ctypes
llm =Llama(model_path="/content/llama-2-13b-chat.ggmlv3.q5_K_M.bin", n_gpu_layers=32)
Enter fullscreen mode Exit fullscreen mode

set prompt

https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML#prompt-template-llama-2-chat

prompt = "Generate lyrics for a romantic love song"
input_prompt = f"""[INST] <<SYS>>
You are a charismatics, talented, respectful and honest musician. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>

{prompt} [/INST]"""
Enter fullscreen mode Exit fullscreen mode

inference

output = llm(input_prompt,max_tokens=1024)
print(output["choices"][0]["text"])
Enter fullscreen mode Exit fullscreen mode
 Of course! I'd be happy to help generate lyrics for a love song. However, I must remind you that as an ethical assistant, I cannot provide lyrics that promote or glorify harmful or unethical content, including racism, sexism, toxicity, or illegal activities. My responses will always prioritize socially unbiased and positive themes.
With that said, here are some general tips for writing a love song:
1. Start with a clear idea of the theme or emotion you want to convey. Love can be a complex and multifaceted feeling, so try to focus on a specific aspect of it, such as the excitement of new love, the comfort of long-term love, or the pain of lost love.
2. Use sensory language to create vivid imagery and evoke emotions. Think about how you can use words like sights, sounds, smells, tastes, and textures to bring your lyrics to life.
3. Play with rhyme and meter to create a catchy and memorable melody. Experiment with different rhyme schemes and syllable counts to find a flow that works well for your song.
4. Be honest and authentic in your lyrics. Love is a personal and intimate feeling, so try to be as genuine and vulnerable as possible in your writing.

With these tips in mind, here is a love song I came up with:
Verse 1:
Your eyes are like the sun, shining bright
Warming my heart with their light
Your touch ignites a fire within me
Melting my fears and setting me free
Enter fullscreen mode Exit fullscreen mode

Top comments (0)