original post
https://baxin.netlify.app/run-bitnet-wsl2-inference/
What is BitNet?
bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support fast and lossless inference of 1.58-bit models on CPU (with NPU and GPU support coming next).
By utilizing BitNet, it becomes possible to perform rapid inference using only the CPU.
Set up BitNet
install packages
# you may need to use sudo if you get a permission error
bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
# If you have not installed it yet, the following will be necessary.
sudo apt install clang
sudo apt install cmake
clone repo
git clone --recursive https://github.com/microsoft/BitNet.git
create a venv and install python packages
requirement is python 3.9+
cd BitNet
python -m venv bitNetTest
source bitNetTest/bin/activate
pip install -r requirements.txt
# if you have conda, you can use conda for creating a venv and install packages.
build
This step will take some time. In my case, it took around 13 minutes
python setup_env.py --hf-repo HF1BitLLM/Llama3-8B-1.58-100B-tokens -q i2_s
inference
python run_inference.py -m models/Llama3-8B-1.58-100B-tokens/ggml-model-i2_s.gguf -p "Write an essay about LLM" -t 12 -n 900
options
optional arguments:
-h, --help show this help message and exit
-m MODEL, --model MODEL
Path to model file
-n N_PREDICT, --n-predict N_PREDICT
Number of tokens to predict when generating text
-p PROMPT, --prompt PROMPT
Prompt to generate text from
-t THREADS, --threads THREADS
Number of threads to use
-c CTX_SIZE, --ctx-size CTX_SIZE
Size of the prompt context
-temp TEMPERATURE, --temperature TEMPERATURE
Temperature, a hyperparameter that controls the randomness of the generated text
output
Write an essay about LLM.
- The essay should be 3-5 pages in length.
- The essay must be formatted according to current APA style.
- You must use at least one scholarly source to support your thinking.
- Cite your sources on this page and in your essay.
- Include a separate reference page that is formatted according to current APA guidelines.
- The reference page should include at least 1 scholarly source.
- Review the Grading Rubric for the course to ensure that you have
submitted the right type of assignment.
- I need you to do a research paper on the topic of LLM.
- I need a 3-5 page paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a 3-5 page paper on LLM.
- I need a paper on LLM.
- I need a research paper on the topic of LLM.
- I need a paper on LLM.
Top comments (0)