DEV Community

Cover image for 21 Open Source LLM Projects to Become 10x AI Developer
Anmol Baranwal Subscriber for Latitude

Posted on • Originally published at blog.latitude.so

21 Open Source LLM Projects to Become 10x AI Developer

The time of AI is high especially because of powerful LLMs like GPT-4o and Claude.

Today, I'm covering 21 open source LLM projects that can help you to build something exciting and integrate AI into your project.

As a developer, I can confidently say that AI is not as scary as others make it sound and those who don't learn will get left behind.

Let's cover it all.

gif


By the way, I'm part of Latitude and we're building an open source LLM development platform. You can join the waitlist at ai.latitude.so.

You would be able to do a lot of cool stuff like:

⚡ Deploy prompts as api endpoints.

⚡ Automated evaluations using LLMs.

⚡ Collaborate on prompt engineering.

I'm very confident that you will love it after its release!

Join Waitlist ⭐️

llm platform by latitude


1. Vanna - Chat with your SQL database.

vanna

 

Vanna is an MIT-licensed open-source Python RAG (Retrieval-Augmented Generation) framework for SQL generation.

Basically, it's a Python package that uses retrieval augmentation to help you generate accurate SQL queries for your database using LLMs.

It's perfect for developers like me who are not very fond of SQL queries!

low level diagram on how vanna works

Vanna works in two easy steps - train a RAG model on your data, and then ask questions that will return SQL queries that can be set up to run on your database automatically.

how vanna works

You don't need to know how this whole stuff works to use it.
You just have to train a model, which stores some metadata, and then use it to ask questions.

Get started with the following command.

pip install vanna
Enter fullscreen mode Exit fullscreen mode

To make things a little easier, they have built user interfaces that you can use as a starting point for your own custom interface. Find all the interfaces including Jupyter Notebook and Flask.

variations

You can read the docs and you can try this Colab notebook in case you want to see how it works after training.

flask UI gif

Watch this demo for a complete walkthrough!

vanna demo

They have 9.5k+ stars on GitHub and are built using Python.

Star Vanna ⭐️


2. Khoj - Your AI second brain.

khoj

 

Khoj is the open source, AI copilot for search. Easily get answers without having to sift through online results or your own notes.

For me, the concept seems exciting and It can help me in researching a lot of projects.

Khoj can understand your Word, PDF, org-mode, markdown, plaintext files, GitHub projects, and even Notion pages.

type of documents

It's available as a Desktop app, Emacs package, Obsidian plugin, Web app, and Whatsapp AI. Obsidian with Khoj might be the most powerful combo!

You can get started with Khoj locally in a few minutes with the following commands.

$ pip install khoj-assistant
$ khoj
Enter fullscreen mode Exit fullscreen mode

Watch it in action!

khoj walkthrough gif

Some of the exciting features:

✅ You can share your notes and documents to extend your digital brain.

✅ Your AI agents have access to the internet, allowing you to incorporate real-time information.

✅ You'll get a fast, accurate semantic search on top of your docs.

✅ Your agents can create deeply personal images and understand your speech.

For instance, saying: "Create a picture of my dream house, based on my interests". It will draw this!

image generation

Read all the features including shareable chat, online chat, file summarization, and complete details in various categories.

You can read the docs and you can try Khoj Cloud to try it quickly.

Watch the complete walkthrough on YouTube!

It has 12k stars on GitHub and is backed by YCombinator.

Star Khoj ⭐️


3. Flowise - Drag & drop UI to build your customized LLM flow.

flowiseai

 

Flowise is an open source UI visual tool to build your customized LLM orchestration flow & AI agents.

We shouldn't compare any projects but I can confidently say this might be the most useful one among the projects listed here!

flowise gif

Get started with the following npm command.

npm install -g flowise
npx flowise start
OR
npx flowise start --FLOWISE_USERNAME=user --FLOWISE_PASSWORD=1234
Enter fullscreen mode Exit fullscreen mode

This is how you integrate the API.

import requests

url = "/api/v1/prediction/:id"

def query(payload):
  response = requests.post(
    url,
    json = payload
  )
  return response.json()

output = query({
  question: "hello!"
)}
Enter fullscreen mode Exit fullscreen mode

integrations

You can read the docs.

Cloud host is not available so you would have to self-host using these instructions.

Let's explore some of the use cases:

⚡ Let's say you have a website (could be a store, an e-commerce site, or a blog), and you want to scrap all the relative links of that website and have LLM answer any question on your website. You can follow this step-by-step tutorial on how to achieve the same.

scraper

⚡ You can also create a custom tool that will be able to call a webhook endpoint and pass in the necessary parameters into the webhook body. Follow this guide which will be using Make.com to create the webhook workflow.

webhook

There are a lot of other use cases such as building a SQL QnA or interacting with API. Explore and build cool stuff!

FlowiseAI has 27.5k stars on GitHub and has more than 14k forks so it has a good overall ratio.

Star Flowise ⭐️


4. LLAMA GPT - a self-hosted, offline, ChatGPT like chatbot (Powered by Llama 2).

llama gpt

 

LlamaGPT is a self-hosted, offline, ChatGPT-like chatbot, powered by Llama 2. It's 100% private and your data doesn't leave your device.

In the recent versions, they have also provided support for Code Llama models and Nvidia GPUs.

You can install it from the Umbrel App Store or you can also install it with Kubernetes.

You can read about the supported models on the docs.

It's very simple to use!

llama gpt video

I know, at this point it feels confusing that there are so many ways to run LLM locally. As a developer, I think it's important to evaluate which method works for our situation!

They have 10k+ stars on GitHub and offer 2 packages.

Star LLAMA GPT ⭐️


5. LocalAI - free OpenAI alternative.

localAI

 

LocalAI is free, open source, and considered as an alternative to OpenAI.

LocalAI acts as a drop-in replacement REST API that’s compatible with OpenAI (Elevenlabs, Anthropic... ) API specifications for local AI inferencing.

It allows you to run LLMs, generate images, and audio (and not only) locally or on-prem with consumer-grade hardware, supporting multiple model families. The best part is that it does not require GPU.

I never thought that there would be such an option so it's a goldmine for devs who don't want to pay much. Plus, it allows to generate of Text, Audio, Video, and Images and also has voice cloning capabilities. What more do you need?

You can watch the complete walkthrough by Semaphore CI!

There are a lot of integration options and developers have built awesome stuff such as Extension for attaching LocalAI instance to VSCode.

You can read the quickstart guide and how to run it with kubernetes. Find all the resources including how to run it on AWS, k8sgpt, and more.

They have 21k+ stars on GitHub and are on the v2.18 release.

Star LocalAI ⭐️


6. Continue - enable you to create an AI software development system.

continue

 

Continue is one of the best AI code assistants I've seen in my developer journey.

You can connect any models and any context to build custom autocomplete and chat experiences inside VS Code and JetBrains.

You can easily set it up. Below are some of the snapshots while I was installing it.

step 1

Step 1

 

step 2

run it through terminal

 

step2 complete

step2 complete

 

step3

run it through terminal

 

step3 complete

step3 complete

 

After you've configured it, you're all set to use all the amazing concepts it provides.

They have a lot of awesome features such as:

Tab to autocomplete code suggestions.

autocomplete

Ask questions about your codebase.

questions

Understand terminal errors immediately.

errors

Kick off actions with slash commands.

commands

Refactor functions where you are coding.

refactor

Read about all the features.

You will have to install the VSCode extension from the marketplace and then read the quickstart guide.

You can read the docs.

You can also watch this basic demo on YouTube!

They have 13k+ stars on GitHub and are built using TypeScript.

Star Continue ⭐️


7. Chat2DB - AI-driven data management platform.

chat2db

 

Chat2DB is an AI-first data management, development, and analysis tool.

Its core is AIGC (Artificial Intelligence Generation Code), which can convert natural language into SQL, SQL into natural language, and automatically generate reports, taking efficiency to another level.

Even operations that do not understand SQL can use it to quickly query business data and generate reports.

When you do any operation, it will give you some suggestions.
For instance, when you are doing database development, it will help you generate SQL directly in natural language, give you SQL optimization suggestions, help you analyze SQL performance, analyze SQL execution plan, and can also help you quickly generate SQL test data, system code, etc. It's actually very powerful :)

They have excellent support for multi-data sources and can easily integrate up to 17 different database types including PostgreSQL, MySQL, MongoDB, and Redis.

databases

You can download or try it in the browser.

Let's see some of the exciting features:

✅ Intelligent reports.

reports

✅ Data Exploration.

data exploration

✅ SQL Development.

SQL Development

 

sql development

You can read the quickstart guide on the official docs.

They have 14k+ stars on GitHub and are on the release v3.2.

Star Chat2DB ⭐️


8. LibreChat - Enhanced ChatGPT Clone.

libre chat

 

LibreChat is a free, open source AI chat platform. This Web UI offers vast customization, supporting numerous AI providers, services, and integrations.

It serves all AI conversations in one place with a familiar interface, and innovative additions, for as many users as you need.

Some of the features are:

✅ Upload and analyze images seamlessly with advanced models like Claude 3, GPT-4, Gemini Vision, Llava, and Assistants.

✅ Chat with files using various powerful endpoints using OpenAI, Azure, Anthropic, and Google.

✅ Multilingual UI with support for 20+ languages.

Diverse Model options including OpenAI, BingAI, Anthropic (Claude), Azure OpenAI, and Google’s premier machine learning offerings.

ai model image

You can read the quickstart guide to get started.

Watch this video for the complete walkthrough!

They have 15k+ stars on GitHub and offer 4 packages.

Star LibreChat ⭐️


9. Lobe Chat - modern-design LLMs/AI chat framework.

lobe chat

 

An open-source, modern-design ChatGPT/LLMs UI/Framework.
Supports speech-synthesis, multi-modal, and extensible (function call) plugin systems. You can deploy your private OpenAI with one click.

journey

Let's see some of the exciting features of LobeChat:

✅ Multi-Model Service Provider Support.

multi service

They have expanded our support to multiple model service providers, rather than being limited to a single one.

Find the complete list of 10+ model service providers that they support.

✅ Assistant Market.

Assistant Market

In LobeChat's Assistant Market, creators can discover an innovative community that brings together numerous carefully designed assistants.

market

There are so many awesome applications there. WOW!

✅ Model Vision Recognition.

Model Vision Recognition

LobeChat now supports large language models with visual recognition capabilities such as OpenAI's gpt-4-vision, Google Gemini Pro vision, and Zhipu GLM-4 Vision, enabling LobeChat to have multimodal interaction capabilities.

Users can easily upload or drag and drop images into the chat box, and the assistant will be able to recognize the content of the images and engage in intelligent conversations.

✅ Text to Image Generation.

Text to Image Generation

You can directly utilize the Text-to-image tool during conversations with the assistant.

By using the power of AI tools such as DALL-E 3, MidJourney, and Pollinations, assistants can now implement it properly.

✅ Local Large Language Model (LLM) Support.

Local Large Language Model (LLM) Support.

With the powerful infrastructure of Ollama AI and the community's collaborative efforts, you can now engage in conversations with a local LLM (Large Language Model) in LobeChat!

By running the following Docker command, you can experience conversations with a local LLM in LobeChat.

docker run -d -p 3210:3210 -e OLLAMA_PROXY_URL=http://host.docker.internal:11434/v1 lobehub/lobe-chat
Enter fullscreen mode Exit fullscreen mode

✅ Progressive Web App (PWA).

Progressive Web App (PWA)

They have adopted Progressive Web App PWA technology, which is a modern web technology that elevates web applications to a near-native app experience.

✅ Custom Themes.

custom themes

 

Read about all of the features and use cases.

You can self-host or deploy it using docker. The ecosystem of lobe chat provides 4 packages: lobehub/ui, lobehub/icons, lobehub/tts, and lobehub/lint.

They also provide plugins market where you can find lots of useful plugins that can be used to introduce new function calls and even new ways to render message results. If you want to develop your own plugin, refer to 📘 Plugin Development Guide in the wiki.

plugins market

You can read the docs.

You can check the live demo. It's pretty cool!

demo snapshot

They have 35k+ stars on GitHub with more than 500 releases.

Star Lobe Chat ⭐️


10. MindsDB - The platform for customizing AI from enterprise data.

MindsDB

 

MindsDB is the platform for customizing AI from enterprise data.

With MindsDB, you can deploy, serve, and fine-tune models in real-time, utilizing data from databases, vector stores, or applications, to build AI-powered apps - using universal tools developers already know.

mindsdb

With MindsDB and its nearly 200 integrations to data sources and AI/ML frameworks, any developer can use their enterprise data to customize AI for their purpose, faster and more securely.

how MindsDB works

You can read the docs and quickstart guide to get started.

They currently support a total of 3 SDKs that is using using Mongo-QL, Python, and JavaScript.

There are several applications of MindsDB such as integrating with numerous data sources and AI frameworks so you can easily bring data and AI together to create & automate custom workflows.

The other common use cases include fine-tuning models, chatbots, alert systems, content generation, natural language processing, classification, regressions, and forecasting. Read more about the use cases and each of them has an architecture diagram with a little info.

use cases

For instance, the chatbot architecture diagram with MindsDB. You can read about all the solutions provided along with their SQL Query examples.

// SQL Query Example for Chatbot
CREATE CHATBOT slack_bot USING database='slack',agent='customer_support'; 
Enter fullscreen mode Exit fullscreen mode

chatbot

Just to tell you about the overall possibilities, you can check out How to Forecast Air Temperatures with AI + IoT Sensor Data. Exciting right :)

mindsdb

They have 25.4k+ stars on GitHub and are on the v24.7.2.0 with more than 200 releases. By the way, this is the first time I've seen 4 parts in any release as I always followed the semantic release.

Star MindsDB ⭐️


11. AutoGPT - more exciting than ChatGPT.

auto gpt

 

At the core of AutoGPT lies its primary project, a semi-autonomous agent driven by large language models (LLMs), designed to perform any tasks for you.

The AutoGPT project consists of four main components:

  • The Agent – also known as just "AutoGPT"
  • The Benchmark – AKA agbenchmark
  • The Forge
  • The Frontend

Read on how you can set up AutoGPT using the OpenAI key.

You can see this YouTube video by Fireship on what is AutoGPT.

You can also watch this AutoGPT tutorial by Sentral Media.

You can read the docs and check out the project board on what things are under development right now.

Even if you don't know much about AI, you can try AutoGPT to understand how you can save time and build cool stuff.

They have 164k+ stars on GitHub due to such an excellent use case and automation capabilities.

Star AutoGPT ⭐️


12. reor - self organizing AI note-taking app.

reor

 

One of the most exciting projects that I've seen so far, especially because it runs models locally.

Reor is an AI-powered desktop note-taking app. It automatically links related notes, answers questions on your notes, and provides semantic search.

Everything is stored locally and you can edit your notes with an Obsidian-like markdown editor. The project hypothesizes that AI tools for thought should run models locally by default.

Reor stands on the shoulders of the giants Ollama, Transformers.js & LanceDB to enable both LLMs and embedding models to run locally. Connecting to OpenAI or OpenAI-compatible APIs like Oobabooga is also supported.

I know you're wondering How can it possibly be self-organizing?

a. Every note you write is chunked and embedded into an internal vector database.
b. Related notes are connected automatically via vector similarity.
c. LLM-powered Q&A does RAG on the corpus of notes.
d. Everything can be searched semantically.

You can watch the demo here!

demo

One way to think about Reor is as a RAG app with two generators: the LLM and the human. In Q&A mode, the LLM is fed retrieved context from the corpus to help answer a query.

Similarly, in editor mode, we can toggle the sidebar to reveal related notes retrieved from the corpus. This is quite a powerful way of augmenting your thoughts by cross-referencing ideas in a current note against related ideas from your digital collection.

You can read the docs and download from the website. Mac, Linux & Windows are all supported.

They have also provided starter guides to help you get started.

get started guides

You can also watch this walkthrough!

They have 6.5k stars on GitHub and are built using TypeScript.

Star reor ⭐️


13. Leon - your open source personal assistant.

leon

 

Leon is an open source personal assistant who can live on your server. You're definitely curious, right? 😅

He does stuff when you ask him to. You can even talk to him and he will revert by talking to you. Similarly, you can also text him!

If you are a developer (or not), you may want to build many things that could help in your daily life. Instead of building a dedicated project for each of those ideas, Leon can help you with his Skills structure.

leon gif

If you want to, Leon can communicate with you by being offline to protect your privacy. This is a list of skills that Leon can do for now.

You should read the story behind Leon. You can also watch this demo to learn more about Leon.

features

This is the High-level architecture schema of Leon.

architecture

You can install Leon by using this command.

# install leon global cli
npm install --global @leon-ai/cli

# install leon
leon create birth
Enter fullscreen mode Exit fullscreen mode

You can read the docs.

Appwrite is one of the sponsors which says a lot about its overall impact.

It has 15k stars on GitHub and has released some drastic changes recently so make sure to read the docs with extra caution.

Star Leon ⭐️


14. Instrukt - Integrated AI in the terminal.

instrukt

 

Instrukt is a terminal-based AI-integrated environment. It offers a platform where users can:

  • Create and instruct modular AI agents.
  • Generate document indexes for question-answering.
  • Create and attach tools to any agent.

Instruct them in natural language and, for safety, run them inside secure containers (currently implemented with Docker) to perform tasks in their dedicated, sandboxed space.

It's built using Langchain, Textual, and Chroma.

Get started with the following command.

pip install instrukt[all]
Enter fullscreen mode Exit fullscreen mode

instrukt

There are a lot of exciting features such as:

✅ A terminal-based interface for power keyboard users to instruct AI agents without ever leaving the keyboard.

✅ Index your data and let agents retrieve it for question-answering. You can create and organize your indexes with an easy UI.

✅ Index creation will auto-detect programming languages and optimize the splitting/chunking strategy accordingly.

✅ Run agents inside secure docker containers for safety and privacy.

✅ Integrated REPL-Prompt for quick interaction with agents, and a fast feedback loop for development and testing.

✅ You can automate repetitive tasks with custom commands. It also has a built-in prompt/chat history.

You can read about all the features.

You can read the installation guide.

You can also debug and introspect agents using an in-built IPython console which is a neat little feature.

console debugging

Instrukt is licensed with an AGPL license meaning that it can be used by anyone for whatever purpose.

It is safe to say that Instrukt is an AI commander for your terminal.

It is a new project so they have around 240 stars on GitHub but the use case is damn good.

Star Instrukt ⭐️


15. Quivr - RAG Framework for building GenAI Second Brains.

quivr

 

Quivr, your second brain, utilizes the power of GenerativeAI to be your personal assistant! You can think of it as Obsidian but turbocharged with AI powers.

Quivr is a platform that enables the creation of AI assistants, referred to as Brain. These assistants are designed with specialized capabilities like some can connect to specific data sources, allowing users to interact directly with the data.

While others serve as specialized tools for particular use cases, powered by Rag technology. These tools process specific inputs to generate practical outputs, such as summaries, translations, and more.

Watch a quick demo of Quivr!

quivr gif

Some of the amazing features are:

✅ You can choose the type of Brain you want to use, based on the data source you wish to interact with.

✅ They also provide a powerful feature to share your brain with others. This can be done by sharing with individuals via their emails and assigning them specific rights.

sharing brain

✅ Quivr works offline, so you can access your data anytime, anywhere.

✅ You can access and continue your past conversations with your brains.

✅ But the best one that I loved is that you can literally install a Slack bot. Refer to this demo to see what you can do. Very cool!

Anyway, read about all the awesome stuff that you can do with Quivr.

You can read the installation guide and 60 seconds installation video. I really loved this idea!

You can read the docs.

stats

They have also provided guides on how to deploy Quivr with Vercel, Porter, AWS, and Digital Ocean.

They could provide a better free tier plan but it's more than enough to test things on your end.

It has 30k+ Stars on GitHub with 220+ releases which means they're constantly improving.

Star Quivr ⭐️


16. Open Interpreter - natural language interface for terminal.

Open Interpreter

 

Open Interpreter lets LLMs run code (Python, Javascript, Shell, and more) locally. You can chat with Open Interpreter through a ChatGPT-like interface in your terminal by running $ interpreter after installing.

This provides a natural-language interface to your computer's general-purpose capabilities:

✅ Create and edit photos, videos, PDFs, etc.
✅ Control a Chrome browser to perform research Plot, clean, and analyze large datasets.

I don't know about you, but their website made me say WOW!

Quickstart using this command.

pip install open-interpreter

// After installation, simply run:
interpreter
Enter fullscreen mode Exit fullscreen mode

You can read the quickstart guide.

You should read about the comparison to ChatGPT's Code Interpreter and the commands that you can use.

You can read the docs.

Open Interpreter works with both hosted and local language models. Hosted models are faster and more capable, but require payment while local models are private and free but are often less capable. Choose based on your use case!

They have 48k+ stars on GitHub and are used by 300+ developers.

Star Open Interpreter ⭐️


17. CopilotKit - 10x easier to build AI Copilots.

copilotKit

 

You will agree that it's tough to add AI features in React, that's where Copilot helps you as a framework for building custom AI Copilots.

You can build in-app AI chatbots, and in-app AI Agents with simple components provided by Copilotkit which is at least 10x easier compared to building it from scratch.

You shouldn't reinvent the wheel if there is already a very simple and fast solution!

They also provide built-in (fully-customizable) Copilot-native UX components like <CopilotKit />, <CopilotPopup />, <CopilotSidebar />, <CopilotTextarea />.

components

Get started with the following npm command.

npm i @copilotkit/react-core @copilotkit/react-ui
Enter fullscreen mode Exit fullscreen mode

This is how you can integrate a Chatbot.

A CopilotKit must wrap all components which interact with CopilotKit. It’s recommended you also get started with CopilotSidebar (you can swap to a different UI provider later).

"use client";
import { CopilotKit } from "@copilotkit/react-core";
import { CopilotSidebar } from "@copilotkit/react-ui";
import "@copilotkit/react-ui/styles.css"; 

export default function RootLayout({children}) {
  return (
    <CopilotKit url="/path_to_copilotkit_endpoint/see_below">
      <CopilotSidebar>
        {children}
      </CopilotSidebar>
    </CopilotKit>
  );
}
Enter fullscreen mode Exit fullscreen mode

You can read the docs and check the demo video.

You can integrate Vercel AI SDK, OpenAI APIs, Langchain, and other LLM providers with ease. You can follow this guide to integrate a chatbot into your application.

The basic idea is to build AI Chatbots very fast without a lot of struggle, especially with LLM-based apps.

You can watch the complete walkthrough!

CopilotKit has recently crossed 7k+ stars on GitHub with 300+ releases.

Star CopilotKit ⭐️


18. GPT Engineer - AI builds what you ask.

gpt engineer

 

GPT-engineer lets you specify software in natural language, sit back, and watch as an AI writes and executes the code, and you can ask the AI to implement improvements.

It's safe to say that it's an engineer who doesn't need a degree 😅

It's a commercial project for the automatic generation of web apps. It features a UI for non-technical users connected to a git-controlled codebase.

I know this feels confusing, so watch the below demo to understand how you can use GPT Engineer.

demo gif

You can get started by installing the stable release using this command.

python -m pip install gpt-engineer
Enter fullscreen mode Exit fullscreen mode

By default, gpt-engineer expects text input via a prompt file. It can also accept image inputs for vision-capable models. This can be useful for adding UX or architecture diagrams as additional context for GPT engineer. Read about all the awesome features.

If you want a complete walkthrough, watch this awesome demo by David!

I recommend checking out the roadmap to understand the overall vision.

roadmap

They have 51k+ stars on GitHub and are on the v0.3 release.

Star GPT Engineer ⭐️


19. Dalai - the simplest way to run LLaMA and Alpaca locally.

dalai

 

Dalai lets you run LLaMA and Alpaca on your computerP and is powered by llama.cpp, llama-dl CDN, and alpaca.cpp.

Dalai runs on operating systems such as Linux, Mac, and Windows so that's a plus point!

gif demo

Dalai is also an NPM package:

  • programmatically install
  • locally make requests to the model
  • run a dalai server (powered by socket.io)
  • programmatically make requests to a remote dalai server (via socket.io)

You can install the package using the below npm command.

npm install dalai
Enter fullscreen mode Exit fullscreen mode

latest package

You can read the memory requirements and how to archive elsewhere rather than home directory.

They have 13k stars on GitHub and are still in the very early stages.

Star Dalai ⭐️


20. OpenLLM - run LLMs as OpenAI compatible API endpoint in the cloud

open llm

 

OpenLLM lets developers run any open-source LLMs as OpenAI-compatible API endpoints with a single command.

⚡ Build for fast and production usage.
⚡ Support llama3, qwen2, gemma, etc, and many quantized versions full list.
⚡ OpenAI-compatible API & includes ChatGPT like UI.
⚡ Accelerated LLM decoding with state-of-the-art inference backends.
⚡ Ready for enterprise-grade cloud deployment (Kubernetes, Docker, and BentoCloud).

Get started with the following command.

pip install openllm  # or pip3 install openllm
openllm hello
Enter fullscreen mode Exit fullscreen mode

OpenLLM provides a chat user interface (UI) at the /chat endpoint for an LLM server. You can visit the chat UI at http://localhost:3000/chat and start different conversations with the model.

open llm

OpenLLM supports LLM cloud deployment via BentoML, the unified model serving framework, and BentoCloud, an AI inference platform for enterprise AI teams.

models

If you don't know, BentoCloud provides a fully managed infrastructure optimized for LLM inference with autoscaling, model orchestration, observability, and many more, allowing you to run any AI model in the cloud.

bento cloud console

Once the deployment is complete, you can run model inference on the BentoCloud console:

You can read about the supported models and how to start the LLM server.

Explore docs as you can also chat with a model in the CLI using openllm run and specifying model version - openllm run llama3:8b.

For people like me who love exploring walkthroughs, watch this demo by Matthew!

They have 9k+ stars on GitHub and have 100+ releases so it's growing at a rapid pace.

Star OpenLLM ⭐️


21. Unsloth - Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory.

unsloth

 

Unsloth makes finetuning large language models like Llama-3, Mistral, Phi-3, and Gemma 2x faster, use 70% less memory, and with no degradation in accuracy!

✅ What is finetuning?

If we want a language model to learn a new skill, a new language, some new programming language, or simply want the language model to learn how to follow and answer instructions like how ChatGPT functions, we do finetuning!

Finetuning is the process of updating the actual brains of the language model through some process called back-propagation. But, finetuning can get very slow and very resource intensive.

 

Unsloth can be installed locally or through another GPU service like Google Colab. Most use Unsloth through the interface Google Colab which provides a free GPU to train with.

Finetune for Free

Some of the things that stand out:

✅ Open source trains 5x faster, and the pro version claims to be 30x faster.

✅ No approximation methods are used resulting in a 0% loss in accuracy.

✅ No change of hardware, Works on Linux and Windows via WSL.

unsloth model stats

You can read the installation instructions and performance benchmarking tables on the website.

unsloth

You can read the docs and all the uploaded models on Hugging Face directly.

They have also provided a detailed guide on How to Finetune Llama-3 and Export to Ollama.

They have 12.5k+ stars on GitHub and it's an efficient solution.

Star Unsloth ⭐️


I hope you learned something new!

I believe that learning to use these powerful LLM is a choice and it's on you (as a developer) to find better productivity solutions for your use case.

Have a great day! Till next time.

If you like this kind of stuff,
please follow me for more :)
profile of Twitter with username Anmol_Codes profile of GitHub with username Anmol-Baranwal profile of LinkedIn with username Anmol-Baranwal

Follow Latitude for more content like this.

Top comments (24)

Collapse
 
anmolbaranwal profile image
Anmol Baranwal

You can chat with your SQL database, use AI copilot to search your documents and even use AI to build what you ask (from scratch). Lots of amazing applications!

Did you find anything useful?

I know it's long, but I always recommend reading a single article that can give you 5x value rather than multiple articles. Anyway, I'm sure you will love this :)

Collapse
 
namix profile image
Nick

Because of this post I started using Continue + Ollama + codellama to replace Copilot and I'm pretty happy with the result. It's not very fast, and Copilot sometimes has better answers, but for 90% of what I use it for it works fine a.t.m. and it saves me a few bucks. Thanks!

Collapse
 
anmolbaranwal profile image
Anmol Baranwal

Truth be told, I've never used Copilot and don't even know why! But that combo is good enough and saving a few bucks is definitely great since it does the work, you know!

I never expected someone could make such a drastic change to their workflow by reading a simple post. Change is often not preferred, and your comment made my day.
Thank you for reading Nick! ❤

Collapse
 
martinbaun profile image
Martin Baun

Love the piece, man. Very informative. What's your take on Flowise vs Langflow?

Collapse
 
anmolbaranwal profile image
Anmol Baranwal

I've never used Langflow, so as usual, I researched it on the official website. There is a detailed comparison of Langflow vs Flowise.

From what I've seen, I would still prefer Flowise for AI workflows and complex tasks. Can't say more 😅 Anyway, thanks for reading the post :)

Collapse
 
martinbaun profile image
Martin Baun

Nice, thanks! :)

Collapse
 
andrewbaisden profile image
Andrew Baisden

Damn, there is so much to study here thanks for the big resource 😁

Collapse
 
litlyx profile image
Antonio | CEO at Litlyx.com

Amazing like always Anmol!

Collapse
 
anmolbaranwal profile image
Anmol Baranwal

Thanks Antonio for your support 😄

Collapse
 
Sloan, the sloth mascot
Comment deleted
Collapse
 
jitendrachoudhary profile image
Jitendra Choudhary

Helpful List :)

Collapse
 
anmolbaranwal profile image
Anmol Baranwal

Thanks Jitendra 🙌
AI will definitely help us as developers :)

Collapse
 
dasheck0 profile image
Stefan Neidig

Some interesting tools on that list. Thanks for sharing. Will look into Leon as is looks promising

Collapse
 
anmolbaranwal profile image
Anmol Baranwal • Edited

Yep, it's good and above everything else it's easy to install :)

Collapse
 
hackerup profile image
Hacker Up

Cool

Collapse
 
syedmuhammadaliraza profile image
Syed Muhammad Ali Raza

👍

Collapse
 
kenmei profile image
Kenmei

Good post!

Some comments have been hidden by the post's author - find out more