I built an AI Agent to validate my PR without actually doing it myself 🚀⚡

TL; DR

In Composio, we review tens of pull requests every week.

That takes a lot of time, so I tried to involve an AI to help us validate all the pull requests before reviewing them.

I built an AI agent to review incoming pull requests, saving me time and energy.

Here is how I did it:

• We track a repository when somebody raises a pull request.
• Forward the pull request event data to an event listener.
• Use an LLM to review the PR.
• Finally, post the review as a comment in the PR and send a summary in Slack.

What are AI Agents?

But before anything, let's get familiar with AI agents.

AI agents are systems powered by AI models that can autonomously perform tasks, interact with their environment, and make decisions based on their programming and the data they process.

Your AI agent tooling platform 🛠️

Composio is an open-source platform that offers over 150 production-ready tools and integrations such as GitHub, Slack, Code Interpreter, and more to empower AI agents to accomplish complex real-world workflows.

Please help us with a star. 🥹

It would help us to create more articles like this 💖

Star the Composio.dev repository ⭐

Let's Get Started 🔥

As with every Python project, first, create a virtual environment.

python -m venv pr-agent
cd pr-agent
source bin/activate

Now, install the required libraries.

pip install composio-core composio-langchain /
langchain-openai/
python-dotenv

A brief description of libraries

• The composio-core is the main library for accessing and configuring tools and integrations. It also has a CLI API to manage integrations and triggers conveniently.
• The composio-langchain is the LangChain plug-in for Composio. It lets you use all the LlamaIndex functionalities with Composio tools.
• The langchain-openai is an additional library from LangChain that enables you to use OpenAI models within its framework.
• python-dotenv loads environment variables from a .env file into your Python project's environment, making it easier to manage configuration settings.

Next, Create a .env file and add environment variables for the OpenAI API key.

OPENAI_API_KEY=your API key

Configure the Integrations*🔧*

Composio allows you to configure SlackBot and GitHub without writing any code for the integration. Composio handles all the user authentication and authorization flows, so you can focus on shipping faster.

You can either do it from the CLI or use the dedicated dashboard.

But before that, log in to Composio from the CLI and update apps by running the following commands.

composio login
composio apps update

Complete the login flow to use Composio CLI API.

From CLI

The Composio core SDK allows you to manage integrations from the terminal.

Execute the following command to configure a Slackbot and a GitHub user account.

composio add slackbot
composio add github

Complete the authentication flow to add both integrations to your Composio account.

From Dashboard

You can also manually add integrations from the dashboard.

First, go to the tools catalogue section, find the integration you want to add, and click on it. In this case, it is the Slackbot.

Now, click on Setup SlackBot integration on the right.

Click on save to add it to your account and connect to the SlackBot app.

Once you finish the integration flow, your live integration will appear in the Integrations section.

Repeat the same process to add the GitHub integration.

Add Triggers for New GitHub PRs ⚙️

Newt, we set up a trigger for our GitHub integration to fetch event data whenever there is a new pull request.

Again, you can do it from either the CLI or the dashboard.

From CLI, you only need to execute the following command.

composio triggers enable github_pull_request_event

Also, you can add triggers from the dashboard from the GitHub page.

Go to the trigger section and add the triggers you need.

Building the Agentic Workflow 🏗️

Now that we have finished setting up integrations and triggers let's hop on to the coding part.

📦 Import Packages and Define Tools 🛠️

Create a new Python file and paste the following codes.

import os
from dotenv import load_dotenv
from composio_langchain import Action, ComposioToolSet
from langchain import hub
from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain_openai import ChatOpenAI

from composio.client.collections import TriggerEventData

load_dotenv()

# Initialize the ComposioToolSet
composio_toolset = ComposioToolSet()

# Define the tools
tools = composio_toolset.get_actions(
    actions=[
        Action.GITHUB_GET_CODE_CHANGES_IN_PR,
        Action.GITHUB_PULLS_CREATE_REVIEW_COMMENT,
        Action.SLACKBOT_CHAT_POST_MESSAGE,
    ]
)

Here is what is going on in the code above

• We imported the packages and modules needed for the project.
• Set up .env variables to the environment variable with load_dotenv().
• Initialized the ComposioToolSet().
• Then, we defined the tool's variable with a few actions, each performing a specific task.
• Action.GITHUB_GET_CODE_CHANGES_IN_PR: Get the code changes in a GitHub PR.
• Action.GITHUB_PULLS_CREATE_REVIEW_COMMENT: Create a review comment in the PR.
• Action.SLACKBOT_CHAT_POST_MESSAGE: Posts a message in Slack via the configured SlackBot.

Define the Agent 🤖

Next, we define our LangChain Agent and the LLM.

# Initialize the language model
llm = ChatOpenAI(model="gpt-4o")
# Define the code review assistant prompt
code_review_assistant_prompt = f"""
        You are an experienced code reviewer.
        Your task is to review the provided file diff and give constructive feedback.

        Follow these steps:
        1. Identify if the file contains significant logic changes.
        2. Summarize the changes in the diff in clear and concise English within 100 words.
        3. Provide actionable suggestions if there are any issues in the code.

        Send the summary of the PR review to the"  "" +os.environ['CHANNEL_ID']+"  "" channel on Slack. Slack doesn't have Markdown, so send a plain text message.
        Also, add the comprehensive review to the PR as a comment.
"""

prompt = hub.pull("hwchase17/openai-functions-agent")
combined_prompt = prompt+code_review_assistant_prompt
query_agent = create_openai_functions_agent(llm, pr_agent_tools, combined_prompt)
agent_executor = AgentExecutor(agent=query_agent, tools=pr_agent_tools, verbose=True)

In the above code,

• We initialized the language model with GPT-4o.
• We defined a detailed prompt for the Agent specifying what steps to take.
• We pulled the system prompt from the LangChain hub for the openai-functions-agent.
• Then, we initialized create_openai_functions_agent with the language model, tools, and prompts.
• Finally, we initialized the agent_executor with the agent and tools we defined earlier.

Note: You can find the Slack channel ID from its URL, which usually starts with "C*"; for example, the channel ID in the URL https://app.slack.com/client/T074SRB4FGS/C073RUF0UQ5 is C074RUF0UQ5.

The Agent will post the summary of the PR in the specified channel.

Define the Event Listener

The next step is to set up the event listener. This will receive the payloads from the trigger events in Slack.

The payloads contain the required event information, such as PR changes, timestamps, etc.

@listener.callback(filters={"trigger_name": "github_pull_request_event"})
def review_new_pr(event: TriggerEventData) -> None:
    # Using the information from Trigger, execute the agent
    code_to_review = str(event.payload)
    query_task = f "Review the following code changes: {code_to_review}"
    # Execute the agent
    res = agent_executor.invoke({"input": query_task})
    print(res)

print("Listener started!")
print("Create a pr to get the review")
listener.listen()

In the above code block,

• The callback function review_new_pr is invoked when the trigger event in GitHub matches github_pull_request_event.
• The code changes are extracted from the event payload.
• The Code changes are passed to the agent_executor to execute the detailed steps mentioned in the code_review_assistant_prompt using the provided language model and the Actions.

Now, run the Python file to set up the event listener.

The agent will act when you make a new PR in the configured GitHub repository.

It will retrieve and review the code, post the review in the PR as a comment, and post the summary of the PR review to the configured Slack channel.

Let's connect! 🔌

You can join our community to engage with maintainers and contribute as an open-source developer. Don't hesitate to visit our GitHub repository to contribute and create issues related to Composio. Dev.

The source for this tutorial is available here:

https://github.com/ComposioHQ/composio/tree/master/python/examples/pr_agent/pr_agent_langchain

Thank you for reading!