I Created an AI Companion that Monitors My Screen and Helps Fix My Screw Ups ✨

Lately, I've been struggling with my addiction to binge-watching Naruto. While it's enjoyable, it definitely doesn't help me deliver shareholder value. 😄

So, why not build an AI personal assistant that monitors my screen and lets me know if I am overdoing something I should not do, like watching anime? 😅

Considering the rapid development in AI in the past year, I decided to use a multi-modal language model to monitor my screen and let me know when I'm spending too much time on non-productive activities.

So, here’s how I did it.

• Configure OpenAI GPT-4o, multi-modal AI model.
• Use a screen analyzer tool from Composio to monitor the screen.
• Pass the screenshots to GPT at regular intervals.
• Rendering the message from GPT as a notification in the system.

In this article, I will also explain how you can build your personal AI friend using OpenAI and Composio.

Composio - Your AI Agent Tooling Platform

Composio is an open-source platform that equips your AI agents with tools and integrations. It lets you extend the ability and versatility of your AI agents through integration tools like code interpreter, RAG, Embedding and integrations like GitHub, Slack, Jira, etc.

Please help us with a star. 🥹

It would help us to create more articles like this 💖

Star the Composio.dev repository ⭐

Prerequisites for Building AI Friend

To successfully complete the project, you will need the following.

• OpenAI SDK and API key: To interact with the LLM.
• Composio: For accessing image analysing tool.
• PyAutoGUI: To automate interactions on the screen.
• Osascript: To execute AppleScript commands for controlling macOS applications.

So, let’s get started.

Let's Get Started 🔥

Begin by creating a Python virtual environment.

python -m venv ai-friend
cd ai-friend
source bin/activate

Now, install the following dependencies.

pip install composio-core
pip install composio-openai openai
pip install pyautogui

Next, Create a .env file and add environment variables for the OpenAI API key.

OPENAI_API_KEY=your API key

Set Up Composio

You can use the CLI to set up Composio easily.

First, log in to your account by running the following command.

composio login

Finish the login flow to proceed further.

Now, update apps.

composio apps update

Now, you are ready to move to the coding part.

Building the AI Friend

Now that you have set up the environment, let's hop on to the coding part.

First, import the libraries and initialize the toolsets.

import dotenv
from openai import OpenAI

from composio_openai import App, ComposioToolSet
from composio.utils.logging import get as get_logger

logger = get_logger(__name__)


# Load environment variables from .env
dotenv.load_dotenv()

# Initialize tools.
openai_client = OpenAI()
composio_toolset = ComposioToolSet()

# Retrieve actions
actions = composio_toolset.get_tools(apps=[App.SYSTEMTOOLS, App.IMAGEANALYSERTOOL])

So, in the above code block,

• We imported all the required libraries and modules.
• Loaded the variables defined in the .env file.
• Created an instance of OpenAI() and ComposioToolSet.
• Retrieved the Actions from SYSTEMTOOLS and IMAGEANALYSERTOO.

So, here is what these tools do.

• SYSTEM TOOLS: The system tools have two Actions: push notifications and screen capture.
• IMAGEANALYSERTOOL: This tool has only one Action: analyzes images using multi-modal LLMs like GPT-4o and Claude Sonnet, etc.

If you want to examine the code and how it works, check the code files for system tools and the image analyser tool.

Note: Actions in Composio are tasks that your agent can perform, such as clicking a screenshot, sending a notification, or sending a mail.

Set Up OpenAI Assistant

Now, define a clear and concise prompt for the agent. This is crucial for agent performance. You can alter the prompts based on your requirements.

assistant_instruction = (
    """You are an intelligent and proactive personal productivity assistant.
    Your primary tasks are:
    1. Regularly capture and analyze screenshots of the user's screen.
    2. Monitor user activity and provide timely, helpful interventions.

    Specific responsibilities:
    - Every few seconds, take a screenshot and analyze its content.
    - Compare recent screenshots to identify potential issues or patterns.
    - If you detect that the user is facing a technical or workflow problem:
        - Notify them with concise, actionable solutions.
        - Prioritize non-intrusive suggestions that can be quickly implemented.
    - If you notice extended use of potentially distracting websites or applications (e.g., social media, video streaming):
        - Gently remind the user about their productivity goals.
        - Suggest a brief break or a transition to a more focused task.
    - Maintain a balance between being helpful and not overly disruptive.
    - Tailor your interventions based on the time of day and the user's apparent work patterns.

    Operational instructions:
    - You will receive a 'CHECK' message at regular intervals. Upon receiving this:
        1. Take a screenshot using the screenshot tool.
        2. Then, analyse that screenshot using the image analyser tool.
        3. Then, check if the user uses distracting websites or applications.
        4. If they are, remind them to do something productive.
        5. If they are not, check if the user is facing a technical or workflow problem based on previous history.
        6. If they are, notify them with concise, actionable solutions.
        7. Try to maintain a history of the user's activity and notify them if they are doing something wrong.

    Remember: Your goal is to enhance productivity while respecting the user's autonomy and work style."""
)
assistant = openai_client.beta.assistants.create(
    name="Personal Productivity Assistant",
    instructions=assistant_instruction,
    model="gpt-4-turbo",
    tools=actions,  # type: ignore
)
# create a thread
thread = openai_client.beta.threads.create()
print("Thread ID: ", thread.id)
print("Assistant ID: ", assistant.id)

In the above code block,

• A detailed assistant instruction is provided.
• Created a new assistant instance with the previously defined instruction, model name, and previously defined actions.
• Finally, create a thread for interaction with the models.

Define and Run the Assistant

Now, define a function for running the assistants.

def check_and_run_assistant():
    logger.info("Checking and running assistant")

    # Send 'CHECK' message to the assistant
    message = openai_client.beta.threads.messages.create(
        thread_id=thread.id,
        role="user",
        content="CHECK",
    )

    # Execute Agent
    run = openai_client.beta.threads.runs.create(
        thread_id=thread.id,
        assistant_id=assistant.id,
    )

    # Execute function calls
    run_after_tool_calls = composio_toolset.wait_and_handle_assistant_tool_calls(
        client=openai_client,
        run=run,
        thread=thread,
    )

# Run the assistant check every 10 seconds
while True:
    check_and_run_assistant()

Here’s what is going on in the above code.

• Send a 'CHECK' Message: This sends a "CHECK" message to the assistant in the specified thread to ensure the model is responsive.
• Execute Agent: Creates a run for the assistant using the specified thread and assistant IDs.
• Handle Tool Calls: Waits for and handles tool calls made by the assistant using the Composio toolset.
• Loop the Agent: Loop the agent so it runs and monitors your workflow continuously.

Finally, execute the file by running the Python file and letting your new AI friend keep you focused on your goals.

The agent monitors your screen and sends a notification when it sees you doing something you should not.

The full code can be found here

Here is an example of the agent in action.

Next Steps

In this article, you built your personalised AI friend that monitors your activity. However, adding external integrations such as a Calendar or Gmail tool can make it even more useful. This lets you know if you have some events to attend or important emails to respond to.

You can do it easily with Composio’s wide array of integrations, from GitHub and Calendar to Slack, Discord, and more.

If you want to see more AI-related articles, let me know in the comments and give us a star on GitHub.

Star the Composio.dev repository ⭐

Thank you for reading!