DEV Community

Cover image for Xpenser - AI Expense Manger using Twilio & Gemini AI with WhatsApp Integration
Amit Wani
Amit Wani

Posted on

Xpenser - AI Expense Manger using Twilio & Gemini AI with WhatsApp Integration

This is a submission for the Twilio Challenge

What I Built

Xpenser is not just a simple expense manager. It is an advanced expense management tool designed to simplify logging and analyzing your spending habits. With robust features, including seamless WhatsApp integration powered by Twilio, Xpenser stands out as a unique solution for personal finance management.

Demo

Checkout the Demo on YouTube at:

Experience Xpenser live https://xpenser.amitwani.dev.

Explore the project on GitHub:

GitHub logo mtwn105 / xpenser-whatsapp-bot

AI Expense Manger with Twilio & Gemini AI

Xpenser

AI Expense Manager with WhatsApp

Built using Twilio & Gemini AI

Xpenser2

Demo

You can find Xpenser live here: https://xpenser.amitwani.dev.

Join the Twilio Sandbox WhatsApp Number: +14155238886 by sending join per-mud

Tech Stack

Backend: NodeJS, Express

Frontend: Angular, Typescript

Database: MongoDB

Deployment: Docker, Northflank

AI: Gemini 1.5 Flash

Twilio: Verify, Functions, WhatsApp

Architecture

WhatsApp Integration

WhatsApp Integration

Portal

Xpenser Portal




To test the WhatsApp integration, join the Twilio Sandbox WhatsApp Number: +14155238886 by sending join per-mud

Workflow

User Registration

Users register on the web portal, verifying their phone numbers via OTP (Powered by Twilio Verify).

User Registration

User Registration OTP

Dashboard

The dashboard offers detailed analytics, including AI-driven insights, category-wise distribution, and average spending.

Dashboard

Expenses

Users can easily add, manage, and delete expenses. The AI automatically categorizes expenses upon saving.

Add Expense

Expense List

WhatsApp Integration

Join the Twilio Sandbox WhatsApp Number: +14155238886 by sending join per-mud

WhatsApp Join

Log new expenses effortlessly via WhatsApp. The AI interprets user messages to gather details for saving the expense.

WhatsApp Log Expense

Request insights on spending, such as top categories or average spending. The AI provides answers based on the last 100 expenses.

WhatsApp Expense Analysis

Tech Stack

Backend: NodeJS, Express
Frontend: Angular, Typescript
Database: MongoDB
Deployment: Docker, Northflank
AI: Gemini 1.5 Flash
Twilio: Verify, Functions, WhatsApp

Architecture

WhatsApp Integration

WhatsApp Integration

Portal

Xpenser Portal

Twilio and AI

Twilio

Xpenser leverages three Twilio services: Verify, Functions and WhatsApp Messaging

Verify

Twilio's Verify service sends OTPs for user authentication.

Send OTP

const verification = await client.verify.v2.services(otpServiceId)
      .verifications
      .create({ to: req.body.phonenumber, channel: 'sms' });
Enter fullscreen mode Exit fullscreen mode

Verify OTP

 const verification = await client.verify.v2.services(otpServiceId)
      .verificationChecks
      .create({ to: req.body.phonenumber, code: req.body.code })
Enter fullscreen mode Exit fullscreen mode

Functions

Twilio Functions handles incoming WhatsApp messages, interfacing with the backend and Gemini AI.

Generated functions using

 twilio serverless:init twilio --typescript
Enter fullscreen mode Exit fullscreen mode
export const handler: ServerlessFunctionSignature = async function (
  context: Context<MyContext>,
  event: MyEvent,
  callback: ServerlessCallback
) {
  // Handle Incoming Message 
}
Enter fullscreen mode Exit fullscreen mode

WhatsApp Messaging

Twilio's Programmable WhatsApp Messaging facilitates user interaction through WhatsApp.

Send WhatsApp Message using TwiML

const twiml = new Twilio.twiml.MessagingResponse();
twiml.message("Please ask about spends you have done or spend you want to log.");
callback(null, twiml);
Enter fullscreen mode Exit fullscreen mode

AI

Gemini 1.5 Flash, a Google AI Model, enhances Xpenser with advanced analytics and predictions.

Expense Analytics

Analyzing user expense history to provide insights and respond to user queries.

const analyticsModel = genAI.getGenerativeModel({
    model: "gemini-1.5-flash",
    systemInstruction: "Based on below expenses done by user give answer to user. Optimize as WhatsApp message reply and add emojis.\n\nExpense History:\n" + expenseCsvText,
  });

const generationConfig = {
  temperature: 0.6,
  topP: 0.95,
  topK: 64,
  maxOutputTokens: 8192,
  responseMimeType: "application/json",
};

const chatSession = model.startChat({
  generationConfig
});

const result = await chatSession.sendMessage(event.Body);
const aiResponse = result.response.text()

Enter fullscreen mode Exit fullscreen mode

Expense Categorization

Categorizing expenses based on user-provided descriptions.

const analyticsModel = genAI.getGenerativeModel({
    model: "gemini-1.5-flash",
    systemInstruction: "detect the category and subcategory from below\n1. Housing - Rent, Taxes, Insurance, Utilities, Repairs, Improvement, Fees\n2. Transportation - Payments, Fuel, Insurance, Repairs, Public, Parking, Tolls, Licensing\n3. Food - Groceries, Dining, Coffee, Delivery, Snacks, etc.",
  });

const generationConfig = {
  temperature: 0.6,
  topP: 0.95,
  topK: 64,
  maxOutputTokens: 8192,
  responseMimeType: "application/json",
};

const chatSession = model.startChat({
  generationConfig
});

const result = await chatSession.sendMessage(event.Body);
const aiResponse = result.response.text()

Enter fullscreen mode Exit fullscreen mode

Additional Prize Categories

Xpenser qualifies for the Twilio Times Two prize category by utilizing multiple Twilio features: Verify, Functions, and WhatsApp Messaging.

Future Scope

  • Spending Habits: Use AI to identify spending patterns and suggest ways to optimize expenses based on historical data.
  • Mobile Applications: Develop native iOS and Android applications for enhanced accessibility and a more seamless user experience.
  • Custom Reports: Allow users to generate customizable reports based on various parameters such as date range, categories, or tags.
  • Investment Tracking: Add features to track investments and provide insights into portfolio performance.
  • Manage Categories: Create and manage custom categories and sub-categories for a more personalized experience.

Conclusion

It was a great experience to work on this project while exploring Twilio services and integrating AI to enhance the user experience. Thanks to Dev.to team for this amazing hackathon.

Top comments (2)

Collapse
 
rishabk7 profile image
Rishab Kumar

Love what you have built!

Collapse
 
thegeekyamit profile image
Amit Wani

Thanks a lot 😁