DEV Community

Cover image for Meet Casper, the Astral Librarian ✨, A Discord Bot for Searching the Web
Andrew Luchuk
Andrew Luchuk

Posted on

Meet Casper, the Astral Librarian ✨, A Discord Bot for Searching the Web

What I built

Casper, the Astral Librarian, is a Discord bot for searching the internet. Unlike other web searching discord bots, Casper does not restrict search results to a single search engine, instead returning results from a whole host of search engines.

Out of the box, Casper supports a number of features provided by the underlying search engine, SearXNG, including searching specific engines by formatting queries like this:

!<ENGINE_NAME> my search
Enter fullscreen mode Exit fullscreen mode

For example,

!reddit nvidia 3000 series review
Enter fullscreen mode Exit fullscreen mode

These features are described in greater detail in the SearXNG documentation

Here is a list of some of the search engines Casper currently supports:

  • google
  • bing
  • duckduckgo
  • qwant
  • reddit
  • stackoverflow
  • github
  • bitbucket
  • gitlab
  • docker hub
  • deviant art
  • imdb
  • pypi
  • npm
  • packagist
  • rubygems
  • yahoo
  • vimeo
  • youtube
  • twitter
  • wolfram alpha

In addition to these features, I plan to add some exciting features like allowing users to add their own search engines to Casper as well as user and channel based customization options.

If things go really well, I may also port Casper to other messaging platforms too!

Category Submission:

  • Wacky Wildcards
  • Integration Innovators

App Link

Discord invite link:
Invite Casper to your Server

Source code:
https://github.com/speratus/indexor-discord-bot

Screenshots

Here is a series of screenshots demonstrating Casper in Use:

Slash command suggestion

Typing a search query

Results from the search

Screenshots of Kubernetes config

List of nodes in Kubernetes cluster according to Kubernetes control panel (identifying information has been removed):

Nodes in Kubernetes resource pool

Same list, but showing the nodes running as instances running in Linode's control panel (identifying information has been removed):

Linodes incorporated into Kubernetes resource pool

Description

Casper is a Discord bot built with Python, discord.py, Docker, Kubernetes, and SearXNG.. The main goal of Casper is to allow users to search for web results directly from within Discord. Secondary goals include a high degree of flexibility, enabling users to search using their preferred search engine, and, to that end, supporting the widest array of search engines possible.

Finally, Casper allows users to share search results with each other, enabling them to establish trust and common sources of information as well as a discussion of the quality of sources.

There are several major components of Casper:

Link to Source Code

Main Casper source code:

GitHub logo speratus / indexor-discord-bot

A Discord Bot that searches everywhere on the internet

Casper (Formerly, Indexor)

Casper is a Discord bot that searches the web for you. Casper gets the best results from around the web and returns them directly to your Discord channel.

Unlike other web searching Discord bots, Casper is not restricted to getting results from a single search engine. Instead Casper is powered by the metasearch engine SearXNG to get results from many sources simultaneously.

In addition to Casper's current features, there are many exciting features planned for future updates, including:

  • Multiple pages of results
  • Results caching for faster response times
  • Support for getting results from specific search engines
  • User-Added search engines (Anything powered by the OpenSearch standard)
  • Server and User based custom configurations
  • Direct messaging the bot to get answers from anywhere on the web
  • Support for other message platforms such as Slack, Telegram, Matrix.org, and others.

Installation

The easiest way to get Access to Casper is to follow…

Search functionality core library:

GitHub logo speratus / indexor-core

Supporting library for the Casper Discord bot

Indexor Core

This library provides the core searching functionality required by the Casper discord bot. At the moment, its features are fairly minimal, but the majority of Casper's planned features will be added here.

Examples of which features will be supported by this library:

  • Results caching
  • Improved support for getting results from specific search engines
  • User based customization options
  • User-added search engines

Installation

You can install indexor_core by running the following command:

pip install git+https://github.com/speratus/indexor-core.git#egg=indexor_core
Enter fullscreen mode Exit fullscreen mode

Usage

Here is a basic usage example:

from indexor_core.config import Config
from indexor_core.searcher import search

search_xng_instance_url = 'https://example.com'

c = Config(engine_url=search_xng_instance_url)

results = search('hello world', c)
Enter fullscreen mode Exit fullscreen mode





The Indexor core library contains the primary search feature and will contain other planned features such as results caching and user based searching preferences.

Permissive License

MIT License

Background

I came up with the idea for Casper on a whim, thinking how interesting it would be to get search results directly in a Discord server.

I also saw this as an opportunity learn a couple of technologies I had been interested in, but had never created any projects with: Docker and Kubernetes.

How I built it

I deployed Casper on top of Linode's Kubernetes engine. I chose Linode Kubernetes engine for this project so that it would be easy to scale, update and rollback as necessary. This also lets me exercise my interest in learning Kubernetes and deploying an application using it.

To deploy Casper on Linode's Kubernetes engine, I had to overcome several obstacles:

  • Packaging the source code as a Docker image to which I could transfer necessary project information while keeping it out of the source code for the sake of privacy
docker build -t casper:1.0.2
Enter fullscreen mode Exit fullscreen mode
  • Creating a deployment for the SearXNG instances that had the configuration required to be compatible with the Casper source code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: searxng-deploy
spec:
  replicas: ...
  selector:
    matchLabels:
      app: searxng
  template:
    metadata:
      labels:
        app: searxng
    spec:
      containers:
        - name: searxng
          image: searxng/searxng
          env:
            - name: INSTANCE_NAME
              value: 'casper-engine'
            - name: BASE_URL
              value: 'http://n.n.n.n'
          volumeMounts:
            - name: settings
              mountPath: /etc/searxng/settings.yml
              subPath: settings.yml
      volumes:
        - name: settings
          configMap:
            name: searxng-settings
Enter fullscreen mode Exit fullscreen mode
  • Creating a Kubernetes service for the SearXNG instances so that they could be load balanced as well as simultaneously enabling the Casper source to communicate with the instances and preventing the SearXNG instances from being accessible outside of the Kubernetes Cluster
apiVersion: v1
kind: Service
metadata:
  name: lb-searxng
spec:
  selector:
    app: searxng
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
Enter fullscreen mode Exit fullscreen mode
  • Creating a configmap for the SearXNG instance that could be read as the settings.yml file and which also includes the required values to enable Casper to function.
kubectl create configmap searxng-settings --from-file=PATH/TO/FILE/searxng/settings.yml
Enter fullscreen mode Exit fullscreen mode
  • Creating a Kubernetes secret to store the Discord bot's token without exposing it to outside access.
kubectl create secret generic ...
Enter fullscreen mode Exit fullscreen mode
  • Creating a deployment for the bot itself which drew from the necessary secrets and had the proper environment variables to communicate with the SearXNG instances.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: casper-deploy
spec:
  replicas: ...
  selector:
    matchLabels:
      app: casper
  template:
    metadata:
      labels:
        app: casper
    spec:
      containers:
        - name: casper-bot
          image: speratus/casper:1.0.2
          env:
            - name: DISCORD_TOKEN
              valueFrom:
                secretKeyRef:
                  name: ...
                  key: ...
            - name: ENGINE_URL
              value: 'http://n.n.n.n'
Enter fullscreen mode Exit fullscreen mode

Once I had tested out each of these steps in detail and identified working settings for each component, deployment was as easy as running a few commands. Once that was done, Casper was live with no headaches.

NOTE: In these samples, I replaced IP address with n to indicate that anybody wishing to follow along should use their own information instead. I also used ... to indicate regular values that need to be replaced with the information specific to your context as well.

Additional Resources/Info

SearXNG's documentation for creating and running an instance:
https://docs.searxng.org/admin/index.html

SearXNG Source:

GitHub logo searxng / searxng

SearXNG is a free internet metasearch engine which aggregates results from various search services and databases. Users are neither tracked nor profiled.



Privacy-respecting, hackable metasearch engine

Searx.space lists ready-to-use running instances.

A user, admin and developer handbook is available on the homepage.

SearXNG install SearXNG homepage SearXNG wiki AGPL License Issues commits weblate SearXNG logo


Contact

Ask questions or chat with the SearXNG community (this not a chatbot) on

IRC
#searxng on libera.chat which is bridged to Matrix.
Matrix
#searxng:matrix.org

Setup

Translations

Help translate SearXNG at Weblate

Contributing

Are you a developer? Have a look at our development quickstart guide, it's very easy to contribute. Additionally we have a developer documentation.

Codespaces

You can contribute from your browser using GitHub Codespaces:

  • Fork the repository
  • Click on the <> Code green button
  • Click on the Codespaces tab instead…

Planned Features

  • Multiple pages of results
  • Results caching for faster response times
  • Improved support for getting results from specific search engines
  • User-added search engines (Any sites supporting the OpenSearch standard)
  • Server and User based settings
  • Direct messaging Casper to get results in addition to using the slash command
  • Support for other messaging platforms such as Slack, Telegram, Matrix.org, and others.
Unsupported features

Currently, Casper does not support SearXNG's special queries features.

Top comments (0)