What I built
Casper, the Astral Librarian, is a Discord bot for searching the internet. Unlike other web searching discord bots, Casper does not restrict search results to a single search engine, instead returning results from a whole host of search engines.
Out of the box, Casper supports a number of features provided by the underlying search engine, SearXNG, including searching specific engines by formatting queries like this:
!<ENGINE_NAME> my search
For example,
!reddit nvidia 3000 series review
These features are described in greater detail in the SearXNG documentation
Here is a list of some of the search engines Casper currently supports:
- bing
- duckduckgo
- qwant
- stackoverflow
- github
- bitbucket
- gitlab
- docker hub
- deviant art
- imdb
- pypi
- npm
- packagist
- rubygems
- yahoo
- vimeo
- youtube
- wolfram alpha
In addition to these features, I plan to add some exciting features like allowing users to add their own search engines to Casper as well as user and channel based customization options.
If things go really well, I may also port Casper to other messaging platforms too!
Category Submission:
- Wacky Wildcards
- Integration Innovators
App Link
Discord invite link:
Invite Casper to your Server
Source code:
https://github.com/speratus/indexor-discord-bot
Screenshots
Here is a series of screenshots demonstrating Casper in Use:
Screenshots of Kubernetes config
List of nodes in Kubernetes cluster according to Kubernetes control panel (identifying information has been removed):
Same list, but showing the nodes running as instances running in Linode's control panel (identifying information has been removed):
Description
Casper is a Discord bot built with Python, discord.py, Docker, Kubernetes, and SearXNG.. The main goal of Casper is to allow users to search for web results directly from within Discord. Secondary goals include a high degree of flexibility, enabling users to search using their preferred search engine, and, to that end, supporting the widest array of search engines possible.
Finally, Casper allows users to share search results with each other, enabling them to establish trust and common sources of information as well as a discussion of the quality of sources.
There are several major components of Casper:
- The bot itself
- The search library containing the main search features. This library is what will enable Casper to be ported to other messaging platforms
- The SearXNG Instance, powering web-wide searches
Link to Source Code
Main Casper source code:
speratus / indexor-discord-bot
A Discord Bot that searches everywhere on the internet
Casper (Formerly, Indexor)
Casper is a Discord bot that searches the web for you. Casper gets the best results from around the web and returns them directly to your Discord channel.
Unlike other web searching Discord bots, Casper is not restricted to getting results from a single search engine. Instead Casper is powered by the metasearch engine SearXNG to get results from many sources simultaneously.
In addition to Casper's current features, there are many exciting features planned for future updates, including:
- Multiple pages of results
- Results caching for faster response times
- Support for getting results from specific search engines
- User-Added search engines (Anything powered by the OpenSearch standard)
- Server and User based custom configurations
- Direct messaging the bot to get answers from anywhere on the web
- Support for other message platforms such as Slack, Telegram, Matrix.org, and others.
Installation
The easiest way to get Access to Casper is to follow…
Search functionality core library:
speratus / indexor-core
Supporting library for the Casper Discord bot
Indexor Core
This library provides the core searching functionality required by the Casper discord bot. At the moment, its features are fairly minimal, but the majority of Casper's planned features will be added here.
Examples of which features will be supported by this library:
- Results caching
- Improved support for getting results from specific search engines
- User based customization options
- User-added search engines
Installation
You can install indexor_core
by running the following command:
pip install git+https://github.com/speratus/indexor-core.git#egg=indexor_core
Usage
Here is a basic usage example:
from indexor_core.config import Config
from indexor_core.searcher import search
search_xng_instance_url = 'https://example.com'
c = Config(engine_url=search_xng_instance_url)
results = search('hello world', c)
The Indexor core library contains the primary search feature and will contain other planned features such as results caching and user based searching preferences.
Permissive License
Background
I came up with the idea for Casper on a whim, thinking how interesting it would be to get search results directly in a Discord server.
I also saw this as an opportunity learn a couple of technologies I had been interested in, but had never created any projects with: Docker and Kubernetes.
How I built it
I deployed Casper on top of Linode's Kubernetes engine. I chose Linode Kubernetes engine for this project so that it would be easy to scale, update and rollback as necessary. This also lets me exercise my interest in learning Kubernetes and deploying an application using it.
To deploy Casper on Linode's Kubernetes engine, I had to overcome several obstacles:
- Packaging the source code as a Docker image to which I could transfer necessary project information while keeping it out of the source code for the sake of privacy
docker build -t casper:1.0.2
- Creating a deployment for the SearXNG instances that had the configuration required to be compatible with the Casper source code
apiVersion: apps/v1
kind: Deployment
metadata:
name: searxng-deploy
spec:
replicas: ...
selector:
matchLabels:
app: searxng
template:
metadata:
labels:
app: searxng
spec:
containers:
- name: searxng
image: searxng/searxng
env:
- name: INSTANCE_NAME
value: 'casper-engine'
- name: BASE_URL
value: 'http://n.n.n.n'
volumeMounts:
- name: settings
mountPath: /etc/searxng/settings.yml
subPath: settings.yml
volumes:
- name: settings
configMap:
name: searxng-settings
- Creating a Kubernetes service for the SearXNG instances so that they could be load balanced as well as simultaneously enabling the Casper source to communicate with the instances and preventing the SearXNG instances from being accessible outside of the Kubernetes Cluster
apiVersion: v1
kind: Service
metadata:
name: lb-searxng
spec:
selector:
app: searxng
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8080
- Creating a
configmap
for the SearXNG instance that could be read as thesettings.yml
file and which also includes the required values to enable Casper to function.
kubectl create configmap searxng-settings --from-file=PATH/TO/FILE/searxng/settings.yml
- Creating a Kubernetes secret to store the Discord bot's token without exposing it to outside access.
kubectl create secret generic ...
- Creating a deployment for the bot itself which drew from the necessary secrets and had the proper environment variables to communicate with the SearXNG instances.
apiVersion: apps/v1
kind: Deployment
metadata:
name: casper-deploy
spec:
replicas: ...
selector:
matchLabels:
app: casper
template:
metadata:
labels:
app: casper
spec:
containers:
- name: casper-bot
image: speratus/casper:1.0.2
env:
- name: DISCORD_TOKEN
valueFrom:
secretKeyRef:
name: ...
key: ...
- name: ENGINE_URL
value: 'http://n.n.n.n'
Once I had tested out each of these steps in detail and identified working settings for each component, deployment was as easy as running a few commands. Once that was done, Casper was live with no headaches.
NOTE: In these samples, I replaced IP address with n
to indicate that anybody wishing to follow along should use their own information instead. I also used ...
to indicate regular values that need to be replaced with the information specific to your context as well.
Additional Resources/Info
SearXNG's documentation for creating and running an instance:
https://docs.searxng.org/admin/index.html
SearXNG Source:
searxng / searxng
SearXNG is a free internet metasearch engine which aggregates results from various search services and databases. Users are neither tracked nor profiled.
Privacy-respecting, hackable metasearch engine
Searx.space lists ready-to-use running instances.
A user, admin and developer handbook is available on the homepage.
Contact
Ask questions or chat with the SearXNG community (this not a chatbot) on
- IRC
- #searxng on libera.chat which is bridged to Matrix.
- Matrix
- #searxng:matrix.org
Setup
- A well maintained Docker image, also built for ARM64 and ARM/v7 architectures.
- Alternatively there are up to date installation scripts.
- For individual setup consult our detailed Step by step instructions.
- To fine-tune your instance, take a look at the Administrator documentation.
Translations
Help translate SearXNG at Weblate
Contributing
Are you a developer? Have a look at our development quickstart guide, it's very easy to contribute. Additionally we have a developer documentation.
Codespaces
You can contribute from your browser using GitHub Codespaces:
- Fork the repository
- Click on the
<> Code
green button - Click on the
Codespaces
tab instead…
Planned Features
- Multiple pages of results
- Results caching for faster response times
- Improved support for getting results from specific search engines
- User-added search engines (Any sites supporting the OpenSearch standard)
- Server and User based settings
- Direct messaging Casper to get results in addition to using the slash command
- Support for other messaging platforms such as Slack, Telegram, Matrix.org, and others.
Unsupported features
Currently, Casper does not support SearXNG's special queries features.
Top comments (0)