Understanding an Open Source Search Platform: Swirl

The team behind Swirl created a platform capable of connecting to multiple databases and searching through a single unified source.

That's Swirl Search for you.

⭐️ Swirl on GitHub

This blog covers the second part of Swirl Security Overview (User Data, Metadata, and Credentials). This is a three-part series that guides you about Swirl’s security.

This blog is the second part of the Swirl Security overview. We have the first part, which discusses and explains Swirl in general. Read the first part here. 👇

Swirl Search: Open Source Enterprise Search 🔍 to Securely 🔐 Search your Data.

Saurabh Rai for SWIRL ・ Nov 22 '23

#opensource #python #productivity #programming

Credential Storage

Swirl securely stores user credentials in its database to enhance the metasearch experience. This storage includes credentials for SearchProviders and OAUTH2 client details. Swirl also logs user activities in the same database and can save these logs to a file if configured.
Built on the Python/Django platform, Swirl ensures all portal URLs and APIs are accessed via HTTPS authentication.

To set up Swirl:

1. Begin by creating an administrator account. Instructions can be found in the Swirl Super User Creation Guide.

2. You can add and manage more users through the Django admin console. For a comprehensive guide, visit Django Admin Documentation.

Object Ownership

In Swirl, every object, such as SearchProviders, Search, and Results, is owned by a specific user. Typically, users, even those in the admin group, can only see their own Search and Result objects. However, there’s an exception: if a SearchProvider object owned by an admin has its ‘shared’ property set to ‘True,’ all users can view it. Rest assured, Swirl ensures any static credentials from such providers are automatically hidden.

If you want detailed insights on object ownership, visit the Swirl Admin Guide on Object Ownership.

Credential Sharing

Swirl supports the sharing of static credentials such as bearer tokens. These credentials can be shared by the admin user, as noted above. OAUTH2 credentials cannot be shared between users.

Securing the Swirl database so that the admin cannot access any data directly gives additional protection; please contact Swirl support for help.

Logging

In production mode, with logging level INFO, no search or result information is present in the logs. The output associated with a user running a search will be similar to:

2023-08-24 14:45:30,360 INFO     admin search 545 FULL_RESULTS_READY 20 16.4

2023-08-24 14:45:30,366 INFO     admin results 545 RelevancyMixer 20

Should an error occur, there will be diagnostic information; however, it may be limited unless Swirl is run in debug mode.

User Provisioning

For scenarios involving multiple users, Swirl offers the following provisions:

• User Permissions: Swirl equips users with the necessary permissions.
• Sharing SearchProviders: Users can access SearchProviders owned by the admin account.
• Access Methods: These tasks can be achieved either through the Django admin portal or the provided built-in APIs.
• Permission Requirement: To utilize Swirl’s features, users must possess the relevant permissions mentioned below.
• Support for OIDC: Swirl Metapipe and Swirl Enterprise support user provisioning via Open ID Connect (OIDC). This is facilitated through Microsoft SSO, leveraging the same APIs and processes.

User Permissions

The following Django permissions are required to use Swirl:

User Permissions Table

Swirl recommends granting these permissions to a group and then adding new users to that group. Additional permissions (like add and change querytransform and searchprovider) can be granted to a power user or developer group.

For more information on Swirl permissions, see Permissioning Normal Users.

Metasearch Authentication

SearchProviders

SearchProvider Configuration:

A SearchProvider is essentially a configuration of a Connector.

While administrators typically create and share these configurations with standard users, those with the right permissions can also create their own.

If you would like detailed insights, you can visit SearchProvider Object Reference.

Executing a Search:

Once logged in, users can initiate a search using one of the following methods:

1. A GET request with either the ?q= or ?qs= parameter.
2. A POST request with JSON data.
3. The integrated Galaxy UI.

Initiating a search generates a ‘Search’ object within the Swirl database. This object:

• Keeps track of the search’s progress, including any status messages from the configured SearchProviders.
• Is owned by the user who initiated the search.

For a deeper dive, refer to:

• Search Object Reference
• Result Objects Reference.

Supported Authentication Schemes

When Swirl sends a query to a configured source, the following schemes are supported:

• Plain-text credentials, such as incorporating them in a URL
• HTTP Basic, Digest, and Proxy authentication
• Bearer tokens
• API tokens
• OAuth2

Where Swirl Stores Credentials

Swirl saves credentials, including plain text, API, and bearer tokens, within SearchProviders. These SearchProviders are specific records in the Swirl database. When non-admin users view shared SearchProviders owned by an administrator, the credential details are concealed for security.

For more details, please refer to Shared SearchProviders and Query Transformations.

Where Swirl Stores OAuth2 Data
In Swirl, when OAuth2 SearchProviders are set up:

1. The user’s email address is saved in the Django User database.
2. This email and a unique session key are used to obtain an OAuth2 token.
3. This token is stored in the Swirl database if the “subscribe” feature is activated.
4. If the “subscribe” feature isn’t needed, the token is temporarily kept in the session cache and isn’t saved elsewhere.

Search Expiration Service

For applications that handle sensitive data:

• The “Retention” property on Search objects can be adjusted to a value above 1.
• Setting this value ensures that the Search and its associated Result objects will be permanently removed after a specified duration, such as an hour, day, week, month, or even longer.

For detailed insights, visit The search Expiration Service Guide.

Dive Deeper into Swirl’s Security

If you haven’t already or need a refresher, read the previous blog in our series for a comprehensive understanding of Swirl’s security measures.

Support & Explore Swirl on GitHub: Visit Swirl’s GitHub page and consider giving us a 🌟.

⭐️ Swirl on GitHub