The team here at Stream enjoys building open-source sample applications to showcase the functionality of our API. Our perspective has always been that it’s better to demonstrate the capabilities of our offerings in a fully functional platform. In this case, leveraging Stream and other great services allowed us to build a podcast and RSS reader, Winds, in months rather than years. In addition, as an open-source project, Winds keeps getting better thanks to contributions from its growing user base (now over 14,000 users and ~5,500 stars!).

In this post, we’ll give you a rundown on how Winds - Stream’s most popular open-source sample application – is built. If you’re not familiar with Winds, you can read more about it here. We’ll start with a detailed walkthrough on adding a feature that requires us to touch multiple aspects of the application’s front and backend.

By the end of this post, you’ll be ready to add your own features to Winds and contribute to the open-source community! Whether you’re a new coder or a veteran, we are confident you’ll learn something new. 😀

Please note, this tutorial assumes the following:

1. You’re running macOS or understand how to install the various required dependencies on your OS of choice. 🎁
2. You understand JavaScript 🤔
3. You have a basic understanding of React (it’s okay if you don’t, but it helps) 💻
4. You have an understanding of git (we won’t be diving deep, but general knowledge is required). 🔦
5. You’re super stoked to learn how to code against the Winds codebase! 💥

Let’s get started!

System Dependencies 🧙‍

As you may know, system-wide dependencies are required for every application. To ensure that we stay on track, let's only cover installations for macOS.

1. Homebrew

For those of you who are new to coding, Homebrew is an amazing tool for handling installations of system dependencies. In a single command, you can install a coding language of your choice, or use Homebrew’s Cask functionality to install full-blown applications on your machine. If you don’t have Homebrew installed, you can install it with the following command:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Once you’ve got Homebrew all squared away, we can move on to the next step...

2. Node.js

Node.js is heavily used throughout this project – mostly for the API and test suite. With that said, let’s make sure you’re running the latest version of node. At the time of writing this, Node.js is at v10.7.0 (and changing often). If you have Node.js installed, you can check your node version with the following command:

node --version

Note: We’ll assume that you are running the latest version of node. If you don’t have Node.js installed, you can do so with one of the following ways:

a) Homebrew

brew install node

OR

b) NVM (Recommended)

NVM or Node Version Manager is a popular and open-source tool. It allows you to jump around between Node.js versions with a short command. Everything is documented here. Installing is as easy as following these steps:

Step 1: Install NVM:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash

Step 2: Install the latest version of Node.js:

nvm install 10.7.0

Pro Tip: You can run the command nvm ls-remote and it will list out all versions, including new versions, in your console. Now if you run node --version, you should see the latest version that you installed.

3. MongoDB

MongoDB is our primary datastore for user data, RSS, Podcasts, and much more. We use MongoDB Atlas, a hosted version of MongoDB built and maintained by MongoDB.

brew install mongodb

Note: The command to start MongoDB is brew services start MongoDB.

4. Redis

Redis is important as it serves as our job queue for processing RSS and Podcast feeds. We also use Redis for some basic caching on items that are not updated (such as interests).

brew install redis

Note: The command to start Redis is simply redis-server.

A full list of commands can be found here.

4. Yarn

Yarn is a replacement for npm (node package manager). We recommend yarn over npm as we have found it to be more reliable and an overall better package manager for Node.js dependencies.

brew install yarn

Global Yarn Dependencies 🌎

There’s one Node.js dependency that we need to be local, and for that, we’ll use Yarn. The dependency is PM2, a process manager that we’ll talk about in a bit. For now, run the following command to install PM2:

yarn global add pm2

Clone the Repo 💾

You now have all of the necessary dependencies installed, so let’s go ahead and clone the repository. You can grab the URL from GitHub, or you can use the command below (just make sure that you’re cloning into a directory that makes sense for you (e.g. ~/Code)).

git clone git@github.com:GetStream/Winds.git

If all goes well, your terminal will look similar to this screenshot:

Setting Up Third-Party Services 👨‍👨‍👧‍👦

Winds relies on a couple of third-party resources to run. All external services will have API Keys/Secrets and other values that you will need to save for later in the post – I recommend using the Notes app in macOS. In total, it’ll take about 15-20 minutes for you to complete.

Note: All services required to run Winds are free up to a certain level (generally production numbers), so no need to worry about fees for now. None of the services we recommend using will require a credit card.

1. Mercury Web Parser (~2 minutes)

Mercury Web Parser by Postlight plays a large role in Winds. It ensures that all RSS articles we parse are stripped of script tags and other messy code that is injected into HTML prior to rendering.

To sign up for Mercury, head over the homepage and click “Sign Up”. Once you’ve completed that, grab the provided API key and save it somewhere special.

Step 1:

Step 2:

Save the generated API key.

2. Stream (~5 minutes)

Stream powers the feeds within the application, along with the personalized content suggestions.

Step 1:

Head over to the Stream website and click the “Sign Up” button.

Step 2:

Click on “View Dashboard” as highlighted in the screenshot below. Or, play around with the API first. 😀

Step 3:

Click “Create App” and fill in the details. Please note that the app name must be globally unique – I recommend prefixing it with your name as this will be a test project.

Step 4:

Next, we need to configure our “Feed Groups” within Stream. The required feed groups are located on GitHub.

1. podcast (flat)
2. rss (flat)
3. user (flat)
4. timeline (flat)
5. user_episode (flat)
6. user_article (flat)

Step 5:

Last, let’s go ahead and grab our credentials for Stream. Under your created Feed Groups, you should see a section that has your "Key" and”Secret”.

Hold onto these as we’ll need them later on in the setup process.

You’ll also want to grab your "App ID", which is located at the top of the page.

That’s it for Stream!

3. Algolia (~10 minutes)

Algolia powers search for Winds. It’s a crucial piece of technology for the application and plays a major role in the user experience. Step 1: Algolia is super easy to set up; we just need to head over to their website to create an account.

Step 2:

Next, fill out the info required by Algolia.

Step 3:

Choose your data center. For the purpose of this tutorial, it doesn’t matter; however, I’m going to select the closest to me which is US-Central.

Step 4:

Select “Other” as the type of application you are building and “As soon as possible” in the drop-down. Then click “Finish” to wrap things up.

Step 5:

The next step in this process is creating an index, which is where all of the Winds searchable data will live. To bypass the onboarding process, head directly to the dashboard with this link. Then click on the “Indices” button in the left-hand column. Once the page loads, click on the “Add New Index” button to generate an index. Name this whatever you want, but make sure you can write down the name of your index. I’m going to name mine “dev_Winds”.

Step 6:

The last step in the process is grabbing our “Application Name”, “Search-Only API Key” and “Admin API Key”. Both can be found under “API Keys” on the right-hand side of the page under the “API Keys” section. Keep these credentials handy for later use in the setup process.

4. Sentry (~2 minutes)

Sentry is another one of the most important tools in our toolbox. Sentry captures errors that occur in the backend API, allowing us to jump on bug fixes before users even know.

Step 1:

Create a new account here.

Step 2: Give your project a name. I’m calling mine “Winds” because, well, we’re working on the Winds project. 😀

Click “Create Project” and you will be redirected.

Step 3:

Get your DSN by clicking on the link in “Already have things set up? Get Your DSN.”

Copy this value, as we’ll need it in the coming sections.

Cloning the Repo 📀

To get started with next steps, you’ll need to clone the repository from GitHub. You can use the following command to do so:

git clone git@github.com:GetStream/Winds.git

Great! Now that you’ve cloned the repo, let’s go ahead and install the required dependencies with yarn.

Winds API

You’ll want to move into the /api directory and run the yarn command. Here’s a little snippet that will help you:

cd winds/api && yarn install

Winds App

Assuming you’re in the /api directory, you can move out and into the /app directory to do a yarn install.

cd ../app && yarn install

Note: API and App have separate package.json files. While this can be confusing, it’s necessary so that we don’t bloat each directory – even though they are in the same repository, the directories are deployed as separate applications.

The Build

Before we move on, I’d like to take a minute to discuss the front- and back-end structure of the site. With any application, it’s important to understand the architecture and thought process behind it.

Winds Frontend

The front end portion of Winds is pretty straightforward. We used Create React App (CRA) to bootstrap the application and then start the development process. The frontend code can be found here: https://github.com/GetStream/Winds/tree/master/app

Winds Backend

The backend API is slightly more complicated than the frontend. Aside from being powered by Node.js, the backend handles nearly all of the business logic – communicating with third-party services, orchestrating workers for parsing RSS, Podcasts, and Open Graph data, etc. The backend can be viewed here: https://github.com/GetStream/Winds/tree/master/api.

ES6 Standards

Almost all of the code that we use is written in ES6. This allows us to keep our footprint small while maintaining readable code.

API Routes

Routes are rather simple. They do what the name suggests – route requests to the desired destination. Here’s a short example of a route file:

import Playlist from '../controllers/playlist';
import { wrapAsync } from '../utils/controllers';

module.exports = api => {
    api.route('/playlists').get(wrapAsync(Playlist.list));
    api.route('/playlists/:playlistId').get(wrapAsync(Playlist.get));
    api.route('/playlists').post(wrapAsync(Playlist.post));
    api.route('/playlists/:playlistId').put(wrapAsync(Playlist.put));
    api.route('/playlists/:playlistId').delete(wrapAsync(Playlist.delete));
};

Note: The route is wrapped in a wrapAsync() function. This function captures any errors that bubble up and sends them to Sentry.

API Controllers

The controllers are called by the route files and contain most, if not all of the business logic within the API. The controllers communicate with the models, which allow them to talk to the database.

API Models

Models are, essentially, the core foundation of the API. They provide the structure for the backend datastore (MongoDB) by enforcing what are known as “schemas”.

Schemas contain various types, such as “String”, “Boolean”, etc. Here’s a short example of our user schema (I removed some of the helper functions to shorten the example, so be sure to look at the code to see them):

import mongoose, { Schema } from 'mongoose';
import bcrypt from 'mongoose-bcrypt';
import timestamps from 'mongoose-timestamp';
import mongooseStringQuery from 'mongoose-string-query';

import FollowSchema from './follow';
import PinSchema from './pin';
import ListenSchema from './listen';

import PlaylistSchema from './playlist';
import jwt from 'jsonwebtoken';
import config from '../config';
import gravatar from 'gravatar';
import { getStreamClient } from '../utils/stream';

export const UserSchema = new Schema({
    email: {
        type: String,
        lowercase: true,
        trim: true,
        index: true,
        unique: true,
        required: true
    },
    username: {
        type: String,
        lowercase: true,
        trim: true,
        index: true,
        unique: true,
        required: true
    },
    password: {
        type: String,
        required: true,
        bcrypt: true
    },
    name: {
        type: String,
        trim: true,
        required: true
    },
    bio: {
        type: String,
        trim: true,
        default: ''
    },
    url: {
        type: String,
        trim: true,
        default: ''
    },
    twitter: {
        type: String,
        trim: true,
        default: ''
    },
    background: {
        type: Number,
        default: 1
    },
    interests: {
        type: Schema.Types.Mixed,
        default: []
    },
    preferences: {
        notifications: {
            daily: {
                type: Boolean,
                default: false
            },
            weekly: {
                type: Boolean,
                default: true
            },
            follows: {
                type: Boolean,
                default: true
            }
        }
    },
    recoveryCode: {
        type: String,
        trim: true,
        default: ''
    },
    active: {
        type: Boolean,
        default: true
    },
    admin: {
        type: Boolean,
        default: false
    }
});

UserSchema.plugin(bcrypt);
UserSchema.plugin(timestamps, {
    createdAt: { index: true },
    updatedAt: { index: true }
});
UserSchema.plugin(mongooseStringQuery);

UserSchema.index({ email: 1, username: 1 });

module.exports = exports = mongoose.model('User', UserSchema);

For a full list of Schema Types, have a look at the Mongoose website.

API Workers

The workers perform very special tasks that would otherwise be blocking processes. For example, we use dedicated tasks for processing RSS feeds, Podcast feeds, Open Graph Images, and more. Without having dedicated processes for these tasks, our API would quickly come to a halt and users would not receive a response message in a timely manner – the API would likely timeout.

Our workers use Bull Queue, a queueing infrastructure for Redis. Basically, our API inserts a call to Redis using the Bull Node.js library, then our workers pick up the job and process it asynchronously.

For example, here’s the code from the Podcast.js Controller that adds a podcast after a user adds it to the system (notice how we add a high priority of 1):

let scrapingPromise = PodcastQueueAdd(
    {
        podcast: p._id,
        url: p.feedUrl,
    },
    {
        priority: 1,
        removeOnComplete: true,
        removeOnFail: true,
    },
);

From there, the following things happen:

1. The conductor picks up on the task that needs to be processed
2. The file podcast.js is notified it has a job to do (process the incoming job)
3. The database is filled with populated episodes
4. The User is notified that new podcasts are available

CLI Commands

The commands directory holds onto the code for specific Winds related tasks – it’s a simple, yet powerful CLI for the Winds API – and is especially helpful when you need to debug RSS feeds. If you’re interested, the getting started along with all of the commands are listed out here.

Example output from running winds rss https://techcrunch.com/feed/:

API Tests

Tests are written with Mocha and Chai. You’re welcome to run the test suite at any time (it never hurts to find something that needs to be fixed). At this time, only Workers and API have coverage – and we’re still working on getting to the 100% mark; however, frontend coverage with jest will be coming soon!

Winds ENV 🌪️

There are two places that require a .env (dotenv) file for running the application: /app/.env as well as /api/tests (assuming you are going to be writing tests). You’ll need to create a .env file inside of /app in order for the application to work. Here’s a boilerplate .env file to help you get started:

DATABASE_URI=mongodb://localhost/WINDS # This value can remain as is
CACHE_URI=redis://localhost:6379 # This value can remain as is
JWT_SECRET=YOUR_JWT_SECRET # This should be a 256-bit random string. You can generate one here: https://randomkeygen.com/

API_PORT=8080 # This can remain as is
REACT_APP_API_ENDPOINT=http://localhost:8080 # This can remain as is, unless you're hosting on a server
STREAM_API_BASE_URL=https://windspersonalization.getstream.io/personalization/v1.0 # This can remain as is

STREAM_APP_ID=YOUR_STREAM_APP_ID # This should be the saved value that you wrote down earlier
REACT_APP_STREAM_APP_ID=YOUR_STREAM_APP_ID # this needs to be included twice, once for the backend, and once for the frontend to make realtime connections directly to Stream
STREAM_API_KEY=YOUR_STREAM_API_KEY # This should be the saved value that you wrote down earlier
STREAM_API_SECRET=YOUR_STREAM_API_SECRET # This should be the saved value that you wrote down earlier

REACT_APP_ALGOLIA_APP_ID=YOUR_ALGOLIA_APP_ID # This should be the saved value that you wrote down earlier
REACT_APP_ALGOLIA_SEARCH_KEY=YOUR_ALGOLIA_SEARCH_ONLY_API_KEY # This should be the saved value that you wrote down earlier
ALGOLIA_WRITE_KEY=YOUR_ALGOLIA_ADMIN_API_KEY # This should be the saved value that you wrote down earlier

MERCURY_KEY=YOUR_KEY_HERE # This should be the saved value that you wrote down earlier

Note: There are inline notes to help guide you through the setup of your .env file

Running PM2 🏃

PM2 is a process manager and we use it extensively for Winds. It’s an extremely powerful tool and we’re big fans of the project, as well as the maintainers. They are quick to respond if a bug arises, and most importantly, it works very well for what we need to do.

Node.js is single threaded by design. This has its ups and downs – it’s extremely fast, but bound to a single I/O operation at a given time. Under the hood, PM2 uses the Node.js cluster module so that the scaled application’s child processes can automatically share server ports. The cluster mode allows networked Node.js applications to be scaled across all CPUs available, without any code modifications. This greatly increases the performance and reliability of the application at hand, depending on the number of CPUs available.

I’d recommend learning the commands for PM2 if you are going to be developing on Winds, or if you plan on using PM2 for your own application. In all honesty, the best feature is the watch command that is built-in – it automatically watches for changes and reloads the app when necessary. Here are a few commands that I use daily:

• pm2 start process_dev.json (Starts the processes via commands set in the process_dev.json file)
• pm2 list (Lists all running processes)
• pm2 restart all (Restarts all running processes managed by pm2)
• pm2 log (Tails the logs that the various processes are spitting out)

Note: Use ctrl+c to get out of the tailing logs.

Let’s Get Started 👯

You’ve made it this far. Congratulations! All of the dependencies are installed, repo is cloned, your .env is set up… we’re ready to go!

Create a New Branch

Inside of your working directory, create a new branch called “feature”. Here’s the code for it if you need:

git checkout -b feature

Start MongoDB

Now that you have the code cloned to your machine, let’s go ahead and get MongoDB up and running. You can use the following command in a separate terminal.

brew services start mongodb

Note: If you’re looking for a decent GUI for MongoDB, have a look at Compass. You can download it here or run brew cask install mongodb-compass.

Start Redis

Similarly to MongoDB, let’s go ahead and get Redis up and running. For this, I like to use the native command (from your command line):

redis-server

Once started, you should see the Redis logo in the terminal (as shown above).

Start the Winds API & Workers

MongoDB is up and running alongside Redis. Now it’s time to start Winds. Head to the base root of the Winds directory and run the following command:

pm2 start process_dev.json

You should see the following once the application spins up:

Let’s Start the Winds UI

With Winds, we provide two ways to start the application UI: The first method starts the application inside of an Electron wrapper:

cd app && yarn start

The second option starts the application in a Chrome browser, which is much easier for debugging purposes:

cd app && yarn dev

Feel free to choose whichever one you like! I’ll be using the browser version as it’s easier to navigate the DOM and seems to reload faster. Woo! You’ve successfully set up and started Winds on your machine! 🎉

Adding a New Feature 🔔

We’ve covered a lot so far, but nothing concrete when it comes to adding new features to the platform. Since this is our first time showing off how to add a new feature, we’re going to keep it simple – we’ll add a social button to the frontend. Before moving ahead with development, please create an account by selecting 3 or more interests and following the guided steps.

Blank State

Don’t be alarmed when you log in. You’ll see a rather blank screen as we haven't added any content yet.

This is easily solved with an OPML file import 😀.

Click here to download the OPML file, then follow the instructions below to import it into Winds.

Click “New” > “New OPML” and a dialog will appear:

Once the dialog appears, drag and drop the downloaded OPML file into the drop zone.

Click “Add RSS”. Reload the page and you should see a list of articles!

If you’re wondering why the “Featured on Winds” and “Discover” section are empty, it’s for two reasons:

1. The Featured on Winds requires that a MongoDB database flag is set to true. For example, it must say “featured: true” on an RSS feed or a Podcast feed.
2. The Discover recommendation feature is powered by our machine learning. Machine learning takes time, as it learns from your interactions with content. The more that you interact with your content, the better.

Note: For this tutorial, it’s not necessary to get these two components up and running. But if you want to experiment, I’d suggest jumping into MongoDB Compass and setting an RSS feed’s “featured” key to the boolean value of true.

Starting to Code

As mentioned, we’re going to add a social button to the frontend. For the purpose of this exercise, we’ll add it to the top level RSS feeds. First, click on the RSS section header:

Next, have a look at each element. Notice how they are missing a Twitter logo? We’re going to add that.

Pro Tip: The easiest way to find what you’re looking for is to find the class name and then search for it in your editor. For this component, we’re going to be looking for two classes – “far fa-bookmark”.

You can search for this in your editor, or you can simply go to “app/src/components/ArticleListItem.js” – line number 57.

First, we need to include a module called is-electron. This module ensures that we’re only showing an icon (and using functionality) in the web environment. The package is already installed, you just need to add it to the imports at the top of the file like so:

import isElectron from 'is-electron';

Between the following ’s on line 59 and line 60, we’re going to add our Twitter button!

{!isElectron() ? (
    <span>
        <a
            href="#"
            onClick={e => {
                e.preventDefault();
                e.stopPropagation();
                this.handleTweet(
                    this.props.title,
                    this.props.url,
                );
            }}
        >
            <i className="fab fa-twitter" />
        </a>
    </span>
) : null}

After adding the code snippet above, your code should look like this:

We’re calling the function tweet(), so we’ll want to make sure we create that as well. Just before the render method, create a new method called “tweet”. You can copy and paste the following code:

tweet(title, url) {
    const getWindowOptions = function() {
        const width = 500;
        const height = 350;
        const left = window.innerWidth / 2 - width / 2;
        const top = window.innerHeight / 2 - height / 2;

        return [
            'resizable,scrollbars,status',
            'height=' + height,
            'width=' + width,
            'left=' + left,
            'top=' + top,
        ].join();
    };

    const shareUrl = `https://twitter.com/intent/tweet?url=${url}&text=${title}&hashtags=Winds`;
    const win = window.open(shareUrl, 'Share on Twitter', getWindowOptions());

    win.opener = null;
}

Now, try clicking on the Twitter logo in the UI. If all went well, you should see a Tweet dialog open with the title of the article, alongside the URL with the hashtag Winds!

Woo! You created your first feature on Winds – hopefully, one of many! Time to celebrate! 🍾🥂

If you’re still a little fuzzy on the process, run git stash and try it all over again. It doesn’t hurt to do things more than once 😉 Feeling like you have everything down? Let’s see some code! Here are a few ideas that may help you get started:

• Facebook Like Buttons
• Bookmark Support
• Dark mode to support macOS Mojave
• Likes (our API already provides support for them)
• General CSS cleanup
• General JavaScript cleanup
• Test coverage for the API and Workers

Note_: We’re also hiring a Full-time (contract) developer to join the team to work on Winds. Think you have what it takes? Ping me here with your LinkedIn profile and any examples of coding work that you’ve worked on as the sole contributor.

Final Thoughts 🤔

Winds is the most popular open-source application of its type – and we couldn’t be more excited. Free desktop applications are available for macOS, Linux, and Windows, and a web version is available as well. The application features several pieces of functionality, notably feeds and personalized content recommendations, all of which are powered by Stream, the leader in API based news feeds, activity streams, and personalization as a service.

Thank you for sticking around and learning a bit about Winds! We hope to see some PRs from you in the near future!

Happy Coding!