Tony Vu for XDevelopers

Posted on Oct 8, 2020 • Edited on Nov 15, 2021

Top 5 things you can learn about remote developer job postings with the new Twitter API

#twitter #react #career #tutorial

The code for this tutorial is available on GitHub.

In a previous tutorial, you learned how to get Tweets containing remote developer job listings in real-time using the Twitter API.

In this follow up tutorial, you will learn how to build an app to answer five must-know things about remote developer job openings posted on Twitter in the last seven days including:

How many Tweets about remote developer job openings were posted in the last seven days in total?
What day of the week had the most remote developer jobs Tweeted in the past seven days?
What are the most in-demand programming languages based on those Tweets?
Which Tweet received the most engagement via retweets, likes, replies, and quotes?
What do some of these Tweeted jobs look like?

To answer these questions, you will be building an app that uses the recent search endpoint, one of the first endpoints of the new Twitter API. Armed with answers to these questions, the aspiring job seeker can devise a strategy to optimize their job search and help land their next job!

Setup

To get started here’s what you will need:

You must have an developer account. If you don’t have one already, you can sign up for one. Access is available with active keys and tokens for a developer App that is attached to a Project created in the developer portal.
Node.js
Npm (This is automatically installed with Node. Make sure you have npm 5.2 or higher.)
Npx (Included with npm 5.2 or higher)

First, install Node.js. Check out the Downloads section from Node’s website and download the source code or installer of your choice. Alternatively, if you are running on a Mac you can install the Node package using the Brew package manager

Open a terminal window and bootstrap your React app using create-react-app by using npx.

npx create-react-app remote-dev-jobs-analytics

After create-react-app has finished executing, change to the newly created remote-dev-job-analytics directory and replace the scripts block in your package.json with the following script block in your package.json. These lines will provide a command shortcut to concurrently run your client and server backend code in development or production as needed.

cd remote-dev-jobs-analytics

package.json

  "scripts": {
    "start": "npm run development",
    "development": "NODE_ENV=development concurrently --kill-others \"npm run client\" \"npm run server\"",
    "client": "react-scripts start",
    "server": "nodemon server/server.js",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },

After updating the scripts section, your package.json should now look as follows.

Next, remove all files within the src/ subdirectory.

rm src/*

Then, create a new file within the src/ subdirectory called index.js. The code for this file will be as follows.

import React from "react";
import ReactDOM from "react-dom";
import App from "./components/App";

ReactDOM.render(<App />, document.querySelector("#root"));

Credentials

Connecting to the recent search endpoint requires you to authenticate using a bearer token from your app in the Twitter developer portal. To utilize your bearer token, you will need to have the following environment variable set. You can do so by issuing the following command in your terminal window assuming you are using bash as your shell. Replace , including the left and right angle brackets, with your bearer token.

export TWITTER_BEARER_TOKEN=<YOUR BEARER TOKEN HERE>

Server-Side Code

First, you will need to get started with implementing the Node server, which will be responsible for making the actual requests to the Twitter API. This Node server will serve as a proxy between your browser-based React client and the Twitter API. On your Node server, you will need to create API endpoints that connect to the recent search endpoint. In turn, requests from your React client will be proxied through to your local Node server.
Before you go any further, cd to the project root directory and install the following dependencies

npm install concurrently express body-parser util request http path http-proxy-middleware axios react-router-dom react-twitter-embed react-chartjs-2

Next, while still within your project root directory, create a new subdirectory called “server” and a new file within that subdirectory called “server.js”.

mkdir server
touch server/server.js

This source code file will contain all of your backend logic for connecting to and receiving Tweets from the recent search endpoint. The contents of your server.js file will be as follows.

server.js

const axios = require("axios");
const express = require("express");
const bodyParser = require("body-parser");
const moment = require("moment");

const app = express();
let port = process.env.PORT || 3000;

const BEARER_TOKEN = process.env.TWITTER_BEARER_TOKEN;

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

const searchURL = "https://api.twitter.com/2/tweets/search/recent";
const query =
  '(developer OR software) remote (context:66.961961812492148736 OR context:66.850073441055133696) -is:retweet -"business developer"';
const maxResults = 100;

const requestConfig = {
  headers: {
    Authorization: `Bearer ${BEARER_TOKEN}`,
  },
  params: {
    max_results: maxResults,
    query: query,
    "tweet.fields": "context_annotations,created_at,public_metrics",
  },
};

const authMessage = {
  title: "Could not authenticate",
  detail: `Please make sure your bearer token is correct. 
      If using Glitch, remix this app and add it to the .env file`,
  type: "https://developer.twitter.com/en/docs/authentication",
};

app.get("/api/search/recent", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(401).send(authMessage);
  }

  try {
    const response = await getSearchResults();
    res.send(response);
  } catch (e) {
    console.log(e);
  }
});

const getSearchResults = async (config = requestConfig) => {
  try {
    const response = await axios.get(searchURL, config);
    return response.data;
  } catch (e) {
    console.log(e);
  }
};

const getAllTweets = async () => {
  let response = await getSearchResults();
  let tweets = [];

  while (response.meta.next_token) {
    let config = {
      ...requestConfig,
      params: {
        ...requestConfig.params,
        next_token: response.meta.next_token,
      },
    };

    response = await getSearchResults(config);
    tweets = tweets.concat(response.data);
  }

  return tweets;
};

const getCount = async () => {
  let response = await getSearchResults();
  let resultCount = response.meta.result_count;

  while (response.meta.next_token) {
    let config = {
      ...requestConfig,
      params: {
        ...requestConfig.params,
        next_token: response.meta.next_token,
      },
    };

    response = await getSearchResults(config);
    resultCount = resultCount + response.meta.result_count;
  }

  return resultCount;
};

const countsByDay = async () => {
  let tweets = await getAllTweets();

  return tweets.reduce(
    (counts, tweet) => ({
      ...counts,
      [moment(tweet.created_at).format("ddd - MM/DD")]:
        (counts[moment(tweet.created_at).format("ddd - MM/DD")] || 0) + 1,
    }),
    {}
  );
};

const countsByLanguage = async () => {
  let counts = {};

  const languages = [
    "javascript",
    "JavaScript",
    "android",
    "frontend",
    "ios",
    "backend",
    "node",
    "nodejs",
    "python",
    "react",
    "scala",
    "c#",
    "rails",
    "ruby",
    "php",
    "java",
    "blockchain",
    ".net",
    "sql",
    "java",
    "php",
    "golang",
    "go",
    "wordpress",
  ];

  const tweets = await getAllTweets();

  for (tweet of tweets) {
    for (language of languages) {
      if (
        tweet.text.includes(language) ||
        tweet.text.includes(language.toUpperCase())
      ) {
        counts[language] = (counts[language] || 0) + 1;
      }
    }
  }

  if (counts["JavaScript"]) {
    counts["javascript"] += counts["JavaScript"];
    delete counts.JavaScript;
  }

  if (counts["node"]) {
    counts["nodejs"] += counts["node"];
    delete counts.node;
  }

  if (counts["golang"]) {
    counts["go"] += counts["golang"];
    delete counts.node;
  }

  return counts;
};

const sortCounts = (counts, keyName = "name") => {
  let sortedCounts = Object.keys(counts).map((language) => ({
    [keyName]: language,
    total: counts[language],
  }));

  sortedCounts.sort((a, b) => {
    return b.total - a.total;
  });

  return sortedCounts;
};

app.get("/api/search/recent/top", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(401).send(authMessage);
  }

  const tweets = await getAllTweets();
  let tweetsByEngagement = {};
  for (tweet of tweets) {
    const total_engagement = Object.values(tweet.public_metrics).reduce(
      (total_engagement, public_metric) => total_engagement + public_metric
    );
    tweetsByEngagement[tweet.id] = total_engagement;
  }

  res.send({ result: sortCounts(tweetsByEngagement, "id")[0] });
});

app.get("/api/search/recent/count", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(401).send(authMessage);
  }

  const results =
    req.query.group === "day" ? await countsByDay() : await getCount();

  res.send({ count: results });
});

app.get("/api/search/recent/language", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(401).send(authMessage);
  }

  try {
    let results = await countsByLanguage();
    results = sortCounts(results);
    res.send({ count: results.slice(0, 10) });
  } catch (e) {
    console.log(e);
  }
});

if (process.env.NODE_ENV === "production") {
  app.use(express.static(path.join(__dirname, "../build")));
  app.get("*", (request, res) => {
    res.sendFile(path.join(__dirname, ".../build", "index.html"));
  });
} else {
  port = 3001;
}

app.listen(port, () => console.log(`Listening on port ${port}`));

In the server-side code the following endpoints are being built

The /api/search/recent/count endpoint, by default, returns the total number of jobs for the last seven days. Passing in the group query parameter, with one of the following values will display one of the following
- group=day will return the number of jobs broken down by day in the last seven days-
- group=language will return the number of jobs broken down by programming language mentioned in the Tweet text, if present, in the last seven days
The /api/search/recent/top endpoint returns the Tweet receiving the most engagement. This endpoint uses the public metrics field to return likes, favorites, retweets, and quotes in the Tweet payload. Using these statistics, you can determine which Tweets are receiving the most engagement or attention.
The /api/search/recent endpoint returns the Tweets matching the following search query

(developer OR software) remote (context:66.961961812492148736 OR context:66.850073441055133696) -is:retweet -"business developer”

This search query instructs the recent search endpoint to match on Tweets containing the keywords “developer” or “software” and with the keyword “remote” present in the Tweet text. Additionally, this search query uses the “context” operator to match on Tweets containing specific domain and entity names.

      "context_annotations": [
        {
          "domain": {
            "id": "65",
            "name": "Interests and Hobbies Vertical",
            "description": "Top level interests and hobbies groupings, like Food or Travel"
          },
          "entity": {
            "id": "847544972781826048",
            "name": "Careers",
            "description": "Careers"
          }
        },
        {
          "domain": {
            "id": "66",
            "name": "Interests and Hobbies Category",
            "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
          },
          "entity": {
            "id": "961961812492148736",
            "name": "Recruitment",
            "description": "Recruitment"
          }
        }

The context operator follows the format context:.. As seen in the example payload above, the domain ids 65 and 66 represent the “Interests and Hobbies Category”. The entity ID 961961812492148736 represents the “Recruitment” entity and the entity ID 847544972781826048 represents the “Career” entity. For a complete list of domains, the Tweet Annotations documenation contains a table with 50+ domain names.

Finally, the operators “-is:retweet” and ”-business developer” can be used to exclude retweets from the search results and to exclude any Tweets containing “business developer”. Retweets are excluded to avoid duplicates in the search results and Tweets containing the terms “business developer” are excluded since that is irrelevant.

Client-Side Code

The next step is to work on the following React components to display the information mentioned above.

App.js - The parent component that will, in turn, render all other components

Tweet.js - Displays a Tweet containing a job posting

Day.js - Displays a bar chart of the number of Tweets posted by day for the last seven days

Top.js - Renders the Tweet that has received the most engagement in the past seven days

Tweets.js - Placeholder component that displays the top ten programming languages posted, the Top.js component, the Day.js component, and renders multiple Tweet.js components

Spinner.js - Renders a loading indicator for any pending API calls

Now you will need to get started with creating the React components. Under your /src subdirectory, create a directory called “components”. The source code files above will be stored in this new directory. First, create the parent most component of the application. This component will be responsible for rendering all other components.

App.js

import React from "react";
import { BrowserRouter, Route } from "react-router-dom";

import Tweets from "./Tweets";

const App = () => {
  return (
    <div className="ui container">
      <div className="introduction"></div>

      <h1 className="ui header">
        <img
          className="ui image"
          src="/Twitter_Logo_Blue.png"
          alt="Twitter Logo"
        />
        <div className="content">
          Remote Developer Job Analytics
          <div className="sub header">Powered by Twitter data</div>
        </div>
      </h1>

      <div className="ui grid">
        <BrowserRouter>
          <Route exact path="/" component={Tweets} />
        </BrowserRouter>
      </div>
    </div>
  );
};

export default App;

Next, create the parent component for rendering a sample of Tweets containing job postings.

Tweets.js

import React, { useEffect, useState } from "react";
import axios from "axios";
import Tweet from "./Tweet";
import Top from "./Top";
import Day from "./Day";
import Spinner from "./Spinner";

const initialState = {
  tweets: [],
};

const Tweets = () => {
  const [tweets, setTweets] = useState([]);
  const [tweetCount, setTweetCount] = useState(0);
  const [topTweetId, setTopTweetId] = useState(null);
  const [error, setError] = useState(null);
  const [isLoading, setIsLoading] = useState(false);

  useEffect(() => {
    getTweets();
    getTweetCount();
    getTopTweetId();
  }, []);

  const getTweets = async () => {
    try {
      setIsLoading(true);
      const response = await axios.get("/api/search/recent");
      setTweets(response.data.data);
      setIsLoading(false);
    } catch (e) {
      setError(e.response.data);
      setIsLoading(false);
    }
  };
  const getTweetCount = async () => {
    try {
      const response = await axios.get("/api/search/recent/count");
      console.log(response);
      setTweetCount(response.data.count);
    } catch (e) {
      setError(e.response.data);
      setIsLoading(false);
    }
  };

  const getTopTweetId = async () => {
    const response = await axios.get("/api/search/recent/top");
    setTopTweetId(response.data.result.id);
  };

  const errors = () => {
    if (error) {
      return (
        <div className="sixteen wide column">
          <div className="ui message negative">
            <div className="header">{error.title}</div>
            <p key={error.detail}>{error.detail}</p>
            <em>
              See
              <a href={error.type} target="_blank" rel="noopener noreferrer">
                {" "}
                Twitter documentation{" "}
              </a>
              for further details.
            </em>
          </div>
        </div>
      );
    }
  };

  const dashboard = () => {
    if (!isLoading) {
      if (!error) {
        return (
          <React.Fragment>
            <div className="sixteen wide column">
              <div className="ui segment">
                <div className="ui header center aligned ">
                  Total number of Tweets
                </div>
                <div className="ui header center aligned ">{tweetCount}</div>
              </div>
            </div>
            <div className="eight wide column">
              <div className="ui segment">
                <Top />
              </div>
            </div>
            <div className="eight wide column">
              <div className="ui segment">
                <Day />
              </div>
            </div>
            <div className="eight wide column">
              <div className="ui header">Top Tweet</div>
              <Tweet key={topTweetId} id={topTweetId} />
            </div>
            <div className="eight wide column">
              <div className="ui basic segment">
                <div className="ui header">Recent Tweets</div>
                {tweets.map((tweet) => (
                  <Tweet key={tweet.id} id={tweet.id} />
                ))}
              </div>
            </div>
          </React.Fragment>
        );
      }
    } else {
      return <Spinner />;
    }
  };

  return (
    <React.Fragment>
      {errors()}
      {dashboard()}
    </React.Fragment>
  );
};

export default Tweets;

Next, create the component for rendering the Tweet receiving the most engagement.

Top.js

import React, { useEffect, useState } from "react";
import axios from "axios";
import Spinner from "./Spinner";

const Top = () => {
  const [countByLanguage, setCountByLanguage] = useState([]);
  const [isLoading, setIsLoading] = useState(false);

  const style = {
    fontSize: "17px",
  };

  useEffect(() => {
    getTopLanguages();
  }, []);

  const getTopLanguages = async () => {
    setIsLoading(true);
    const response = await axios.get("/api/search/recent/language");
    setCountByLanguage(response.data.count);
    setIsLoading(false);
  };

  const capitalize = (word) => {
    const first_letter = word.slice(0, 1).toUpperCase();
    return first_letter + word.slice(1);
  };

  const displayTopLanuguages = () => {
    {
      if (!isLoading) {
        return countByLanguage.map((count, i) => (
          <div style={style} className="item">
            {i + 1}. {capitalize(count.name)}
          </div>
        ));
      } else {
        return <Spinner />;
      }
    }
  };

  return (
    <React.Fragment>
      <div className="ui header">Top Programming Languages</div>
      <ul className="ui relaxed list"> {displayTopLanuguages()}</ul>
    </React.Fragment>
  );
};

export default Top;

Next, create the component for rendering an individual Tweet.

Tweet.js

import React from "react";
import { TwitterTweetEmbed } from "react-twitter-embed";

const Tweet = ({ id }) => {
  const options = {
    cards: "hidden",
    align: "left",
    width: "550",
    conversation: "none",
  };

  return <TwitterTweetEmbed options={options} tweetId={id} />;
};

export default Tweet;

Finally, create a component to display a loading indicator during any pending API calls.

import React from "react";

const Spinner = () => {
  return (
    <div>
      <div className="ui active centered large inline loader">
        <img
          className="ui image"
          src="/Twitter_Logo_Blue.png"
          alt="Twitter Logo"
        />
      </div>
    </div>
  );
};

export default Spinner;

Proxy Setup

The final step is to proxy requests from your client to your backend server. To do this, from within your src/ directory, create a new file called “setupProxy.js” and add the following code.

setupProxy.js

const { createProxyMiddleware } = require("http-proxy-middleware");

// This proxy redirects requests to /api endpoints to
// the Express server running on port 3001.
module.exports = function (app) {
  app.use(
    ["/api"],
    createProxyMiddleware({
      target: "http://localhost:3001",
    })
  );
};

You can now start up both the server and client by going to the project root directory and typing the following:

npm start

After this command completes, your default web browser should automatically launch and navigate to http://localhost:3000 where you can see a sample of Tweets containing job postings from the last seven days along with the information displayed to answer all of the questions raised in the introduction.

Conclusion

Using the recent search endpoint, you created an app to answer some questions about remote developer job postings from the last seven days. Answers to these questions could be very helpful to the aspiring developer or a developer that already has a job and wants to be more strategic about how they should approach their search for their next one.

Have you found interesting ways to extend this app? Follow me on Twitter and send me a Tweet to let me know. I used several libraries beyond the Twitter API to make this tutorial, but you may have different needs and requirements and should evaluate whether those tools are right for you.

Top comments (6)

Dendi Handian • Oct 9 '20 • Edited

I'm confused by looking at developer.twitter.com/en/docs/auth..., so which one is the bearer token,

access token or token secret?

Tony Vu • Oct 9 '20

The bearer token is neither of those. It's automatically generated for you within the developer portal or you can do so from the command line. More info here developer.twitter.com/en/docs/auth...