Building an app to stream Tweets in real-time using the Twitter API
By

August 12, 2020

Building an app to stream Tweets in real-time using the Twitter API

_The code for this tutorial is available on GitHub_. *You can also check out the live demo of the app on Glitc*h

The first endpoints of the new Twitter API are out and waiting for you to explore and build apps with in order to understand what’s happening in the public conversation.

This tutorial will walk you through how to build your own real-time Tweet streaming app using the filtered stream endpoints and Tweet annotations to listen for Tweets based on your own topics of interest.

After building the app, you will learn about how it can be applied to some real-life examples to inspire you to get started such as.

Setup

To get started here’s what you will need:

First, install Node.js. Check out the Downloads section from Node’s website and download the source code or installer of your choice. Alternatively, if you are running on a Mac you can install the Node package using the Brew package manager

Open a terminal window and bootstrap your React app using create-react-app by using npx.

After create-react-app has finished executing, change to the newly created real-time-tweet-streamer directory and replace the scripts block in your package.json with the following script block in your package.json. These lines will provide a command shortcut to concurrently run your client and server backend code in development or production as needed.

package.json

After updating the scripts section, your package.json should now look as follows.

Next, remove all files within the src/ subdirectory.

Then, create a new file within the src/ subdirectory called index.js. The code for this file will be as follows.

Credentials

Connecting to the filtered stream endpoints requires you to authenticate using a bearer token from your app in the Twitter developer portal. To utilize your bearer token, you will need to have the following environment variable set. You can do so by issuing the following command in your terminal window assuming you are using bash as your shell. Replace , including the left and right angle brackets, with your bearer token.

Server Side Code

First, you will need to get started with implementing the Node server, which will be responsible for making the actual requests to the Twitter API. This Node server will serve as a proxy between your browser-based React client and the Twitter API. On your Node server, you will need to create API endpoints that connect to the filtered stream endpoints. In turn, requests from your React client will be proxied through to your local Node server.

Before you go any further, cd to the project root directory and install the following dependencies

Next, while still within your project root directory, create a new subdirectory called “server” and a new file within that subdirectory called “server.js”.

This source code file will contain all of your backend logic for connecting to and receiving Tweets from the filtered stream endpoint. The contents of your server.js file will be as follows.

server.js

Filtering Tweets on the filtered stream endpoint using rules

Filtered stream has two endpoints, a streaming endpoint, to receive data and a rules endpoint that is used to create and delete rules. The filtered stream endpoints require you to define search queries called rules, in order for it to know what kind of Tweets to send to you*. *Rules allow you to narrow down to only the Tweets you are looking for by using a set of operators. You will see some example use cases and corresponding rules you can use later once you finish building the app.

The other filtered stream endpoint is the streaming endpoint, which uses a simple GET connection. Once a connection is established, Tweets are delivered in JSON format through a persistent HTTP streaming connection. You will only receive Tweets matching your rules while connected to the stream.

Client-Side Code

The next step is to work on the following React components

App.js - The parent component that be will, in turn, render all other components

NavBar.js - Displays the navigation bar for navigating between the Tweet feed and managing rules

Tweet.js - Displays a Tweet on the page

TweetFeed.js - Renders multiple Tweet components at once in a “feed” like fashion

Rule.js - Renders an individual rule on your stream

RuleList.js - Renders multiple Rule components and displays an input field to add a rule

ErrorMessage.js - Renders any status or error messages to the screen

Spinner.js - Renders a loading indicator for any pending API calls

Now you will need to get started with creating the React components. Under your /src subdirectory, create a directory called “components”. The source code files above will be stored in this new directory. First, create the parent most component of the application. This component will be responsible for rendering all other components.

App.js

Next, create the component for the navigation bar.

Navbar.js

Next, create the parent component for rendering all job listings.

TweetFeed.js

Next, create the child component for the previous component that renders an individual Tweet.

Tweet.js

Next, create the component responsible for rendering all the rules on our stream as well as displaying input controls for creating new rules. In this case, we will only be using one rule.

RuleList.js

Next, create the child component of RuleList.js responsible for displaying a single rule and deleting a rule.

Rule.js

Next, create a component for displaying any status or error messages.

ErrorMessage.js

Finally, create a component to display a loading indicator during any pending API calls.

Spinner.js

Proxy Setup

The final step is to proxy requests from your client to your backend server. To do this, from within your src/ directory, create a new file called “setupProxy.js” and add the following code.

setupProxy.js

You can now start up both the server and client by going to the project root directory and typing the following.

After this command completes, your default web browser should automatically launch and navigate to http://localhost:3000. You will then be taken to the rules management section of the app.

Now that you have an app in place to listen for any kind of Tweet you want, let’s walk through some real-life examples of how this app can be used such as

For each of the examples and accompanying rules listed below, you can navigate to the rules section of the app and simply copy and paste the rule into the input field to add it to your stream. Once the rule has been added, it will take  effect within seconds and only Tweets matching the criteria of that rule will be sent to you.

In this first example, let’s say you are interested in finding remote developer job

openings. To surface these kinds of Tweets, you can use the following rule.

To understand what this rule is doing, you can break it down into two parts. The keywords part and the Tweet annotations part.

Using keyword operators

The keywords part of the rule will match Tweets containing the keywords “developer” or “engineer” and the keyword “remote”. These keywords alone will certainly match Tweets containing remote developer job postings, but it will also match on irrelevant Tweets. For example, the Tweet below will match this rule.

https://twitter.com/EraserFarm/status/1220013392766947332

Since this is not the desired Tweet, you need to take this rule a step further. Though this rule matches irrelevant Tweets, it is also successful in matching Tweets with actual job postings. For example, the rule you have so far will also result in matching the Tweet below.

https://twitter.com/plataformatec/status/1225460423761317888

Tweet annotations: Using context operators

The challenge you have now is, though you are receiving Tweets containing job postings you will still have to go through irrelevant Tweets. If only there was a way to only match on Tweets that contain job postings as best as possible. Are there operators you can use in your rule that only match these relevant Tweets?

Fortunately, this is where the power of Tweet annotations comes in. Take a closer look at an example Tweet object payload that was sent for this Tweet on your filtered stream endpoint. Within the payload, the nested “context_annotations” field is present. Context annotations are delivered as a “context_annotations” field in the payload. These annotations are inferred based on the Tweet text and result in the domain and/or entity labels, which can be used to discover Tweets on topics that may have been previously difficult to surface. Note, that these fields will only be included if data is present since not all Tweets will contain this data.

To match on the domain and entity ids within the context_annotations field, you can use the “context” operator. The “context” operator can be used to instruct your filtered stream endpoint to match on Tweets containing specific domain and entity names. Here’s what that would look like.

The operators above follow the format “context:.”. As seen in the example payload above, the domain id 66 represents the “Interests and Hobbies Category”. The entity ID 961961812492148736 represents the “Recruitment” entity and the entity ID 850073441055133696 represents the “Job search” entity. For a complete list of domains, the Tweet annotations docs contain a table with 50+ domain names.

With the operator explanation out of the way, this 2nd part of the rule matches Tweets containing the entity names “Recruitment” or “Jobs search”.

In summary, taking both parts of this rule together, it will match on Tweets that contain the keywords “developer” or “engineer” and the keyword “remote”, but only if those Tweets also contain the entity names “Recruitment” or “Jobs search”

If you need new music video suggestions, you can start by using a simple rule that matches on Tweets containing the keywords “song” and “YouTube”. You will also want Tweets that actually link out to external videos

Taking a closer look at the payload of this Tweet, you see that it has some annotations on it that can help you match more relevant Tweets. Notice the Annotation with an entity label of “Pop” and a domain name of “Music Genre”

To make this rule better and narrow down your Tweets to be even more relevant, you can update your rule as follows.

This will take the original rule you used a step further by narrowing down to only the Tweets that are labeled with the Music Genre domain label and the Pop entity label.

As a final example, let’s say you are interested in learning about personal finance and you can be savvier about your spending and savings. You also only want original Tweets that contain links to online articles to learn more.

Going through a similar process as you did with earlier examples if you simply add the following rule, only Tweets containing the words “personal”, “finance” and “savings” will be sent to you.

Taking a look at one of the Tweet payloads, the nested annotations contains an entity label about personal finance that will help you narrow down your Tweets to the most relevant ones

Using the context operator to match on Tweets containing this label, you can revise your rule to look as follows

Conclusion

Using the filtered stream endpoints to stream publicly available Tweets to your server and annotations, you created an app to more easily surface Tweets around a topic of interest. The filtered stream endpoint gives you the haystack of data in the form of Tweets and the Tweet annotations help you find the needle in that haystack.

Have you found other interesting ways to use this app? Follow me on Twitter and send me a Tweet to let me know. I used several libraries beyond the Twitter API to make this tutorial, but you may have different needs and requirements and should evaluate whether those tools are right for you.