DEV Community

Suhem Parack for TwitterDev

Posted on • Updated on

Understanding the new Tweet payload in the Twitter API v2

Twitter recently announced the new Twitter API v2, rebuilt from the ground up to deliver new features faster to better serve all developers. The latest release includes new endpoints to listen to and analyze the public conversation. This new API allows for a new way of requesting the Tweet payload, giving you greater control by allowing you to request specific fields that you might be interested in. The new Tweet payload also includes additional new fields that are not part of the v1.1 payload (that you get from endpoints such as statuses/filter and statuses/show).

In this article, we will take a close look at the v1.1 Tweet payload (from v.1.1) and compare it with the new v2 Tweet payload (v2).

Comparison of the old Tweet payload and the new Tweet payload

If you have used endpoints that are part of the v.1.1 Twitter API, then you may have seen the Tweet payload that is part of the API response. The new Twitter API provides a new way to build the Tweet payload that you want returned, giving you more control over the specific data that you need. By default, the new endpoints only return the Tweet ID and text. Any additional fields can be explicitly specified and requested using expansions and fields. You can learn more about how to use expansions and fields from our documentation. Here is an example of the new Tweet payload when fully hydrated:

{
 "data": {
   "author_id": "2244994945",
   "context_annotations": [
     {
       "domain": {
         "id": "46",
         "name": "Brand Category",
         "description": "Categories within Brand Verticals that narrow down the scope of Brands"
       },
       "entity": {
         "id": "781974596752842752",
         "name": "Services"
       }
     },
     {
       "domain": {
         "id": "47",
         "name": "Brand",
         "description": "Brands and Companies"
       },
       "entity": {
         "id": "10045225402",
         "name": "Twitter"
       }
     },
     {
       "domain": {
         "id": "65",
         "name": "Interests and Hobbies Vertical",
         "description": "Top level interests and hobbies groupings, like Food or Travel"
       },
       "entity": {
         "id": "848920371311001600",
         "name": "Technology",
         "description": "Technology and computing"
       }
     },
     {
       "domain": {
         "id": "66",
         "name": "Interests and Hobbies Category",
         "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
       },
       "entity": {
         "id": "848921413196984320",
         "name": "Computer programming",
         "description": "Computer programming"
       }
     }
   ],
   "created_at": "2020-07-02T17:48:57.000Z",
   "entities": {
     "annotations": [
       {
         "start": 23,
         "end": 29,
         "probability": 0.4636,
         "type": "Organization",
         "normalized_text": "Twitter"
       }
     ],
     "hashtags": [
       {
         "start": 106,
         "end": 117,
         "tag": "TwitterAPI"
       }
     ],
     "urls": [
       {
         "start": 188,
         "end": 211,
         "url": "https://t.co/WvjuEWCa6G",
         "expanded_url": "https://blog.twitter.com/developer/en_us/topics/tools/2020/a-year-with-twitter-developer-labs.html",
         "display_url": "blog.twitter.com/developer/en_u…",
         "images": [
           {
             "url": "https://pbs.twimg.com/news_img/1278747527043362816/7HQRkQeV?format=jpg&name=orig",
             "width": 1600,
             "height": 600
           },
           {
             "url": "https://pbs.twimg.com/news_img/1278747527043362816/7HQRkQeV?format=jpg&name=150x150",
             "width": 150,
             "height": 150
           }
         ],
         "status": 200,
         "title": "A year with Twitter Developer Labs: What we've learned and changed",
         "description": "Labs has been invaluable in helping us understand what works well and what doesn’t, what you liked and what you didn’t.",
         "unwound_url": "https://blog.twitter.com/developer/en_us/topics/tools/2020/a-year-with-twitter-developer-labs.html"
       }
     ]
   },
   "id": "1278747501642657792",
   "lang": "en",
   "possibly_sensitive": false,
   "public_metrics": {
     "retweet_count": 64,
     "reply_count": 13,
     "like_count": 148,
     "quote_count": 19
   },
   "source": "Twitter Web App",
   "text": "It's been a year since Twitter's Developer Labs launched.\n\nAs we build towards the next generation of the #TwitterAPI (coming VERY soon), see what we've learned and changed along the way. https://t.co/WvjuEWCa6G"
 },
 "includes": {
   "users": [
     {
       "created_at": "2013-12-14T04:35:55.000Z",
       "description": "The voice of Twitter's #DevRel team, and your official source for updates, news, & events about Twitter's API. \n\n#BlackLivesMatter",
       "entities": {
         "url": {
           "urls": [
             {
               "start": 0,
               "end": 23,
               "url": "https://t.co/3ZX3TNiZCY",
               "expanded_url": "https://developer.twitter.com/en/community",
               "display_url": "developer.twitter.com/en/community"
             }
           ]
         },
         "description": {
           "hashtags": [
             {
               "start": 23,
               "end": 30,
               "tag": "DevRel"
             },
             {
               "start": 113,
               "end": 130,
               "tag": "BlackLivesMatter"
             }
           ]
         }
       },
       "id": "2244994945",
       "location": "127.0.0.1",
       "name": "Twitter Dev",
       "pinned_tweet_id": "1255542774432063488",
       "profile_image_url": "https://pbs.twimg.com/profile_images/1267175364003901441/tBZNFAgA_normal.jpg",
       "protected": false,
       "public_metrics": {
         "followers_count": 507830,
         "following_count": 1867,
         "tweet_count": 3565,
         "listed_count": 1562
       },
       "url": "https://t.co/3ZX3TNiZCY",
       "username": "TwitterDev",
       "verified": true
     }
   ]
 }
}
Enter fullscreen mode Exit fullscreen mode

Updates to existing fields

Tweet Text

The v1.1 payload returned the truncated Tweet text (for Tweets over 140 characters) by default, and also returned a Boolean field to identify whether the Tweet is truncated or not. In order to get the full Tweet text, you needed to use the tweet_mode = extended parameter and then do a conditional check on the truncated field. With the new v2 Tweet payload, you do not need to do this check. The Tweet text for all Tweets (including 140+ characters) is returned in the 'text' field and can be used directly.

Tweet Id

Old Tweet Payload New Tweet Payload
Returns the Tweet ID in two fields:
1. 'id' which is an integer
2. 'id_str' which is the string form of the ID
Returns a single 'id' field as a string

Entities

If a Tweet contains a URL, you can request information about it in the new v2 Tweet payload using 'entities' as a value for the tweet.fields parameter. This will provide a 'urls' object on your Tweet object (nested in the 'data' object). For a URL in the Tweet, you will be able to get the URL, the title, description, and unwounded_url as shown below:

{
    "entities": {
        "mentions": [
            {
                "start": 117,
                "end": 129,
                "username": "suhemparack"
            }
        ],
        "urls": [
            {
                "start": 253,
                "end": 276,
                "url": "https://t.co/qwVOgw0zSV",
                "expanded_url": "https://twittercommunity.com/t/using-twitter-developer-labs-filtered-stream-to-study-covid-19/138613",
                "display_url": "twittercommunity.com/t/using-twitte…",
                "images": [
                    {
                        "url": "https://pbs.twimg.com/news_img/1270417574266900490/bIXurp5s?format=jpg&name=orig",
                        "width": 480,
                        "height": 360
                    },
                    {
                        "url": "https://pbs.twimg.com/news_img/1270417574266900490/bIXurp5s?format=jpg&name=150x150",
                        "width": 150,
                        "height": 150
                    }
                ],
                "status": 200,
                "title": "Using Twitter Developer Labs filtered stream to study COVID-19",
                "description": "If you don’t have access to the covid-19 stream endpoint, you can still study some of this data using the Twitter Developer Labs filtered stream endpoint and the covid-19 annotation. Check out the video tutorial below, that teaches you how to get a sample of the data about covid-19 using the filtered stream endpoint and the covid-19 annotation. Watch this video here: Note: The filtered stream endpoint only provides ~1% of the conversation that matches your filtering rule, so the volume is o...",
                "unwound_url": "https://twittercommunity.com/t/using-twitter-developer-labs-filtered-stream-to-study-covid-19/138613"
            }
        ]
    }
}
Enter fullscreen mode Exit fullscreen mode

Updates to field names

Updates have been made to naming of certain fields as well. These include:

Old Payload New Payload
favorite_count like_count
screen_name username
statuses_count tweet_count
friends_count following_count

Updated Tweet payload structure

As you can see in the example above, the overall structure of both the payloads is different. The v1.1 payload has one single object that contains information about the Tweet such as the text and id, as well the user object, with information such as the id and name. The new v2 payload is structured differently. It primarily contains two objects: 'data' and 'includes.'

The sample below showcases the structural differences between the old and new payload, highlighting the breakdown of the new payload by 'data' and 'includes:'

Sample Tweet (old payload) structure Sample Tweet (new payload) structure
{
 "id": 1270417572001976322,
 "id_str": "Sample Tweet ID",
 "text": "Sample Tweet Text",
 "user": {
   ...
 },
 "place": {
   ...
 }
}
{
 "data": {
   "id": "1270417572001976322",
   "text": "Sample Tweet Text"
 },
 "includes": {
   "users": [
     {
       ...
     }
   ],
   "places": [
     {
       ...
     }
   ]
 }
}

User

Unlike the v1.1 Tweet payload, the User object is not included directly in the main payload, and has to be requested separately using the expansions parameter when using the Twitter API v2. The requested User object, along with the requested fields is returned nested in the 'includes' object.

Place

Similarly, in the v2 of the API, the information for the Place object can also be requested separately, and if present, is also returned nested in the 'includes' object of the new Tweet payload. An example of this object is shown below:

{
    "places": [
        {
            "country": "United States",
            "country_code": "US",
            "full_name": "Manhattan, NY",
            "geo": {
                "type": "Feature",
                "bbox": [
                    -74.026675,
                    40.683935,
                    -73.910408,
                    40.877483
                ],
                "properties": {}
            },
            "id": "01a9a39529b27f36",
            "name": "Manhattan",
            "place_type": "city"
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Quoted Tweet

In the v1.1 payload, a 'quoted status' object was returned in the Tweet payload for a Tweet object that was a Quote Tweet. In the new v2 Tweet payload, it can be requested separately, and is returned nested in the 'includes' object of the new Tweet payload.

Public Metrics

In the v1.1 Tweet payload, you can get metrics about the Tweet such as favorite_count and retweet_count from the Tweet object itself. In the new v2 Tweet payload, you can request a 'public_metrics'; object.' In addition to retweet_count and like_count (renamed from favorite_count), this will include additional fields such as quote_count and reply_count. See the example below:

{
    "public_metrics": {
        "retweet_count": 13,
        "reply_count": 3,
        "like_count": 39,
        "quote_count": 1
    }
}
Enter fullscreen mode Exit fullscreen mode

Note : the quote_count field in the public_metrics object is a new field that identifies the number of Quote Tweets. More information on this new field can be found in our forums.

Similarly, the user object in the v1.1 payload contains fields like followers_count and friends_count. With the new v2 payload, you can request the public_metrics object on the user object that will include fields such as:

{
    "public_metrics": {
        "followers_count": 507944,
        "following_count": 1865,
        "tweet_count": 3560,
        "listed_count": 1546
    }
}
Enter fullscreen mode Exit fullscreen mode

New objects introduced with Twitter API v2

Non-public & organic metrics

The new v2 Tweet payload includes the 'non_public_metrics,' which includes 'metrics' totals such as number of impressions, video view quartiles, 'organic_metrics' and 'promoted_metrics' objects (all of which require OAuth 1.0a User Context authentication). Learn more about available metrics here.

Polls

The new Tweet payload includes a 'polls' object where applicable, which includes information about Tweets with polls, such as the number of voters per option, and duration of the poll. This information is not provided in the v1.1 Tweet payload.

Context annotations

The new Tweet payload includes a new object called 'context_annotations' that can be requested using the 'context_annotations' value for the tweet.fields parameter. This provides contextual information to help you understand what the Tweet is about without needing to do custom entity recognition.

For example, a Tweet payload about the NFL may include the following object for context_annotations:

{
    "context_annotations": [
        {
            "domain": {
                "id": "47",
                "name": "Brand",
                "description": "Brands and Companies"
            },
            "entity": {
                "id": "10042758256",
                "name": "NFL"
            }
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Each object within context_annotations contains a 'domain' object and an 'entity' object, and each of those have an id and name property. The domain indicates the high level category or topic under which the Tweet falls, and the entity indicates the person, place, or organization that is recognized from the Tweet text. More details on available annotations can be found on our documentation page.

Getting the original Tweet for 'in_reply_to'

One major difference between the v1.1 and new v2 Tweet payloads is the payload structure for Tweets that are 'in reply to' another Tweet.

  • The v1.1 Tweet payload does not include the Tweet information for the original Tweet in reply to which this Tweet was posted. So, if you wanted to get the Tweet information for both the Tweet and the Tweet in reply to it, you would have to make another API call, using the in_reply_to_status_id
  • With the new v2 Tweet payload, you can request the original Tweet and the reply, as both objects will be part of the returned Tweet payload, with one single call, using the referenced_tweets.id expansions.

Below is an example of the new Tweet payload for a Tweet that was in reply to another Tweet:

{
   "data": {
       "author_id": "2244994945",
       "context_annotations": [
           {
               "domain": {
                   "id": "123",
                   "name": "Ongoing News Story",
                   "description": "Ongoing News Stories like 'Brexit'"
               },
               "entity": {
                   "id": "1220701888179359745",
                   "name": "COVID-19"
               }
           },
           {
               "domain": {
                   "id": "65",
                   "name": "Interests and Hobbies Vertical",
                   "description": "Top level interests and hobbies groupings, like Food or Travel"
               },
               "entity": {
                   "id": "848920371311001600",
                   "name": "Technology",
                   "description": "Technology and computing"
               }
           },
           {
               "domain": {
                   "id": "66",
                   "name": "Interests and Hobbies Category",
                   "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
               },
               "entity": {
                   "id": "848921413196984320",
                   "name": "Computer programming",
                   "description": "Computer programming"
               }
           }
       ],
       "created_at": "2020-04-29T17:01:40.000Z",
       "entities": {
           "urls": [
               {
                   "start": 243,
                   "end": 266,
                   "url": "https://t.co/j725kKXuXs",
                   "expanded_url": "https://docs.google.com/forms/d/e/1FAIpQLSdX3znRD87_j1E3aOUHAZtxzJfHwXzpFPKZTBiB90_9vJLscA/viewform",
                   "display_url": "docs.google.com/forms/d/e/1FAI…",
                   "images": [
                       {
                           "url": "https://pbs.twimg.com/news_img/1269516220459515905/TZm5wJPQ?format=jpg&name=orig",
                           "width": 1200,
                           "height": 630
                       },
                       {
                           "url": "https://pbs.twimg.com/news_img/1269516220459515905/TZm5wJPQ?format=jpg&name=150x150",
                           "width": 150,
                           "height": 150
                       }
                   ],
                   "status": 200,
                   "title": "Twitter COVID-19 data application",
                   "description": "Please complete this form to apply for access to the COVID-19 stream endpoint. As the conversation around this topic has surged in recent weeks, this dataset is notably large and requires experience working with Twitter data and substantial infrastructure to ingest, process, store, and analyze the volume. Please provide as much detail in English as you can, as this will help speed our review process. Note that we are unable to process applications that do not contain complete and detailed answers to each question listed. Requirements: * You have a Twitter developer account (https://developer.twitter.com/en/portal/petition/essential/basic-info) * You have demonstrable experience working with Twitter data in JSON format * You have a clear plan and ability to support the ingestion, processing, storage, and analysis of streaming data at a scale of tens of millions of Tweets per day * Your project is adherent with our Developer Terms and Restricted Use Cases (https://developer.twitter.com/en/developer-terms) * ",
                   "unwound_url": "https://docs.google.com/forms/d/e/1FAIpQLSdX3znRD87_j1E3aOUHAZtxzJfHwXzpFPKZTBiB90_9vJLscA/viewform"
               }
           ]
       },
       "id": "1255542782099255296",
       "in_reply_to_user_id": "2244994945",
       "lang": "en",
       "possibly_sensitive": false,
       "public_metrics": {
           "retweet_count": 35,
           "reply_count": 17,
           "like_count": 66,
           "quote_count": 6
       },
       "referenced_tweets": [
           {
               "type": "replied_to",
               "id": "1255542774432063488"
           }
       ],
       "source": "Twitter Web App",
       "text": "The volume of conversation about COVID-19 is tremendous, which means it requires expertise and computational resources to process. \n\nDevelopers and researchers with that capability and intent to support the public good can apply for access. \n\nhttps://t.co/j725kKXuXs"
   },
   "includes": {
       "users": [
           {
               "created_at": "2013-12-14T04:35:55.000Z",
               "description": "The voice of Twitter's #DevRel team, and your official source for updates, news, & events about Twitter's API. \n\n#BlackLivesMatter",
               "entities": {
                   "url": {
                       "urls": [
                           {
                               "start": 0,
                               "end": 23,
                               "url": "https://t.co/3ZX3TNiZCY",
                               "expanded_url": "https://developer.twitter.com/en/community",
                               "display_url": "developer.twitter.com/en/community"
                           }
                       ]
                   },
                   "description": {
                       "hashtags": [
                           {
                               "start": 23,
                               "end": 30,
                               "tag": "DevRel"
                           },
                           {
                               "start": 113,
                               "end": 130,
                               "tag": "BlackLivesMatter"
                           }
                       ]
                   }
               },
               "id": "2244994945",
               "location": "127.0.0.1",
               "name": "Twitter Dev",
               "pinned_tweet_id": "1255542774432063488",
               "profile_image_url": "https://pbs.twimg.com/profile_images/1267175364003901441/tBZNFAgA_normal.jpg",
               "protected": false,
               "public_metrics": {
                   "followers_count": 507676,
                   "following_count": 1863,
                   "tweet_count": 3563,
                   "listed_count": 1552
               },
               "url": "https://t.co/3ZX3TNiZCY",
               "username": "TwitterDev",
               "verified": true
           }
       ],
       "tweets": [
           {
               "author_id": "2244994945",
               "context_annotations": [
                   {
                       "domain": {
                           "id": "123",
                           "name": "Ongoing News Story",
                           "description": "Ongoing News Stories like 'Brexit'"
                       },
                       "entity": {
                           "id": "1220701888179359745",
                           "name": "COVID-19"
                       }
                   },
                   {
                       "domain": {
                           "id": "46",
                           "name": "Brand Category",
                           "description": "Categories within Brand Verticals that narrow down the scope of Brands"
                       },
                       "entity": {
                           "id": "781974596752842752",
                           "name": "Services"
                       }
                   },
                   {
                       "domain": {
                           "id": "47",
                           "name": "Brand",
                           "description": "Brands and Companies"
                       },
                       "entity": {
                           "id": "10045225402",
                           "name": "Twitter"
                       }
                   }
               ],
               "created_at": "2020-04-29T17:01:38.000Z",
               "entities": {
                   "annotations": [
                       {
                           "start": 54,
                           "end": 60,
                           "probability": 0.6094,
                           "type": "Product",
                           "normalized_text": "Twitter"
                       }
                   ],
                   "urls": [
                       {
                           "start": 264,
                           "end": 287,
                           "url": "https://t.co/BPqMcQzhId",
                           "expanded_url": "https://blog.twitter.com/developer/en_us/topics/tools/2020/covid19_public_conversation_data.html",
                           "display_url": "blog.twitter.com/developer/en_u…",
                           "images": [
                               {
                                   "url": "https://pbs.twimg.com/news_img/1274730688919613440/SEsqlE16?format=jpg&name=orig",
                                   "width": 1600,
                                   "height": 600
                               },
                               {
                                   "url": "https://pbs.twimg.com/news_img/1274730688919613440/SEsqlE16?format=jpg&name=150x150",
                                   "width": 150,
                                   "height": 150
                               }
                           ],
                           "status": 200,
                           "title": "Enabling study of the public conversation in a time of crisis",
                           "description": "To further support our company’s ongoing efforts, we want to enable developers and researchers to study the public conversation on COVID-19 in real-time.",
                           "unwound_url": "https://blog.twitter.com/developer/en_us/topics/tools/2020/covid19_public_conversation_data.html"
                       }
                   ]
               },
               "id": "1255542774432063488",
               "lang": "en",
               "possibly_sensitive": false,
               "public_metrics": {
                   "retweet_count": 327,
                   "reply_count": 308,
                   "like_count": 710,
                   "quote_count": 120
               },
               "source": "Twitter Web App",
               "text": "During these unprecedented times, what’s happening on Twitter can help the world better understand & respond to the pandemic. \n\nWe're launching a free COVID-19 stream endpoint so qualified devs & researchers can study the public conversation in real-time. https://t.co/BPqMcQzhId"
           }
       ]
   }
}
Enter fullscreen mode Exit fullscreen mode

Summary of the contrast between fields and objects of the v1.1 and new v2 Tweet payloads

Below is a summary of the comparison of objects and fields in the v1.1 Tweet payload and the new v2 Tweet payload:

Objects/Fields v1.1 Tweet Payload New Tweet Payload
user Part of the returned Tweet object Requested with expansions and is nested in 'includes' object of the returned Tweet object
places Part of the returned Tweet object Requested with expansions and is nested in 'includes' object of the returned Tweet object
quoted_status Part of the returned Tweet object Requested with expansions and is nested in 'includes' object of the returned Tweet object
polls Not supported Requested with expansions and is nested in 'includes' object of the returned Tweet object
Original Tweet (in reply to) Not supported Requested with expansions and is nested in 'includes' object of the returned Tweet object
context_annotations Not supported Provided in the 'data' object of the returned Tweet object
public_metrics Some available as individual fields as part of the returned Tweet object A new public_metrics object is provided in the 'data' object of the returned Tweet object
non_public_metrics & organic_metrics Not supported A new non_public_metrics object is provided in the 'data' object of the returned Tweet object

These are some of the key differences between the v1.1 Tweet payload and the new payload available with v2. With the introduction of expansion and fields, we aim to provide developers a higher degree of customization to scope and limit the fields you need based on your use case. Below are some resources to help you get started with exploring the new v2 Tweet payload. Reach out to me on Twitter @suhemparack or @TwitterDev or on our community forums with feedback or questions.

Resources to help you get started

  • Check out the video tutorial explaining the differences between the v1.1 Tweet payload and the new v2 Tweet payload.
  • A complete list of all the fields that are part of the new v2 Tweet payload and what each of those can be used for can be found on our documentation for the data dictionary for the object model.
  • Learn more about how to use fields and expansions to get the data you want returned in the Tweet payload.
  • A list of code samples for getting started with the new Twitter API (that return the new Tweet payload) can be found on Github.
  • Check out this hydrator tool on Glitch that retrieves the new v2 Tweet payload for a given Tweet ID.

Latest comments (0)