Zero to SaaS - Bookmarksy.io Working around Twitter API Limitations

#webdev #javascript #beginners #programming

One of the most important features of bookmarksy.io is it’s ability to reach out to Twitters API and retrieve a given users bookmarks.

Getting a list of users bookmarks is the easy bit. You can make a call to

https://api.twitter.com/2/users/:id/bookmarks

and what is returned is a list that looks something like this:

{
    "data": [
        {
            "created_at": "2022-11-11T18:18:36.000Z",
            "edit_controls": {
                "edits_remaining": 5,
                "is_edit_eligible": false,
                "editable_until": "2022-11-11T18:48:36.000Z"
            },
            "text": "Build a $50 info-product in one day using this process:\n\n(Probably shouldn't give this away for free bc later I'm going to sell a guide to this same process for ~$50.)",
            "public_metrics": {
                "retweet_count": 242,
                "reply_count": 85,
                "like_count": 1171,
                "quote_count": 29
            },
            "source": "Tweet Hunter Pro",
            "conversation_id": "1591133306069843979",
            "lang": "en",
            "id": "1591133306069843979",
            "reply_settings": "everyone",
            "possibly_sensitive": false,
            "edit_history_tweet_ids": [
                "1591133306069843979"
            ]
        }
    ],
    "includes": {
        "users": [
            {
                "name": "Rob Lennon 🗯 | Audience Growth",
                "username": "thatroblennon"
            }
        ],
        "tweets": [
            {
                "created_at": "2022-11-11T18:18:36.000Z",
                "edit_controls": {
                    "edits_remaining": 5,
                    "is_edit_eligible": false,
                    "editable_until": "2022-11-11T18:48:36.000Z"
                },
                "text": "Build a $50 info-product in one day using this process:\n\n(Probably shouldn't give this away for free bc later I'm going to sell a guide to this same process for ~$50.)",
                "public_metrics": {
                    "retweet_count": 242,
                    "reply_count": 85,
                    "like_count": 1171,
                    "quote_count": 29
                },
                "source": "Tweet Hunter Pro",
                "conversation_id": "1591133306069843979",
                "lang": "en",
                "id": "1591133306069843979",
                "reply_settings": "everyone",
                "possibly_sensitive": false,
                "edit_history_tweet_ids": [
                    "1591133306069843979"
                ]
            }
        ]
    },
    "meta": {
        "result_count": 1,
        "next_token": "zldjwdz3w6sba13nbs0z2lzml73coscn3mc18c0avxs"
    }
}

And that’s all fine and well. But if I’m dealing with a thread, which most bookmarks tend to be, how do I retrieve all the related tweets?

The twitter documentation speaks of use of recent search or archive search for tweets. This isn’t viable for two reasons:

recent search only works on tweets less than seven days old
archive search is enrollment based and EXPENSIVE!.

So what’s the solution? Here’s what I’ve implemented to work around this:

assumptions:

A bookmark is either a single tweet, or a collection (conversation: twitter language) of tweets.
If a bookmark is a collection of tweets, it can be assumed that this bookmark is a thread.
Thread tweets are typically created around the same time (roughly < 10 seconds between tweets).

Solution:

Leverage the

https://api.twitter.com/2/users/:id/tweets

endpoint with a request body as such:

{
    since_id: <ID OF BOOKMARKED TWEET>,
    end_time: <CREATED_AT OF BOOKMARKED TWEET + 5 MINS>,
    limit: 100
}

this will retrieve the first 100 tweets published within the first five minutes after the initial tweet.

Then we can filter on those tweets by conversation_id to retrieve the ones associated with our bookmark!

With this solution, we work around the 7 day and archive restrictions. Is it ideal? of course not, but it’s been reliable so far! And unless your bookmark is a thread of over one hundred tweets, there doesn’t appear to be any data loss.

The only drawback is that I will likely end up querying redundant tweets which would eat into my monthly query limit, but we will cross that bridge when we get to it!

https://bookmarksy.io is coming along nicely so far. Hit a couple of snag’s but I will be looking for beta testers soon!

Follow the journey on twitter bookmarksy_io or brandonkpbailey and join the waitlist! https://brandonkpbailey.substack.com/