ETOOBUSY 🚀 minimal blogging for the impatient
Thread of tweets via API
TL;DR
We finally download a whole thread of tweets using the Twitter API, yay!
We have discussed a lot about getting Twitter threads lately:
-
Scrape a Thread of Tweets shows a poor man’s way of doing this by scraping data from a page generated by an online service to unroll threads;
-
Getting started with MojoX::Twitter introduced us to MojoX::Twitter, a handy module to consume the Twitter API;
-
last, Tweets from a user paved the way to getting a thread by coding a way to get all tweets since a specific one.
In particular, we’re leveraging the code from the last post here!
Filtering tweets in a thread
The following function filters an input anonymous array (such the one returned
by get_tweets_since) of tweets to select only those belonging to a thread:
1 sub filter_thread ($tweets) {
2 my @thread = ($tweets->[0]);
3 my %in_thread = ($thread[0]{id} => 1);
4 for my $tweet ($tweets->@*) {
5 defined(my $rid = $tweet->{in_reply_to_status_id}) or next;
6 $in_thread{$rid} or next;
7 push @thread, $tweet;
8 $in_thread{$tweet->{id}} = 1;
9 } ## end for my $tweet ($tweets->...)
10 return \@thread;
11 } ## end sub filter_thread
The thread is assumed to begin at the first tweet in the array, which is
consistent with get_tweets_since that we introduced in Tweets from a
user. Hence, the first tweet is both collected in
array @thread (which is later returned in line 10, as an array reference)
and tracked as being %in_thread.
Input tweets are iterated to find interesting ones, depending on two conditions (lines 5 and 6):
- they MUST have a defined
in_reply_to_status_id, because follow-ups in a thread are always referred to some previous tweet, and - the
in_reply_to_status_idMUST be part of the current thread.
If this is the case, the tweet is put in the @thread and its identifier is
flagged as being %in_thread, for getting possible follow-ups later.
The complete program
#!/usr/bin/env perl
use 5.024;
use warnings;
use experimental qw< postderef signatures >;
no warnings qw< experimental::postderef experimental::signatures >;
use Mojo::JSON 'j';
use Mojo::File 'path';
use MojoX::Twitter;
my $id = shift // '1215710451343904768';
my $credentials = j path('twitter-credentials.json')->slurp;
my $client = MojoX::Twitter->new(
consumer_key => $credentials->{'api-key'},
consumer_secret => $credentials->{'api-secret-key'},
access_token => $credentials->{'access-token'},
access_token_secret => $credentials->{'access-token-secret'},
);
my $tweets = get_tweets_since($client, $id);
my $thread = filter_thread($tweets);
say j $thread;
sub get_tweets_since ($client, $id) {
my $tweet = $client->request( # needed to get the user
GET => "statuses/show/$id",
{tweet_mode => 'extended'}
);
my @tweets;
my %options = (
user_id => $tweet->{user}{id},
since_id => $id,
count => 200, # max value possible
tweet_mode => 'extended',
);
while ('necessary') {
my $chunk =
$client->request(GET => 'statuses/user_timeline', \%options);
my @chunk = sort { $a->{id} <=> $b->{id} } $chunk->@*;
pop @chunk if exists $options{max_id}; # remove duplicate
last unless @chunk; # no more available
$options{max_id} = $chunk[0]{id}; # remark for next iteration
unshift @tweets, @chunk; # older ones in front
} ## end while ('necessary')
unshift @tweets, $tweet; # the starting one...
return \@tweets;
} ## end sub get_tweets_since
sub filter_thread ($tweets) {
my @thread = ($tweets->[0]);
my %in_thread = ($thread[0]{id} => 1);
for my $tweet ($tweets->@*) {
defined(my $rid = $tweet->{in_reply_to_status_id}) or next;
$in_thread{$rid} or next;
push @thread, $tweet;
$in_thread{$tweet->{id}} = 1;
} ## end for my $tweet ($tweets->...)
return \@thread;
} ## end sub filter_threadSee the local version if the above snippet from GitLab is not available.
Wrap-up for now
Did you enjoy this tour-de-force about getting a thread of tweets? Do you have comments? By all means use the machinery below!