Consuming the new Twitter 1.1 API with Feeds and friends

Update: This post now contains a feature that you can import in D7 to see the Twitter feed in action.

The new Twitter 1.1 API kicked in recently, which meant a new cycle of maintenance for anyone consuming their data programmatically. My own Feeds + Views demo site streams #drupal, using Feeds and complementary modules. I had to make a few changes to the importer to adapt to the new API:

  • Authorization using OAuth
  • Parsing JSON instead of XML

OAuth authentication

The new Twitter 1.1 API requires OAuth authentication/authorization for every request. Fortunately, I had already written Feeds OAuth to solve for feeds that require OAuth, so I just had to plug this in. Well, not "just", because it took choosing among the several authorization options that Twitter provides, and fixing a couple of bugs in the module itself.

Twitter provides several options for authorization, depending on the needs of the consumer (not listed on this page is the Application-only authentication). I ended up choosing the method that required the least work on the Feeds OAuth module, namely obtaining OAuth tokens from dev.twitter.com as a pre-processing step. To do this, I manually added an entry to the feeds_oauth_access_tokens table, with the tokens that were handed to me by Twitter on my application page. This way, Feeds OAuth would not have to ask me to login to Twitter in order to make the API call. Obviously, this is a temporary hack and I will work on enhancing the module support for different authentication options.

JSON parsing

Twitter 1.1 API only returns JSON results. To parse JSON instead of RSS/Atom, I used Feeds JSONPath Parser. It does the job as advertised, but the only challenge here was to retrieve the tweet URL for each result. The Twitter search API itself does not return tweet URLs, for some unfathomable reason. My setup needs the tweet URL to pass it to oEmbed, which renders the tweet on the view. Tweet URLS are of the form https://twitter.com/<user_screen_name>/status/<tweet_id>.

To get the URL, I had to resort to coding. Here's how I did it:

  • First convince Feeds JSONPath Parser to retrieve the user's screen name. To do this, I had to map it to some field - I chose the node body, although a better solution would be to use a NULL target field, just to fool the parser into returning the value.

  • In a custom module, I created a new programmatic source field called "Tweet URL" that synthesizes the URL:

/**
 * Implements hook_feeds_parser_sources_alter().
 */ 
function demo_feeds_parser_sources_alter(&$sources, $content_type) {
  $sources['tweet_url'] = array(
    'name' => t('Tweet URL'),
    'description' => t('The URL of a tweet.'),
    'callback' => 'demo_feeds_tweet_url',
  );
}

/**
 * Populates the "tweet_url" field for each result.
 */
function demo_feeds_tweet_url(FeedsSource $source, FeedsParserResult $result, $key) {
  $item = $result->currentItem();
  // jsonpath_parser:2 corresponds to user screen name in my importer.
  // jsonpath_parser:0 corresponds to tweet ID in my importer.
  return 'https://twitter.com/' . $item['jsonpath_parser:2'] . '/status/' . $item['jsonpath_parser:0'];
}
  • Finally, I mapped this new source field to my target field, the URL of a Link that renders using oEmbed.

Conclusion

This little exercise took a good couple of hours - and that's just for a demo. API changes are always painful, but at least the Feeds OAuth module got some love and fixes in the process.

Addendum: Using the feature

  • Enable the module twitter_feed_custom.
  • Copy the ping.php_.txt file to your Drupal root folder and rename it to ping.php. Also edit the file to point the DRUPAL_ROOT definition to your actual Drupal root folder.
  • Copy the Consumer key and Consumer secret strings of the Twitter app to the Fetcher > HTTPS OAuth Fetcher Settings > Consumer key and Consumer secret settings, respectively.
  • Create a new node of type Twitter feed with your query URL (e.g. #drupal). Make a note of this node's nid.
  • Edit the (badly-named) hashdrupal view included in this feature, such that the filter Feeds item: Owner feed nid refers to the nid noted above.
AttachmentSize
twitter_feed_custom.tar10 KB
twitter_feed-7.x-0.2.tar40 KB
ping.php_.txt968 bytes

Comments

Hello, I am trying to get this to work on my site. I am up to the part where I need to authorize with Twitter. I click on the link to "authenticate first" and twitter gives me a PIN to "complete the authorization process".... however I see no place to enter this pin. And so even though I keep authenticating and getting pins, it will never actually go out and get the files, I keep getting an error that I have not authenticated.

Is this perhaps an issue because my site is in DEV and not publicly accessible? I have a dummy url to another site that I own...

Thanks so much, I have been trying to get tweets into nodes for weeks now... Really need to solve this issue!

Thanks!

April

Excellent solution, it works fine. I have a question: if i want to change the twitter app of the oauth authentication (after the first time), how had i to procede? Thanks

Hi, Thanks for this module, very usefull. I have a question about interation with OAuth module. After the first Twitter authentication (when i create a twitter_feed content type), i can't change the Twitter App. If i change the Twitter key the import generate error (because the authentication fails). Could you help me please? Luke

Glad it's working for you. If I understand your question correctly, you want to change the Consumer key/secret pair that is used to connect to Twitter. In that case, open the Feeds importer that you created, edit the OAuth fetcher settings, and change the keys there. Hope this helps!

Thanks for the response. But if i open the Feeds importer, edit the OAuth fetcher settings and change the keys, then the import fails.

Please check the logs for hints on the failure.

Sorry for the double posting by I replied in another comment by accident instead of writing a new one.

Hello and thanks for this module.

I am trying to make twitter and feeds_oauth work in Drupal 7 for my site.

I follow the installation instruction of the module and have already created an app in twitter.

I fill the oauth parameters and requested in HTTP OAuth fetcher.

Then I use import to load the tweets of an account.

I get everytime the the following error: Download of https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=x failed with code 400.

Everything is installed according to the instructions, any help on how to proceed?

Are you sure you authenticated against Twitter before starting the import? According to Twitter doc, requests without authentication are considered invalid and will yield this response (400).

To authenticate, you should be seeing the following message on the import page: "Could not find OAuth access tokens for site twitter. You should probably authenticate first to access protected information." Just click the link to authenticate with Twitter.

Excellent article. This solution is just for Drupal 7? I have a site still on 6 at the moment and looking for a solution for making nodes from tweets. Have worked this into your Feeds OAuth module?

It's for Drupal 7, I haven't backported the fixes to Drupal 6. You can try the same recipe and submit feature requests to backport whatever fixes are needed.

I'm having trouble getting the custom source to work. I have the attached custom module saved into sites/all/modules in a twitter_feed_custom directory, but nothing is appearing in the Sources dropdown at /admin/structure/feeds/twitter/mapping.

This is the module file:

 t('Tweet URL'),
    'description' => t('The URL of a tweet.'),
    'callback' => 'twitter_feed_custom_tweet_url',
  );
}

/**
 * Populates the "tweet_url" field for each result.
 */
function twitter_feed_custom_tweet_url(FeedsSource $source, FeedsParserResult $result, $key) {
  $item = $result->currentItem();
  // jsonpath_parser:4 corresponds to user screen name for me
  // jsonpath_parser:1 corresponds to tweet ID for me
  return 'https://twitter.com/' . $item['jsonpath_parser:4'] . '/status/' . $item['jsonpath_parser:1'];
}

Oooh I've got it working! Just created a new custom module.

Code below:

 t('Tweet URL'),
'description' => t('Tweet URL from screen name and tweet ID'),
'callback' => 'twitterfeed_customsource_tweet_url',
);
}

/**
 * Populates the "tweet_url" field for each result.
 */
function twitterfeed_customsource_tweet_url(FeedsSource $source, FeedsParserResult $result, $key) {
  $item = $result->currentItem();
  // jsonpath_parser:4 corresponds to user screen name for me
  // jsonpath_parser:1 corresponds to tweet ID for me
  return 'https://twitter.com/' . $item['jsonpath_parser:4'] . '/status/' . $item['jsonpath_parser:1'];
}

The new module/new module name seemed to kickstart my Drupal into action, and I now have a lovely oEmbed powered twitter view. Thank you so much for this tutorial and your work on your contributed modules!

(and the module is definitely enabled at admin/modules!)

Not sure why the Syntax highlighter/something else has put '<!--?php' at the top of the module code, it should just be '<?php'

and

'name' =--> t('Tweet URL'),

should be

'name' => t('Tweet URL'),

Thanks for this writeup, I found it very helpful.

I've got everything (mostly) working in my client's D6 environment. OAuth seems to be fine (at least I'm not getting any authentication errors that I know of). The problem seems to be more with Twitter's 1.1 API.

I'm trying to use the search API to import tweets with a particular hashtag. It just requires a GET, but when I pass something like https://api.twitter.com/1.1/search/tweets.json?q=%23scaserve&count=100 as the feed URL, I don't get anything back because Twitter is returning an error 400. Using the Twitter for Mac console (yes, the Mac client has a dev console!), if I make that GET I actually get a 400 along with some JSON:

{ "errors": [ { "code": 25, "message": "Query parameters are missing" } ] }

When I make that same request via apigee.com's console, it works just fine. Not sure what's happening here, any ideas? I'm wondering if it's an issue with the Twitter API wanting the query parameters in the headers rather than in the URL ( http://stackoverflow.com/questions/13169953/php-use-twitter-api-1-1-to-s... ) or something?

Cancel that. Reviewed the site log and realized I'm getting auth errors:

{"errors":[{"message":"Bad Authentication data","code":215}]}

Not sure what I've done wrong. In the "Settings for HTTS OAuth 2.0 Fetcher" I'm sure I've got the consumer key, consumer secret, access token URL and authorize URL set to what I see on dev.twitter.com for my application. I've left "scope" blank, is that correct?

Then in my database I manually added a row to feeds_oauth_access_tokens, I set the uid = 1, the site_id to what I have as "site idenitifier" in the admin, and then the oauth_token and oauth_token_secret values set to what my application gives me at dev.twitter.com. My access level is Read-only (I'm just GETing a search) and "Sign in with Twitter" is set to No.

Anything obvious I'm doing wrong?

Twitter uses the HTTP OAuth Fetcher (for OAuth v1), not HTTPS OAuth 2.0 Fetcher. Which one are you using?

Okay, nope, the wddx error was related to some crummy code from a previous developer, stuck into a block. I've eliminated it.

Still getting the aforementioned HTTP 500 error. Sigh.

Please check the Drupal log and the web error log for any clue. Also, it's better to submit a support request on the Feeds OAuth issue queue.

Browsing through the logs, looks like it may be this:

wddx_deserialize(): Expecting parameter 1 to be a string or a stream in /home/dev/sites/thesca/includes/common.inc(1696) : eval()'d code on line 6.

I was using HTTPS OAuth 2.0. Oops.

Changed to HTTP OAuth Fetcher. Now I just get an error when initiating the import:

An error has occurred. Please continue to the error page An HTTP error 500 occurred. /batch?id=2&op=do

Not terribly helpful....

Seeing this in the logs: [Wed Jul 10 22:14:03 2013] [error] [client 75.173.82.226] PHP Fatal error: Class 'OAuth' not found in /home/dev/sites/thesca/sites/all/modules/feeds_oauth/feeds_oauth.module on line 68

I've got php-proauth-read-only in the feeds_oauth module directory, which is where the D6 version of feeds_oauth seems to expect it.

Just a quick comment on getting the tweet url, thanks for those functions. I was able to map the user.screen_name to username and that seems to work fine for getting it passed in.

Thanks for this. I'm pretty sure, after hours of Googling, that you're the only guy writing about this right now. I'm trying to get this to work after the v1 API was shut-off, but I'm having issues. I won't use this as a support forum, but was wondering if you had thought about writing this up as a tutorial or walk-through.

I'm sure lots of people would find it invaluable. Thanks!

Thanks for the suggestion. I attached a feature that shows the necessary configuration and updated the post with an explanation of how to set it up. Hope this helps!

Also feel free to submit a support request if something remains unclear.