Ingesting the Facebook stream using Feeds and OAuth 2.0 (updated for Drupal 7)

Now that Twitter 1.1 and Feeds are buddies, time to move to other data sources. Next up: Facebook. Using trusty Feeds and friends, I was able to ingest my own Facebook home feed. Here's how to replicate this:

For the impatient, attached is a feature that should get you set up quickly. You'll need the following modules:

  • Feeds latest HEAD from 7.x-2.x branch.
  • Feeds JSONPath Parser version 7.x-1.0-beta2 - make sure to install the needed JSONPath library as per the instructions on the module page.
  • Feeds OAuth latest HEAD from 7.x-1.x branch.
  • php-proauth library that you install in sites/all/libraries module as such:
git clone https://github.com/infojunkie/php-proauth.git

The idea behind the setup is to create a Feeds pipeline that:

  • Fetches the given resource URL (a Facebook Graph API URL) using Feeds OAuth 2.0 Fetcher. This fetcher checks for OAuth 2.0 access token for the current user and performs authorization if it's not found. It alerts the user to missing access tokens during feed creation.
  • Parses the result using Feeds JSONPath Parser, since Facebook Graph API uses JSON.
  • Maps the result to nodes using the standard Node Processor.

Setting up the Facebook app

Create a new Facebook application. You need to add two specific settings to it:

  • Basic > Website with Facebook Login > Site URL: enter the callback URL that is reported in the Feed importer's Fetcher > HTTPS OAuth 2.0 Fetcher Settings > Site identifier description.
  • Permissions > Extended Permissions: add the read_stream permission.

Configuring the feature

  • You will need to copy the App ID and App Secret strings of the Facebook app to the Fetcher > HTTPS OAuth 2.0 Fetcher Settings > Consumer key and Consumer secret settings, respectively.
  • Set the fetcher's Method to GET.
  • Then create a new node of type Facebook feed with the Graph API URL (e.g. your home feed). Make a note of this node's nid.
  • Finally, edit the facebook view included in this feature, such that the filter Feeds item: Owner feed nid refers to the nid noted above.

That's it. This should cure your feed indigestions!

AttachmentSize
facebook_feed-7.x-0.2.tar40 KB

Comments

Hey there. Very great write up. I am stuck, however, on creating the Facebook app and getting read_stream accepted. Facebook has some verification and review process to allow that in the app. Am I missing something? I filled out the basic information for the app, but Facebook rejected my request for read_stream. Any thoughts or help would be much appreciated. Thanks!

what are the:

Request token URL * Access token URL * Authorize URL *

for facebook the documentation is soooooooo bad - i believe that i am using the sdk that is version 3 (version 4 has even less documentation).

I am trying to pull in an event from facebook - I have managed to get this working in local dev but now I am in drupal and using feeds_oauth and it wants all these urls that I cant supply the app dashboard doesn't fill me in. Anyone have any ideas?

Thanks

O

Since this is OAuth 2, you only need:

Thanks for the reply.

do you know what information should be pushed into the feeds_oauth_access_tokens table in the database drupal?

Thanks

O

Expect a new row to be created with your Facebook site identifier, an oauth_token that Facebook handed you, and an expiry date for the token.

Just in case anyone else starts banging their head against a wall, I was getting some errors along the lines of:

{"error":{"message":"Missing message or attachment.","type":"FacebookApiException","code":100,"error_subcode":1349125}}

I was using the feature above, which sets the request type to POST, simply change it to GET and this solved the problem for me.

Otherwise, really useful article, thanks

Thanks for the note. I update the feature file with the method setting. Sorry about the head banging!

does anybody currently have this working? I've tried everything and I'm not getting anything imported from FB!

Hi,I followed these steps three times and ended up in the same situation each time, with this error:

Attempt to retrieve access token failed: stdClass Object ( [request] => POST /oauth/access_token HTTP/1.0 Content-Type: application/x-www-form-urlencoded User-Agent: Drupal (+http://drupal.org/) Host: graph.facebook.com Content-Length: 562 code=AQAF6EW3INSetg1C4tmZNKhxb7MFwUavuqRy1a2lTXi6-3qdToNaNZ_dSJBIE0ugFb9NzbzQbbspE0NKtAzvZ5IP8ofhF8z1CQtIjTPqFTxViKA5MtOvu8vHHB2L8b_lHwUHAKEJ9gH_y4l-Uvhb6ASKCmmoGXAPOnoxlCB73lZ3Ulqd8uHHfBYyDwiXXQxUnAtF1dIkXMwM2NJIlRR8AOLugUALeJQZVww1yUNr33WhesmAn1GayngxmzwzuI-pCqTIs9h8tyL8O6dZ9zsdxqLbo194TScdbd-AS9DfumI6cC08NaWwbTfDBg7V4AbDb2YtObYftLmrSYfdRKiQU6Jv&client_id=710968228928589&client_secret=3cc08356b64d58e1840d47ca7917c6cb&redirect_uri=http%3A%2F%2Ftopsite.secondlevel.com%2F%3Fq%3Dfeeds%2Foauth2%2Fcallback%2Ffacebook_feed&grant_type=authorization_code [data] => {"error":{"message":"Error validating verification code. Please make sure your redirect_uri is identical to the one you used in the OAuth dialog request","type":"OAuthException","code":100}} [protocol] => HTTP/1.0 [status_message] => Bad Request [headers] => Array ( [access-control-allow-origin] => * [cache-control] => no-store [content-type] => text/javascript; charset=UTF-8 [expires] => Sat, 01 Jan 2000 00:00:00 GMT [pragma] => no-cache [www-authenticate] => OAuth "Facebook Platform" "invalid_code" "Error validating verification code. Please make sure your redirect_uri is identical to the one you used in the OAuth dialog request" [x-fb-rev] => 911604 [x-fb-debug] => oj9dGSkr3c4xpR9v35xgi0rZNtNLJLlTcCjQThU6m8I= [date] => Mon, 19 Aug 2013 03:12:59 GMT [connection] => close [content-length] => 190 ) [code] => 400 [error] => Bad Request )

I am currently being asked for the access token url and the authorize url.

What are these? How do I obtain them?

Has facebook changed anything with their oauth? In the feed_oauth module function feeds_oauth_callback2, the response to the access token URL is in the format:

access_token=(167 char access token)&expires=5183999

So it's not a json encoded string, there's no access_token property, access_token_secret property, expires_in property. So naturally, it isn't storing the access token in the database.

It's sending the request: POST /oauth/access_token HTTP/1.0 Content-Type: application/x-www-form-urlencoded User-Agent: Drupal (+http://drupal.org/) Host: graph.facebook.com

And sending the following post data: Array ( [code] => (code) [client_id] => (app client id) [client_secret] => (app secret) [redirect_uri] => http://(hostname)/feeds/oauth2/callback/facebook_feed [grant_type] => authorization_code )

Obviously the strings in parenthesis are just placeholders I've taken out.

It seems facebook updated their oauth. I just had to update to the dev branch of the feeds_oauth module.

Thanks for the update. I've made a new beta release for the module.

Hello Sir,

firstly, thanks for all the hard work!

I have replicated, as best I could your settings. After making sure everything is up to date as of today, I seem to be stuck parsing the data, would greatly appreciate your thoughts on where I have gone wrong.

As I am trying to import stream from a fb page(not profile), whre the 'me/home' query is not available. but 'feed' or 'posts' are. So, I get my token stored in the db correctly(assuming), then when i import it replies: "There are no new nodes." but if i call https://graph.facebook.com/138782916317724/feed?access_token=ONE_FROM_DB it displays the json.

Could you shed some light on this?

thanks, Arif

  • Look into the Drupal "Recent log messages"
  • Look into the feed node's "Log" tab
  • In your Feeds importer, turn on debugging in the JSONPath parser settings then go to your feed node and import manually from the "Import tab"

Hopefully you can get a hint as to what's going on from the above. Otherwise, please submit a support request.

Thank you. I will look into this a bit more in the next days. With your module package it's working great! For some reason I couldn't replicate the importer to work without using your module. If I figure why I will post it here. Aside from no notification when a pre auth token exired, everything is clockwork. next up, twitter.. Thanks again, Arif

I have implemented this following your guide lines without the attached module and everything is working great! My original problem was an user id issue, and thanks to your telling me where to look, an easy fix!

The only issue I have is with the cron. When importing manually from 'import/{$fetcher_id} when the facebook token is expired, from the feeds module file 'OAuth2HTTPSFetcher.inc' the form on line 160( sourceForm() ) is seemingly called. This prints the form: "Could not find OAuth access tokens for site facebook. You should probably authenticate first to access protected information.". the link provided 'feeds/oauth2/authenticate/{$fetcher_id}' works as advertised and stores a new token!

When the cron is run manually from the cron page, and the token is expired, the same form and link are presented. So my question is, would there be a way to run the feeds_oauth_authenticate2() automatically, bypassing the link so the cron can grab the new token?

Thank you, Arif

Well, I just read the comments below this, and you have already answered this. it is not possible without logging into facebook.

Sorry for the repeated question! feel free to delete the last comment!

This module is exactly what I needed. I had a query though. Will it be able to trigger on its own after lets say, every 1 hour? I want to import feeds from many facebook pages and I want to do it programatically, i.e. I do not want to click the import button everytime. I am okay with saving my facebook userid and password in DB if that is all that'a needed. Does this module do that? Please guide me through.

Thanks Prince

The feed importer has a setting that specifies the time interval between automatic imports. On the feed admin screen, find the "Basic settings" > "Refresh" option.

A really dumb question. The tokens expire after sometime. Don't they? In that case a new token will have to be created which will need my presence. No?

That's right, you need to re-login.

What if I store my facebook username and password in db? Will it be possible to auto login?

Not that I know. The whole point of OAuth is that you only use your username/password in an interactive setting, not programmatically.

Right! My bad :-( Thanks for bearing with me though! My requirement is such that I want to aggregate content from facebook pages. And I want to do that programatically because I cannot be around all the time. Can you guide me as to how I would do that?

The OAuth token that this module stores in the DB is all that's needed to keep the feed working in the background. Facebook might expire them after a number of days, and I haven't researched refreshing them beyond this point.

Okkay. Thanks a ton for this info.

Great post! I'm already using this on two sites.

My only hangup now is getting the Feeds Image Grabber module working. I believe I have everything setup but it doesn't seem to be working. Has anyone had any success with this?

I would just store the URL but it looks like some of the images are authenticated so I would get broken images on the frontend.

The description still works - thanks for the writeup.

The only thing that took me a while to find out was, that the "scope" within feeds_oauth should include "read_stream" if you want to read /me/home. To reset the oauth, just truncate the oauth_token table from the database.

{ "data": [ { "from": { "name": "Disaster.TV", "category": "News/media", }, "message": "Disaster TV expects Japan to raise", "comments": { "data": [ { "from": { "name": "Renjay September", }, "message": "Thanx guys,,,Bye!", }, { "from": { "name": "Ranjith Jey", }, "message": "Gd nyt", } ], } }, ], }

Hello. I have a json like this. Context $data.* contains from,message,comments. What will be the values of JSONPath expression to put in JSONPath Parser Settings for parsing $data.from.name and $data.comments.data.from.name and $data.comments.data.message as cck fields. Also is it possible to parse those two section of $data.comments.data.* as cck fields.

hi can you help me? pls. i have this problem. notice: Undefined property: stdClass::$title in /home/khawshik/public_html/sites/all/modules/feeds/feeds.module on line 345. Could not retrieve title from feed. And heres a debug msg And this is how the debug looks. context: array ( 'id' => '100001562085366_150851048310297', 'from' => array ( 'name' => 'Alif Hasan', 'id' => '100001562085366', ), 'message' => 'CINTA is VALOBASA....GooD MorninG...@ALL FriendS..', 'type' => 'status', 'application' => array ( 'name' => 'Mobile', 'id' => '186529194704290', ), 'created_time' => '2011-03-25T00:37:50+0000', 'updated_time' => '2011-03-25T00:37:50+0000', ) jsonpath_parser:0: 'natok 3................joy' jsonpath_parser:1: '100001533788745_188096421233245' jsonpath_parser:2: 'bangla eid natok 2008 december'

Make sure you have associated:

  • node title with message
  • guid with id
  • node body with description

This is what's working for me. If you're using a new version of Feeds, there might be some changes because I haven't tested this recipe since November 2010.

In your attach file name facebook_feed-6.x-1.0.tar creates a content type Facebook Feed don't have title and body field. Thats creates the problem. i just replace that with a new one.

Thanks. i already fix that. in importer basic setting i Attached new content type .. and title is ok description is ok .. and its work ..

this great stuff. I have been working with a not-for-profit organization with an active facebook page and we would like to be able to show the content from the page on the org's own website. This module here looks like just the ticket. I am more of a drupal builder than a php dev and don't have much personal experience with feeds or json, but I get the general idea.

Wondering, as this post here is several months old, if you had made any advancement or had plans to turn this into an official d.o module?? I put it in my queue to explore this though.

Thanks!

Thanks for your comment. I am not planning to turn this into a module, since it just involves assembling and configuring components. I haven't worked on this since then, but please post your experience here and I will gladly provide assistance.

Thanks for a great post and sharing your code. I am almost there in making it work. This is really awesome to me.