Home

Question to my readers: Drupal queue task sequencing

Sun, 04/07/2013 - 23:03 — infojunkie

I asked this question on StackExchange:

Is there a way to use Drupal Queue API to sequence the execution of tasks, like in a pipeline?

My use case is pretty simple: I have a number of tasks executing in the background, doesn't matter their order because they are self-contained. However, I'd like a single task (of a different type) to execute after all the others are done.

How would this be done?

Comments

Mon, 04/08/2013 - 12:04 — Tom Phethean (not verified)

I created a proposal for item priorisation in d8, which would allow you set a heaviest item which would process last. It was rejected for core but could easily be implemented for contrib by overriding the system queue class. You'd need to implement an alternative queue class which has its own queue table with a priority column, and then override the claim, release, delete and create methods to use the new table

Alternatively, what it really sounds like you need is batch api ( which in d7 uses queues ) where you can define a "final" task to execute when all the batch items are finished. See more on batch api at http://drupal.org/node/180528

Mon, 04/08/2013 - 12:19 — infojunkie

Thanks for the info. Batch API would not work, because I need this processing to happen in the background.

Mon, 04/08/2013 - 12:38 — Tom Phethean (not verified)

You can do background batch process either with the background process module - http://drupal.org/project/background_process - or by implementing a drush command to set and process the batch.

I think there's also a flag you can set on batch api to force its execution into the background but I can't find a link right now.

Mon, 04/08/2013 - 12:40 — infojunkie

Didn't know about that. Thanks again!

Mon, 04/08/2013 - 12:05 — Tom Phethean (not verified)

The queue priority issue is here to see what was proposed http://drupal.org/node/1913504

Mon, 04/08/2013 - 08:35 — Nick_vh (not verified)

Together with a bunch of other people we created the queue runner and it does support this use-case and we use it internally. You can add a collector task that includes all your small tasks and you add a second collector task so it is executed after the whole lot of small tasks. See the API file for the queue runner to figure out how you should do this exactly. It works with the default Drupal queue class so you can even extend it with external queue systems such as http://drupal.org/project/aws_sqs (amazon queue class) or others.

http://drupal.org/project/queue_runner Works for D6 and D7

Mon, 04/08/2013 - 08:38 — infojunkie

That's great! I will try it very soon.

Mon, 04/08/2013 - 08:07 — Bryce Fisher-Fleig (not verified)

Great question! I'm also curious about the Drupal Queue API use cases. The Examples project, especially the queue_example module in that project, has some great examples of the API usage, but I couldn't find any high level documentation of what the Queue API actually does.

I just took a deeper look at the API docs though and came across this overview page, but I'm not sure of the difference between the different queues in Drupal...

Mon, 04/08/2013 - 08:10 — infojunkie

I received some interesting replies on StackExchange. I suggest you follow there.