Data transport and persistence for task processing

[thekaveman]

We need a resource that will be used to manage tasks, their processing state, and store any success or failure results.

There are many task processing libraries and integrations available for Django. Some considerations:

• The difference between "backends" (that store the results of a task) and "brokers" (that transport messages for pending task processing)
• Using an Amazon service (e.g. SQS) vs. using another storage service (e.g. Redis) vs. using a database in the Postgres cluster
• Monitoring and analysis of task processing -- where does this happen, how easy/difficult is it?

Many different options and more details can be found on the Django packages page for Workers, Queues, and Tasks