Task Manager¶
Spirit ships with several tasks that run asynchronously. The supported task managers to run these tasks are Huey, and Celery.
While Spirit can run without a task manager, some of the tasks may not run, as they are too expensive to run as part of the request-response cycle (web server), or they are require to be run periodically.
Functionality that requires a task manager includes: search indexation.
Huey¶
Read the Huey docs for Django.
Configuration¶
Set Huey as the Spirit task manager:
# settings.py (dev.py or prod.py)
# extend installed apps with +=
INSTALLED_APPS += [
'huey.contrib.djhuey',
]
ST_TASK_MANAGER = 'huey'
The settings/prod.py and settings/dev.py files include sample
configuration for Huey. The prod.py settings set Redis as the backend,
and the dev.py settings set SQLite as the backend. Redis is recommended
for production environments, and it’s a good option to use as the Django cache
as well.
How to install Redis depends on the OS, for example it can be installed as a Snap package in Ubuntu, or it can be run with docker. So, it’s out of the scope of these docs.
Run the tasks consumer¶
To start the task manager, run:
python manage.py run_huey
This can be run with supervisord, or systemd in production.
The caveats of SQLite as a backend¶
SQLite only works in single server instances, where both the web-server and the task manager (Huey) are running in the same machine, sharing the same SQLite data base.
For sufficiently large forum instances, it’s recommended to run the task manager on its own machine, while the web-server can scale up and down independently. This means Redis must be used instead of SQLite.
Scaling Huey¶
I’m not aware of how well Huey scales horizontally. It seems
it may just work fine.
However, all Huey instances but one must disable the periodic
task functionality to avoid running periodic tasks in more than
one machine. This can be done by passing the --no-periodic
parameter to the task manager command.
Celery¶
Read the Celery docs for Django.
Configuration¶
Set Celery as the Spirit task manager:
# mysite/settings/settings.py (dev.py or prod.py)
ST_TASK_MANAGER = 'celery'
Import Celery at Django startup:
# mysite/__init__.py
from .celery import app as celery_app
__all__ = ('celery_app',)
Run the tasks consumer¶
To start the task manager, run:
celery -A mysite worker -l info
This can be run with supervisord, or systemd in production.
To start the periodic task manager, run:
celery -A mysite beat -l info
Note this will just enqueue tasks that will later be consumed by the worker.
Celery does not work¶
Celery likes to break all the things in every major version.
The celery configuration (including prod.py, dev.py, and
mysite/celery.py) were tested on v4.4.7.
I’m not a Celery user. These docs and the Celery support are a community effort. Please, send a PR if something breaks.