|
| 1 | +# Laravel 5 Queue Worker for Elastic Beanstalk |
| 2 | + |
| 3 | +*Use your L5 application as a worker to consume queues on AWS Elasticbeanstalk* |
| 4 | + |
| 5 | +Laravel provides a [wonderful array](https://laravel.com/docs/5.1/queues) of drivers for consuming queues within your application as well as [some documentation](https://laravel.com/docs/5.1/queues#supervisor-configuration) on how to manage your application with [Supervisord](http://supervisord.org/) when it is acting as a worker. |
| 6 | + |
| 7 | +Unfortunately that's where the documentation ends. There is no guidance on how to manage multiple workers from a devops context which is a huge bummer. But don't worry fam I've got your covered. |
| 8 | + |
| 9 | +**This package enables your L5 application to manage itself, as a worker, in an [AWS Elasticbeanstalk](https://aws.amazon.com/elasticbeanstalk/) environment.** |
| 10 | + |
| 11 | +**It provides these features:** |
| 12 | + |
| 13 | +* **Automated installation of supervisor on first-time deployment** |
| 14 | +* **Automatic updating of supervisor configuration upon deployment** |
| 15 | +* **Parsing of EB environmental variables to generate supervisor config** |
| 16 | + |
| 17 | +# Let's get down to business |
| 18 | + |
| 19 | + |
| 20 | +## Installation |
| 21 | + |
| 22 | +Require this package |
| 23 | + |
| 24 | +```php |
| 25 | +composer require "foxxmd/laravel-elasticbeanstalk-queue-worker" |
| 26 | +``` |
| 27 | + |
| 28 | +**After installing the package you can either:** |
| 29 | + |
| 30 | +Add the ServiceProvider to the providers array in `config/app.php` |
| 31 | + |
| 32 | +```php |
| 33 | +FoxxMD\LaravelElasticBeanstalkQueueWorker\ElasticBeanstalkQueueWorkerProvider::class |
| 34 | +``` |
| 35 | + |
| 36 | +Then, publish using artisan |
| 37 | + |
| 38 | +```php |
| 39 | +php artisan vendor:publish --tag=ebworker |
| 40 | +``` |
| 41 | + |
| 42 | +**OR** |
| 43 | + |
| 44 | +Copy everything from `src/.ebextensions` into your own `.ebextensions` folder manually |
| 45 | + |
| 46 | +**Note:** This library only consists of the EB deploy steps -- the provider is only for a convenience -- so if you want to you can modify/consolidate the `.ebextensions` folder if you're not into me overwriting your stuff. |
| 47 | + |
| 48 | + |
| 49 | +## Configuration |
| 50 | + |
| 51 | +### Enable Worker Mode |
| 52 | + |
| 53 | +In order for worker deployment to be active you **must** add this environmental to your elasticbeanstalk environment configuration: |
| 54 | + |
| 55 | +``` |
| 56 | +IS_WORKER = true |
| 57 | +``` |
| 58 | + |
| 59 | +**If this variable is false or not present the deployment will not run** |
| 60 | + |
| 61 | +### Set Queue Driver |
| 62 | + |
| 63 | +Set the [driver](https://laravel.com/docs/5.1/queues#introduction) in your your EB envronmental variables: |
| 64 | + |
| 65 | +``` |
| 66 | +QUEUE_DRIVER = [driver] |
| 67 | +``` |
| 68 | + |
| 69 | +**Note: If no `QUEUE_DRIVER` key is present in your EB envronmental variables then `beanstalkd` will be used.** |
| 70 | + |
| 71 | +### Add Queues |
| 72 | + |
| 73 | +All queues are configured using EB envronmental variables with the following syntax: |
| 74 | + |
| 75 | +**Note**: keys/values are currently **case-sensitive** |
| 76 | + |
| 77 | +**Note**: brackets are placeholders only, do not use them in your actual configuration |
| 78 | + |
| 79 | +``` |
| 80 | +queue[QueueName] = [queueName] # Required. The name of the queue that should be run. |
| 81 | +[QueueName]NumProcs = [value] # Optional. The number of instances supervisor should run for this queue. Defaults to 1 |
| 82 | +[QueueName]Tries = [value] # Optional. The number of times supervisor should attempt to run in the event an unexpected exit code occurs. Defaults to 5 |
| 83 | +[QueueName]Sleep = [value] # Optional. The number of seconds supervisor should sleep if no new jobs are in the queue. Defaults to 5 |
| 84 | +[QueueName]StartSecs = [value] # Optional. How long a job should run for to be considered successful. Defaults to 1 |
| 85 | +``` |
| 86 | + |
| 87 | +Add one `queue[QueueName] = [queueName]` entry in your EB environmental variables for each queue you want to run. The rest of the parameters are optional. |
| 88 | + |
| 89 | +That's it! On your next deploy supervisor will have its configuration updated/generated and start chugging along on your queues. |
| 90 | + |
| 91 | +# But how does it work? |
| 92 | + |
| 93 | +Glad you asked. It's a simple process but required a ton of trial and error to get right (kudos to AWS for their lack of documentation) |
| 94 | + |
| 95 | +EB applications can contain a [folder](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html) that provides advanced configuration for an EB environment, called `.ebextensions`. |
| 96 | + |
| 97 | +This package uses AWS commands files in this folder to detect, install, and update supervisor and its configuration and then run it for you. |
| 98 | + |
| 99 | +### 1. Ingress Supervisor rules |
| 100 | + |
| 101 | +Supervisor requires port 9001 to be open if you want to access its web monitor. This is an optional step and can be removed if you don't need it by deleting `00supervisordIngress.config` |
| 102 | + |
| 103 | +### 2. Parse Queue Configuration |
| 104 | + |
| 105 | +`parseConfig.php` looks for a json file generated earlier that contains all of the environmental variables configured for elastic beanstalk. It then parses out any queue configurations found (see `Add Queues`) section above and generates a supervisor program for each. The program to be generated looks like this: |
| 106 | + |
| 107 | +``` |
| 108 | +[program:$queue] |
| 109 | +command=sudo php artisan doctrine:queue:work $connection --queue=$queue --tries=$tries --sleep=$sleep --daemon |
| 110 | +directory=/var/app/current/ |
| 111 | +autostart=true |
| 112 | +autorestart=true |
| 113 | +process_name=$queue-%(process_num)s |
| 114 | +numprocs=$numProcs |
| 115 | +startsecs=$startSecs |
| 116 | +``` |
| 117 | + |
| 118 | +After parsing all queues it then appends the programs to a clean `supervisord.conf` file in the same directory. |
| 119 | + |
| 120 | +### 3. Deploy Supervisor |
| 121 | + |
| 122 | +Now a bash script `workerDeploy.sh` checks for `IS_WORKER=TRUE` in the EB environmental variables: |
| 123 | + |
| 124 | +* If none is found the script does nothing and exists. |
| 125 | +* If it is found |
| 126 | + * And there is no `init.d` script |
| 127 | + * Supervisor is installed using pip and the custom `supervisord` init script in this project is copied to `/etc/init.d` |
| 128 | + * The generated configuration is copied over the default one at `/etc/supervisord.conf` |
| 129 | + * Supervisor is started |
| 130 | + * Supervisor is set to start at boot |
| 131 | + * And there is an `init.d` script |
| 132 | + * Supervisor is stopped |
| 133 | + * The generated configuration is copied over the old one at `/etc/supervisord.conf` |
| 134 | + * Laravel artisan is used to restart the queue to refresh the daemon |
| 135 | + * Supervisor is restarted with the new configuration |
| 136 | + |
| 137 | + |
| 138 | +# Caveats |
| 139 | + |
| 140 | +This is almost verbatim how I have things setup for another project so some usage is limited because of how it was originally written: |
| 141 | + |
| 142 | +* Queue driver defaults to beanstalkd if not explicitly set |
| 143 | +* Queue parameters in the EB environental variables are case-sensitive |
| 144 | +* There is no way to generate a supervisor program without `--queue=[queue]` right now |
| 145 | +* There is no way to use a pre-generated `supervisord.conf` file right now |
| 146 | + |
| 147 | +All of these are simple fixes though! Check out issues to see these and more and if you need them please make a PR! |
| 148 | + |
| 149 | +## Contributing |
| 150 | + |
| 151 | +Make a PR for some extra functionality and I will happily accept it :) |
| 152 | + |
| 153 | +## License |
| 154 | + |
| 155 | +This package is licensed under the [MIT license](https://github.com/FoxxMD/laravel-elasticbeanstalk-queue-worker/blob/master/LICENSE.txt). |
0 commit comments