Introduction
Recently, I had to migrate my queue driver on a project from Redis to SQS and for some weird reason the information provided on the official Laravel Queues documentation didn't do justice to this and I couldn't find an article online that could help, that's why I'm writing this, with the hope that you don't spend two hours of your Saturday morning figuring stuff that should have been documented. I'd try to keep this article as brief as possible, I'd attach links to other articles that explain some steps so I can focus on the important parts.
Prerequisites
- An existing Laravel application
- An AWS account
- An understanding of Laravel queues
- Some AWS knowledge
Step One - Install the AWS SDK
Run the following command to install the SDK:
composer require aws/aws-sdk-php
Step Two - Get your AWS keys
Click on this link to watch the guide.
When you get to the point where you need to set permissions, search for AmazonSQSFullAccess
and select it.
If you already have existing credentials, modify its policies and do the same.
Step Three - Create a queue
Head over to your SQS console, click on create queue
.
Out of the box, Laravel dispatches jobs to the default
queue, therefore, your first queue should be named default
on SQS.
If you would be dispatching to a queue other than
default
, the queue name should be the same as the queue you're dispatching to.
Leave everything else at its default, scroll to the bottom and click on the create queue
button.
Step Four - Set .env
variables
Open your .env
with your preferred text editor. Add your AWS keys in the following format:
AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION=YOUR_SQS_REGION
Ensure you replace YOUR_*
with the right credentials. Next, set your SQS Prefix:
SQS_PREFIX=https://sqs.<your-region>.amazonaws.com/<your-account-id>
| I don't think I need to explain what <your-region>
means
You can get <your-account-id>
on your AWS console, click on your account name on the top right of your dashboard.
you'd find your account ID just beside My Account
as shown above. Finally, change the queue driver to SES.
QUEUE_CONNECTION=sqs
Step Five - Start queue worker
Run the following command to start your queue worker:
php artisan queue:work sqs
Dispatch your job on the default queue and it should be processed.
Modification for Laravel horizon
If you use Laravel Horizon to manage your queues, you'd be making some modifications to your horizon.php
file. For each worker defined, change the queue connection from redis
which is the default to env('QUEUE_CONNECTION')
.
From this:
<?php
...
'defaults' => [
'queue_name' => [
...
'connection' => 'redis',
],
...
]
...
To this:
<?php
...
'defaults' => [
'queue_name' => [
...
'connection' => env('QUEUE_CONNECTION'),
],
...
]
...
After these modifications have been made, restart horizon and all should be good.
Conclusion
I have nothing else to add, have fun, keep building.
Top comments (7)
The Horizon docs at laravel.com/docs/8.x/horizon#insta... say:
"Laravel Horizon requires that you use Redis to power your queue. Therefore, you should ensure that your queue connection is set to redis in your application's config/queue.php configuration file."
So I wonder how this configuration (Horizon+SQS) is actually running,
Great article. Worthy of note are the limitations associated with using SQS such as maximum job payload size, maximum job delay, maximum job life span, etc.
This will not provide full Horizon functionality.
Horizon uses an extended RedisQueue class with additional methods used to provide queue metrics and extra functionality.
You must extend from SqsQueue and add the missing pieces and register the new driver for Horizon to achieve complete functionality.
Here is an example of a full alternative driver for queues on RabbitMQ, which also has complete support for Horizon: github.com/vyuldashev/laravel-queu...
With the above you might have broken jobs and you'll have to monitor your error log constantly to catch what Horizon will not.
I found the docs quite lacking too so wrote it up here for personal notes:
snippets.snhd.co/posts/neat-secure...
I actually forget to start the queue runner in my post, but it was more about configuring things within the AWS Ecosystem.
Sounds like a two hours well spent! I guess also now we have a time to beat?
Horizon does not play well with SQS
You don't necessarily have to name your queue "default" on AWS, you can set any name and specify it in your Laravel project's env/config