DEV Community

Cover image for Configuring Laravel Queues with AWS SQS
Michael Okoh
Michael Okoh

Posted on

Configuring Laravel Queues with AWS SQS

Introduction

Recently, I had to migrate my queue driver on a project from Redis to SQS and for some weird reason the information provided on the official Laravel Queues documentation didn't do justice to this and I couldn't find an article online that could help, that's why I'm writing this, with the hope that you don't spend two hours of your Saturday morning figuring stuff that should have been documented. I'd try to keep this article as brief as possible, I'd attach links to other articles that explain some steps so I can focus on the important parts.

Prerequisites

  • An existing Laravel application
  • An AWS account
  • An understanding of Laravel queues
  • Some AWS knowledge

Step One - Install the AWS SDK

Run the following command to install the SDK:



composer require aws/aws-sdk-php


Enter fullscreen mode Exit fullscreen mode

Step Two - Get your AWS keys

Click on this link to watch the guide.

When you get to the point where you need to set permissions, search for AmazonSQSFullAccess and select it.

If you already have existing credentials, modify its policies and do the same.

Step Three - Create a queue

Head over to your SQS console, click on create queue.

create queue

Out of the box, Laravel dispatches jobs to the default queue, therefore, your first queue should be named default on SQS.

If you would be dispatching to a queue other than default, the queue name should be the same as the queue you're dispatching to.

default queue

Leave everything else at its default, scroll to the bottom and click on the create queue button.

Step Four - Set .env variables

Open your .env with your preferred text editor. Add your AWS keys in the following format:



AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION=YOUR_SQS_REGION


Enter fullscreen mode Exit fullscreen mode

Ensure you replace YOUR_* with the right credentials. Next, set your SQS Prefix:



SQS_PREFIX=https://sqs.<your-region>.amazonaws.com/<your-account-id>


Enter fullscreen mode Exit fullscreen mode

| I don't think I need to explain what <your-region> means

You can get <your-account-id> on your AWS console, click on your account name on the top right of your dashboard.

account ID

you'd find your account ID just beside My Account as shown above. Finally, change the queue driver to SES.



QUEUE_CONNECTION=sqs


Enter fullscreen mode Exit fullscreen mode

Step Five - Start queue worker

Run the following command to start your queue worker:



php artisan queue:work sqs


Enter fullscreen mode Exit fullscreen mode

Dispatch your job on the default queue and it should be processed.

Modification for Laravel horizon

If you use Laravel Horizon to manage your queues, you'd be making some modifications to your horizon.php file. For each worker defined, change the queue connection from redis which is the default to env('QUEUE_CONNECTION').

From this:



<?php
...
'defaults' => [
    'queue_name' => [
        ...
        'connection' => 'redis',
    ],
...
]
...


Enter fullscreen mode Exit fullscreen mode

To this:



<?php
...
'defaults' => [
    'queue_name' => [
        ...
        'connection' => env('QUEUE_CONNECTION'),
    ],
...
]
...


Enter fullscreen mode Exit fullscreen mode

After these modifications have been made, restart horizon and all should be good.

Conclusion

I have nothing else to add, have fun, keep building.

Top comments (7)

Collapse
 
oliworx profile image
Oliver Kurmis

The Horizon docs at laravel.com/docs/8.x/horizon#insta... say:

"Laravel Horizon requires that you use Redis to power your queue. Therefore, you should ensure that your queue connection is set to redis in your application's config/queue.php configuration file."

So I wonder how this configuration (Horizon+SQS) is actually running,

Collapse
 
chygoz2 profile image
Chigozie Ekwonu

Great article. Worthy of note are the limitations associated with using SQS such as maximum job payload size, maximum job delay, maximum job life span, etc.

Collapse
 
madalinignisca profile image
Madalin Ignisca

This will not provide full Horizon functionality.
Horizon uses an extended RedisQueue class with additional methods used to provide queue metrics and extra functionality.
You must extend from SqsQueue and add the missing pieces and register the new driver for Horizon to achieve complete functionality.

Here is an example of a full alternative driver for queues on RabbitMQ, which also has complete support for Horizon: github.com/vyuldashev/laravel-queu...

With the above you might have broken jobs and you'll have to monitor your error log constantly to catch what Horizon will not.

Collapse
 
seanhood profile image
Sean Hood

I found the docs quite lacking too so wrote it up here for personal notes:
snippets.snhd.co/posts/neat-secure...

I actually forget to start the queue runner in my post, but it was more about configuring things within the AWS Ecosystem.

Collapse
 
llbbl profile image
Logan Lindquist

Sounds like a two hours well spent! I guess also now we have a time to beat?

Collapse
 
ankurk91 profile image
Ankur K

Horizon does not play well with SQS

Collapse
 
tommie profile image
ToMmiE DaRkU

You don't necessarily have to name your queue "default" on AWS, you can set any name and specify it in your Laravel project's env/config