Background jobs with Symfony messenger component

Let’s say your client comes up with this request: whenever an image is uploaded, do something with it. Change background color, convert it to some other format, add watermark… Whatever it is, image processing is often a slow process that will add significant time to your ‘upload image’ action. 



Do it later

But, if you were to think about it, there is no reason why it should be executed during the request. The user doesn’t really care about that. He will upload the image, continue with his day and image processing will be done somewhere.. in the background. 


Flow of any background job looks like this: create job + store necessary data somewhere so it can be executed later + execute it



Symfony messenger component, introduced in version 4.1, handles this for you!



Create job

Think about the data necessary for job execution. In this case, we should remember some kind of identifier that will let us fetch the image later on. Depending on your requirements, it can be id of an entity stored in the database, path to the image or something else.

In the Message folder of your project, create a class that will encapsulate that data (and do nothing else).

namespace App\Message;

class SetBackgroundColorToBlack
{
    /**
     * @var string
     */
    private $imageIdentifier;

    public function __construct(string $imageIdentifier)
    {
        $this->imageIdentifier = $imageIdentifier;
    }

    public function getImageIdentifier(): string
    {
        return $this->imageIdentifier;
    }
}

Inject MessageBusInterface into a service that handles image uploading. After the image is uploaded, create your message and dispatch it.

 public function __construct(MessageBusInterface $messageBus)
    {
        $this->messageBus = $messageBus;
    }

 $message = new SetBackgroundColorToBlack($image->getIdentifier());
 $this->messageBus->dispatch($message);

Congrats! You've just told symfony... well, nothing much.

Where do all the messages go to?

You've created your message and dispatched it. What now? Store it somewhere so the class that knows how to handle it can pick it.

In this case, it's called transport. Symfony provides multiple transport options as well as retry strategies, so read the docs and decide what's best for your case. I used doctrine transport that won't be suitable for large projects.

Two transports are defined: async, that handles messages of SetBackgroundColorToBlack type and failure transport.

framework:
    messenger:
        failure_transport: failed
        transports:
            async:
                dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
            failed:
                dsn: 'doctrine://default?queue_name=failed'

        routing:
            'App\Message\SetBackgroundColorToBlack': async

It's best to define your transports as env variables.

###> symfony/messenger ###
MESSENGER_TRANSPORT_DSN=doctrine://default
###< symfony/messenger ###

Do the job

Someone should do the job. In MessageHandler directory of your project, create class SetBackgroundColorToBlackHandler.

This class must implement MessageHandlerInterface.

class SetBackgroundColorToBlackHandler implements MessageHandlerInterface
{
    /**
     * @var ImageProcessingService
     */
    private $imageProcessingService;

    /**
     * @var ImageRepository
     */
    private $imageRepository;

    public function __construct(ImageProcessingService $imageProcessingService, 
ImageRepository $imageRepository)
    {
        $this->imageProcessingService = $imageProcessingService;
        $this->imageRepository = $imageRepository;
    }

    public function __invoke(SetBackgroundColorToBlack $setBackgroundColorToBlack): void
    {
        $image = $this->imageRepository
                 ->get($setBackgroundColorToBlack->getImageIdentifier());

        if ($image === null) {
            return;
        }

        $this->imageProcessingService->setBackgroundColorToBlack($image);

    }

Notes:

• Symfony is smart enough to connect message to the handler. It's enough to type hint variable of __invoke() method signature in the handler class.
• Handlers are services which means that you can inject other services.
• Maybe the image can't be fetched because someone deleted it in the meantime. Depending on your domain, maybe the image must exist and you want to throw an exception in that case.
• Maybe imageProcessingService throws exception (it probably does). Remember we defined failure transport? By default, all messages will be retried 3 times before they end up in failure transport.

Work, work, work, work, work

Define worker that is gonna consume those messages.
php bin/console messenger:consume async

Install supervisor on production that will watch out for your workers.

How do you like symfony messenger component? What do you use to handle background jobs? Do you think they are necessary in the first place?