How to use LiipImagineBundle to manage thumbnails through Amazon S3

Configure LiipImagineBundle to create thumbnails of images stored on Amazon S3 and save a cache version of them again on S3 may be really painful.

In this post I’ll try to guide you step by step in the configuration process, to have a full configured data flow to create thumbnails stored on Amazon S3 using LiipImagineBundle.

Which is our goal

Our goal is to configure the following data flow:

1. Get a source image file from AWS S3 (from the [bucket_name]/images folder);
2. Manipulate it in some ways with LiipImagineFilters (for example, the thumbnail filter);
3. Save the manipulated image again to AWS S3 (into the bucket_name/cache folder).

As the configuration of the full data flow could be difficult and prone to errors, we will use a simple configuration to reduce to minimum possibilities of issues.

So we will try to get an already uploaded image from the images folder in our bucket on S3 (hardcoding the path in the Twig template), resize it applying custom filters test_medium_thumb and test_small_thumb (that we will configure), then we will save again the resized image in the cache folder into same bucket on S3.

Before starting with the real configuration of the LiipImagineBundle we have to prepare our test bucket and our test route to simplify things.

We will, in fact, create a dedicate action in a controller to test in the tiniest way our filters. This will help us to test in a simpler manner the funcionalities of LiipImagineBundle.

Install the required bundles: LiipImagineBundle and KnpGaufretteBundle

To make all things work we need to install and activate, besides LiipImagineBundle, also KnpGaufretteBundle.

So, install and activate LiipImagineBundle and install and activate KnpGaufretteBundle.

Gaufrette is a PHP5 library that provides a filesystem abstraction layer.

Gaufrette is required to make LiipImagineBundle to get and save files from and to Amazon S3 buckets.

For the moment believe me and install the bundles: continuing in reading of this post you’ll better understand why we need also KnpGaufretteBundle.

Prepare our app for the test

Now that we have installed and activated our required bundles, lets prepare our app to test the data flow:

1. Create in your bucket the folder images;
2. Into this images folder upload a big image (I used this);
3. Create a cache folder in your bucket;
4. Create a DefaultController::TestThumbAction in your DefaultController (or in another controller: this action will be used to just render the test image and its thumbnail) that return the path of the just MANUALLY uploaded test image:

    /**
     * @Route("/test_thumb", name="thumb")
     * @Template()
     */
    public function testThumbAction()
    {
        $image = 'https://s3-'.
            $this->getParameter('amazon.s3.region').
            '.amazonaws.com/'.
            $this->getParameter('amazon.s3.bucket').
            '/images/symfony2.png';

        return [
            'image' => $image
        ];
    }

1. Create the template for the DefaultController::TestThumbnailAction:

<h1>Test resized with <code>test_small_thumb</code></h1>
{# NOTE THAT FOR THE MOMENT THE THUMBNAIL IS COMMENTED #}
{# <img src="{{ image | imagine_filter('test_small_thumb') }}" alt="" /> #}

<h1>Test resized with <code>test_medium_thumb</code></h1>
{# NOTE THAT FOR THE MOMENT THE THUMBNAIL IS COMMENTED #}
{# <img src="{{ image | imagine_filter('test_medium_thumb') }}" alt="" /> #}

<h1>Original Image</h1>
<img src="{{ image }}" alt="" />

Now go to http://127.0.0.1:8000/test_thumb and check that the test image is loaded correctly.

If the test image is not loaded, may be you have to set it as public on AWS S3 (select the image and in the Actions dropdown select Make Public).

Once you can see the test image in its original size, remove comments from the image with test_small_thumb filter applied (only from this for the moment!) and reload the page.

You’ll see this Twig_Error_Runtime exception

An exception has been thrown during the rendering of a template (“Could not find configuration for a filter: test_small_thumb”) in src/AppBundle/Resources/views/Default/testThumb.html.twig at line 2.

500 Internal Server Error – Twig_Error_Runtime

1 linked Exception: NonExistingFilterException »

Ok, we are now ready to start configuring LiipImagineBundle to create our thumbnails of images stored on Amazon S3 and save the cached thumbnail again on Amazon S3. Lets start!

Understand the dataflow of LiipImagineBundle

The dataflow followed by LiipImagineBundle is described in the documentation. I suggest you to read it to fully understand what we need to make the thumbnails being generated and saved to S3.

The short story is this:

1. STEP 1: LiipImagineBundle retrieve the original image from S3 using a DataLoader. Our DataLoader is the StreamLoader (that uses Gaufrette to work).
2. STEP 2: The source image is then processed with filters. The filters we will configure are test_medium_thumb and test_small_thumb.
3. STEP 3: The resulting image file is then saved (cached) to S3 using a CacheResolver. The CacheResolver we’ll use is the AwsS3Resolver.

STEP 1: Get the image source file from AWS S3 with Gaufrette

So, as told, the first step is to get the source image from the images folder in our bucket on AWS S3.

To do this, LiipImagnineBundle requires a DataLoader.

LiipImagnineBundle comes with some bundled DataLoaders. The default DataLoader is the FilesystemLoader as described also in the Configuration chapter of the documentation.

To load images from Amazon S3, we need another DataLoader, the StreamLoader:

The Liip\ImagineBundle\Binary\Loader\StreamLoader allows to read images from any stream (http, ftp, and others…) registered thus allowing you to serve your images from literally anywhere.

LiipImagineBundle Documentation, Stream Loader

The easier way to use this loader is configuring it to use Gaufrette.

So we have to first configure Gaufrette and then use it with the LiipImagineBundle‘StreamLoader.

Understand how Gaufrette works

At this point Gaufrette should be already installed as you should have already installed and activated KnpGaufretteBundle.

But, before continuing, just a little bit of theory to understand how Gaufrette works.

As told before,

Gaufrette is a PHP 5.3+ library providing a filesystem abstraction layer. This abstraction layer allows you to develop applications without needing to know where all their media files will be stored or how.

In other words, you interact with Gaufrette’s API that emulates actions you can take on a filesystem to manage files, so, when you decide to switch from the local filesystem to a cloud environment, you have to simply change some parameters in the configuration. Clean, simple and easy!

If you like, read some more details on the Knp Labs’ blog.

Anyway, to make your application “storage agnostic”, Gaufrette uses two main concepts: Adapters and Filesystems.

Your application will ever communicate with a Filesystem object. An Filesystem object uses a Adapter to communicate with the “storage” and perform the real actions on it.

Here the details about this simple process.

Introducing Gaufrette StreamWrapper

To make LiipImagineBundle StreamLoader work with images we need to understand another concept used by Gaufrette, the last one: StreamWrapper.

As we don’t simply want to interact with the filesystem, but we want also to manipulate images, we need a StreamWrapper that we’ll then use as DataLoader in LiipImagineBundle.

A StreamWrapper permits to register a new stream wrapper so we’ll can get the contents of the image and use them to create a new image.

Configuring KnpGaufretteBundle to work with AWS S3

So, after the theory, finally the practice! Lets configure KnpGaufretteBundle.

So, we need to configure those three things:

1. An Adapter: we’ll use the AwsS3Adapter
2. A Filesystem that uses the AwsS3Adapter: we’ll call it filesystem_aws_S3_images
3. A StreamWrapper (we don’t have to do anything more than writing a directive in our config.yml file to configure it)

Configure the AwsS3Adapter for KnpGaufretteBundle

The AwsS3Adapter needs the credentials to access the bucket on Amazon AWS S3, so, we first configure these credentials, then we’ll configure the S3Client (that uses the credentials):

shq.amazon.s3Credentials:
    class: Aws\Credentials\Credentials
    arguments: ["%amazon.s3.key%", "%amazon.s3.secret%"]

shq.amazon.s3:
    class: Aws\S3\S3Client
    arguments:
        - version: %amazon.s3.version%
          region: %amazon.s3.region%
          credentials: "@shq.amazon.s3Credentials"

Now, we can configure the AwsS3Adapter, the Filesystem and the StreamWrapper:

# config.yml
# KnpGaufretteBundle Configuration
knp_gaufrette:
    adapters:
        adapter_aws_s3_images:
            aws_s3:
                service_id: "shq.amazon.s3" # ! ! ! Without @ ! ! !
                bucket_name: "%amazon.s3.bucket%"
                options:
                    directory: 'images'
                    create: true
    filesystems:
        filesystem_aws_s3_images:
            adapter: adapter_aws_s3_images
    stream_wrapper: ~

As told in the documentation,

The stream_wrapper settings allow you to register filesystems with a specified domain and then use as a stream wrapper anywhere in your code like: gaufrette://domain/file.txt

GaufretteBundle Documentation, Stream Wrappers

Testing that filesystem_aws_s3_images works

Now that we have configured our Adapter and our Filesystem, we should test that it really works as expected.

Once configured, Filesystems are accessibile from the container, so your DefaultController::testThumbAction() write the following code:

    /**
     * @Route("/test_thumb", name="thumb")
     * @Template()
     */
    public function testThumbAction()
    {
        $filesystem = $this->get('knp_gaufrette.filesystem_map')->get('filesystem_aws_s3_images');
        $file = $filesystem->get('symfony2.png');

        /*
         * Note the use of the dump() function.
         * If you don't have the VarDumperComponent installed, use var_dump().
         * @see http://symfony.com/doc/current/components/var_dumper/introduction.html
         */
        dump($file);die;

        $image = 'https://s3-'.

        ...
    }

If the dump() shows you a Gaufrette\File object, then… CONGRATULATIONS! Gaufrette is correctly configured!

As we know that Gaufrette works, we can now use it with LiipImgineBundle!

Using Gaufrette as DataLoader for LiipImagineBundle

As told, we need a DataLoader to make LiipImagineBundle get the real content of the file so it can create a thumbnails.

Now that we have a working Gaufrette, we need to use it to configure the StreamLoader.

We have to alternatives to configure a DataLoader: using the factory and using a defined service.

We’ll use the factory method, so we have all configuration in one place (I find this more comfortable, but you can use the method you like).

So, to define the DataLoader using the factory method, put this in your config.yml file:

liip_imagine:
    loaders:
        loader_aws_s3_images:
            stream:
                # This refers to knp_gaufrette filesystems configuration
                wrapper: gaufrette://filesystem_aws_s3_images/

Perfect: we have done. Now we have to configure custom filters.

STEP 2: Define filters to resize images and create the thmbnails

Now that we can get images from AWS S3, it’s time to define our custom filters to manipulate them.

We will create two filters: test_medium_thumb and test_small_thumb.

To define filters we have to edit the config.yml file:

liip_imagine:
    loaders:
...
    filter_sets:
        test_medium_thumb:
            data_loader: loader_aws_s3_images
            # We don't yet have a cache resolver configured
            cache: cache_resolver_aws_s3
            quality: 75
            filters:
                thumbnail: { size: [400, 400], mode: outbound }
        test_small_thumb:
            data_loader: loader_aws_s3_images
            # We don't yet have a cache resolver configured
            cache: cache_resolver_aws_s3
            quality: 75
            filters:
                thumbnail: { size: [100, 100], mode: outbound }

Done! Really simple, doesn’t it?

But, as you can see, we have the directive cache in each of the two defined filters: these directives use the CacheResolver called cache_resolver_aws_s3.

So, as it may be yet clear, we have to define the cache_resolver_aws_s3. We are fastly moving toward the STEP 3!

STEP 3: Define the CacheResolver AwsS3Resolver

As we want to save our generated thumbnails to AWS S£, we need to use the CacheResolver AwsS3Resolver.

In this case, too, we can decide to define our resolver as a service or using the factory.

In this case, too, I’ll use the factory, but you can choose to use the method you like.

So, in the config.yml file:

liip_imagine:
    loaders:
        ...
    resolvers:
       cache_resolver_aws_s3:
          aws_s3:
              client_config:
                  credentials:
                      key: %amazon.s3.key%
                      secret: %amazon.s3.secret%
                  region: %amazon.s3.region%
                  version: %amazon.s3.version%
              get_options:
                  Scheme: 'https'
              put_options:
                  CacheControl: 'max-age=86400'
    filter_sets:
        ...

Test that all works well

And now that we have all configured, it’s time to test!

Edit DefaultController::TestThumbAction

First: change the code into the action:

    /**
     * @Route("/test_thumb", name="thumb")
     * @Template()
     */
    public function testThumbAction()
    {
        //$filesystem = $this->get('knp_gaufrette.filesystem_map')->get('filesystem_aws_s3_images');
        //$file = $filesystem->get('symfony2.png');

        /*
         * Note the use of the dump() function.
         * If you don't have the VarDumperComponent installed, use var_dump().
         * @see http://symfony.com/doc/current/components/var_dumper/introduction.html
         */
        //dump($file);die;

        /*$image = 'https://s3-'.
            $this->getParameter('amazon.s3.region').
            '.amazonaws.com/'.
            $this->getParameter('amazon.s3.bucket').
            '/images/symfony2.png';*/

        $image = 'symfony2.png';

        return [
            'image' => $image
        ];
    }

Activate filters in the template

Now activate the filters in the template removing comments:

<h1>Test resized with <code>test_small_thumb</code></h1>
<img src="{{ image | imagine_filter('test_small_thumb') }}" alt="" />

<h1>Test resized with <code>test_medium_thumb</code></h1>
<img src="{{ image | imagine_filter('test_medium_thumb') }}" alt="" />

<h1>Original Image</h1>
<img src="{{ image }}" alt="" />

Test!

Reload your page at http://127.0.0.1:8000/test_thumb and, if you’ve done well all the steps, you’ll see something like this:

Obviously, “Original image” is no longer visible as the path isn’t correct (we commented it in the controller!).

Done: now you have LiipImagineBundle up and running. You can now save your images to Amazon AWS S3, get them and resize to thumbnail sizes and save them again on AWS S3!

Some things to note: what happened behind the scenes

The first thing you should note is the URLs of the thumbnails.

The first time you load the page the resulting URLs are those:

<h1>Test resized with <code>test_small_thumb</code></h1>
<img src="http://127.0.0.1:8000/media/cache/resolve/test_small_thumb/symfony2.png" alt="" />

<h1>Test resized with <code>test_medium_thumb</code></h1>
<img src="http://127.0.0.1:8000/media/cache/resolve/test_medium_thumb/symfony2.png" alt="" />

<h1>Original Image</h1>
<img src="symfony2.png" alt="" />

As you can note, they are in this form: media/cache/resolve/[filter_name]/.

Try to reload the page.

The resulting URLs, this time, are those:

<h1>Test resized with <code>test_small_thumb</code></h1>
<img src="https://s3-eu-west-1.amazonaws.com/trustback-me-dev/test_small_thumb/symfony2.png" alt="" />

<h1>Test resized with <code>test_medium_thumb</code></h1>
<img src="https://s3-eu-west-1.amazonaws.com/trustback-me-dev/test_medium_thumb/symfony2.png" alt="" />

<h1>Original Image</h1>
<img src="symfony2.png" alt="" />

They refers to the public URL on AWS S3: https://s3-eu-west-1.amazonaws.com/[bucket_name]/[filter_name/.

In fact, the first time you load the images, if LiipImagineBundle cannot find a cached version, points to a controller defined in vendor/liip/imagine-bundle/Resources/config/routing.xml

The URL you see the first time points to the controller that applies the filters. Once the resulting image is cached, the next time LiipImagineBundle uses directly the URL of the bucket on AWS S3.

The second thing you should note, is that LiipImagineBundle automatically creates a folder for each filter in your bucket:

In practice, for each configured filter, the resulting images are saved into a dedicated folder.

Troubleshooting

And finally, some advices to debug in case of issues:

1. Keep attention to differences between UPPERCASE and lowercase letters: maybe you can write filesystem_aws_s3_images in one place and filesystem_aws_S3_images in another place. Spot the difference!
2. As the bundle is usually usedfrom inside a template, you’ll don’t get exceptions or errors. If something doesn’t work, go to the logs! There there are all the errors: find them looking for strings like “gaufrette” or “liip_imagine” or “liip/imagine-bundle” or “request.ERROR” or “php.DEBUG” and you’ll find all the info you need to understand what went wrong.

And finally, before to close this long post, remember that in the Issues section on GitHub there are a lot of useful information to better understand how to use the bundle.

Here are a couple of interesting issues:

• Complete S3 Setup: Original Images and cached Images on AWS S3
• need https option for aws stream

Remember to “Make. Ideas. Happen.”.

I wish you flocking users, see you soon!

L'articolo How to use LiipImagineBundle to manage thumbnails through Amazon S3 proviene da ÐΞV Experiences by Serendipity HQ.