DEV Community

Dhaval Mehta
Dhaval Mehta

Posted on

Automating AWS Batch Jobs with Amazon EventBridge Scheduler

Introduction

Amazon EventBridge Scheduler, introduced by AWS in November 2022, is a powerful serverless scheduler that allows you to create, run, and manage tasks across various AWS services from one central, managed platform. This highly scalable service enables you to schedule millions of tasks, invoking any AWS service as a target, without the need for provisioning or managing underlying infrastructure. In this blog post, we will explore how to use EventBridge Scheduler to automate AWS Batch jobs, ensuring smooth and efficient task execution.

What is Amazon EventBridge Scheduler?

Amazon EventBridge Scheduler is a serverless scheduler introduced by AWS to facilitate the easy creation, execution, and management of tasks across multiple AWS services. With support for over 200 services and more than 6,000 APIs, the scheduler allows you to configure schedules with a minimum granularity of one minute, ensuring precise control over task execution. It also offers at-least-once event delivery to targets, along with the ability to set delivery windows, retries, retention times, and dead letter queues.

How does EventBridge Scheduler work?

EventBridge Scheduler enables you to create one-time or recurring schedules, making it flexible enough to handle various use cases. Whether you need to trigger a task just once or run it periodically, EventBridge Scheduler has you covered. The scheduler requires no event bus, streamlining the process of setting up schedules and reducing complexity. Here are some key features of EventBridge Scheduler:

  • Quota on schedules: Up to 1 million schedules per AWS account.
  • Event invocation throughput: Capable of supporting throughput in thousands of transactions per second (TPS).
  • Targets: Over 270 services and more than 6,000 API actions with AWS SDK targets.
  • Time expressions and time-zones: Supports at(), cron(), and rate() expressions in all time-zones and with Daylight Saving Time (DST) adjustments.
  • One-time schedules: Allows the creation of single, one-off schedules with precise execution times.
  • Time window schedules: Enables the configuration of schedules with a time window for more flexibility.
  • Rule quota consumption: No rule quota consumption; soft limit of 1 million schedules.

Scheduling AWS Batch Jobs with EventBridge Scheduler

Now, let's dive into the process of automating AWS Batch jobs using Amazon EventBridge Scheduler. Below are the steps to achieve this:

Creating the Required Policies

To get started, we need to create a policy that grants the necessary permissions for the Scheduler to execute our AWS Batch job. The policy should include permissions for "batch:SubmitJob", "batch:DescribeJobDefinitions", and "batch:DescribeJobQueues".

# CloudFormation code snippet for creating the BatchSubmitPolicy
BatchSubmitPolicy:
  Type: AWS::IAM::ManagedPolicy
  Properties:
    Description: Policy to allow Scheduler to execute AWS Batch jobs
    ManagedPolicyName: YourBatchSubmitPolicy
    PolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: 'Allow'
          Action: 
            - "batch:SubmitJob"
            - "batch:DescribeJobDefinitions"
            - "batch:DescribeJobQueues"
          Resource: 
            - YourBatchJobDefinitionARN
            - YourBatchJobQueueARN
Enter fullscreen mode Exit fullscreen mode

Creating the required role

Next, we create an IAM role that the Scheduler will assume to execute the tasks. This role will have the previously created BatchSubmitPolicy attached.

# CloudFormation code snippet for creating the EventBridgeRole
EventBridgeRole:
  Type: 'AWS::IAM::Role'
  Properties:
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: 'Allow'
          Principal:
            Service:
              - 'scheduler.amazonaws.com'
          Action: 'sts:AssumeRole'
    Path: "/"
    ManagedPolicyArns:
      - !Ref BatchSubmitPolicy
    Description: EventBridge role for AWS Batch job execution
    RoleName: YourEventBridgeRole
Enter fullscreen mode Exit fullscreen mode

Configuring the Schedule

Now we create the EventBridge schedule, specifying the name, description, and the desired ScheduleExpression. In this example, we use a rate expression to trigger the job every 8 hours.

# CloudFormation code snippet for creating the EventSchedule
EventSchedule:
  Type: AWS::Scheduler::Schedule
  Properties: 
    Description: 'EventBridge schedule to trigger the AWS Batch job'
    FlexibleTimeWindow: 
      Mode: "OFF"
    Name: YourBatchExecSchedule
    ScheduleExpression: 'rate(8 hours)'
    State: 'ENABLED'
Enter fullscreen mode Exit fullscreen mode

Defining the Target

The TargetArn needs to be defined in a specific format, and in this case, we'll set it to invoke the "batch:SubmitJob" API action of AWS Batch.

# CloudFormation code snippet for defining the Target
Target: 
  Arn: arn:aws:scheduler:::aws-sdk:batch:submitJob
  RoleArn: !GetAtt EventBridgeRole.Arn     
  Input: !Sub |
    {
      "JobName": "scheduled-batch-job",
      "JobDefinition": "YourBatchJobDefinitionARN",
      "JobQueue": "YourBatchJobQueueARN",
      "ContainerOverrides": {
          "Environment": [ 
            { 
                "Name": "LoggingLevel",
                "Value": "${LoggingLevel}"
            },
            {
              "Name": "Prefix",
              "Value": "${Prefix}"
            }
          ]
        }            
    }
Enter fullscreen mode Exit fullscreen mode

Complete CloudFormation Code

Here's the complete CloudFormation template to create the EventBridge schedule for your AWS Batch job.

AWSTemplateFormatVersion: "2010-09-09"
Description: CloudFormation template to create an EventBridge schedule for AWS Batch jobs

Parameters:
  LoggingLevel:
    Type: String
    AllowedValues: [NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL]    
    Default: "INFO"     

  Prefix:
    Type: String
    Default: "prefix-value"   

Resources:
  BatchSubmitPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: Policy to allow Scheduler to execute AWS Batch jobs
      ManagedPolicyName: YourBatchSubmitPolicy
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: 'Allow'
            Action: 
              - "batch:SubmitJob"
              - "batch:DescribeJobDefinitions"
              - "batch:DescribeJobQueues"
            Resource:
              - YourBatchJobDefinitionARN
              - YourBatchJobQueueARN

  EventBridgeRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: 'Allow'
            Principal:
              Service:
                - 'scheduler.amazonaws.com'
            Action: 'sts:AssumeRole'
      Path: "/"
      ManagedPolicyArns:
        - !Ref BatchSubmitPolicy
      Description: EventBridge role for AWS Batch job execution
      RoleName: YourEventBridgeRole

  EventSchedule:
    Type: AWS::Scheduler::Schedule
    Properties: 
      Description: 'EventBridge schedule to trigger the AWS Batch job'
      FlexibleTimeWindow: 
        Mode: "OFF"
      Name: YourBatchExecSchedule
      ScheduleExpression: 'rate(8 hours)'
      State: 'ENABLED'
      Target: 
        Arn: arn:aws:scheduler:::aws-sdk:batch:submitJob
        RoleArn: !GetAtt EventBridgeRole.Arn     
        Input: !Sub |
          {
            "JobName": "scheduled-batch-job",
            "JobDefinition": "YourBatchJobDefinitionARN",
            "JobQueue": "YourBatchJobQueueARN",
            "ContainerOverrides": {
                "Environment": [ 
                  { 
                      "Name": "LoggingLevel",
                      "Value": "${LoggingLevel}"
                  },
                  {
                    "Name": "Prefix",
                    "Value": "${Prefix}"
                  }
                ]
              }            
          }

Enter fullscreen mode Exit fullscreen mode

Conclusion

Amazon EventBridge Scheduler offers a powerful and flexible solution for automating tasks across various AWS services. In this blog post, we explored how to use EventBridge Scheduler to schedule and trigger AWS Batch jobs. By following the steps outlined in this guide, you can efficiently manage your tasks and improve operational efficiency.

References

  1. Amazon EventBridge Scheduler Documentation
  2. Supported Universal Targets for EventBridge Scheduler
  3. AWS Batch API Reference - Submit Job
  4. AWS CloudFormation - AWS::Scheduler::Schedule Resource Type

Top comments (1)

Collapse
 
sreerajuv profile image
Sreeraj

How to use !Ref arn value in
"JobName": "scheduled-batch-job",
"JobDefinition": "YourBatchJobDefinitionARN",
"JobQueue": "YourBatchJobQueueARN",