This article is about migrating Keycloak on Fargate, but it also describes how to scale out Fargate according to load and how to scale out according to the time of day, controlling scale out on Fargate It is also a reference for those who want to.
What is Keycloak
Open source identity and access management software for single sign-on and API access authentication and authorization control.
What is Fargate
It is an AWS management service that can execute containers. There are also ECS on EC2 that executes containers on EC2 and AWS EKS, which is a Kubernetes management service, but AWS Fargate is recommended for environments that can be executed simply without maintaining the container execution infrastructure.
About this story
This story describes the story of migrating Keycloak services built on ECS on EC2 to Fargate and migrating Keycloak from v6 to v19.
The story of migrating Keycloak's services that were built on ECS on EC2 to Fargate
There are three advantages of migrating ECS on EC2 to Fargate, and we migrated it before the version upgrade described below.
- No need to maintain container infrastructure
- Minimizes the cost of scaling out
- Faster startup time during scale-out, making it easier to follow spikes
The story of migrating Keycloak from v6 to v19
Keycloak basically only needs to be started with a new version because it has a function to automatically migrate at startup, but there are some incompatibility problems due to DB constraints. Therefore, each time an error occurs, you need to investigate how to respond and adjust for inconsistencies.
In the following example, SELECT REALM_ID, NAME, COUNT() FROM KEYCLOAK_GROUP WHERE PARENT_GROUP IS NULL GROUP BY REALM_ID, NAME HAVING COUNT() > 1; can detect duplicate group names.
ERROR [org.keycloak.connections.jpa.updater.liquibase.conn.DefaultLiquibaseConnectionProvider] (ServerService Thread Pool -- 67) Change Set META-INF/jpa-changelog-9.0.1.xml::9.0.1-KEYCLOAK-12579-add-not-null-constraint::keycloak failed. Error: Duplicate entry 'school- -ks' for key 'SIBLING_NAMES' [Failed SQL: UPDATE authdbdev.KEYCLOAK_GROUP SET PARENT_GROUP = ' ' WHERE PARENT_GROUP IS NULL]
FATAL [org.keycloak.services] (ServerService Thread Pool -- 67) java.lang.RuntimeException: Failed to update database
It is also important to note that the environment variables to be set have changed due to the migration from WildFly to Quarks.
| WildFly | Quarks |
|---|---|
| DB_DATABASE | KC_DB_URL_DATABASE |
| DB_HOST | KC_DB_URL_HOST |
| DB_PASSWORD | KC_DB_PASSWORD |
| DB_USER | KC_DB_USERNAME |
When configuring a multi-node cluster with Infinispan defined in standalone-ha.xml in Wildfly, the following environment variables must be set in Quarks after v17.
KC_CACHE="ispn"
KC_CACHE_CONFIG_FILE="cache-ispn-jdbc-ping.xml"
The cache-ispn-jdbc-ping.xml performs the following description (when MySQL is selected for RDS): owners sets the number of nodes on which the cache is kept.
If you are scaling out while running with at least two nodes to maintain availability, you must determine the number of nodes while considering the number of nodes that will degenerate simultaneously when scaling in. (Since you cannot control the nodes when scaling in, you need to devise a way to prevent the cache from being lost by deleting the nodes that hold the cache all at once.)
Also, realms and users max-count affect performance. If you keep a session that exceeds max-count, communication with the DB will occur, so it is better to increase max-count as much as memory allows.
However, when starting in Duplicated mode instead of Replicated mode, it is necessary to thoroughly test with a load test tool using Distributed Load Testing on AWS, etc. so that the cache is rebalanced when scaling in, resulting in out-of-memory. For details of the parameters, see Configuring Infinispan caches and urn:infinispan:config:11.0.
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
xmlns="urn:infinispan:config:11.0">
<jgroups>
<stack name="jdbc-ping-tcp" extends="tcp">
<JDBC_PING connection_driver="com.mysql.cj.jdbc.Driver"
connection_username="${env.KC_DB_USERNAME}" connection_password="${env.KC_DB_PASSWORD}"
connection_url="${env.KC_DB_URL}"
initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, ping_data VARBINARY(255), constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name));"
info_writer_sleep_time="500"
remove_all_data_on_view_change="true"
stack.combine="REPLACE"
stack.position="MPING" />
</stack>
</jgroups>
<cache-container name="keycloak">
<transport lock-timeout="60000" stack="jdbc-ping-tcp"/>
<local-cache name="realms">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<local-cache name="users">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<distributed-cache name="sessions" owners="3">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="authenticationSessions" owners="3">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineSessions" owners="3">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="clientSessions" owners="3">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineClientSessions" owners="3">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="loginFailures" owners="3">
<expiration lifespan="-1"/>
</distributed-cache>
<local-cache name="authorization">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<replicated-cache name="work">
<expiration lifespan="-1"/>
</replicated-cache>
<local-cache name="keys">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="3600000"/>
<memory max-count="1000"/>
</local-cache>
<distributed-cache name="actionTokens" owners="3">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="-1" lifespan="-1" interval="300000"/>
<memory max-count="-1"/>
</distributed-cache>
</cache-container>
</infinispan>
The Dockerfile is as follows:
FROM quay.io/keycloak/keycloak:19.0.3
COPY conf/keycloak.conf /opt/keycloak/conf/keycloak.conf
COPY conf/cache-ispn-jdbc-ping.xml /opt/keycloak/conf/cache-ispn-jdbc-ping.xml
RUN /opt/keycloak/bin/kc.sh build --cache-config-file=cache-ispn-jdbc-ping.xml
WORKDIR /opt/keycloak
ENTRYPOINT [ "/opt/keycloak/bin/kc.sh" ]
The definition of ECS in Terraform is as follows: Please understand that the part marked with _xxxx_ is a constant passed in a variable.
resource "aws_ecs_cluster" "keycloak" {
name = "clustername"
setting {
name = "containerInsights"
value = "enabled"
}
}
resource "aws_ecs_service" "keycloak" {
cluster = aws_ecs_cluster.keycloak.id
deployment_maximum_percent = 200
deployment_minimum_healthy_percent = 100
desired_count = _keycloak_desired_count_min_
enable_ecs_managed_tags = false
enable_execute_command = true
health_check_grace_period_seconds = 180
name = _servicename_
platform_version = "LATEST"
propagate_tags = "TASK_DEFINITION"
scheduling_strategy = "REPLICA"
task_definition = aws_ecs_task_definition.keycloak.arn
capacity_provider_strategy {
capacity_provider = "FARGATE"
base = 2
weight = 1 // After the third unit, it will be started with FARGATE at a rate of 25%
}
capacity_provider_strategy {
capacity_provider = "FARGATE_SPOT"
base = 0
weight = 3 // After the third unit, it starts with FARGATE_SPOT at a rate of 75%
}
deployment_circuit_breaker {
enable = false
rollback = false
}
deployment_controller {
type = "ECS"
}
load_balancer {
container_name = "keycloak"
container_port = aws_alb_target_group.keycloak.port
target_group_arn = aws_alb_target_group.keycloak.arn
}
network_configuration {
assign_public_ip = true
security_groups = [
aws_security_group.keycloak.id
]
subnets = _cluster_subnets_
}
timeouts {}
lifecycle {
ignore_changes = [desired_count]
}
}
resource "aws_ecs_task_definition" "keycloak" {
container_definitions = jsonencode(
[
{
cpu = 0
command = ["start --optimized"]
disableNetworking = false
portMappings = [
{
containerPort = aws_alb_target_group.auth.port
hostPort = aws_alb_target_group.auth.port
protocol = "tcp"
}
]
environment = [
{
name = "KC_DB_URL_DATABASE"
value = _KC_DB_URL_DATABASE_
},
{
name = "KC_DB_URL_HOST"
value = _KC_DB_URL_HOST_
},
{
name = "KC_DB_URL"
value = _KC_DB_URL_
},
{
name = "KC_DB_PASSWORD"
value = _KC_DB_PASSWORD_
},
{
name = "KC_DB_USERNAME"
value = _KC_DB_USERNAME_
},
{
name = "JAVA_OPTS"
value = _JAVA_OPTS_
},
{
name = "KC_CACHE"
value = "ispn"
},
{
name = "KC_HOSTNAME"
value = _keycloak_fqdn_
},
{
name = "KC_HOSTNAME_STRICT_BACKCHANNEL"
value = "true"
},
{
name = "KC_CACHE_CONFIG_FILE"
value = "cache-ispn-jdbc-ping.xml"
},
]
essential = true
healthCheck = {
command = [
"CMD-SHELL",
"curl -f http://localhost:${_keycloak_port_}/auth/ || exit 1",
]
interval = 30
retries = 3
timeout = 5
}
image = _ecr_repo_url_
stopTimeout = 120
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = aws_cloudwatch_log_group.keycloak.name
awslogs-region = "ap-northeast-1"
awslogs-stream-prefix = "ecs"
}
}
mountPoints = []
name = "keycloak"
volumesFrom = []
},
]
)
cpu = _keycloak_cpu_
task_role_arn = aws_iam_role.ecs_task_role.arn
execution_role_arn = aws_iam_role.execution_role.arn
family = _service_name_
memory = _keycloak_memory_
network_mode = "awsvpc"
requires_compatibilities = [
"FARGATE",
]
}
resource "aws_alb_target_group" "keycloak" {
deregistration_delay = "115"
load_balancing_algorithm_type = "round_robin"
name = _clustername_
port = _keycloak_port_
protocol = "HTTP"
protocol_version = "HTTP1"
slow_start = 0
target_type = "ip"
vpc_id = _cluster_vpc_id_
health_check {
...
}
stickiness {
cookie_duration = 86400
enabled = false
type = "lb_cookie"
}
}
resource "aws_iam_role" "execution_role" {
name = "ecs-execution-role"
managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"]
assume_role_policy = jsonencode({
"Version" : "2008-10-17",
"Statement" : [
{
"Sid" : "",
"Effect" : "Allow",
"Principal" : {
"Service" : "ecs-tasks.amazonaws.com"
},
"Action" : "sts:AssumeRole"
}
]
})
}
resource "aws_iam_role" "ecs_task_role" {
name = "ecs-task-role"
assume_role_policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Sid" : "",
"Effect" : "Allow",
"Principal" : {
"Service" : "ecs-tasks.amazonaws.com"
},
"Action" : "sts:AssumeRole"
}
]
})
inline_policy {
name = "SessionManagerRoleForECS"
policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Action" : [
"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel"
],
"Resource" : "*"
}
]
})
}
}
resource "aws_cloudwatch_log_group" "keycloak" {
name = "/ecs/${_keycloak_service_name_}"
retention_in_days = 180
}
Scaling policies can be realized by defining them as follows.
resource "aws_appautoscaling_target" "keycloak" {
service_namespace = "ecs"
resource_id = "service/${aws_ecs_cluster.keycloak.name}/${aws_ecs_service.keycloak.name}"
scalable_dimension = "ecs:service:DesiredCount"
min_capacity = _keycloak_desired_count_min_
max_capacity = _keycloak_desired_count_max_
lifecycle {
ignore_changes = [min_capacity, max_capacity]
}
}
resource "aws_appautoscaling_policy" "keycloak_scale_out" {
name = "keycloak_scale_out"
policy_type = "StepScaling"
service_namespace = aws_appautoscaling_target.keycloak.service_namespace
resource_id = aws_appautoscaling_target.keycloak.id
scalable_dimension = aws_appautoscaling_target.keycloak.scalable_dimension
step_scaling_policy_configuration {
adjustment_type = "ChangeInCapacity"
cooldown = 30
metric_aggregation_type = "Maximum"
step_adjustment {
metric_interval_lower_bound = 0
metric_interval_upper_bound = local.KeycloakCpuHightThreshold
scaling_adjustment = _keycloak_desired_count_scaleout_policy_
}
step_adjustment {
metric_interval_lower_bound = local.KeycloakCpuHightThreshold
scaling_adjustment = _keycloak_desired_count_scaleout_policy_ * 2
}
}
}
resource "aws_appautoscaling_policy" "keycloak_scale_in" {
name = "keycloak_scale_in"
policy_type = "StepScaling"
service_namespace = aws_appautoscaling_target.keycloak.service_namespace
resource_id = aws_appautoscaling_target.keycloak.id
scalable_dimension = aws_appautoscaling_target.keycloak.scalable_dimension
step_scaling_policy_configuration {
adjustment_type = "ChangeInCapacity"
cooldown = 60
metric_aggregation_type = "Average"
step_adjustment {
metric_interval_upper_bound = 0
scaling_adjustment = -1
}
}
}
When scaling in advance by time zone, it can be realized by defining the following.
resource "aws_appautoscaling_scheduled_action" "keycloak_time_scaling_start" {
name = "keycloak_time_caling_start"
service_namespace = aws_appautoscaling_target.keycloak.service_namespace
resource_id = aws_appautoscaling_target.keycloak.id
scalable_dimension = aws_appautoscaling_target.keycloak.scalable_dimension
schedule = _keycloak_desired_count_time_scaling_start_
scalable_target_action {
min_capacity = _keycloak_desired_count_min_ * _keycloak_desired_count_time_scaling_scale
max_capacity = _keycloak_desired_count_max_ * _keycloak_desired_count_time_scaling_scale
}
}
resource "aws_appautoscaling_scheduled_action" "keycloak_time_scaling_stop" {
name = "keycloak_time_caling_stop"
service_namespace = aws_appautoscaling_target.keycloak.service_namespace
resource_id = aws_appautoscaling_target.keycloak.id
scalable_dimension = aws_appautoscaling_target.keycloak.scalable_dimension
schedule = _keycloak_desired_count_time_scaling_stop_
scalable_target_action {
min_capacity = _keycloak_desired_count_min_
max_capacity = _keycloak_desired_count_max_
}
depends_on = [aws_appautoscaling_scheduled_action.keycloak_time_scaling_start]
}
That's all you can control with Keycloak with Fargate.
Top comments (0)