Bartłomiej Kuciński
Technical Leader | Expert Software Engineer

Bartłomiej is a Software Developer and TL in GrapeUp with 5+ years of professional experience. Experienced in complex application development in various environments including overall app architecture design. He cooperates closely with customers providing inestimable knowledge-sharing support.

Blog

Read articles

Software development

Running Langfuse v3 on ECS Fargate: architecture, hardening and operations

Introduction

Langfuse is an open-source LLM observability platform: traces, prompts, evaluations, datasets. Traces hold prompts, completions and often personal data, so many teams self-host it.

Langfuse v2 was one container and a Postgres database. v3 adds a worker, ClickHouse, Redis and S3 (architecture comparison), and v4 keeps the same components. This guide shows how to run all of it on AWS with ECS Fargate and managed services, without Kubernetes. The setup meets strict requirements:

•     no public endpoints apart from a CDN,

•     read-only root filesystems,

•     customer-managed KMS keys,

•     images only from the account's own registry, behind a scan gate.

The guide describes a single environment; code is Terraform plus some shell, shortened to what matters.

Everything here reflects Langfuse 3.224.2 and ClickHouse 26.5.7, as of October 2026. Langfuse v4 changes the ClickHouse data model and will get a separate article.

1. Architecture

1.1 Components

‍

Langfuse v3 on AWS: CloudFront, internal load balancers, ECS Fargate and managed data stores

1.2 How data flows

Ingestion. An application posts a batch to /api/public/ingestion. Web writes it to the S3 events bucket and queues only a reference in Redis; the worker reads the events from S3 and writes them to ClickHouse. A load spike fills a queue instead of hitting ClickHouse.

Reading. The UI and read APIs query ClickHouse for traces, observations and scores, and Postgres for the rest.

1.3 Network and access

Everything runs in private subnets. Outbound traffic leaves through a NAT gateway and AWS Network Firewall with a domain allow-list; Langfuse needs it for LLM APIs, the identity provider if you use SSO, and AWS APIs without a VPC endpoint.

CloudFront is the only public endpoint. Both load balancers behind it are internal, which gives two ways in:

•     users through CloudFront with AWS WAF, connected to an internal ALB by a VPC origin,

•     applications in the VPC through a second internal ALB.

Web and worker reach ClickHouse through a private Cloud Map DNS name.

2. Setting up the components

2.1 Shared secrets

Web and worker must share ENCRYPTION_KEY (encrypts secrets stored in Postgres, such as LLM connection keys; 256 bits as 64 hex characters), SALT (salts hashed API keys) and NEXTAUTH_SECRET (signs sessions).

Generate them once (ENCRYPTION_KEY with random_id of 32 bytes, rendered as .hex) and store all three in one Secrets Manager secret. Both task definitions read from it:

secrets = [
  { name = "ENCRYPTION_KEY",  valueFrom = "${var.langfuse_app_secret_arn}:encryption_key::" },
  { name = "SALT",            valueFrom = "${var.langfuse_app_secret_arn}:salt::" },
  { name = "NEXTAUTH_SECRET", valueFrom = "${var.langfuse_app_secret_arn}:nextauth_secret::" },
]

:key:: selects one field of a JSON secret. Unlike environment, secrets keeps values out of the task definition, which anyone with ecs:DescribeTaskDefinition can read.

2.2 Postgres

Langfuse gets its own database and user in an Aurora cluster that is reachable only from the VPC's private subnets, so Terraform cannot connect to it directly. A one-off Lambda inside the VPC, invoked by Terraform through aws_lambda_invocation, runs as the master user:

CREATE USER langfuse WITH ENCRYPTED PASSWORD '...';
GRANT langfuse TO root;
CREATE DATABASE langfuse WITH ENCODING='UTF8' OWNER=langfuse;

Do not pass a DATABASE_URL. The Langfuse images build it from four variables, so the password stays in Secrets Manager:

environment = [
  { name = "DATABASE_HOST", value = var.aurora_endpoint },
  { name = "DATABASE_NAME", value = "langfuse" },
]
secrets = [
  { name = "DATABASE_USERNAME", valueFrom = "${var.db_secret_arn}:username::" },
  { name = "DATABASE_PASSWORD", valueFrom = "${var.db_secret_arn}:password::" },
]

A scheduled Lambda rotates the password, then redeploys web and worker, because ECS resolves secrets only at task start.

2.3 Redis (Valkey)

Langfuse needs Redis 7+ or Valkey. We use ElastiCache Serverless (Valkey 7) with a usage limit and a password-authenticated ACL user. Serverless caches always run in cluster mode with TLS, so:

environment = [
  { name = "REDIS_CLUSTER_ENABLED", value = "true" },
  { name = "REDIS_CLUSTER_NODES",   value = "${aws_elasticache_serverless_cache.langfuse.endpoint[0].address}:6379" },
  { name = "REDIS_TLS_ENABLED",     value = "true" },
  { name = "REDIS_USERNAME",        value = "langfuse-user" },
]
secrets = [
  { name = "REDIS_AUTH", valueFrom = "${var.redis_secret_arn}:password::" },
]

‍

This is a deliberate deviation. The Langfuse documentation requires maxmemory-policy=noeviction; on ElastiCache Serverless it is fixed at volatile-lru. Once nothing evictable (keys with a TTL) is left at the usage limit, writes fail and Langfuse cannot enqueue ingestion jobs. An alarm on BytesUsedForCache well below the limit buys time but is not the same guarantee. If you need it, use a node-based cluster with a parameter group that sets noeviction.

2.4 ClickHouse

ClickHouse runs as a single Fargate task with its data on an encrypted EFS file system (throughput_mode = "elastic"). A replicated setup is possible but needs more components, such as ClickHouse Keeper and storage per replica; section 3.6 outlines it.

Storage

Mount the file system through an access point that presents every file as the clickhouse user (uid/gid 101) of the official image:

resource "aws_efs_access_point" "clickhouse" {
  file_system_id = aws_efs_file_system.clickhouse.id
 
  root_directory {
    path = "/clickhouse"
    creation_info {
      owner_uid   = 101
      owner_gid   = 101
      permissions = "0755"
    }
  }
 
  posix_user {
    uid = 101
    gid = 101
  }
}

A file system policy allows mounting only by the ClickHouse task role, through this access point, over TLS; NFS (2049) is open only to the ClickHouse task.

Image

Build a thin image on top of the official one. Section 3.3 shows how the pipeline resolves BASE_REF.

# clickhouse/clickhouse-server@sha256:...
ARG BASE_REF
FROM ${BASE_REF}
 
# patch OS packages, keep the pinned ClickHouse version
USER root
RUN set -eux; \
    apt-get update; \
    apt-mark hold clickhouse-server clickhouse-client clickhouse-common-static || true; \
    apt-get upgrade -y; \
    rm -rf /var/lib/apt/lists/*
 
COPY config.d/paths.xml /etc/clickhouse-server/config.d/paths.xml
# the access point already presents files as uid 101
ENV CLICKHOUSE_DO_NOT_CHOWN=1
COPY docker-entrypoint.sh /docker-entrypoint.sh
ENTRYPOINT ["/docker-entrypoint.sh"]

config.d/paths.xml sets the data, temp, PID and log paths below and enables console logging, so the logs reach CloudWatch. With a read-only root filesystem, each of these paths is a volume:

‍

Because users.d is a volume, files copied into it at build time are hidden at runtime. User-profile settings therefore have to be written at startup, by a wrapper around the official entrypoint. ClickHouse 26.5.x with Langfuse 3.224.2 needs one such setting. Without it, the lazy-materialization optimizer fails on Langfuse's scores query with Code: 10 NOT_FOUND_COLUMN_IN_BLOCK:

#!/usr/bin/env bash
set -euo pipefail
 
mkdir -p /var/tmp/clickhouse /var/run/clickhouse
chown 101:101 /var/tmp/clickhouse /var/run/clickhouse
 
mkdir -p /etc/clickhouse-server/users.d
cat > /etc/clickhouse-server/users.d/zzz-langfuse.xml <<'EOF'
<clickhouse>
    <profiles>
        <default>
            <query_plan_optimize_lazy_materialization>0</query_plan_optimize_lazy_materialization>
        </default>
    </profiles>
</clickhouse>
EOF
 
exec /entrypoint.sh

Task and service

volume {
  name = "clickhouse-data"
  efs_volume_configuration {
    file_system_id     = aws_efs_file_system.clickhouse.id
    transit_encryption = "ENABLED"
    authorization_config {
      access_point_id = aws_efs_access_point.clickhouse.id
      iam             = "ENABLED"
    }
  }
}
 
container_definitions = jsonencode([{
  name                   = "clickhouse"
  image                  = "${aws_ecr_repository.clickhouse.repository_url}:latest"
  readonlyRootFilesystem = true
  portMappings = [
    { containerPort = 8123 },   # HTTP: Langfuse
    { containerPort = 9000 },   # native protocol: migrations
  ]
  # mountPoints: the volumes from the table above
  ulimits = [{ name = "nofile", softLimit = 262144, hardLimit = 262144 }]   # container-level field, not part of linuxParameters
  healthCheck = { command = ["CMD-SHELL", "wget -qO- 127.0.0.1:8123/ping || exit 1"] }
}])

The service must never run two tasks against the same data:

resource "aws_ecs_service" "clickhouse" {
  desired_count                      = 1
  deployment_minimum_healthy_percent = 0     # stop the old task first
  deployment_maximum_percent         = 100
  availability_zone_rebalancing      = "DISABLED"
  # + Cloud Map registration: clickhouse-internal.<private zone>
}

Every ClickHouse deploy therefore has a shortdowntime. On the Langfuse side, set CLICKHOUSE_CLUSTER_ENABLED=false, because a single node cannot run ON CLUSTER statements:

environment = [
  { name = "CLICKHOUSE_URL",             value = "http://clickhouse-internal.${var.private_zone}:8123" },
  { name = "CLICKHOUSE_MIGRATION_URL",   value = "clickhouse://clickhouse-internal.${var.private_zone}:9000" },
  { name = "CLICKHOUSE_CLUSTER_ENABLED", value = "false" },
]
# + CLICKHOUSE_USER, CLICKHOUSE_PASSWORD, CLICKHOUSE_DB from Secrets Manager

2.5 S3

Langfuse uses three buckets: events (every ingested batch lands here first), media and batch exports. Each has versioning, blocked public access and SSE-KMS with a customer-managed key. The bucket policy denies non-TLS requests and any PutObject that does not ask for SSE-KMS with our key:

statement {
  sid       = "DenyPutWithWrongKmsKey"
  effect    = "Deny"
  actions   = ["s3:PutObject"]
  resources = ["${each.value.arn}/*"]
  principals {
    type        = "AWS"
    identifiers = ["*"]
  }
  condition {
    test     = "StringNotEquals"
    variable = "s3:x-amz-server-side-encryption-aws-kms-key-id"
    values   = [aws_kms_key.langfuse_s3.arn]
  }
}

With this policy in place, Langfuse has to send the encryption headers, or every upload is denied. Set the following on both containers:

{ name = "LANGFUSE_S3_EVENT_UPLOAD_BUCKET",         value = local.buckets.events },
{ name = "LANGFUSE_S3_EVENT_UPLOAD_REGION",         value = var.aws_region },
{ name = "LANGFUSE_S3_EVENT_UPLOAD_SSE",            value = "aws:kms" },
{ name = "LANGFUSE_S3_EVENT_UPLOAD_SSE_KMS_KEY_ID", value = aws_kms_key.langfuse_s3.arn },
# the same four for LANGFUSE_S3_MEDIA_UPLOAD_* and LANGFUSE_S3_BATCH_EXPORT_*,
# plus LANGFUSE_S3_BATCH_EXPORT_ENABLED=true

No access keys are needed; the task role needs s3:GetObject, s3:PutObject and s3:ListBucket on the buckets, and kms:GenerateDataKey and kms:Decrypt on the key.

2.6 Web and worker on ECS Fargate

Web

The web task has an init container, Langfuse on port 3000 and an nginx sidecar for TLS on port 8453. With a read-only root filesystem, the Next.js cache (/app/web/.next/cache) needs a volume. Fargate creates that volume owned by root, while Langfuse runs as nextjs, so an init container fixes the ownership first:

{
  name                   = "fix-next-cache-perms"
  image                  = local.langfuse_web_image   # same image, so nextjs:nodejs exist
  essential              = false
  user                   = "0"
  readonlyRootFilesystem = true
  entryPoint             = ["/bin/sh", "-c"]          # skip the Langfuse entrypoint
  command                = ["chown nextjs:nodejs /app/web/.next/cache"]
  mountPoints            = [{ sourceVolume = "next-cache", containerPath = "/app/web/.next/cache" }]
},
{
  name                   = "langfuse-web"
  image                  = local.langfuse_web_image
  readonlyRootFilesystem = true
  dependsOn              = [{ containerName = "fix-next-cache-perms", condition = "SUCCESS" }]
  mountPoints            = [{ sourceVolume = "next-cache", containerPath = "/app/web/.next/cache" }]
  environment = [
    { name = "HOSTNAME",     value = "0.0.0.0" },
    { name = "NEXTAUTH_URL", value = "https://${var.langfuse_domain}" },
  ]
  healthCheck = {
    command = ["CMD-SHELL", "wget -q -O /dev/null localhost:3000/api/public/health || exit 1"]
  }
}

The entryPoint override is required: the image's entrypoint exits without database variables, and otherwise runs migrations first.

Langfuse serves plain HTTP and does not terminate TLS itself, so an nginx sidecar does. It generates a self-signed certificate at startup, listens on 8453 with TLS and proxies to http://127.0.0.1:3000 with proxy_read_timeout 25m. An ALB does not validate target certificates, so traffic stays encrypted into the task without any certificate management. The service attaches the nginx container to two target groups (section 2.7) and uses the deployment circuit breaker with rollback. Target tracking scales it between 2 and 4 tasks at 50% average CPU, Langfuse's saturation point (section 3.6).

Worker

The worker is one container with the same store settings as web, plus LANGFUSE_ENABLE_BACKGROUND_MIGRATIONS=true and a health check on http://localhost:3030/api/health. It needs no load balancer or inbound rules.

2.7 Access

Users

CloudFront reaches the internal ALB through an aws_cloudfront_vpc_origin (origin_protocol_policy = "https-only", TLS 1.2), referenced from the distribution's vpc_origin_config. The distribution uses the managed policies Managed-CachingDisabled and Managed-AllViewerExceptHostHeader and an ACM certificate from us-east-1. The ALB is internal = true, and its security group accepts port 443 only from CloudFront:

data "aws_ec2_managed_prefix_list" "cloudfront_origin_facing" {
  name = "com.amazonaws.global.cloudfront.origin-facing"
}
 
resource "aws_security_group_rule" "alb_from_cloudfront" {
  security_group_id = aws_security_group.alb_external.id
  type              = "ingress"
  protocol          = "tcp"
  from_port         = 443
  to_port           = 443
  prefix_list_ids   = [data.aws_ec2_managed_prefix_list.cloudfront_origin_facing.id]
}

The WAF web ACL (scope CLOUDFRONT) has a per-IP rate limit and the AWS managed rule groups. The OAuth callback carries a long query string and base64 payloads, which some size-restriction and cross-site-scripting rules of the common rule set flag. Switch only those rules to count with rule_action_override, and keep them effective elsewhere: add a rule that blocks on their labels for every path except /api/auth/callback/*.

SSO is optional: Langfuse also supports email and password, and OAuth/OIDC providers such as Google, GitHub, Okta or Microsoft Entra ID. This setup disables passwords and uses SSO, here with Entra ID (redirect URI https://<langfuse domain>/api/auth/callback/azure-ad):

environment = [
  { name = "NEXTAUTH_URL",                   value = "https://${var.langfuse_domain}" },
  { name = "AUTH_DISABLE_USERNAME_PASSWORD", value = "true" },
]
# + AUTH_AZURE_AD_CLIENT_ID, AUTH_AZURE_AD_CLIENT_SECRET, AUTH_AZURE_AD_TENANT_ID from Secrets Manager

Applications

Applications that send traces or fetch prompts use a second internal ALB with an ACM certificate for a name in the private zone. Its security group accepts port 443 only from the application subnets. Machine traffic stays inside the VPC and bypasses WAF rules tuned for browsers, which would otherwise block large ingestion batches.

Webhooks

As SSRF protection, Langfuse refuses private webhook targets unless allow-listed. List only the receiving service's subnets:

{ name = "LANGFUSE_WEBHOOK_WHITELISTED_IP_SEGMENTS", value = join(",", var.webhook_target_subnet_cidrs) }

Give the target a certificate the worker trusts, such as an internal ALB with ACM. Do not set NODE_TLS_REJECT_UNAUTHORIZED=0: it disables certificate checks for every TLS connection of the worker.

3. Operating it

3.1 Images: mirror, scan, deploy

ECS pulls only from ECR in its own account, with IMMUTABLE tags and scan on push. A weekly CI pipeline authenticates with its OIDC token via sts:AssumeRoleWithWebIdentity, for a role that trusts only the pipeline project's main branch.

It separates publishing from approval: scheduled jobs push langfuse/langfuse:3 and langfuse/langfuse-worker:3 under a timestamp and their exact version and gate them on the scan; the manual deploy job (section 3.2) moves latest and updates the service:

# --- scheduled jobs ---
DATE="$(date "+%Y%m%d%H%M%S")"
VERSION="$(docker image inspect langfuse/langfuse:3 \
  --format '{{index .Config.Labels "org.opencontainers.image.version"}}')"   # e.g. 3.224.2
docker tag langfuse/langfuse:3 "$REPOSITORY:$DATE"    && docker push "$REPOSITORY:$DATE"
docker tag langfuse/langfuse:3 "$REPOSITORY:$VERSION" && docker push "$REPOSITORY:$VERSION"
 
aws ecr wait image-scan-complete --repository-name "$NAME" --image-id imageTag="$DATE"
CRITICAL=$(aws ecr describe-image-scan-findings --repository-name "$NAME" --image-id imageTag="$DATE" \
  --query "imageScanFindings.findingSeverityCounts.CRITICAL" --output text | head -n 1)
if [ -n "$CRITICAL" ] && [ "$CRITICAL" -gt 0 ]; then echo "CRITICAL: $CRITICAL"; exit 1; fi
 
# --- manual deploy job ---
# immutable tags: delete latest, then push it again
aws ecr batch-delete-image --repository-name "$NAME" --image-ids imageTag=latest || true
docker tag langfuse/langfuse:3 "$REPOSITORY:latest" && docker push "$REPOSITORY:latest"
# REVISION: describe-task-definition -> jq (image = $REPOSITORY:$DATE) -> register-task-definition
aws ecs update-service --cluster "$CLUSTER" --service "$NAME" --task-definition "$NAME:$REVISION"
aws ecs wait services-stable --cluster "$CLUSTER" --services "$NAME"

With enhanced scanning, the CI role also needs inspector2:ListFindings and inspector2:ListCoverage. Terraform owns the same task definition and references :latest. Since latest moves only in the manual job, a later terraform apply cannot roll out an unapproved image or its migrations.

With continuous scanning, Inspector re-evaluates deployed images and sends two EventBridge event types (source aws.inspector2):

•     Inspector2 Scan, sent once after the initial scan,

•     Inspector2 Finding, sent whenever a finding is created, updated or closed, including new CVEs in running images. Match this one:

event_pattern = jsonencode({
  source        = ["aws.inspector2"],
  "detail-type" = ["Inspector2 Finding"],
  detail = {
    severity  = ["CRITICAL"],
    status    = ["ACTIVE"],
    resources = { type = ["AWS_ECR_CONTAINER_IMAGE"] }
  }
})

If you automate deleting images instead, note that ECS cannot start new tasks from a deleted image.

3.2 Upgrading Langfuse

A release can change the Postgres schema, the ClickHouse schema, or both. In 3.224.2:

‍

Deploying an older image does not revert a migration, so the deploy job is when: manual while the scheduled jobs only publish and scan.

Right before it, take an on-demand recovery point of the ClickHouse file system. AWS Backup copies EFS file by file without freezing it, and AWS recommends pausing writers for a consistent copy, so stop ClickHouse meanwhile. Web keeps accepting ingestion into S3 and the queue; check that the queue drains afterwards.

aws ecs update-service --cluster "$CLUSTER" --service clickhouse --desired-count 0
aws ecs wait services-stable --cluster "$CLUSTER" --services clickhouse
 
JOB=$(aws backup start-backup-job \
  --backup-vault-name "$BACKUP_VAULT" \
  --resource-arn "arn:aws:elasticfilesystem:${AWS_REGION}:${ACCT}:file-system/${FS_ID}" \
  --iam-role-arn "$BACKUP_ROLE_ARN" \
  --query BackupJobId --output text)
 
DEADLINE=$(( $(date +%s) + 3600 ))
while true; do
  STATE=$(aws backup describe-backup-job --backup-job-id "$JOB" --query State --output text)
  case "$STATE" in
    COMPLETED)              break ;;
    FAILED|ABORTED|EXPIRED) echo "backup $JOB ended in $STATE"; exit 1 ;;
  esac
  [ "$(date +%s)" -lt "$DEADLINE" ] || { echo "backup $JOB timed out in $STATE"; exit 1; }
  sleep 30
done
 
aws ecs update-service --cluster "$CLUSTER" --service clickhouse --desired-count 1
aws ecs wait services-stable --cluster "$CLUSTER" --services clickhouse

Then deploy web, which runs the migrations; Aurora point-in-time recovery covers Postgres.

To roll back, scale web and worker to zero, restore ClickHouse (section 3.4) and Aurora to the same moment, since they reference each other by ID, and deploy the previous image. Rehearse this before you need it.

3.3 Upgrading ClickHouse

ClickHouse has its own weekly pipeline that follows a release line (CLICKHOUSE_RELEASE_LINE: "26.5"): it finds the highest stable patch, builds the image from section 2.4 on its digest, pushes it as <version>-<timestamp> through the same scan gate, and deploys it in a manual job after the snapshot from section 3.2.

TAGS="$(curl -fsS "https://hub.docker.com/v2/repositories/clickhouse/clickhouse-server/tags?page_size=100&name=${CLICKHOUSE_RELEASE_LINE}.")"
CLICKHOUSE_VERSION="$(echo "$TAGS" | jq -r --arg line "$CLICKHOUSE_RELEASE_LINE" '
  [ .results[].name | select(test("^[0-9]+\\.[0-9]+\\.[0-9]+$")) | select(startswith($line + "."))
    | {name: ., parts: (split(".") | map(tonumber))} ] | sort_by(.parts) | last.name // empty')"
 
docker pull "clickhouse/clickhouse-server:${CLICKHOUSE_VERSION}"
BASE_REF="$(docker inspect --format='{{index .RepoDigests 0}}' "clickhouse/clickhouse-server:${CLICKHOUSE_VERSION}")"
docker build --build-arg BASE_REF="$BASE_REF" -t "$REPOSITORY:${CLICKHOUSE_VERSION}-${DATE}" .

A newer release line only triggers a notification, and changing the line is a manual decision, because the dependency works both ways: a Langfuse release can require a newer ClickHouse (v4 needs at least 25.12, with 26.4 recommended), and a newer ClickHouse can break a Langfuse query, as the users.d override in section 2.4 shows.

3.4 Backups

‍

The backup plan has a copy_action to a vault in the second region; both vaults use customer-managed KMS keys.

The daily backup runs while ClickHouse writes, so it may be inconsistent; the pre-deploy backup is the reliable restore point, or use ClickHouse's own BACKUP ... TO S3(...).

To restore ClickHouse:

1.    Restore the recovery point to a new file system. AWS Backup never restores in place: the data lands in /aws-backup-restore_<datetime>/clickhouse.

2.    Either move that directory back to /clickhouse with a one-off task, or create an access point whose root_directory.path points at it.

3.    Point the task definition's EFS volume at the new file system and access point, add mount targets and the file system policy, and redeploy.

3.5 Monitoring

3.6 Scaling

Where this setup stands. It handles a modest load: around 200 traces a day with about 40 observations each, LLM-as-a-judge evaluations on top, and about 80 sessions. Web stays below 10% CPU and the worker below 10% of its memory, so the worker could start at Langfuse's minimum of 2 CPU and 4 GiB. ClickHouse sits at a few percent, apart from short daily peaks of up to about 90% CPU and 80% memory during large data transfers. Size ClickHouse for those peaks, not for the average, and keep the memory alarm (section 3.5) above them.

When load grows, the Langfuse scaling guide recommends:

Worker: scale out on queue depth, not CPU. With ENABLE_AWS_CLOUDWATCH_METRIC_PUBLISHING=true (task role: cloudwatch:PutMetricData), Langfuse publishes langfuse.queue.ingestion.depth.type_waiting to CloudWatch; put a scaling policy on it. Each task runs up to LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY (default 20) and LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY (default 25) jobs, adding load on ClickHouse and Redis.

Web: scale out, then split. The documented minimum is 2 CPU and 4 GiB per container, saturated above 50% CPU. Under heavy ingestion, a second web service for /api/public/ingestion*, /api/public/media* and /api/public/otel* keeps the UI responsive; here, a path-pattern listener rule on the internal ALB.

Redis: shard the queues. Above 90% Redis CPU, set LANGFUSE_INGESTION_QUEUE_SHARD_COUNT and LANGFUSE_TRACE_UPSERT_QUEUE_SHARD_COUNT to two to three times the Redis shard count and lower the per-worker concurrency in proportion. Never reduce them later. ElastiCache Serverless shows one virtual node, so watch ECPU instead.

ClickHouse: vertical first (at least 16 GiB for larger deployments; Fargate allows 16 vCPU and 120 GB per task). Beyond that:

•     Replicas add availability, not write capacity. Langfuse supports only a single shard and recommends at least 3 replicas with 3 ClickHouse Keeper nodes. Moving there means CLICKHOUSE_CLUSTER_ENABLED=true, storage per replica and tables recreated with replicated engines: a data migration, not a configuration change.

•     For larger deployments, Langfuse recommends ClickHouse Cloud or BYOC, which separate compute from storage; CLICKHOUSE_READ_ONLY_URL can route reads to separate compute.

4. Summary

4.1 What we built

Langfuse v3 runs on AWS without Kubernetes, behind one public endpoint, with customer-managed encryption, read-only containers and approved, scanned images. Most of the work is in details the documentation covers briefly:

•     secrets that web and worker must share,

•     Valkey in cluster mode with TLS,

•     ClickHouse with a read-only root filesystem and a users.d setting for 26.5.x,

•     S3 encryption that Langfuse has to request explicitly,

•     an init container for the Next.js cache.

4.2 What you take on

Self-hosting means owning weekly image updates, irreversible migrations behind a manual gate and a fresh snapshot, ClickHouse upgrades, backups of two stores restored together, and scaling.

4.3 Alternatives

‍

Each self-hosted variant can add the commercial Enterprise license, which brings SSO enforcement, project-level RBAC, audit logs and support.

This guide fits teams already on ECS who do not want Kubernetes for one tool.

A follow-up article will cover connecting applications to Langfuse and managing its data over time, as well as the move to Langfuse v4.

Thank you for taking the time to read.

‍

Read more
Software development
AI

Model Context Protocol in Java and Python: from a single server to an MCP proxy

Introduction

Whatever exactly an AI agent is, the part this article cares about is a model calling tools - to read data, such as a search index, or to act, such as opening a pull request. Wiring one tool in is easy. Connecting an agent to the five systems a company actually runs - each built by a different team, some in Java, some in Python - is where things get interesting. The Model Context Protocol (MCP) promises to standardize that: one way for an agent to discover and call tools, no matter who built them.

Much of the AI tooling lives in Python, and so does one of the official MCP SDKs. On paper,Java is ready too: there is an official MCP Java SDK, and Spring AI wraps it in starters and annotations. But how much of the protocol do these libraries actually give you, what do they quietly leave out, and what happens when one agent talks to several servers at once?

To find out, I built the same MCP server three times - on the bare Java SDK, on Spring AI and in Python - connected to them with the client libraries of both languages, and recorded what happened. Below I walk through the setup, show the results that surprised me - Spring AI, for example, by default renames tools depending on connection order - and explain what an MCP proxy is and what it takes over.

One disclaimer upfront: the MCP ecosystem moves fast. Everything here reflects the state as of October 2026 - specification revision 2026-07-28, MCP Java SDK 2.0.1 and MCP Python SDK 2.3.0. In section 5, I show how to check the current state yourself.

1. What is MCP, and why should you care?

1.1 What is MCP?

MCP defines three roles. The host is the application with the model in it. The client lives inside the host and holds a connection to exactly one server. The server exposes the capabilities of one system. Servers offer tools (actions the model calls), resources (data for the context) and prompts (templates for the user); most integrations, and this article, are about tools.

Underneath, it is plain JSON-RPC 2.0 over one of two transports: stdio, where the host starts the server as a child process, or Streamable HTTP for remote servers. The older HTTP+SSE transport is deprecated.

Why care? Because MCP is designed to turn M agents × N systems into M + N integrations.

1.2 What actually goes over the wire

The quickest wayto understand MCP is to look at the frames. Here is one session with one of my servers, trimmed:

--> initialize
     { "protocolVersion": "2025-11-25", "capabilities": {},
       "clientInfo": { "name": "nameless-hero", "version": "1.0.0" } }
 <-- result
     { "protocolVersion": "2025-11-25",
       "capabilities": { "logging": {}, "tools": { "listChanged": true } },
       "serverInfo": { "name": "fire-mages", "version": "1.0.0" } }
 --> notifications/initialized
 --> tools/list
 <-- result   { "tools": [ { "name": "askMages", "inputSchema": {...}, "outputSchema": {...} }, ... ] }
 --> tools/call
     { "name": "askMages", "arguments": { "question": "Who is the Sleeper?", "maxAnswers": 1 } }
 <-- result
     { "content": [ { "type": "text", "text": "Fire Circle answers: [Sleeper, confidence 0.95] ..." } ],
       "isError": false,
       "structuredContent": { "mages": "Fire Circle",
                              "answers": [ { "topic": "Sleeper", "confidence": 0.95, ... } ] } }

Two things are worth noticing. The handshake agrees on a protocol version, here 2025-11-25. And a tool result comes in two forms: content for the model and structuredContent for code. Errors in tool execution come back the same way, with isError: true, so the model can read them.

To record them, I wrap the Java client transport, which has exactly two places messages passthrough: sendMessage (towards the server) and the handler given to connect (towards the client). Wrapping both gives you a tap on every JSON-RPC message, and it works the same for stdio and HTTP:

public final class WireTap implements McpClientTransport {
 
    private final McpClientTransport delegate;
     private final String channel;
     private final WireLog log;   // prints a message as JSON
 
    @Override
     public Mono<Void> connect(Function<Mono<JSONRPCMessage>, Mono<JSONRPCMessage>> handler) {
         return delegate.connect(inbound -> handler.apply(
                 inbound.doOnNext(message -> log.record(INBOUND, channel, message))));
     }
 
    @Override
     public Mono<Void> sendMessage(JSONRPCMessage message) {
         log.record(OUTBOUND, channel, message);
         return delegate.sendMessage(message);
     }
 
    // closeGracefully(), close(), unmarshalFrom(), protocolVersions()
     // and setExceptionHandler() simply delegate
 }

Keep 2025-11-25 in mind. It will matterl ater.

1.3 What is an MCP proxy?

The term has no single definition. The MCP specification knows only hosts, clients and servers; its latest revision prepares for intermediaries - Mcp-Method and Mcp-Name headers let them route without parsing the body - but does not name them. In practice, “MCP proxy” is used for three different things:

·      a transport bridge - stdio on one side, HTTP on the other, one server, no logic,

·      an aggregator - many servers behind one endpoint, with name spaces and policy; several vendors call this a gateway,

·      an interceptor - something in the path that inspectsor modifies traffic.

In this article, an MCP proxy means the second kind: an MCP server towards the agent and an MCP client towards the real servers. The agent sees one ordinary server; the servers behind it see one ordinary client.

In this setup, the proxy owns routing and cross-cutting policy, and domain behaviour stays in the upstream servers. Servers can enforce permissions and keep audits too; here the split is:

2. Setup overview

2.1 Setup description

To keep the examples readable, I borrowed the setting from my previous article on multi-layered guardrailing: the world of Gothic. My three servers are three sources of knowledge in the Colony, and each answers questions differently:

·      the Fire Circle - cheap and fast, but narrow and dogmatic; it answers every question with full confidence, including the ones it gets wrong,

·      the Water Mages - broader and more experimental; they offer hypotheses and mark them with lower confidence,

·      Xardas - the leader of the twelve mages who raised the Barrier, the most knowledgeable and the most expensive to consult.

All three servers implement the same functional contract - askMages(question, maxAnswers) and describeMages() - though, as section 3.3 shows, not with identical parameter names. Xardas has one more tool, summonDemon, which changes the state of the world - and is therefore the one a read-only policy should stop. His tower also requires a password (a bearer token), so any client connecting to it directly needs a secret. Retrieval behind the tools is deliberately trivial keyword matching.

The questions in this article fall into two groups. Some are about the protocol and the libraries - what a schema looks like, how a version is negotiated, what a client library hands to the model. Those need no model, so their results are exact and repeatable. Others are about the model itself: which tool it picks, and what it does when a tool is off-limits. For those I use a real one - ClaudeSonnet 5 on Amazon Bedrock - and run every variant ten times. The agent with the model is a Java one (Spring AI); the Python client appears only in the questions that need no model, so this is not a comparison of Java and Python agents. Keeping the groups apart lets me attribute each result to its actual source. The caveat: one model, one prompt, ten runs - observations, not a benchmark.

‍

Architecture: agent, MCP proxyand three MCP servers

2.2 Tech stack

·      Java 25 (LTS)

·      MCP Java SDK 2.0.1 - implements protocol revision 2025-11-25

·      Spring AI 2.0.1 on Spring Boot 4.1.1

·      Jetty 12.1 (EE11) - hosting the servlet transport of the bare SDK

·      Python 3.14 with MCP Python SDK2.3.0 - implements 2026-07-28

·      Claude Sonnet 5 on Amazon Bedrock, called through Spring AI’s spring-ai-bedrock-converse

3. Code snippets

3.1 Fire Circle: the bare MCP Java SDK

On the bare SDK, a tool is a pair: a Tool definition (name, description, schemas) and a handler. Everything is written by hand:

private static final String ASK_INPUT_SCHEMA = """
         {
           "type": "object",
           "properties": {
             "question":   { "type": "string",  "description": "Question for the mages" },
             "maxAnswers": { "type": "integer", "description": "How many answers to return",
                             "minimum": 1, "maximum": 5, "default": 3 }
           },
           "required": ["question"]
         }""";
 
McpSchema.Tool tool = McpSchema.Tool.builder("askMages", MAPPER, ASK_INPUT_SCHEMA)
         .description("Asks the Fire Circle, the mages of Innos in the Old Camp. Cheap and fast; answers with the Circle's official doctrine.")
         .outputSchema(MAPPER, ASK_OUTPUT_SCHEMA)
         .annotations(McpSchema.ToolAnnotations.builder().readOnlyHint(true).idempotentHint(true).build())
         .build();
 
SyncToolSpecification askMages = SyncToolSpecification.builder()
         .tool(tool)
         .callHandler((exchange, request) -> {
             Map<String, Object> args = request.arguments();   // a raw map, no binding to types
             String question = String.valueOf(args.getOrDefault("question", ""));
             int maxAnswers = ((Number) args.getOrDefault("maxAnswers", 3)).intValue();
             List<FireLore.Entry> hits = FireLore.consult(question, maxAnswers);
             return McpSchema.CallToolResult.builder()
                     .addTextContent(render(hits))         // for the model
                     .structuredContent(structured(hits))  // for code
                     .build();
         })
         .build();

The Streamable HTTP transport is a plain HttpServlet, so the server can be mounted in any servlet container - here, embedded Jetty.

In SDK 2.x,arguments are validated against the schema before the handler runs - but nothing checks whether schema and handler still agree.

3.2 Xardas: Spring AI

In Spring AI, the same contract is a bean with annotated methods:

@McpTool(name = "askMages",
         description = "Asks Xardas, the necromancer who led the twelve mages that raised the Barrier. Broad and reliable, but expensive.",
         annotations = @McpTool.McpAnnotations(readOnlyHint = true, idempotentHint = true),
         generateOutputSchema = true)
 public XardasLore.Consultation askMages(
         @McpToolParam(description = "Question for the mages", required = true) String question,
         @McpToolParam(description = "How many answers to return (1-5)", required = false) Integer maxAnswers) {
     return lore.consult(question, maxAnswers == null ? 3 : Math.clamp(maxAnswers, 1, 5));
 }
 
@McpTool(name = "summonDemon",
         description = "Summons a demon. Changes the state of the world.",
         annotations = @McpTool.McpAnnotations(readOnlyHint = false, destructiveHint = true))
 public XardasLore.Summoning summonDemon(
         @McpToolParam(description = "What the demon is summoned for", required = true) String purpose) {
     return lore.summon(purpose);
 }

With spring-ai-starter-mcp-server-webmvc on the class path, the transport and the handshake are configuration:

pring:
   ai:
     mcp:
       server:
         name: xardas-mage
         version: 1.0.0
         protocol: streamable
         type: sync
         streamable-http:
           mcp-endpoint: /mcp

‍

Auto-configuration finds the @McpTool beans on its own. Two things are easy to miss: the output schema is opt-in (without generateOutputSchema = true it is null), and the input schema is built from the method signature - more on that in Test case C.

3.3 Water Mages: Python

mcp = MCPServer(name="water-mages", version="1.0.0",
                 instructions="Water Mages. Broad knowledge, cheap answers, hypotheses marked with low confidence.")
 
class Consultation(BaseModel):
     mages: str
     answers: list[Answer]
     total_cost_gold: int
 
@mcp.tool(name="askMages",
           description="Asks the Water Mages of the New Camp, who study the magic ore and the Barrier. Cheap; marks uncertain answers as hypotheses.",
           annotations=ToolAnnotations(readOnlyHint=True, idempotentHint=True))
 def ask_mages(
     question: Annotated[str, Field(description="Question for the mages")],
     max_answers: Annotated[int, Field(description="How many answers to return", ge=1, le=5)] = 3,
 ) -> Consultation:
     ...
 
mcp.run(transport="streamable-http", port=8083)

‍

Names, constraints, the default value and the output schema all come from the signature and the Pydantic types. MCPServer is the 2.x name of FastMCP. To be fair: Python also has a low-level Server with hand-written definitions, so the right pairs are bare SDK vs. low-level Server and Spring AI vs. MCPServer.

Notice also the parameter name: max_answers, not maxAnswers. Pydantic took the name straight from the Python signature, so on the wire this parameter differs from the Java servers.

3.4 Client side

A Java client takes three steps - transport, configuration, initialize():

McpSyncClient xardas = McpClient.sync(HttpClientStreamableHttpTransport.builder("http://localhost:8082")
                 .endpoint("/mcp")
                 .httpRequestCustomizer((request, method, uri, body, context) ->
                         request.header("Authorization", "Bearer " + towerPassword))
                 .build())
         .clientInfo(McpSchema.Implementation.builder("nameless-hero", "1.0.0").build())
         .requestTimeout(Duration.ofSeconds(20))
         .build();
 xardas.initialize();

In Python, the client is an async context manager:

async with Client("http://localhost:8081/mcp", mode="auto") as client:
     result = await client.call_tool("askMages", {"question": "Who is the Sleeper?", "maxAnswers": 1})

mode="auto" matters in Testcase D.

‍

3.5 MCP proxy

My proxy is built on the bare Java SDK: an McpServer facing the agent, whose tools forward to McpSyncClient instances facing the upstream servers. Each upstream is configured with a prefix and, where needed, credentials - so secrets live in the proxy, not in the agent.

When the proxybuilds its catalog, it puts the prefix into the name, so a call can address the tool, and into the description, so the model can choose it:

String exposedName = upstream.prefix() + "." + tool.name();        // askMages -> xardas.askMages
 McpSchema.Tool exposed = McpSchema.Tool.builder(exposedName, tool.inputSchema())
         .description("[%s] %s".formatted(upstream.name(), tool.description()))
         .outputSchema(tool.outputSchema())
         .annotations(tool.annotations())
         .build();

Every exposedtool gets the same call handler, where policy, routing and audit meet:

private McpSchema.CallToolResult handle(McpSchema.CallToolRequest request) {
     Map<String, Object> args = request.arguments() == null ? Map.of() : request.arguments();
     RoutedTool routed = catalog.route(request.name()).orElse(null);
     if (routed == null) {
         return error("Unknown tool: " + request.name());
     }
     // read-only mode: anything not marked readOnlyHint=true is refused, including tools with no annotations
     if (policy.readOnly() && !routed.isReadOnly()) {
         audit.record(routed.exposedName(), args, DENIED);
         return error("Denied: the proxy is read-only and this tool is not marked readOnlyHint=true");
     }
     McpSchema.CallToolResult result = clients.get(routed.upstreamName()).callTool(
             McpSchema.CallToolRequest.builder(routed.originalName()).arguments(args).build());
     audit.record(routed.exposedName(), args, Boolean.TRUE.equals(result.isError()) ? ERROR : OK);
     return result;
 }

‍

A refusal is adomain error (isError: true), not a protocol error, so the model gets a readable message. A blocked tool also stays in tools/list, with [BLOCKED BY POLICY - read-only session] prepended to its description. The idea is that the model knows the tool exists and why it will not run, instead of looking for a workaround - Test case B checks whether tha tholds.

Note what this policy rests on: readOnlyHint is the server’s own declaration, not proof that a tool has no side effects. The specification tells clients to treat annotations as untrusted unless the server is trusted. That is acceptable for servers you run yourself; for third-party servers, the policy should be an explicit allow or deny list of tool names.

4. Test phase

For the model runs in Test cases A and B, Spring AI talks to Claude on Amazon Bedrock, and every tool is wrapped in a decorator that records each call:

ChatClient chat = ChatClient.create(BedrockProxyChatModel.builder()
         .region(Region.EU_CENTRAL_1)
         .credentialsProvider(DefaultCredentialsProvider.builder().build())
         .options(BedrockChatOptions.builder().model("eu.anthropic.claude-sonnet-5").maxTokens(1024).build())
         .build());
 
String answer = chat.prompt()
         .system("You are an assistant to a traveller in the Colony. Use the available tools when they help. Answer briefly.")
         .user("Who is the Sleeper?")
         .tools(ToolCallbackProvider.from(recording(tools)))   // delegates to the MCP tool, logs every call
         .call()
         .content();

‍

Test case A - one client, three servers, three tools named askMages

Every server exposes askMages and describeMages. This is not contrived: the specification asks only that tool names be unique within a server. Names collide whenever servers share a contract - two instances of one server, such as two Jira sites, or several teams implementing one company-wide interface, as here. My question was: what do the real client libraries do when one agent connects to all three - and does the model care? In Java, the standard way to hand MCP tools to a Spring AI agent is SyncMcpToolCallbackProvider:

ToolCallback[] tools = SyncMcpToolCallbackProvider.builder()
         .mcpClients(fire, water, xardas)
         .build()
         .getToolCallbacks();

In Python, the SDK offers ClientSessionGroup:

async with ClientSessionGroup() as group:
     for server in (fire, water, xardas):
         await group.connect_to_server(server)

Here is what the libraries did:

== Spring AI: default settings =============================================
   askMages               <- Asks the Fire Circle, the mages of Innos...
   describeMages          <- Returns the mages' specialization and th...
   alt_1_askMages         <- Asks the Water Mages of the New Camp, wh...
   alt_2_describeMages    <- Returns the mages' specialization and th...
   alt_3_askMages         <- Asks Xardas, the necromancer who led the...
   alt_4_describeMages    <- Returns the mages' specialization and th...
   summonDemon            <- Summons a demon. Changes the state of th...
 
== Spring AI: the same servers connected in reverse order ==================
   askMages               <- Asks Xardas, the necromancer who led the...
   describeMages          <- Returns the mages' specialization and th...
   summonDemon            <- Summons a demon. Changes the state of th...
   alt_1_askMages         <- Asks the Water Mages of the New Camp, wh...
   alt_2_describeMages    <- Returns the mages' specialization and th...
   alt_3_askMages         <- Asks the Fire Circle, the mages of Innos...
   alt_4_describeMages    <- Returns the mages' specialization and th...
 
== Python: ClientSessionGroup ==============================================
   MCPError: {'askMages', 'describeMages'} already exist in group tools.

I expected a collision. Spring AI avoids one silently: duplicates become alt_1_askMages, alt_3_askMages and so on, depending on connection order - askMages is the FireCircle in one run and Xardas in the other. That is the default: the provider’s builder accepts your own naming scheme through toolNamePrefixGenerator(...). Python refuses to connect until you decide how to name things:

async with ClientSessionGroup(component_name_hook=lambda name, server: f"{server.name}.{name}") as group:
     ...
 # tools: ['fire-mages.askMages', 'fire-mages.describeMages', ..., 'xardas-mage.summonDemon']

Then, in this controlled scenario, I gave the model questions that do not say whom to ask, ten runs per variant. A run can ask several sources; “asked” means the source’s askMages was called. A prime marks the same setup with one thing changed.

No knowledge ofthe game is needed here: each server’s data answers differently, and onlyXardas’s contains the fact I count - that the Sleeper is an archdemon ofBeliar. The Fire Circle calls it a delusion; the Water Mages do not know. Icounted an answer when it contained both “archdemon” and “Beliar”. Withouttools it never appeared, so the fact came from the servers. Connection orderchanges both the names and the list order. With the Fire Circle first, themodel simply went down the list. With Xardas first, as plain askMages, it got less tidy: in 4runs it started with describeMages. Two runs nevercalled Xardas’s askMages and omitted theexpected fact. The ore mound question went to the Water Mages alone, althoughtheir tool was called alt_1_askMages - only thedescription pointed to them, and it was enough.

There are twomore catches. The three describeMages tools haveidentical descriptions, so for those the model has nothing to choose by. Andthis is the complete definition Spring AI passes to the model for summonDemon:

name:        summonDemon
 description: Summons a demon. Changes the state of the world.
 inputSchema: {"type":"object","properties":{"purpose":{"type":"string","description":"What the demon is summoned for"}},"required":["purpose"]}

Spring AI’s ToolDefinition consists of a name, adescription and an input schema - nothing else. readOnlyHint=false and destructiveHint=true do not reach themodel at all.

Takeaway:

·      a question that matched one description went to that server alone; one that matched all of them went to nearly all,

·      Spring AI’s default names depend on connection order: askMages was the Fire Circle in one run and Xardas in the other, so log and allow-list entries become ambiguous; changing the order also changed how the model searched,

·      Python fails fast and makes you choose a naming scheme,

·      Spring AI does not pass tool annotations to the model, so rules like “do not write” must be enforced elsewhere.

Test case B - the same servers behind an MCP proxy

Now the agent connects to the proxy only: one configuration entry, no secrets, and the proxy runs in read-only mode. Its tools/list looks like this:

fire.askMages          [fire] Asks the Fire Circle, the mages of Innos in t...
   water.askMages         [water] Asks the Water Mages of the New Camp, who st...
   xardas.askMages        [xardas] Asks Xardas, the necromancer who led the tw...
   xardas.summonDemon     [BLOCKED BY POLICY - read-only session] [xardas] Sum...
   ... plus fire.describeMages, water.describeMages, xardas.describeMages

The same questions, plus a request for the one tool the proxy blocks. In B2’ the proxy lists that tool without the note, and in B3 a deny rule removes it from the list altogether:

Through the proxy, the model went to Xardas first in 9 of 10 runs and never skipped him -compared with 0/10 in A1 and 4/10 in A2. I cannot attribute this to the names alone: the host connects to one MCP endpoint instead of three, and the model receives a catalog with different names and description prefixes.

B2 ran in two more series of 10 - one with an earlier version of the descriptions: 0 call sand an explanation of the block in all 30 runs.

A typical B2 answer: “I’m unable to do that - demon summoning is disabled in this session (it’s a world-changing action blocked by policy, read-only mode).” In B2’, the model called the tool once per run, got the proxy’s refusal and reported it just as accurately. With the tool hidden (B3), none of the 10 answers said the action was not permitted: four said the assistant cannot summon anything, the rest advised against it. The handler’s check does the enforcing: in B2’, the proxy returned this exact error ten times. A direct tools/call adds the audit entry:

<-- isError=true  "Denied: the proxy is read-only and this tool is not marked readOnlyHint=true"
 audit: DENIED  xardas.summonDemon  0 ms  {purpose=guard the tower}

The costs: one more hop, one more process, and tool definitions about 3% larger (1,182estimated tokens versus 1,147 directly).

Takeaway:

·      through the proxy the search was more orderly - names,description prefixes or both; this setup cannot tell,

·      in this scenario, listing a blocked tool beat hiding it: every answer explained the block, versus 16 extra calls and no clear “notallowed”,

·      the note in the description saved one refused call per run; the check in the handler is what enforces the policy.

A tool name on its way to the model

The MCP specification allows dots in tool names - its own example is admin.tools.list. Model APIs are stricter: Amazon Bedrock accepts only [a-zA-Z0-9_-], up to 64characters. Spring AI bridges the gap silently: it removes every other character, turns hyphens into underscores and keeps the last 64 characters. The Fire Circle’s askMages along the way:

The last one still needs cleaning up before it reaches a model API. If your audit log or allow-lists refer to tools by name, decide which of these names they use.

Test case C - one parameter, three schema generators

The same maxAnswers parameter, as each server publishes it in tools/list, followed by a call with an out-of-range value of 9:

== Fire (bare SDK) =========================================================
   "maxAnswers": {"type":"integer","description":"How many answers to return","minimum":1,"maximum":5,"default":3}
 == Xardas (Spring AI) ======================================================
   "maxAnswers": {"type":"integer","format":"int32","description":"How many answers to return (1-5)"}
 == Water (Python) ==========================================================
   "max_answers": {"default":3,"description":"How many answers to return","maximum":5,"minimum":1,"title":"Max Answers","type":"integer"}
 
+--------------------+------------------------------+---------------------------+
 | server             | min/max/default in schema    | answers=9                 |
 +--------------------+------------------------------+---------------------------+
 | Fire (bare SDK)    | yes                          | rejected                  |
 | Xardas (Spring AI) | no (only in the description) | accepted, clamped in code |
 | Water (Python)     | yes                          | rejected                  |
 +--------------------+------------------------------+---------------------------+

Spring AI does not emit minimum, maximum or default for flat method parameters.The range exists only in the description - that is, in the prompt - and the SDK’s validator has nothing to enforce, so the value reaches the handler and gets clamped there. The model never learns it asked for too much.

Spring AI generates schemas with victools, without its Jakarta Validation module, so @Min and @Max are ignored. Swagger’s @Schema(minimum = "1", maximum ="5") works only on a field of a record parameter. This is not a bug: the documentation lists the annotations the generator reads - for descriptions and required parameters - and constraints are not among them.

The input schema also depends on a compiler flag. Parameter names are not kept in byte code unless you compile with -parameters. This is the schema of the same method compiled without it:

{"type":"object","properties":{"arg0":{"type":"string","description":"Question for the mages"},
  "arg1":{"type":"integer","format":"int32","description":"How many answers to return (1-5)"}},"required":["arg0"]}

The Spring BootGradle plugin and the spring-boot-starter-parent POM set the flag for you, which is why this is easy to forget - until a module is built without them.

Takeaway:

·      look at the schema your framework actually emits - the model and the validator see only that,

·      in Spring AI, constraints need a record with Swagger annotations and names need -parameters; in Python both come from the signature.

Test case D - Java and Python talking to each other

First, the Java client connects to the Python server and calls askMages with maxAnswers=1 for a question that matches four topics:

== Java client (SDK 2.0.1) -> Python server (mcp 2.3.0) ====================
 -->  [water] initialize
 <--  [water] result
   negotiated protocol: 2025-11-25
   sent maxAnswers=1 -> isError=false, answers returned: 3

Then the Python client connects to the Java server:

== Python client (mcp 2.3.0, mode=auto) -> Java server (SDK 2.0.1) =========
   --> server/discover            protocol 2026-07-28
   <-- HTTP 400
   --> initialize                 protocol 2025-11-25
   <-- HTTP 200
   --> notifications/initialized  protocol 2025-11-25
   negotiated protocol: 2025-11-25

‍

The Python client opens with server/discover - the stateless way to start a conversation in 2026-07-28. The Java server answers with HTTP 400, so the client falls back to initialize and both sides settle on 2025-11-25. That fallback is the client’s mode: "legacy" always uses initialize, a version string pins a modern revision, and the default, "auto", tries server/discover first. The Python server simply accepts the older handshake.

Takeaway:

·      interoperability works: the newer side falls back,

·      the price: the whole setup speaks the older revision, withoutany 2026-07-28 features,

·      the Python server silently ignored the unknown argument: maxAnswers sent where it expects max_answers had no effect, so the default of 3 was used. Agree on argument naming explicitly - @McpToolParam cannot rename a parameter yet (spring-ai#6544).

5. Java and Python as of October 2026

Since February 2026, MCP SDKs have official tiers. Tier 1 SDKs (TypeScript, Python, C#, Go,Rust and Ruby) support a new specification revision when it is published. Javai s Tier 2, which allows up to six months. It shows: the Python SDK 2.3.0 implements 2026-07-28, while the MCPJava SDK 2.0.1 stops at 2025-11-25. The 2026-07-28 work is tracked under the2.2 milestone of the Java SDK, and even the Spring AI 2.1.0-M1 milestone still builds on SDK 2.0.0.

2026-07-28 is a breaking revision. It removes the initialize handshake and protocol-level sessions, adds server/discover, replaces server-initiated requests with multi-round-trip results, introduces extensions such as Tasks and MCP Apps, and deprecates Roots, Sampling and Logging. As Test case D showed, a 2025-11-25 server still talks to the Python 2.3.0 client, so for this pair the gap is a planning risk, not an outage.

The three version numbers - protocol revision, SDK and Spring AI - are independent and easy to mix up: Spring AI issue #6954 asks to backport “MCP Java SDK 2.0.x (MCP 2026-07-28)”, yet SDK 2.0.x does not implement 2026-07-28.

Deprecations differ, too: the Python SDK flags Roots, Sampling and Logging with a warning shown by default, while the Java SDK does not yet know the revision that deprecated them.

Since all of thiswill change, here is how to check it yourself:

# Which protocol revisions does the Java SDK know?
 curl -s https://raw.githubusercontent.com/modelcontextprotocol/java-sdk/main/mcp-core/src/main/java/io/modelcontextprotocol/spec/ProtocolVersions.java | grep 'String MCP_'
 
# Which revision does the Python SDK implement?
 uv run --with mcp python -c "import mcp.types as t; print(t.LATEST_PROTOCOL_VERSION)"
 
# Which MCP SDK does a given Spring AI version pull in?
 curl -s https://repo1.maven.org/maven2/org/springframework/ai/spring-ai-mcp/2.0.1/spring-ai-mcp-2.0.1.pom | grep -A1 '<artifactId>mcp</artifactId>'

6. Summary

On the wire, MCPis JSON-RPC: a handshake (in the 2025-11-25 revision used bythe Java SDK here), a tool list with a schema per tool, and calls, as the frames in section 1.2 show.

Python is further ahead on the protocol: its SDK 2.3.0 implements 2026-07-28, while the Java SDK, a Tier 2 SDK, stops at 2025-11-25. In my code, Pydantic also put into the schema the constraints that Spring AI dropped.

Frameworks make decisions for you, some invisible until you look. In my tests, Spring AI renamed tools depending on connection order, stripped characters from their names, dropped constraints from a schema and did not pass tool annotations to the model, while Python refused to guess and made me decide. Outside the model, the same names make log and allow-list entries ambiguous, and validation never sees a constraint the schema does not carry. The common thread: client libraries make these decisions quietly; a proxy is one place to make them explicitly.

Finally, once agents talk to several servers, somebody has to take care of stable names, rules, secrets and audit. A host can handle the names - Python’s naming hook and Spring AI’s pluggable prefix generator do that - while a proxy handles all four once, for every agent connected to it. That layer does not have to be a product - mine is a few hundred lines on the bare SDK. One design choice is worth trying: tell the model what it may not do instead of hiding it. In my runs, every answer then explained the block, and a note in the description saved the one refused call; the check in the proxy did the enforcing.

There are drawbacks: the specification moves fast, and some of this will be outdated within months - hence the commands in section 5. The protocol gap did not break interoperability, though: the Python client fell back to the older handshake.

Thank you fortaking the time to read.

‍

Read more
Software development

How to set up Kafka integration test

Do you consider unit testing as not enough solution for keeping the application's reliability and stability? Are you afraid that somehow or somewhere there is a potential bug hiding in the assumption that unit tests should cover all cases? And also is mocking Kafka not enough for project requirements? If even one answer is ‘yes’, then welcome to a nice and easy guide on how to set up Integration Tests for Kafka using TestContainers and Embedded Kafka for Spring!

What is TestContainers?

TestContainers is an open-source Java library specialized in providing all needed solutions for the integration and testing of external sources. It means that we are able to mimic an actual database, web server, or even an event bus environment and treat that as a reliable place to test app functionality. All these fancy features are hooked into docker images, defined as containers. Do we need to test the database layer with actual MongoDB? No worries, we have a test container for that. We can not also forget about UI tests - Selenium Container will do anything that we actually need.
In our case, we will focus on Kafka Testcontainer.


What is Embedded Kafka?

As the name suggests, we are going to deal with an in-memory Kafka instance, ready to be used as a normal broker with full functionality. It allows us to work with producers and consumers, as usual, making our integration tests lightweight.

Before we start

The concept for our test is simple - I would like to test Kafka consumer and producer using two different approaches and check how we can utilize them in actual cases.

Kafka Messages are serialized using Avro schemas.

Embedded Kafka - Producer Test

The concept is easy - let's create a simple project with the controller, which invokes a service method to push a Kafka Avro serialized message.

Dependencies:

dependencies {

implementation "org.apache.avro:avro:1.10.1"

implementation("io.confluent:kafka-avro-serializer:6.1.0")

implementation 'org.springframework.boot:spring-boot-starter-validation'

implementation 'org.springframework.kafka:spring-kafka'

implementation('org.springframework.cloud:spring-cloud-stream:3.1.1')

implementation('org.springframework.cloud:spring-cloud-stream-binder-kafka:3.1.1')



implementation('org.springframework.boot:spring-boot-starter-web:2.4.3')

implementation 'org.projectlombok:lombok:1.18.16'



compileOnly 'org.projectlombok:lombok'

annotationProcessor 'org.projectlombok:lombok'

testImplementation('org.springframework.cloud:spring-cloud-stream-test-support:3.1.1')

testImplementation 'org.springframework.boot:spring-boot-starter-test'

testImplementation 'org.springframework.kafka:spring-kafka-test'

}

Also worth mentioning fantastic plugin for Avro. Here plugins section:

plugins {

id 'org.springframework.boot' version '2.6.8'

id 'io.spring.dependency-management' version '1.0.11.RELEASE'

id 'java'

id "com.github.davidmc24.gradle.plugin.avro" version "1.3.0"

}

Avro Plugin supports schema auto-generating. This is a must-have.


Link to plugin: https://github.com/davidmc24/gradle-avro-plugin


Now let's define the Avro schema:

{

"namespace": "com.grapeup.myawesome.myawesomeproducer",

"type": "record",

"name": "RegisterRequest",

"fields": [

{"name": "id", "type": "long"},

{"name": "address", "type": "string", "avro.java.string": "String"

}



]

}

Our ProducerService will be focused only on sending messages to Kafka using a template, nothing exciting about that part. Main functionality can be done just using this line:

ListenableFuture<SendResult<String, RegisterRequest>> future = this.kafkaTemplate.send("register-request", kafkaMessage);

We can’t forget about test properties:

spring:

main:

allow-bean-definition-overriding: true

kafka:

consumer:

group-id: group_id

auto-offset-reset: earliest

key-deserializer: org.apache.kafka.common.serialization.StringDeserializer

value-deserializer: com.grapeup.myawesome.myawesomeconsumer.common.CustomKafkaAvroDeserializer

producer:

auto.register.schemas: true

key-serializer: org.apache.kafka.common.serialization.StringSerializer

value-serializer: com.grapeup.myawesome.myawesomeconsumer.common.CustomKafkaAvroSerializer

properties:

specific.avro.reader: true

As we see in the mentioned test properties, we declare a custom deserializer/serializer for KafkaMessages. It is highly recommended to use Kafka with Avro - don't let JSONs maintain object structure, let's use civilized mapper and object definition like Avro.

Serializer:

public class CustomKafkaAvroSerializer extends KafkaAvroSerializer {

public CustomKafkaAvroSerializer() {

super();

super.schemaRegistry = new MockSchemaRegistryClient();

}



public CustomKafkaAvroSerializer(SchemaRegistryClient client) {

super(new MockSchemaRegistryClient());

}



public CustomKafkaAvroSerializer(SchemaRegistryClient client, Map<String, ?> props) {

super(new MockSchemaRegistryClient(), props);

}

}

Deserializer:

public class CustomKafkaAvroSerializer extends KafkaAvroSerializer {

public CustomKafkaAvroSerializer() {

super();

super.schemaRegistry = new MockSchemaRegistryClient();

}



public CustomKafkaAvroSerializer(SchemaRegistryClient client) {

super(new MockSchemaRegistryClient());

}



public CustomKafkaAvroSerializer(SchemaRegistryClient client, Map<String, ?> props) {

super(new MockSchemaRegistryClient(), props);

}

}

And we have everything to start writing our test.

@ExtendWith(SpringExtension.class)

@SpringBootTest

@AutoConfigureMockMvc

@TestInstance(TestInstance.Lifecycle.PER_CLASS)

@ActiveProfiles("test")

@EmbeddedKafka(partitions = 1, topics = {"register-request"})

class ProducerControllerTest {

All we need to do is add @EmbeddedKafka annotation with listed topics and partitions. Application Context will boot Kafka Broker with provided configuration just like that. Keep in mind that @TestInstance should be used with special consideration. Lifecycle.PER_CLASS will avoid creating the same objects/context for each test method. Worth checking if tests are too time-consuming.

Consumer<String, RegisterRequest> consumerServiceTest;@BeforeEach

void setUp() {

DefaultKafkaConsumerFactory<String, RegisterRequest> consumer = new DefaultKafkaConsumerFactory<>(kafkaProperties.buildConsumerProperties();



consumerServiceTest = consumer.createConsumer();

consumerServiceTest.subscribe(Collections.singletonList(TOPIC_NAME));

}

Here we can declare the test consumer, based on the Avro schema return type. All Kafka properties are already provided in the .yml file. That consumer will be used as a check if the producer actually pushed a message.

Here is the actual test method:

@Test

void whenValidInput_therReturns200() throws Exception {

RegisterRequestDto request = RegisterRequestDto.builder()

.id(12)

.address("tempAddress")

.build();



mockMvc.perform(

post("/register-request")

.contentType("application/json")

.content(objectMapper.writeValueAsBytes(request)))

.andExpect(status().isOk());



ConsumerRecord<String, RegisterRequest> consumedRegisterRequest = KafkaTestUtils.getSingleRecord(consumerServiceTest, TOPIC_NAME);



RegisterRequest valueReceived = consumedRegisterRequest.value();



assertEquals(12, valueReceived.getId());

assertEquals("tempAddress", valueReceived.getAddress());

}

First of all, we use MockMvc to perform an action on our endpoint. That endpoint uses ProducerService to push messages to Kafka. KafkaConsumer is used to verify if the producer worked as expected. And that’s it - we have a fully working test with embedded Kafka.

Test Containers - Consumer Test

TestContainers are nothing else like independent docker images ready for being dockerized. The following test scenario will be enhanced by a MongoDB image. Why not keep our data in the database right after anything happened in Kafka flow?

Dependencies are not much different than in the previous example. The following steps are needed for test containers:

testImplementation 'org.testcontainers:junit-jupiter'

testImplementation 'org.testcontainers:kafka'

testImplementation 'org.testcontainers:mongodb'



ext {

set('testcontainersVersion', "1.17.1")

}



dependencyManagement {

imports {

mavenBom "org.testcontainers:testcontainers-bom:${testcontainersVersion}"

}

}

Let's focus now on the Consumer part. The test case will be simple - one consumer service will be responsible for getting the Kafka message and storing the parsed payload in the MongoDB collection. All that we need to know about KafkaListeners, for now, is that annotation:

@KafkaListener(topics = "register-request")

By the functionality of the annotation processor, KafkaListenerContainerFactory will be responsible to create a listener on our method. From this moment our method will react to any upcoming Kafka message with the mentioned topic.

Avro serializer and deserializer configs are the same as in the previous test.

Regarding TestContainer, we should start with the following annotations:

@SpringBootTest

@ActiveProfiles("test")

@Testcontainers

public class AbstractIntegrationTest {

During startup, all configured TestContainers modules will be activated. It means that we will get access to the full operating environment of the selected source. As example:

@Autowired

private KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;



@Container

public static KafkaContainer kafkaContainer = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:6.2.1"));



@Container

static MongoDBContainer mongoDBContainer = new MongoDBContainer("mongo:4.4.2").withExposedPorts(27017);

As a result of booting the test, we can expect two docker containers to start with the provided configuration.

What is really important for the mongo container - it gives us full access to the database using just a simple connection uri. With such a feature, we are able to take a look what is the current state in our collections, even during debug mode and prepared breakpoints.
Take a look also at the Ryuk container - it works like overwatch and checks if our containers have started correctly.

And here is the last part of the configuration:

@DynamicPropertySource

static void dataSourceProperties(DynamicPropertyRegistry registry) {

registry.add("spring.kafka.bootstrap-servers", kafkaContainer::getBootstrapServers);

registry.add("spring.kafka.consumer.bootstrap-servers", kafkaContainer::getBootstrapServers);

registry.add("spring.kafka.producer.bootstrap-servers", kafkaContainer::getBootstrapServers);

registry.add("spring.data.mongodb.uri", mongoDBContainer::getReplicaSetUrl);

}



static {

kafkaContainer.start();

mongoDBContainer.start();



mongoDBContainer.waitingFor(Wait.forListeningPort()

.withStartupTimeout(Duration.ofSeconds(180L)));

}



@BeforeTestClass

public void beforeTest() {



kafkaListenerEndpointRegistry.getListenerContainers().forEach(

messageListenerContainer -> {

ContainerTestUtils

.waitForAssignment(messageListenerContainer, 1);



}

);

}



@AfterAll

static void tearDown() {

kafkaContainer.stop();

mongoDBContainer.stop();

}

DynamicPropertySource gives us the option to set all needed environment variables during the test lifecycle. Strongly needed for any config purposes for TestContainers. Also, beforeTestClass kafkaListenerEndpointRegistry waits for each listener to get expected partitions during container startup.

And the last part of the Kafka test containers journey - the main body of the test:

@Test

public void containerStartsAndPublicPortIsAvailable() throws Exception {

writeToTopic("register-request", RegisterRequest.newBuilder().setId(123).setAddress("dummyAddress").build());



//Wait for KafkaListener

TimeUnit.SECONDS.sleep(5);

Assertions.assertEquals(1, taxiRepository.findAll().size());



}



private KafkaProducer<String, RegisterRequest> createProducer() {

return new KafkaProducer<>(kafkaProperties.buildProducerProperties());

}



private void writeToTopic(String topicName, RegisterRequest... registerRequests) {



try (KafkaProducer<String, RegisterRequest> producer = createProducer()) {

Arrays.stream(registerRequests)

.forEach(registerRequest -> {

ProducerRecord<String, RegisterRequest> record = new ProducerRecord<>(topicName, registerRequest);

producer.send(record);

}

);

}

}

The custom producer is responsible for writing our message to KafkaBroker. Also, it is recommended to give some time for consumers to handle messages properly. As we see, the message was not just consumed by the listener, but also stored in the MongoDB collection.

Conclusions

As we can see, current solutions for integration tests are quite easy to implement and maintain in projects. There is no point in keeping just unit tests and counting on all lines covered as a sign of code/logic quality. Now the question is, should we use an Embedded solution or TestContainers? I suggest first of all focusing on the word “Embedded”. As a perfect integration test, we want to get an almost ideal copy of the production environment with all properties/features included. In-memory solutions are good, but mostly, not enough for large business projects. Definitely, the advantage of Embedded services is the easy way to implement such tests and maintain configuration, just when anything happens in memory.
TestContainers at the first sight might look like overkill, but they give us the most important feature, which is a separate environment. We don't have to even rely on existing docker images - if we want we can use custom ones. This is a huge improvement for potential test scenarios.
What about Jenkins? There is no reason to be afraid also to use TestContainers in Jenkins. I firmly recommend checking TestContainers documentation on how easily we can set up the configuration for Jenkins agents.
To sum up - if there is no blocker or any unwanted condition for using TestContainers, then don't hesitate. It is always good to keep all services managed and secured with integration test contracts.

Read more
View all