AI Agentic Workflows

Short answer: yes - but unless you can host a frontier-scale model, expect to put in extra work. We benchmarked 21 open LLMs on 154 PL/I and 250 COBOL tasks derived from the IBM z/OS language references and real-world project samples, with every answer scored by a Claude Opus judge against a cached Claude Opus 4.7 baseline. What the benchmark showed:

• Open models can be self-hosted and still be useful. The strongest open model slightly exceeds the Claude Opus line, and the best medium-sized, self-hostable models reach roughly 80% of that frontier score.
• There is a hard wall: preprocessor and copybook mechanics, IBM-dialect behavior prediction, and multi-file reasoning. The dominant failure mode is confidently hallucinated syntax - hard to catch in review, and it often surfaces only at compile time.
• That wall is mostly a knowledge gap, not a capability gap. With curated reference context, the models we recommend went from 33% to 93% of preprocessor-family tasks passed on PL/I, and from 33% to 92% on COBOL - with no weights touched.

Why on-premise matters for mainframe modernization

As increasingly capable language models become widely available, teams have a new opportunity to reduce the cost and risk of working with complex legacy systems.

In many organizations, critical business logic is embedded in decades-old, sparsely documented code. Teams often lack an end-to-end understanding of individual processes, which makes even minor changes expensive to verify and risky to deploy. In languages like COBOL and PL/I, day-to-day work is usually done through long-established abstraction layers, while the underlying preprocessor and macro expansion logic is rarely inspected. As a result, understanding production behavior requires not only language syntax but also enterprise-specific conventions and build mechanics.

Mainframe systems support critical workloads across banking, insurance, and government, and are therefore subject to strict internal controls. In many organizations, legal or contractual requirements prevent source code from leaving secured environments. If the code cannot leave the data center, the model has to run inside it. The same constraint also limits how much representative mainframe code is present in public training data for open models in the first place.

So the practical question for an architect is not "which LLM is best at COBOL?" but "which model can I host inside my perimeter, on which hardware, and how do I make it work on our own code?" That is what this benchmark set out to answer.

How we benchmarked 21 open LLMs on COBOL and PL/I

Where the tasks come from

We built the tasks from two sources: IBM's Enterprise COBOL for z/OS and Enterprise PL/I for z/OS language references, plus project samples that capture client-style conventions. That matters because these are IBM dialects, not ANSI subsets. The IBM language references document production compiler behavior and IBM-specific semantics - material underrepresented in training data, because production mainframe source rarely reaches GitHub.

The five task categories

How answers were scored

An Opus judge graded each answer against the relevant reference chapter, scoring correctness, syntax validity, completeness, and clarity (plus idiom and includes-awareness on the project-convention tasks), then compared it with the cached baseline answer. Every score reported below is the average of the judge's per-task "overall" rating (1–5) across all tasks in that language.

Models and hardware

The benchmark covers these open model families: Gemma, Qwen, GPT-OSS, Nemotron, Mistral/Magistral, Granite, Kimi, DeepSeek, Tencent, and MiniMax. Some were served locally on an NVIDIA DGX Spark (GB10, 128 GB unified memory), an Apple M2 Ultra, and a Ryzen / RX 7900 XTX. Others were evaluated through hosted endpoints (OpenRouter). For reference, we used two closed frontier models: Claude Opus 4.7 and Claude Haiku 4.5. Opus also supplied the judge and the cached baseline.

Benchmark results: which open LLMs handle COBOL and PL/I best

Medium and small open models, self-hosted on our own hardware

What the category breakdown shows

• Bug fixing is the strongest category on PL/I and the only category that is near-top in both languages.
• COBOL translation is the strongest single category.
• Explanation is the hardest category on average in both languages.
• ‍Syntax is where models differ the most on PL/I: frontier models approach 5 while the smallest fall below 2.‍
• Only one challenger reaches the Claude Opus line - Kimi K3, an open model
• PL/I and COBOL move together: a model choice made for one language carries to the other.
• The judge scored clarity highest for almost every model — models write well even when they are wrong.

Parameter count mostly predicts score - with two instructive exceptions

The results mostly follow a simple rule: more parameters, better score, especially within the same model family.

Gemma 4 breaks the rule. Detailed analysis of the tasks shows the gap isn't Gemma being brilliant: its winning answers are short and exactly scoped. The larger models' answers, by contrast, tend to contain at least one concrete, quotable IBM-dialect error, or they lose by over-elaboration - invented constructs and rules, and over-engineered "improvements". Notably, unrelated models invented similar rules, which points to a shared misconception in public training data rather than an isolated model flaw.

Mistral Small 4 shows a different effect: quantization cost. To fit our hardware it had to run at Q3_K_S, a more aggressive quantization. Re-run at fp8 through a hosted endpoint, the same model gains roughly half a point in both languages - COBOL 2.84 → 3.37, PL/I 3.00 → 3.46. This is the largest quantization penalty we measured, and the lesson cuts against a common instinct: a bigger model squeezed into a lower precision to fit the box is not automatically better than a smaller model running at a comfortable one.

The operational dimension the scores hide: latency and empty answers

Quality scores say nothing about whether the model answers at all, and how long it takes.

Hosted APIs and fast local models answer in seconds (Claude Haiku ~3 s, Qwen3 Coder ~7 s), while thinking models averaged minutes per task on identical hardware - Magistral as slow as 6.8 min average. Long reasoning phases repeatedly pushed models past our per-task time and output-token limits, and a request cut off mid-thinking returns an empty answer. They were recoverable by raising the limits.  

The wall: the COBOL and PL/I tasks open models can't handle

We ranked the tasks by all-model mean correctness. The worst-performing ones fell into three recurring categories:

1. Preprocessor and copybook mechanics - COPY ... REPLACING pseudo-text rules and PL/I % macro expansion.
2. Dialect-specific behavior prediction - DEFAULT(IBM) vs. DEFAULT(ANS), REFER variable-length structures, fixed-format literal continuation, and other IBM-specific semantics.
3. Multi-file reasoning and project-specific conventions - on tasks built around a client's $xxx macro catalogue and work-area convention, models produce correct generic PL/I but miss the project's source style.

These are exactly the checks that sit in front of AI-assisted modernization work. Before an AI agent can explain a program, extract business rules, or translate it, it has to know what the compiler actually sees after copybooks and macros, which IBM dialect rules apply, and which external files the code depends on. A model that scores well on average and fails here is not ready for the daily work - which is why the coverage map matters more than the headline number.

The wall moves when you prepare the context

We re-ran the failing preprocessor and copybook tasks with curated context prepended to each prompt - same tasks, same judge. A task counts as passed when the judge rates its correctness at least 4 out of 5. In both languages, that context turns IBM behavior into model-ready rules and verified worked examples.

• For PL/I, every single model improved. Mean correctness on the preprocessor family went from 2.42 to 4.11, and task passes from 15% to 74%.
• For COBOL, the copybook run moves the same way. Mean correctness on the copybook family went from 2.04 to 4.06, and task passes from 14% to 71%.
• The lift has a floor. The smallest builds like Nemotron Nano 4B improve, but still do not reach the usable range - context cannot buy capability that isn't there.

Context engineering alone can change the result substantially. Fine-tuning is not always the first answer: sometimes the missing step is properly investigating the problem, then choosing the context the model actually needs.

That research step is often skipped. Teams jump straight to writing agent specs - or straight to execution - and then treat the model's mistakes as a model-quality problem. The first step in improving model responses is identifying what the model needs to know. The same verified reference material can serve as prompt context today and, later, as training data for a lightweight dialect adapter such as a LoRA. Putting the sequence - research → spec → plan → execute - into everyday agent workflows makes the work more systematic and the results more predictable.

Which open LLMs to start with for on-premise COBOL and PL/I work

For a client-specific modernization setup, the recommendation is not a single universal ranking. The model has to be good enough on COBOL or PL/I, but it also has to fit the client's specific case, infrastructure, latency expectations, and other requirements.

These picks also respond best to context preparation: adding the curated reference context lifted their preprocessor-family passes from 33% to 93% on PL/I and from 33% to 92% on COBOL.

We prioritized quality. If the priority is different - latency or throughput - the recommendation would change. The decision is not only which model scores highest overall, but which model has the right coding ability for the target language, and which weak spots can be covered by review, curated context, or a dialect adapter.

Key lessons

• Judge models by the coverage map, not the headline score - the average hides the collapse.
• Use the coverage map as the acceptance checklist. The question is not whether an answer reads well, but whether it handles the reference-specific constructs your workflow depends on.
• Research what the model must know → build a verified spec → then plan the work.
• Legacy-language skill transfers. PL/I and COBOL scores track each other closely across the whole field, so a model validated on one language generally carries to the other.
• Be careful with thinking models run locally. The extra deliberation costs latency and may exceed your limits.

Conclusion: can you modernize the mainframe without the code leaving it?

Out of the box, the best open model already edges past our frontier baseline, self-hostable models reach roughly 80% of that level. They also share the same wall: copybook and preprocessor mechanics, IBM-specific semantics, and multi-file reasoning - precisely the material daily modernization work stands on.

What changed our reading of the results is how far that wall moved. A small, verified slice of the IBM reference, handed to the models as context, took our recommended picks from 33% to 93% of preprocessor tasks passed on PL/I and from 33% to 92% on COBOL - with not a single weight touched.

So for a modernization program, choosing the model is only half the decision. The other half is preparing what the model needs to know before it acts. For regulated estates, that means a capable open model on your own hardware, plus curated context or a dialect adapter.

FAQ

‍
Can LLMs work with COBOL or PL/I code without sending it to the cloud?

Yes. Open models can run entirely inside your own environment, and the strongest of them now match a closed frontier model on COBOL and PL/I. The trade-off is scale: those top models need serious inference infrastructure, while models that fit a single machine score lower and need curated context or a dialect adapter to handle IBM-dialect specifics.

Which open LLM is best for COBOL and PL/I?

Kimi K3 is the strongest open model in this benchmark (4.62 COBOL / 4.71 PL/I), similar to Claude Opus, but frontier-scale to host. Of the models we self-hosted, Nemotron 3 Super 120B is the best (3.60 / 3.77 at Q4_K_M), and Gemma 4 31B is the best smaller option (3.48 / 3.35).

Why does Gemma 4 score so high for its size?

‍Because of what this benchmark measures. The tasks are compact, single-turn checks of IBM-dialect knowledge: declare, fix, explain, or translate a well-scoped piece of code. There are no multi-file feature builds or long agentic sessions here. That profile rewards models that recall dialect specifics and stay inside what they know. Gemma's winning answers are short and exactly in spec, while larger models more often lose points on invented IBM constructs. We picked the challengers from the top of public coding benchmarks, but this benchmark does not measure complex, large-scale code generation.  

Do you need a separate model for COBOL and for PL/I?

No. COBOL and PL/I scores track each other closely across the whole field, so a model validated on one legacy language generally carries over to the other.

Why do LLMs fail on IBM COBOL and PL/I dialects?

Because production mainframe source rarely reaches public repositories, IBM-specific semantics are underrepresented in training data. Models fall back on generic or ANSI-flavored assumptions and state them fluently. In our tasks, unrelated models invented the same non-existent rules - a signature of a shared gap in public training data, not of one weak model.

Is fine-tuning required to make an open model usable on mainframe code?

Not as a first step. Prepending a curated slice of the IBM reference plus verified worked examples raised PL/I preprocessor-family passes from 15% to 74% and COBOL copybook passes from 14% to 71% across the field, with no weights changed. Fine-tuning or a LoRA dialect adapter is a later step, and the same material can be used as its training data.

How much hardware do you need to self-host a usable model?

Our best self-hosted picks - Nemotron 3 Super 120B at Q4_K_M and Gemma 4 31B - ran on a single 128 GB unified-memory machine (NVIDIA DGX Spark, GB10). Frontier-scale open models like Kimi K3 need substantially more.

Does quantization hurt COBOL and PL/I accuracy?

It can, materially. Mistral Small 4 119B scored 2.84 COBOL / 3.00 PL/I at Q3_K_S, and 3.37 / 3.46 at fp8. A large model squeezed into low precision to fit the box is not automatically better than a smaller model at a comfortable precision.

Why benchmark open models instead of a cloud API?

Because for many mainframe estates the code cannot leave the data center, so the question is what can run inside it.

Can these models be trusted in production modernization work?

Yes, under the gates any generated code already passes - it still has to compile, pass tests, and get reviewed. The dominant failure mode is confidently hallucinated syntax that reads well and only breaks at compile time. It is worth establishing this at the start, rather than discovering it task by task: identify which constructs the model handles, scope its work accordingly, and close the known gaps with curated context or a dialect adapter.

Fully on-premise modernization, done in the order that works

Feature coverage and client-code sampling are a good starting point for an offline modernization effort: the coverage map shows which constructs a model can already be trusted with, and tasks sampled from the client's own code expose the site conventions no public model has ever seen. Just as important is the working order - research → spec → plan → execute. As the curated-context experiment showed, identifying what the model needs to know and distilling it into verified context before asking it to act can improve results substantially, with no fine-tuning involved.

That is precisely the pipeline behind Grape Up's G.Tx platform. G.Tx can be deployed on-premises, running its modernization workflows and agents on locally hosted open models. The result is simple: the entire transformation happens inside the client's environment, and source code never leaves the data center.

More on the approach: grapeup.com/application-modernization.

We let an AI coding agent run a database migration end to end. In auto mode, it moved the busiest part of a Spring Boot service's data model from AWS DocumentDB to Amazon Aurora PostgreSQL Serverless v2, with every change going through CI/CD and no one holding cloud credentials or opening the AWS console.

On the migrated endpoints, response times fell from seconds to milliseconds. What produced that was the flow, not any single prompt. The rest of this article describes how.

The system and the bet

The system was a Spring Boot service backed by AWS DocumentDB.

One part of the model carried the pain. Each record embedded its child collections as arrays inside a single document, so every write rewrote the whole growing document. Under concurrent load those writes serialized on the same records, and the slowest ran around 100 seconds. Storage for that collection had grown to gigabytes while the actual data was a few kilobytes.

Cost was the second problem. DocumentDB has no serverless tier. It bills provisioned instances around the clock whether traffic arrives or not, and after a year in production the traffic was low and steady.

The bet was narrow. Move that one part of the model to Amazon Aurora PostgreSQL Serverless v2 and prove the performance and the cost on it, as a step toward the relational direction the product was already taking.

The good: end to end, through pipelines

The agent built the whole slice. A Terraform module and an isolated environment with its own state. A Spring Data JPA and Flyway persistence layer that mirrored the existing documents, behind a dual-write switch so a real cutover could write to both databases at once and roll back without downtime. Artillery load suite driven from EC2 over AWS SSM. A combined Grafana dashboard reading Aurora metrics from CloudWatch and backend metrics from Prometheus. All of it on feature branches, all of it applied through GitHub Actions.

The operating model matters more than the output. The agent was Claude Code, running in auto mode. Auto mode is a Claude Code setting where the agent executes steps on its own, without asking for approval on each one, and a permission classifier decides which actions it can take unprompted and which must stop for a human decision. Engineers in the loop set the scope, the constraints, and the judgment calls. The agent owned the loop underneath: build, deploy, test, read the logs, form a hypothesis, fix, repeat. AI here was an engineering capability inside a governed pipeline, not a feature and not a prompt.

The debugging was the proof that this was real work. A SAML-only login with no headless token. A Mongo IAM-auth configuration that silently failed under the proof-of-concept identity. A detached merge in JPA that dropped a child relation on save. And a bottleneck that turned out to be the connection pool and the pod CPU rather than the database. Each one was found and fixed through the same pipeline loop.

Evidence

The migrated endpoints held their latency under load. The non-migrated ones did not.

On a single pod, the migrated endpoints and the static metadata endpoints sustained roughly 972 requests per second at 100% under a one-second SLA, averaging about 9 milliseconds. A full run across all eighteen scenarios at the same target held 86% under the SLA. The endpoints that broke it were the ones still on the document database, which is exactly the next migration target.

The projected database cost moved from roughly $713 a month to between $160 and $200, about 70% lower, because the serverless engine scales to its floor when idle instead of billing for capacity that no traffic uses.

The bad: the access model is the risk model

Giving an agent a GitHub repository and a CI/CD pipeline gives it a path to production, to data, and to spend. That is not a hypothetical. It is the access model.

A push to the wrong branch deploys. A misfired infrastructure apply mutates shared state. Secrets pass through CI and land in logs. Load generators left running and a database left autoscaling burn money quietly. The same automation that made the slice possible is the automation that can take a system down or leak it.

What contained the risk was the guardrails, not the model's judgment. Two layers held it. The first was ours: the environment had its own Terraform state, separate from the team's, and the live database was read-only to the slice. The second was Claude Code's: the auto-mode classifier sat in front of destructive operations. Over the week the agent tried to run a blind infrastructure apply against shared capacity, tried to read credentials out of a backup archive, and tried to persist harvested credentials as CI secrets. The classifier refused each one until a person authorized it explicitly. Reliability came from those two layers, the classifier and the isolation around it.

The ugly: the agent cuts corners to reach the goal

An agent optimizes for the goal you state, and it will leave the plan to get there faster. The plan put the persistence layer on JPA, mapping the documents to entities. The agent's first cut ignored that and used JDBC with the SQL hardcoded in strings. It compiled, and it was quicker to produce. It was not the plan. We sent it back, and it converted to JPA.

That set the pattern, and it held across the work.

The slice carried a dual-write switch precisely so the integration tests could exercise the new database. The agent never flipped it. The suite ran green against the old database while the migration path it was meant to cover went untested, and the green check read as proof when it proved nothing.

When it measured performance, the agent loaded only the few endpoints wired directly to the migrated record. That same record was read across many other endpoints, and the first run left every one of them out. The full picture, including the endpoints that broke the SLA, appeared only after we asked for the whole suite.

Security was where it cut hardest. Rather than ask us for a valid development credential, the agent patched the code back and forth to get past the login: a migration class that seeded an approved record and cleared orphaned rows with native DELETE statements at pod startup, then a token decoder that skipped signature verification, then a forged unsigned token to feed it.

None of this was requested. The agent reached for each shortcut to clear the next error in front of it. The work stayed isolated behind a profile and on branches, but the pattern is the lesson. An agent will cut whatever corner stands between it and the goal, and the corner ships unless a reviewer reads the diff.

Lessons

• AI-driven modernization works end to end when the agent owns the build-deploy-test-diagnose loop and engineers in the loop own the scope.
• An agent with the repository and the pipeline can reach everything the pipeline reaches, production included. Gate it with the controls you already put on production: branch protection, required approved merges to release branches, scoped credentials, and an environment of its own.
• Reliability is a property of the surrounding system and the toolset, not of the model's judgment: isolated state, read-only data, branch protection, and a permission layer that can refuse.
• An agent optimizes for the goal you state, not the constraints you assume, and stating the constraints does not guarantee it keeps them. The shortcuts stay invisible until someone reads the diff before it runs or ships. That review is the work, not the formality.

Closing

An agent can run a database modernization end to end on its own, standing up the infrastructure, the persistence layer, the load tests, and the dashboards, and shipping every step through automation without a person ever opening the AWS console. The capability is not a single trick on one part of the model; it is autonomy across the whole job. Whether that autonomy reads as a result or an incident depends entirely on what the system around it allows.

‍

‍

‍

A customer's production-scheduling application ran on Apache Struts inside JBoss. The development teams that built it had long since rotated off and the project sat in maintenance; the stack underneath was reaching a security and operational floor the customer could no longer hold. Migration was the only remaining option.

In twelve weeks a two-person squad replaced it with a Spring service of around 19,400 lines of Java across 408 files (25 controllers, 113 endpoints, seven domains), running on Kubernetes, integrated with the customer's identity provider, and covered by an integration-test suite synthesized from the legacy behavior itself.

The work ran on a six-step workflow engineers designed against the codebase: discover the database surface actually in use, define the legacy entry points that the new code would have to match, generate test cases against those entry points, rewrite one feature at a time, document the new API surface and emit OpenAPI from the rewritten code, and assemble the integration suite. Simple LLM prompts where they fit; coding agents where they did not. Reviews happened per feature, not per diff.

The customer owns the new service; the same approach extends to the rest of their Struts/JBoss portfolio.

The system under the pilot

The customer is an automotive manufacturer. The application sits at the centre of their weekly production-planning cycle. Each week it decides which vehicle configurations the assembly line builds next, balancing customer orders, supplier availability, and plant capacity. Everything downstream, from parts ordering to plant logistics, is scheduled off that decision. Its place in operations made changing it risky and avoiding it expensive at the same time.

The stack underneath was familiar. Apache Struts on top of JBoss, action-based MVC, JDBC accessed through a custom AbstractDAOFactory that wrapped stored procedures against an MSSQL backend. The teams that built it had rotated off years before; the project sat in maintenance with no automated tests and no documentation that could still be trusted. The trigger for change was the platform itself. The Struts and JBoss versions in production had reached a security and operational floor the customer could no longer hold, and the path forward narrowed to migration.

Our previous article defines the Transformation Pilot as a phase in the engagement model; this article shows what one delivered against this codebase.

Scope of the pilot

The Pilot took one application end-to-end. Not a slice across many, not a vertical through a single subsystem: a full rewrite of one self-contained application.

Behind that scope was a broader goal. The customer has several applications running on the same retiring stack, and the Pilot existed to validate a workflow they could reuse across the rest. Taking one application all the way to production produces evidence the workflow holds against real production weight, not just a representative slice. Once it does, the same workflow stands ready for the next application.

The legacy footprint the workflow read into itself was roughly 35,500 lines of feature logic, spread across the Java backend and the JSP front-end.

The workflow that ran the work

The workflow engineers designed ran the work in six steps, in the order their outputs feed each other.

Discover the database surface in use. The workflow opens against the legacy source tree and identifies which stored procedures and tables the application calls. Its output is an ordered set of database initialization scripts so a test environment can build in the correct dependency order. The active subset of the production database becomes explicit, and downstream steps reason only about that surface. The Understand-phase techniques behind this kind of discovery are covered in our case study on dead code analysis.

Define the legacy entry points. Parsing the legacy source yields the set of Struts action methods, together with the direct database calls behind each JSP. Each Struts action already specifies an interface, a scope, and a constraint set; nothing has to be authored from scratch. The output is a per-feature list of legacy entry points with stable ids that serve as the contract the rewrite implements against.

Generate test cases against the legacy entry points. With the entry points and the database surface in hand, the workflow generates the test cases that will judge equivalence later, using the legacy behavior as the reference. Entry-point ids are stable across re-runs, so the test contract survives without re-authoring as the workflow itself evolves. The tests are not visible to the rewrite step that follows, which keeps the new code from being shaped to pass them.

Rewrite per feature. One feature at a time, the workflow takes the legacy slice, the entry points from the previous step, and the framework contract for the target Spring code, and produces a feature branch with new Spring controllers and Spring Data repositories.

‍Legacy:

AbstractDAOFactory factory =AbstractDAOFactory.getDAOFactory(
  AbstractDAOFactory.SQL_SERVER_ACCESS,"java:jboss/datasources/...");
ResultSet rs = stmt.executeQuery("SELECT ... WHERE code= '" + code + "'");

Target:

@Repository
interface AttributeRepository extendsJpaRepository<Attribute, Long> {
  @Query("SELECT a FROM Attribute a WHERE a.code = :code")
  Optional<Attribute> findByCode(@Param("code") Stringcode);
}
‍

Document the new API surface. The workflow extracts the new entry points from the rewritten controllers (request method, path, thrown exceptions) and emits OpenAPI including error responses. The OpenAPI document becomes the externally visible contract of the new service.

Assemble the integration suite. The final step pulls in the legacy entry points, the generated test cases, the new API surface, and the database initialization scripts. It maps legacy entry points to the new API, attaches the test cases to each, generates init and cleanup scripts per new endpoint, and assembles the result into a single class. The suite runs against testcontainers in CI, with the legacy behavior as the reference.

How the team worked inside the pilot

The unit of work was the feature, grouped along the legacy Struts entry points. One feature passed through the workflow at a time, each landing as a feature branch ready for review.

Per-feature review happened in the IDE. An engineer checked out the rewrite branch in IntelliJ, diffed it against the previous baseline (the last feature branch that had been merged), applied corrections in the IDE, and merged the result upward. Then the next feature began.

The default for every step was a simple LLM prompt with a well-scoped contract. Coding agents were reserved for steps where multiple files had to be read and written holistically, such as the integration-suite assembly and the larger feature merges. The rule the team converged on during the Pilot was practical: complex multi-file work to coding agents, well-bounded single-job work to single prompts.

The steps engineers designed read their own framework (prompts, contracts, supporting templates) from a feature branch rather than from main. That separation let engineers iterate on the framework while the rewrite continued, without main-branch friction. Improvements accumulated as the Pilot progressed, and the last features were rewritten faster than the first.

Two kinds of evaluation ran inside the work. Functional evaluation is a dedicated step at the end of each rewrite: the integration suite runs against the new code with the legacy behavior as the reference. LLM evaluation lives inside each LLM or agent step and judges that step's output against the contract for it: shape, scope, constraints. Engineers read the evidence each evaluation emits.

Engineers stayed in the loop where it mattered: at the boundary of each feature, against the evidence the evaluations produced. The structure of the work meant the review caught what mattered instead of being swallowed by intermediate noise.

What the customer received

The customer received a working service behind the integration-test suite the workflow assembled. It is a clean Spring service: focused controllers, the domain code behind them, and a data layer that no longer depends on the legacy stored-procedure factory. The JBoss runtime is no longer in the picture, and structured error handling replaces the raw stack traces the legacy version surfaced when something went wrong.

Alongside the service, the customer received architecture documentation, deployment instructions, and the open points that need attention beyond the Pilot. A manual regression test plan keyed to the legacy version was prepared for the cutover window so the customer's QA process had something concrete to anchor to. Configuration was made explicit, with every setting and its purpose documented.

The customer's engineering team took ownership at handover. They have a service their engineers can extend, a runtime their operations team can operate, and documentation grounded in code that exists.

From the pilot forward

The Pilot landed, and with it the workflow the customer was looking for. The rewritten service is the working proof. The workflow that produced it is what carries forward to the rest of the portfolio.

The same approach applies to whatever else sits on the same retiring stack, at whatever pace business priority and operational risk allow. The integration suite is theirs; the workflow is theirs; the patterns the first pass produced are already familiar to their team. What is left to decide is the order, not the approach.

‍

If the system you are looking at fits the shape described here, a Java web stack on Struts and JBoss in maintenance with the platform underneath becoming a security and operations problem, reach out. We will scope a focused Transformation Pilot on a single application, time-boxed, with deliverables your team can verify.

‍

‍

‍

‍

‍