Uncovering the Seams in Mainframes for Incremental Modernisation

Contextual Background

The Mainframe hosted a diverse range of
services crucial to the client’s business operations. Our programme
specifically focused on the data platform designed for insights on Consumers
in UK&I (United Kingdom & Ireland). This particular subsystem on the
Mainframe comprised approximately 7 million lines of code, developed over a
span of 40 years. It provided roughly ~50% of the capabilities of the UK&I
estate, but accounted for ~80% of MIPS (Million instructions per second)
from a runtime perspective. The system was significantly complex, the
complexity was further exacerbated by domain responsibilities and concerns
spread across multiple layers of the legacy environment.

Several reasons drove the client’s decision to transition away from the
Mainframe environment, these are the following:

1. Changes to the system were slow and expensive. The business therefore had
   challenges keeping pace with the rapidly evolving market, preventing
   innovation.
2. Operational costs associated with running the Mainframe system were high;
   the client faced a commercial risk with an imminent price increase from a core
   software vendor.
3. Whilst our client had the necessary skill sets for running the Mainframe,
   it had proven to be hard to find new professionals with expertise in this tech
   stack, as the pool of skilled engineers in this domain is limited. Furthermore,
   the job market does not offer as many opportunities for Mainframes, thus people
   are not incentivised to learn how to develop and operate them.

High-level view of Consumer Subsystem

The following diagram shows, from a high-level perspective, the various
components and actors in the Consumer subsystem.

The Mainframe supported two distinct types of workloads: batch
processing and, for the product API layers, online transactions. The batch
workloads resembled what is commonly referred to as a data pipeline. They
involved the ingestion of semi-structured data from external
providers/sources, or other internal Mainframe systems, followed by data
cleansing and modelling to align with the requirements of the Consumer
Subsystem. These pipelines incorporated various complexities, including
the implementation of the Identity searching logic: in the United Kingdom,
unlike the United States with its social security number, there is no
universally unique identifier for residents. Consequently, companies
operating in the UK&I must employ customised algorithms to accurately
determine the individual identities associated with that data.

The online workload also presented significant complexities. The
orchestration of API requests was managed by multiple internally developed
frameworks, which determined the program execution flow by lookups in
datastores, alongside handling conditional branches by analysing the
output of the code. We should not overlook the level of customisation this
framework applied for each customer. For example, some flows were
orchestrated with ad-hoc configuration, catering for implementation
details or specific needs of the systems interacting with our client’s
online products. These configurations were exceptional at first, but they
likely became the norm over time, as our client augmented their online
offerings.

This was implemented through an Entitlements engine which operated
across layers to ensure that customers accessing products and underlying
data were authenticated and authorised to retrieve either raw or
aggregated data, which would then be exposed to them through an API
response.

Incremental Legacy Displacement: Principles, Benefits, and
Considerations

Considering the scope, risks, and complexity of the Consumer Subsystem,
we believed the following principles would be tightly linked with us
succeeding with the programme:

• Early Risk Reduction: With engineering starting from the
  beginning, the implementation of a “Fail-Fast” approach would help us
  identify potential pitfalls and uncertainties early, thus preventing
  delays from a programme delivery standpoint. These were:
• Outcome Parity: The client emphasised the importance of
  upholding outcome parity between the existing legacy system and the
  new system (It is important to note that this concept differs from
  Feature Parity). In the client’s Legacy system, various
  attributes were generated for each consumer, and given the strict
  industry regulations, maintaining continuity was essential to ensure
  contractual compliance. We needed to proactively identify
  discrepancies in data early on, promptly address or explain them, and
  establish trust and confidence with both our client and their
  respective customers at an early stage.
• Cross-functional requirements: The Mainframe is a highly
  performant machine, and there were uncertainties that a solution on
  the Cloud would satisfy the Cross-functional requirements.
• Deliver Value Early: Collaboration with the client would
  ensure we could identify a subset of the most critical Business
  Capabilities we could deliver early, ensuring we could break the system
  apart into smaller increments. These represented thin-slices of the
  overall system. Our goal was to build upon these slices iteratively and
  frequently, helping us accelerate our overall learning in the domain.
  Furthermore, working through a thin-slice helps reduce the cognitive
  load required from the team, thus preventing analysis paralysis and
  ensuring value would be consistently delivered. To achieve this, a
  platform built around the Mainframe that provides better control over
  clients’ migration strategies plays a vital role. Using patterns such as
  Dark Launching and Canary
  Release would place us in the driver’s seat for a smooth
  transition to the Cloud. Our goal was to achieve a silent migration
  process, where customers would seamlessly transition between systems
  without any noticeable impact. This could only be possible through
  comprehensive comparison testing and continuous monitoring of outputs
  from both systems.

With the above principles and requirements in mind, we opted for an
Incremental Legacy Displacement approach in conjunction with Dual
Run. Effectively, for each slice of the system we were rebuilding on the
Cloud, we were planning to feed both the new and as-is system with the
same inputs and run them in parallel. This allows us to extract both
systems’ outputs and check if they are the same, or at least within an
acceptable tolerance. In this context, we defined Incremental Dual
Run as: using a Transitional
Architecture to support slice-by-slice displacement of capability
away from a legacy environment, thereby enabling target and as-is systems
to run temporarily in parallel and deliver value.

We decided to adopt this architectural pattern to strike a balance
between delivering value, discovering and managing risks early on,
ensuring outcome parity, and maintaining a smooth transition for our
client throughout the duration of the programme.

Incremental Legacy Displacement approach

To accomplish the offloading of capabilities to our target
architecture, the team worked closely with Mainframe SMEs (Subject Matter
Experts) and our client’s engineers. This collaboration facilitated a
just enough understanding of the current as-is landscape, in terms of both
technical and business capabilities; it helped us design a Transitional
Architecture to connect the existing Mainframe to the Cloud-based system,
the latter being developed by other delivery workstreams in the
programme.

Our approach began with the decomposition of the
Consumer subsystem into specific business and technical domains, including
data load, data retrieval & aggregation, and the product layer
accessible through external-facing APIs.

Because of our client’s business
purpose, we recognised early that we could exploit a major technical boundary to organise our programme. The
client’s workload was largely analytical, processing mostly external data
to produce insight which was sold on to clients. We therefore saw an
opportunity to split our transformation programme in two parts, one around
data curation, the other around data serving and product use cases using
data interactions as a seam. This was the first high level seam identified.

Following that, we then needed to further break down the programme into
smaller increments.

On the data curation side, we identified that the data sets were
managed largely independently of each other; that is, while there were
upstream and downstream dependencies, there was no entanglement of the datasets during curation, i.e.
ingested data sets had a one to one mapping to their input files.
.

We then collaborated closely with SMEs to identify the seams
within the technical implementation (laid out below) to plan how we could
deliver a cloud migration for any given data set, eventually to the level
where they could be delivered in any order (Database Writers Processing Pipeline Seam, Coarse Seam: Batch Pipeline Step Handoff as Seam,
and Most Granular: Data Characteristic
Seam). As long as up- and downstream dependencies could exchange data
from the new cloud system, these workloads could be modernised
independently of each other.

On the serving and product side, we found that any given product used
80% of the capabilities and data sets that our client had created. We
needed to find a different approach. After investigation of the way access
was sold to customers, we found that we could take a “customer segment”
approach to deliver the work incrementally. This entailed finding an
initial subset of customers who had purchased a smaller percentage of the
capabilities and data, reducing the scope and time needed to deliver the
first increment. Subsequent increments would build on top of prior work,
enabling further customer segments to be cut over from the as-is to the
target architecture. This required using a different set of seams and
transitional architecture, which we discuss in Database Readers and Downstream processing as a Seam.

Effectively, we ran a thorough analysis of the components that, from a
business perspective, functioned as a cohesive whole but were built as
distinct elements that could be migrated independently to the Cloud and
laid this out as a programme of sequenced increments.

Seams

Our transitional architecture was mostly influenced by the Legacy seams we could uncover within the Mainframe. You
can think of them as the junction points where code, programs, or modules
meet. In a legacy system, they may have been intentionally designed at
strategic places for better modularity, extensibility, and
maintainability. If this is the case, they will likely stand out
throughout the code, although when a system has been under development for
a number of decades, these seams tend to hide themselves amongst the
complexity of the code. Seams are particularly valuable because they can
be employed strategically to alter the behaviour of applications, for
example to intercept data flows within the Mainframe allowing for
capabilities to be offloaded to a new system.

Determining technical seams and valuable delivery increments was a
symbiotic process; possibilities in the technical area fed the options
that we could use to plan increments, which in turn drove the transitional
architecture needed to support the programme. Here, we step a level lower
in technical detail to discuss solutions we planned and designed to enable
Incremental Legacy Displacement for our client. It is important to note that these were continuously refined
throughout our engagement as we acquired more knowledge; some went as far as being deployed to test
environments, whilst others were spikes. As we adopt this approach on other large-scale Mainframe modernisation
programmes, these approaches will be further refined with our most up to date hands-on experience.

External interfaces

We examined the external interfaces exposed by the Mainframe to data
Providers and our client’s Customers. We could apply Event Interception on these integration points
to allow the transition of external-facing workload to the cloud, so the
migration would be silent from their perspective. There were two types
of interfaces into the Mainframe: a file-based transfer for Providers to
supply data to our client, and a web-based set of APIs for Customers to
interact with the product layer.

Batch input as seam

The first external seam that we found was the file-transfer
service.

Providers could transfer files containing data in a semi-structured
format via two routes: a web-based GUI (Graphical User Interface) for
file uploads interacting with the underlying file transfer service, or
an FTP-based file transfer to the service directly for programmatic
access.

The file transfer service determined, on a per provider and file
basis, what datasets on the Mainframe should be updated. These would
in turn execute the relevant pipelines through dataset triggers, which
were configured on the batch job scheduler.

Assuming we could rebuild each pipeline as a whole on the Cloud
(note that later we will dive deeper into breaking down larger
pipelines into workable chunks), our approach was to build an
individual pipeline on the cloud, and dual run it with the mainframe
to verify they were producing the same outputs. In our case, this was
possible through applying additional configurations on the File
transfer service, which forked uploads to both Mainframe and Cloud. We
were able to test this approach using a production-like File transfer
service, but with dummy data, running on test environments.

This would allow us to Dual Run each pipeline both on Cloud and
Mainframe, for as long as required, to gain confidence that there were
no discrepancies. Eventually, our approach would have been to apply an
additional configuration to the File transfer service, preventing
further updates to the Mainframe datasets, therefore leaving as-is
pipelines deprecated. We did not get to test this last step ourselves
as we did not complete the rebuild of a pipeline end to end, but our
technical SMEs were familiar with the configurations required on the
File transfer service to effectively deprecate a Mainframe
pipeline.

API Access as Seam

Furthermore, we adopted a similar strategy for the external facing
APIs, identifying a seam around the pre-existing API Gateway exposed
to Customers, representing their entrypoint to the Consumer
Subsystem.

Drawing from Dual Run, the approach we designed would be to put a
proxy high up the chain of HTTPS calls, as close to users as possible.
We were looking for something that could parallel run both streams of
calls (the As-Is mainframe and newly built APIs on Cloud), and report
back on their outcomes.

Effectively, we were planning to use Dark
Launching for the new Product layer, to gain early confidence
in the artefact through extensive and continuous monitoring of their
outputs. We did not prioritise building this proxy in the first year;
to exploit its value, we needed to have the majority of functionality
rebuilt at the product level. However, our intentions were to build it
as soon as any meaningful comparison tests could be run at the API
layer, as this component would play a key role for orchestrating dark
launch comparison tests. Additionally, our analysis highlighted we
needed to watch out for any side-effects generated by the Products
layer. In our case, the Mainframe produced side effects, such as
billing events. As a result, we would have needed to make intrusive
Mainframe code changes to prevent duplication and ensure that
customers would not get billed twice.

Similarly to the Batch input seam, we could run these requests in
parallel for as long as it was required. Ultimately though, we would
use Canary
Release at the
proxy layer to cut over customer-by-customer to the Cloud, hence
reducing, incrementally, the workload executed on the Mainframe.

Internal interfaces

Following that, we conducted an analysis of the internal components
within the Mainframe to pinpoint the specific seams we could leverage to
migrate more granular capabilities to the Cloud.

Coarse Seam: Data interactions as a Seam

One of the primary areas of focus was the pervasive database
accesses across programs. Here, we started our analysis by identifying
the programs that were either writing, reading, or doing both with the
database. Treating the database itself as a seam allowed us to break
apart flows that relied on it being the connection between
programs.

Database Readers

Regarding Database readers, to enable new Data API development in
the Cloud environment, both the Mainframe and the Cloud system needed
access to the same data. We analysed the database tables accessed by
the product we picked as a first candidate for migrating the first
customer segment, and worked with client teams to deliver a data
replication solution. This replicated the required tables from the test database to the Cloud using Change
Data Capture (CDC) techniques to synchronise sources to targets. By
leveraging a CDC tool, we were able to replicate the required
subset of data in a near-real time fashion across target stores on
Cloud. Also, replicating data gave us opportunities to redesign its
model, as our client would now have access to stores that were not
only relational (e.g. Document stores, Events, Key-Value and Graphs
were considered). Criterias such as access patterns, query complexity,
and schema flexibility helped determine, for each subset of data, what
tech stack to replicate into. During the first year, we built
replication streams from DB2 to both Kafka and Postgres.

At this point, capabilities implemented through programs
reading from the database could be rebuilt and later migrated to
the Cloud, incrementally.

Database Writers

In regards to database writers, which were mostly made up of batch
workloads running on the Mainframe, after careful analysis of the data
flowing through and out of them, we were able to apply Extract Product Lines to identify
separate domains that could execute independently of each other
(running as part of the same flow was just an implementation detail we
could change).

Working with such atomic units, and around their respective seams,
allowed other workstreams to start rebuilding some of these pipelines
on the cloud and comparing the outputs with the Mainframe.

In addition to building the transitional architecture, our team was
responsible for providing a range of services that were used by other
workstreams to engineer their data pipelines and products. In this
specific case, we built batch jobs on Mainframe, executed
programmatically by dropping a file in the file transfer service, that
would extract and format the journals that those pipelines were
producing on the Mainframe, thus allowing our colleagues to have tight
feedback loops on their work through automated comparison testing.
After ensuring that outcomes remained the same, our approach for the
future would have been to enable other teams to cutover each
sub-pipeline one by one.

The artefacts produced by a sub-pipeline may be required on the
Mainframe for further processing (e.g. Online transactions). Thus, the
approach we opted for, when these pipelines would later be complete
and on the Cloud, was to use Legacy Mimic
and replicate data back to the Mainframe, for as long as the capability dependant on this data would be
moved to Cloud too. To achieve this, we were considering employing the same CDC tool for replication to the
Cloud. In this scenario, records processed on Cloud would be stored as events on a stream. Having the
Mainframe consume this stream directly seemed complex, both to build and to test the system for regressions,
and it demanded a more invasive approach on the legacy code. In order to mitigate this risk, we designed an
adaption layer that would transform the data back into the format the Mainframe could work with, as if that
data had been produced by the Mainframe itself. These transformation functions, if
straightforward, may be supported by your chosen replication tool, but
in our case we assumed we needed custom software to be built alongside
the replication tool to cater for additional requirements from the
Cloud. This is a common scenario we see in which businesses take the
opportunity, coming from rebuilding existing processing from scratch,
to improve them (e.g. by making them more efficient).

In summary, working closely with SMEs from the client-side helped
us challenge the existing implementation of Batch workloads on the
Mainframe, and work out alternative discrete pipelines with clearer
data boundaries. Note that the pipelines we were dealing with did not
overlap on the same records, due to the boundaries we had defined with
the SMEs. In a later section, we will examine more complex cases that
we have had to deal with.

Coarse Seam: Batch Pipeline Step Handoff

Likely, the database won’t be the only seam you can work with. In
our case, we had data pipelines that, in addition to persisting their
outputs on the database, were serving curated data to downstream
pipelines for further processing.

For these scenarios, we first identified the handshakes between
pipelines. These consist usually of state persisted in flat / VSAM
(Virtual Storage Access Method) files, or potentially TSQs (Temporary
Storage Queues). The following shows these hand-offs between pipeline
steps.

As an example, we were looking at designs for migrating a downstream pipeline reading a curated flat file
stored upstream. This downstream pipeline on the Mainframe produced a VSAM file that would be queried by
online transactions. As we were planning to build this event-driven pipeline on the Cloud, we chose to
leverage the CDC tool to get this data off the mainframe, which in turn would get converted into a stream of
events for the Cloud data pipelines to consume. Similarly to what we have reported before, our Transitional
Architecture needed to use an Adaptation layer (e.g. Schema translation) and the CDC tool to copy the
artefacts produced on Cloud back to the Mainframe.

Through employing these handshakes that we had previously
identified, we were able to build and test this interception for one
exemplary pipeline, and design further migrations of
upstream/downstream pipelines on the Cloud with the same approach,
using Legacy
Mimic
to feed back the Mainframe with the necessary data to proceed with
downstream processing. Adjacent to these handshakes, we were making
non-trivial changes to the Mainframe to allow data to be extracted and
fed back. However, we were still minimising risks by reusing the same
batch workloads at the core with different job triggers at the edges.

Granular Seam: Data Characteristic

In some cases the above approaches for internal seam findings and
transition strategies do not suffice, as it happened with our project
due to the size of the workload that we were looking to cutover, thus
translating into higher risks for the business. In one of our
scenarios, we were working with a discrete module feeding off the data
load pipelines: Identity curation.

Consumer Identity curation was a
complex space, and in our case it was a differentiator for our client;
thus, they could not afford to have an outcome from the new system
less accurate than the Mainframe for the UK&I population. To
successfully migrate the entire module to the Cloud, we would need to
build tens of identity search rules and their required database
operations. Therefore, we needed to break this down further to keep
changes small, and enable delivering frequently to keep risks low.

We worked closely with the SMEs and Engineering teams with the aim
to identify characteristics in the data and rules, and use them as
seams, that would allow us to incrementally cutover this module to the
Cloud. Upon analysis, we categorised these rules into two distinct
groups: Simple and Complex.
Simple rules could run on both systems, provided
they fed on different data segments (i.e. separate pipelines
upstream), thus they represented an opportunity to further break apart
the identity module space. They represented the majority (circa 70%)
triggered during the ingestion of a file. These rules were responsible
for establishing an association between an already existing identity,
and a new data record.
On the other hand, the Complex rules were triggered by cases where
a data record indicated the need for an identity change, such as
creation, deletion, or updation. These rules required careful handling
and could not be migrated incrementally. This is because an update to
an identity can be triggered by multiple data segments, and operating
these rules in both systems in parallel could lead to identity drift
and data quality loss. They required a single system minting
identities at one point in time, thus we designed for a big bang
migration approach.

In our original understanding of the Identity module on the
Mainframe, pipelines ingesting data triggered changes on DB2 resulting
in an up to date view of the identities, data records, and their
associations.

Additionally, we identified a discrete Identity module and refined
this model to reflect a deeper understanding of the system that we had
discovered with the SMEs. This module fed data from multiple data
pipelines, and applied Simple and Complex rules to DB2.

Now, we could apply the same techniques we wrote about earlier for
data pipelines, but we required a more granular and incremental
approach for the Identity one.
We planned to tackle the Simple rules that could run on both
systems, with a caveat that they operated on different data segments,
as we were constrained to having only one system maintaining identity
data. We worked on a design that used Batch Pipeline Step Handoff and
applied Event Interception to capture and fork the data (temporarily
until we can confirm that no data is lost between system handoffs)
feeding the Identity pipeline on the Mainframe. This would allow us to
take a divide and conquer approach with the files ingested, running a
parallel workload on the Cloud which would execute the Simple rules
and apply changes to identities on the Mainframe, and build it
incrementally. There were many rules that fell under the Simple
bucket, therefore we needed a capability on the target Identity module
to fall back to the Mainframe in case a rule which was not yet
implemented needed to be triggered. This looked like the
following:

As new builds of the Cloud Identity module get released, we would
see less rules belonging to the Simple bucket being applied through
the fallback mechanism. Eventually only the Complex ones will be
observable through that leg. As we previously mentioned, these needed
to be migrated all in one go to minimise the impact of identity drift.
Our plan was to build Complex rules incrementally against a Cloud
database replica and validate their outcomes through extensive
comparison testing.

Once all rules were built, we would release this code and disable
the fallback strategy to the Mainframe. Bear in mind that upon
releasing this, the Mainframe Identities and Associations data becomes
effectively a replica of the new Primary store managed by the Cloud
Identity module. Therefore, replication is needed to keep the
mainframe functioning as is.

As previously mentioned in other sections, our design employed
Legacy Mimic and an Anti-Corruption Layer that would translate data
from the Mainframe to the Cloud model and vice versa. This layer
consisted of a series of Adapters across the systems, ensuring data
would flow out as a stream from the Mainframe for the Cloud to consume
using event-driven data pipelines, and as flat files back to the
Mainframe to allow existing Batch jobs to process them. For
simplicity, the diagrams above don’t show these adapters, but they
would be implemented each time data flowed across systems, regardless
of how granular the seam was. Unfortunately, our work here was mostly
analysis and design and we were not able to take it to the next step
and validate our assumptions end to end, apart from running Spikes to
ensure that a CDC tool and the File transfer service could be
employed to send data in and out of the Mainframe, in the required
format. The time required to build the required scaffolding around the
Mainframe, and reverse engineer the as-is pipelines to gather the
requirements was considerable and beyond the timeframe of the first
phase of the programme.

Granular Seam: Downstream processing handoff

Similar to the approach employed for upstream pipelines to feed
downstream batch workloads, Legacy Mimic Adapters were employed for
the migration of the Online flow. In the existing system, a customer
API call triggers a series of programs producing side-effects, such as
billing and audit trails, which get persisted in appropriate
datastores (mostly Journals) on the Mainframe.

To successfully transition incrementally the online flow to the
Cloud, we needed to ensure these side-effects would either be handled
by the new system directly, thus increasing scope on the Cloud, or
provide adapters back to the Mainframe to execute and orchestrate the
underlying program flows responsible for them. In our case, we opted
for the latter using CICS web services. The solution we built was
tested for functional requirements; cross-functional ones (such as
Latency and Performance) could not be validated as it proved
challenging to get production-like Mainframe test environments in the
first phase. The following diagram shows, according to the
implementation of our Adapter, what the flow for a migrated customer
would look like.

It is worth noting that Adapters were planned to be temporary
scaffolding. They would not have served a valid purpose when the Cloud
was able to handle these side-effects by itself after which point we
planned to replicate the data back to the Mainframe for as long as
required for continuity.

Data Replication to enable new product development

Building on the incremental approach above, organisations may have
product ideas that are based primarily on analytical or aggregated data
from the core data held on the Mainframe. These are typically where there
is less of a need for up-to-date information, such as reporting use cases
or summarising data over trailing periods. In these situations, it’s
possible to unlock business benefits earlier through the judicious use of
data replication.
When done well, this can enable new product development through a
relatively smaller investment earlier which in turn brings momentum to the
modernisation effort.
In our recent project, our client had already departed on this journey,
using a CDC tool to replicate core tables from DB2 to the Cloud.

While this was great in terms of enabling new products to be launched,
it wasn’t without its downsides.

Unless you take steps to abstract the schema when replicating a
database, then your new cloud products will be coupled to the legacy
schema as soon as they’re built. This will likely hamper any subsequent
innovation that you may wish to do in your target environment as you’ve
now got an additional drag factor on changing the core of the application;
but this time it’s worse as you won’t want to invest again in changing the
new product you’ve just funded. Therefore, our proposed design consisted
of further projections from the replica database into optimised stores and
schemas, upon which new products would be built.

This would give us the opportunity to refactor the Schema, and at times
move parts of the data model into non-relational stores, which would
better handle the query patterns observed with the SMEs.

Upon
migration of batch workloads, in order to keep all stores in sync, you may
want to consider either a write back strategy to the new Primary directly
(what was previously known as the Replica), which in turn feeds back DB2
on the Mainframe (though there will be higher coupling from the batches to
the old schema), or revert the CDC & Adaptation layer direction from the
Optimised store as a source and the new Primary as a target (you will
likely need to manage replication separately for each data segment i.e.
one data segment replicates from Replica to Optimised store, another
segment the other way around).

Conclusion

There are multiple things to consider when offloading from the
mainframe. Depending on the size of the system that you wish to migrate
off the mainframe, this work can take a considerable amount of time, and
Incremental Dual Run costs are non-negligible. How much this will cost
depends on various factors, but you cannot expect to save on costs via
dual running two systems in parallel. Thus, the business should look at
generating value early to get buy-in from stakeholders, and fund a
multi-year modernisation programme. We see Incremental Dual Run as an
enabler for teams to respond fast to the demand of the business, going
hand in hand with Agile and Continuous Delivery practices.

Firstly, you have to understand the overall system landscape and what
the entry points to your system are. These interfaces play an essential
role, allowing for the migration of external users/applications to the new
system you are building. You are free to redesign your external contracts
throughout this migration, but it will require an adaptation layer between
the Mainframe and Cloud.

Secondly, you have to identify the business capabilities the Mainframe
system offers, and identify the seams between the underlying programs
implementing them. Being capability-driven helps ensure that you are not
building another tangled system, and keeps responsibilities and concerns
separate at their appropriate layers. You will find yourself building a
series of Adapters that will either expose APIs, consume events, or
replicate data back to the Mainframe. This ensures that other systems
running on the Mainframe can keep functioning as is. It is best practice
to build these adapters as reusable components, as you can employ them in
multiple areas of the system, according to the specific requirements you
have.

Thirdly, assuming the capability you are trying to migrate is stateful, you will likely require a replica of the
data that the Mainframe has access to. A CDC tool to replicate data can be employed here. It is important to
understand the CFRs (Cross Functional Requirements) for data replication, some data may need a fast replication
lane to the Cloud and your chosen tool should provide this, ideally. There are now a lot of tools and frameworks
to consider and investigate for your specific scenario. There are a plethora of CDC tools that can be assessed,
for instance we looked at Qlik Replicate for DB2 tables and Precisely Connect more specifically for VSAM stores.

Cloud Service Providers are also launching new offerings in this area;
for instance, Dual Run by Google Cloud recently launched its own
proprietary data replication approach.

For a more holistic view on mobilising a team of teams to deliver a
programme of work of this scale, please refer to the article “Eating the Elephant” by our colleague, Sophie
Holden.

Ultimately, there are other things to consider which were briefly
mentioned as part of this article. Amongst these, the testing strategy
will play a role of paramount importance to ensure you are building the
new system right. Automated testing shortens the feedback loop for
delivery teams building the target system. Comparison testing ensures both
systems exhibit the same behaviour from a technical perspective. These
strategies, used in conjunction with Synthetic data generation and
Production data obfuscation techniques, give finer control over the
scenarios you intend to trigger and validate their outcomes. Last but not
least, production comparison testing ensures the system running in Dual
Run, over time, produces the same outcome as the legacy one on its own.
When needed, outcomes are compared from an external observer’s point of
view as a minimum, such as a customer interacting with the system.
Additionally, we can compare intermediary system outcomes.

Hopefully, this article brings to life what you would need to consider
when embarking on a Mainframe offloading journey. Our involvement was at the very first few months of a
multi-year programme and some of the solutions we have discussed were at a very early stage of inception.
Nevertheless, we learnt a great deal from this work and we find these ideas worth sharing. Breaking down your
journey into viable valuable steps will always require context, but we
hope our learnings and approaches can help you getting started so you can
take this the extra mile, into production, and enable your own
roadmap.