Systems Integration — Challenges and Solutions to Assembling Large Systems

1. Systems Integration

1.1 Definition

Systems integration addresses the challenges of assembling a (typically large and complex) solution from components or subsystems that may be independently sourced. Their architectural patterns, low-level design, and technology stack can vary widely.

A large software solution where components A and B, C and D, and E were designed and developed independently and integrated into a target solution with third-party software F and G via external interfaces.

The target system is the set of all components interacting via software interfaces that satisfies predefined business requirements.

The complexity of an integration task arises from the following factors:

The subsystems will become ready and available for testing at different times — efficient task synchronization is required.
The number of parts is significant, with some being black boxes supplied by independent vendors while others might be built in-house — expert technical knowledge must be available.
Faults may occur in any component, and diagnosing the problem is not trivial as it may require inspecting all subsystems associated with the feature — extensive verification and quality assurance effort must be invested.

On top of that, constraints typical of large software programs apply:

Time constraints limit the amount of testing and quality assurance the project can afford.
Budget constraints may impact the allocation of talent (senior technical staff, project managers, testers, DevOps engineers)

These factors combine to produce high levels of project risk that must be adequately managed.

1.2 Project Activities

The following activities occur during a systems integration project:

The first step is acquiring the latest software application, component, or subsystem release. This includes artifacts, documents (functional specifications, interface definition documents, implementation guide, operations guide, release notes), and licenses.
The next step is to install the application, configure it, and run some basic tests to ensure its primary services are running correctly and it is accessible over the network.
In this third step, we smoke-test its interfaces through simulators or command line programs like curl.
Fourthly, we connect the new component to the rest of the system and run our integration tests.
Finally, relevant documentation is updated.

Incremental or mult-stage approach to systems integration — An incremental or multi-stage approach to systems integration

The above is typically an iterative process run for every subsystem, although a single integration approach might work in particular cases. The sequence of activities significantly impacts the integration process’ efficiency and will be the topic of our next discussion.

2. Approaches to Systems Integration

2.1 Single-Stage Approach

In a single-stage approach, we assume the constituent parts will arrive simultaneously, and testing will be carried out once the system has been fully assembled.

This approach can be efficient if the below criteria are met:

The number of subsystems to be integrated is small.
Technical staff are familiar with the tasks and the components to be installed; therefore, the uncertainty and project risk is minimal.

The single-stage approach might be inevitable in some cases, despite being highly inefficient. This scenario can occur under the following conditions:

Technical constraints — Examples might be the unavailability of adequate test labs (like those required for performance testing) or poor quality simulators to stand in for the actual interfaces.
Project constraints — For example, severe limitations on testing resource availability might drive managers to cut down on the verification and validation.

2.2 Multi-Stage Approach

Bringing in experts from overarching domains to integrate, test, and troubleshoot a large system in a single stage is highly inefficient.

Consider the case where three experts integrate three platforms into a single solution. Any time an issue is identified, one expert will probably be working while the other three are waiting on them to complete their analysis. In this simplistic (yet highly realistic) scenario, the efficiency is at only 33%.

If there were four platforms and four experts, the efficiency would drop to 25%. For five experts, it’s 20%.

In a multi-stage (or incremental) approach, we assume the below to be true:

The system is complex enough that a single-stage approach is impractical, and components must be assembled in larger subsystems before putting the entire solution together.
The subsystems involved are developed and tested by independent teams, and no guarantee can be made on which systems will be ready at what time.

In a multi-stage approach, subsystems are assembled and tested incrementally as soon as they are ready. A new subsystem is integrated with the rest during each stage until the target solution is complete.

Task synchronization, dependency management, and stakeholder management are crucial for the success of large integration projects.

2.3 Summary

There are apparent similarities (with subtle differences) between systems integration approaches (single vs multi-stage) and software delivery methodologies (Agile, Waterfall, and DevOps).

To understand better the context in which one approach is superior to another, we shall closely consider this duality (single-stage and Waterfall, multi-stage and Agile).

In both single-stage and Waterfall, we assume great familiarity with the software applications and the integration tasks ahead; uncertainty is relatively low, requirement volatility virtually non-existent, and project risk is minimal.

Operational Excellence is guaranteed in these contexts with expertise, standardized processes, and best practices.

Multi-stage integration processes and Agile are also closely related but with subtle nuances.

In a multi-stage approach, the overhead incurred from additional testing of isolated components or larger subsystems is still smaller than the cost of a single-stage method for the reasons detailed earlier. This cost/benefit analysis is also accurate for Agile, albeit for different reasons.

The underlying concept in Agile (where requirement volatility is high) is that smaller, faster iterations are more efficient than a single, slower cycle since you can incorporate valuable knowledge gained with every iteration while slowly spiralling down towards the target solution.

The same concept is used in multi-stage integration. This time, we are optimising our time usage by kicking off integration on smaller scales as quickly as possible and only bringing in more people (and more subsystems) when confidence in the isolated subsystems peaks.

3. Verification and Validation

Although generic software testing concepts remain valid, this section will elaborate on some of the more relevant ones for large integration projects.

3.1 Systems Integration Testing (SIT)

In the (recommended) multi-stage incremental approach to systems integration, testing works as follows:

A subsystem undergoes verification and validation under lab conditions using black box and unit testing. At this stage, it is isolated from the remaining constituents of the target system. It could even be developed offsite, with developers knowing very little about the other components of the solution. Once developers complete their implementation, the artifacts are supplied to the integrator or client with Functional Specifications and Interface Definition Documents.
Next, the new subsystem is installed and connected to the other components while technical testers verify the system’s interface for accessibility and basic functionality. Failures are identified and fixed in cooperation with the vendor. This stage is known as Systems Integration Testing or SIT.

The scope of an SIT is restricted to connectivity, accessibility, and basic interface functionality tests. However, the lines are becoming increasingly blurred with automation and the ease the latter brings in running entire suites of test cases on demand.

The systems integration test staff must be technically proficient and understand IT systems well.

SITs are typically conducted in lab settings that may differ significantly from production. Test data is almost exclusively used.

During SITs, we are still unsure whether the system is integrated correctly and adheres well to the specifications. Patches and further test iterations are also expected.

End-to-End tests and User Acceptance Testing (UAT) can only start once the system is fully integrated and tested.

3.2 End-To-End Testing

Once all the pieces have been assembled, and the target system emerges, testing shifts to a distinct mode. We are no longer solely interested in ensuring the interfaces are connected and perform as expected. Instead, our primary focus is ascertaining the entire system’s stability, reliability, and resilience through end-to-end testing.

The questions we are interested in can be:

Are all components running smoothly with no exceptions or crashes?
Are all the services behaving as expected, synchronizing tasks, and cooperating to provide the requested end-user business functionality?
Can the system recover from a failure in one of its components or fall back to alternative modes of operation?

In addition to fundamental software testing skills, end-to-end testers:

Have a thorough understanding of the platforms across the entire landscape.
Can pinpoint the specific subsystem responsible for the error.
Are technically proficient; they can differentiate between low-level hardware or software errors and faulty features or implementation.
Might be specialized in one module more than others.
Are familiar with the business requirements and the software development lifecycle of the various products.

Although highly variable between teams and organizations, the scope of end-to-end testing does not cover the entire gamut of functionality but focuses primarily on the system’s essential features, with industry-relevant system certifications and User Acceptance Testing (UAT) covering the rest.

Finally, the data used in end-to-end testing and the third-party or external interfaces are typically more similar to production and less similar to dummy test data.

3.3 The Role of Test Tools and Simulators

Developers have been crafty in creating testing tools, from simple programs that use TCP/IP connections to send dummy messages and receive dummy responses to elaborate transaction simulators that generate production-like data and perform serious validation on responses.

Simulators have played a vital role in allowing interfaces to be tested offline during the implementation stages of the software development lifecycle.

Naturally, sophisticated simulators generate higher-quality production-similar requests and API calls, but they are typically commercial products with a decent learning curve.

Multi-stage incremental integration cycles would have been highly challenging without testing tools like this.

Standardized interface protocols also helped promote the reusability of simulators and made them viable commercial products. Using standardized interfaces (even internally) has advantages by allowing message simulator reuse (rather than redevelopment).