End-to-End Testing: Goals and Specifics of an IT Project Implementation

Functional and integration testing may not be enough to launch a large system. Because of the abundance of technical issues, an aspect may be missed, which is important for business and may delay the launch of an apparently ready product (service, offer). We are talking about business process testing (end-to-end testing or e2e). We recommend using a holistic approach to this, without fragmenting it into sprints and system releases, going through the whole process in a test environment without risk to users and infrastructure.

This article is about preparations for such a test. Product owners (POs) and QA may benefit from learning about possible planning errors and the optimal allocation of resources to the task for multiple teams.

When and Who Needs End-to-End Testing

The PO and QA have tools to understand the state of the product and processes. They test, capture metrics, plan and automate the independent delivery of changes to the user to reduce time-to-market and be ready to implement any business needs.

Let's take the Scaled Agile Framework as an example where there are many interacting internal and external systems and services, different vendors, different tech stacks, different teams, different functionalities, and, therefore, a subjective view of business values, where each team has its own vision of everything.

The business value often does not fit within the competence of one team or even one system. Let's call this value capability (a high-level functional solution that is implemented in one cycle). Who is responsible for ‌capability implementation in multiple systems and teams? To make it a shared responsibility, capability requires a lead team. A new tool is also required — end-to-end testing (e2e) to ensure the delivery of value.

E2e checks the operation of a business process going through different systems, for example, from customer registration to closing a transaction. Everything seems simple: determine the systems, teams and launch dates, plan and implement. But let's see if that's the case.

Implementing e2e: How to Avoid Stepping on the Rake

The main problem with implementing a new tool is resources. More precisely, the evaluation and planning of e2e testing without practice. After all, if we are doing well, no team has surplus resources to implement capability. It's important for the customer to know that everything is ready to start the business process.

So, QA specialists are asked to evaluate the task's "to test" capability and implement it, for example, within four sprints using three teams and four systems. At the same time, you need to evaluate your part of the test, taking into account the available resources. A QA lead, who has experience in testing different systems, is recommended for performing the evaluation. Meanwhile, the PO and analysts have already worked out the dependencies and know who will participate in e2e. Ideally, everyone has experience with e2e-testing, everything is accounted for, and the business knows what it wants or is always willing to discuss. Unfortunately, this is not always the case.

Let's consider the situation where the D system team, which supported the dependency on capability, is making preparations for quarterly planning. This is what PO and QA might say to each other:

"We have e2e planned for transactions in Brazilian reals. Let's evaluate."
"What do you need to check?" QA asks humbly.
"That the transaction is closed in reals."
"And how is this different from other transactions?" QA goes on.
"Currency."

QA is thinking, "Gosh, I'm closing 10 million dollars and 12 million euros worth of transactions every day during the test. There's something wrong here."

But asks, "What is e2e for?"
"The site is localized for Brazil, and their team is the best in e2e."
"Do they have a QA on their team? We'll need the QA's contact info."
"Sure."

The update on the new currencies is in the production environment after the feature toggle (functionality availability setting in the production environment). Let there be 2 story points, taking into account communications and report preparation.

Next, everything will be as it happens when there is no test plan: forgot to include one of the systems in the test, started e2e before fixing bugs and closing the transactions. As a result, the whole sprint was the same scenario repeated again and again for the B system. Testing was completed during the next sprint. Successful end-to-end testing was reported during the demo phase. What did they do wrong?

Firstly, the purpose of testing dissolved, and they were sure that there were no defects. Secondly, the principle of early testing was neglected — the earlier the issue was identified, the cheaper the fix. Thirdly, QAs deny its leading role in e2e. They don't apply test design as part of the whole business process but limit themselves to their system. Coverage can be excessive or insufficient if you continue to consider business process testing fragmented, separated depending on systems requirements.

As a result, such e2e testing not only wastes the resources of several teams, but also fails to achieve the goal. Costs may not be estimated, both when planning and in the end. And while it is not clear what metrics should be used to evaluate ‌e2e effectiveness, it is important to properly prepare for it to reduce losses and risks.

LIFE HACK: we could write a separate article about capability test design, but experience shows that the best way to use pairwise and paired testing. The first will allow you to take into account the maximum requirements in different systems. In this case, ‌coverage will be sufficient, as releases are tested in parallel. Paired testing (QAs from different systems) should pass at least one scenario. A fresh look will allow you to notice things you may have forgotten or missed.

Preparation: How to Make it More Effective

Ideally, preparation for e2e with QA should begin before task planning and evaluation. QAs can request access to test benches of related systems, obtain test devices, and configure tools. It is important to hold a kickoff meeting with QAs from all the teams involved. It is advisable to "go through" the business process with an analyst or with the customer who will immerse QAs and give insight into the business value.

LIFE HACK: if organizing a meeting with the customer is difficult, QAs from different systems can call each other. By combining product knowledge, they will be able to get the information they need for further planning.

The main thing: the lead team's QA should gather the information required for test design and setting goals for testing:

• expectations of the business, features, novelty, benefits of the new process;
• user behavior, their actions as part of the business process outside the system, which must be taken into account;
• peculiarities of interaction between systems, issues and identified requirements;
• an opportunity to go through a similar process or examine all the systems to have an idea of how each of them should be tested, the readiness and the number of future changes in it.

It is necessary to focus on the business process and integration of the systems to verify and determine the sufficient list of the participants. In our experience, teams can play different roles in testing:

• organization, creating test cases, the plan and the report;
• passing scenarios;
• approval of test cases and verification of the results in their systems;
• setting up test benches or preparing test data.

As a result, we have not only coverage and the plan, but also steps to optimize resources. For example, setting up the systems and preparing test data can be delegated to a team that will relieve QA if necessary.

How Access Can Save Resources

It's great to have access to everything. You can enter data into ‌system A and see the result in it as a user after processing the data in the B and C systems, then use a test device under a different role in the D system to continue and complete the process. And, eventually, check its artifacts in A and E. This e2e testing is performed by a single specialist without any time loss related to communication and synchronization between QA from different teams.

However, it is not always possible to obtain access, or it may not be enough to check something complicated. It is important to evaluate the need and benefit of involving in e2e a QA specialist from another team who has access and knowledge of the system, or it would be more efficient to obtain different types of access for your team for later use.

Since vendors and teams without QA can participate in testing, the access types help focus responsibility and make communications functional.

Launch: What You Can't Do Without

So, the test plan is practically ready. We know the business process and have an idea of the testing goal. There are scenarios and participants with their roles at this stage. During quarterly planning, we set up and discussed dependencies, evaluated development, e2e, and test bench setup tasks, and maybe even scheduled a "window" when settings should remain intact. We know:

• the goal of testing relative to the business process;
• what and where will and will not be checked;
• what and where to do to start going through the scenarios;
• the scenarios as they are.

Yes, there can be many unknowns and variables. All issues should be mentioned in the plan. It is important to get answers before the sprint when e2e is scheduled so that the participants can re-evaluate their testing tasks.

LIFE HACK: the test plan is not necessarily a document. You can create a chat, task, or page in Confluence to keep track of what has been agreed and exchange artifacts, or do this in a test management system such as Test Rail. It is important not to neglect communications, not only to follow the plan, but also to adjust it in time.

Based on the number of unknowns, we can determine the e2e completion criteria. For example, it is not necessary for all participants to retest for non-critical defects or to test improvements to one of the systems required based on e2e results. It is important to discuss all this with the participants and include it in the plan. This way, e2e will not become a black hole for resources.

With such a plan, even a canceled e2e will become a job that is not done for a specific reason. Without a plan, you might end up with a worthless report.

Let's Summarize

By involving QA in a timely manner you will avoid business value loss during e2e testing.

In practice, QAs can highlight requirements missed in another system based on previous experience and issues with similar integrations. For example, if there is no validation of client data in the CRM database, but it is available in the front-end systems, where the user serves the client. New applications often do not follow common validation rules and may store incorrect data in the CRM, creating a load on users of other applications. If QAs are involved in time, before the start of e2e testing, they will advise what should be taken into account.

Immersing QA in the business process will allow you to plan e2e in an optimal way:

• define the necessary teams,
• define the scope of their participation and preparations,
• allocate resources by obtaining different types of access,
• agree on the criteria for completeness.

Based on the plan, each team will evaluate and plan their work. This way, we get a flexible process at the project level rather than a mere formality.

At the team level: the resources of one team will not be wasted on testing the functionality by the other, as the scope of participation and criteria for the start (readiness) are defined. Definition of Done (DoD) will allow you to understand at what point the overall task is ready, and revisions and corrections in the individual team begin.

As a result, product owners receive information on both the status of the individual product and the readiness of the entire capability for launch.

There are other interesting cases on our website.