Test Automation Tool Overview

The Test Automation Tool is embedded in SAP S/4HANA Cloud Public Edition to support automated testing of business processes. The Test Execution Service is the engine behind the tool that performs actions in a simulated user interface on behalf of the virtual test user. This service lives in SAP Business Technology Platform and is visible in the customer's SAP Cloud Identity Authentication Service, because even virtual users need to be authenticated by the customer's identity provider.

Within the tool, there are preconfigured test automates, which correspond to the manual test scripts for each business process in SAP Signavio Process Navigator. If customizations are made that change a business process flow, or the fields that display in the UI of an app, the test automate needs to be edited to align with the customized business process. For business processes that have no customizations, SAP offers a library of Post Upgrade tests they will run on behalf of the customer in their system after each release upgrade (if consent is provided). This only works for processes that have not been changed from the standard process flow. The combination of Post Upgrade tests and flexibility of the Test Automation Tool significantly reduces the testing workload after release upgrades.

Because many implementation teams will experience at least one release upgrade during their project, we recommend setting up the corresponding test automate as you finish configuring the actual business process in the customer's test system. This makes it easy for partner LoB configuration experts to quickly test processes after a release upgrade, and sets the customers up for success in maintaining their SAP S/4HANA Cloud systems after go-live. While setting-up and running test plans should be done directly in SAP S/4HANA Cloud, the progress of executed tests are visible in SAP Cloud ALM along with your other manual test plans.

Adapt Test Automates

The Manage Your Test Processes app is where you view the standard test process automates, and create your own custom test process automates. Within the app, there are three types of test processes:

• Standard test automates/processes that are created and delivered by SAP to align with the manual test script for the SAP Best Practices business process. Standard automates cannot be edited or deleted.
• Custom test automates/processes that are copied from a standard process or built from scratch.
• Post upgrade test automates/processes that are created and run by SAP on behalf of the customer after a release upgrade.

To align a standard test automate with a customized business process, first make a copy of the standard process. All further edits should be made in the copied process. After the process is copied, the type changes to custom, and by default, the process is not visible in the Test Your Processes app. After making your edits to the process, you can change the visibility to make the process available to be assigned to a test plan.

1. Select the Edit button to choose which process step (test procedure on the test script) you need to change.
2. Select the checkbox of the process step and choose Change Type to make the step custom (instead of standard). In the Release Compatibility column, the checkmark changes to a question mark, which indicates this process step will need to be validated after each release upgrade.
3. Select Save.
4. Click the custom process step to view the individual actions. These correspond to the test steps on the test script. Select the Editbutton in the lower right corner.
   • Actions that are not optional (slider set to NO) are mandatory when executed in a test plan. If there is a failure on any mandatory action (e.g. if the value you've set up in the test process to select isn't available to choose), the entire test plan stops at the failed action and does not continue to the run any actions that follow it.
   • Some values are bound to others because the selection of one field (e.g. United States) determines the options available to choose in another field (e.g. List of states within the U.S.).
5. In this example, we added some custom fields to the UI of the Create Customer Projects app, and want to capture those additional fields within the test automate. I look through the list of actions and select the checkbox of the action where I want to add my custom action above or below.
6. A new browser tab will open with the actual Create Customer Projects app. I navigate back to the original tab to select Record, which begins the screen capture of my keystrokes on the browser tab with the Create Customer Projects app.

Try it yourself!

Learn about the features of the Manage Your Test Processes app in this tutorial.

Recording new actions

On the new browser tab, I see a Recording Panel that includes buttons for Check, Read, Stop, and Pause. Click through the app as you normally would to capture your custom keystrokes. The Check and Read buttons should be used in specific use cases:

• Use the Check button when trying to capture a value, error message, or label that is static on the screen.
• Use the Read button when trying to capture a value or non-error message on the screen.

For both the Check and Read buttons, the value or text captured when you pressed the button is stored as a variable or parameter that can be used later for data binding. Data binding passes the data read from one executable action to another action. During test execution, these values are validated. If the value or text captured during test execution does not match what you captured when editing the actions of the process, the action step will fail. If the action is mandatory (not optional), the entire test will fail at this step.

After capturing my keystrokes, the tool generates the relevant actions and the values I selected (e.g. select field, choose value, etc.). When finished, I can save my changes and change the visibility of the test process so it can be added to a test plan.

Learn more about managing your test processes in the SAP Help Portal.

Try it yourself!

Learn how to adapt SAP Best Practices test automates in this tutorial.

Test Data Containers

To ensure there is always data available to run test processes with, you can bind the process to a Test Data Container (TDC). These are containers of reusable sample data relevant for different fields within business processes. Test data containers can serve as a single source of test data to ensure a test process doesn't fail in execution simply because there wasn't any available data to fetch. There are standard TDC available and you can create your own custom TDCs.

Learn more about Test Data Containers in the SAP Help Portal.

Try it yourself!

Learn how to create a test data container and it's variants in this tutorial.

Create a Test User

The Test Your Processes app is where you create virtual test user(s), test plans that include one or more test processes, and provide consent for SAP to run Post Upgrade tests on the customer's behalf. Before a test plan can be executed, a virtual test user needs to be created who will complete the actions of each process in the plan.

There are three types of test users that can be created under the two different options in the Test Your Processes app:

• Maintain Roles
  • Business User
  • Communication User
• Maintain URL for Authentication
  • Conditional Authentication User

Maintain Roles

The Business User role type should be used for non-integration business processes where the process flow is completed entirely within SAP S/4HANA Cloud. It's important to create this user just as you would any other human business user (with the Manage Workforce app, then import the user info into SAP Cloud Identity Authentication to validate the user and set a password), but this user would be assigned all the available business roles so they have access to all the apps necessary to complete the different test process automates. The name of this user should be DEFAULT, because all the test process automates in the Manage Your Test Processes app already have a column forRole with a prepopulated DEFAULT user to test each process step. If you create your user with this same name, and assign the required business role permissions, it will have the ability to test the non-integration business process automates. Find a summary of the business processes and the roles required for each in the SAP Help Portal.

Try it yourself!

Learn how to create a test business user in this tutorial.

Try it yourself!

Learn how to create a test communication user in this tutorial.

Maintain URL for Authentication

The Conditional Authentication User role type should be used if the customer organization has configured a non-SAP Identity provider (IdP), such as Microsoft Azure AD or Ping. The Test Automation Tool only supports login with an SAP IdP, so if the customer is using a non-SAP IdP and conditional authentication is not configured, the test plans will not execute. Learn how to configure conditional authentication in the SAP Help Portal.

Create Test Plans

In the Test Your Processes app, select the plus (+) icon to create a new test plan and assign test process(es). If you select more than one process, the order in which you click the checkboxes determines the order in which they are added to the plan. If you click into the test process assigned to the plan, you can see where the custom process step was added and the question mark in the Release Compatibility column, which needs to be validated after each release. Non-customized process steps have the original check mark.

Learn more about testing your processes in the SAP Help Portal.

Try it yourself!

Learn about the features of the Test Your Process app in this tutorial.

Data Variants in Test Plans

Before executing the test plan, navigate to the Variants tab to select which country/region and company code the plan should be executed in. This determines the data that will be selected for different fields during the process step execution.

A default variant is populated already, so if you do not make a variant section, the tool will still allow you to run the test. This also applies for the TDCs (Test Data Containers) tab, which has a default container populated, but also allows you to add additional containers.

Try it yourself!

Learn how to work with data variants in test plans in this tutorial.

Try it yourself!

Learn how to execute a test plan that has integration steps in this tutorial.

Provide Consent for Post-Upgrade Testing

The customer needs to explicitly provide consent for SAP to run post-upgrade tests on their behalf in the Test Your Processes app. Navigate to the Post-Upgrade Tests tab, and select the badge icon to open the consent screen. After consent is provided for post-upgrade testing, SAP will execute the applicable standard test automates in the customer's system using test data automatically fetched from the customer's system, or from a Test Data Container. Post-upgrade tests (PUTs) are generally focused on business processes where the customer has made no customizations to the standard process, however SAP has been moving towards making PUTs more customer-driven with additional options to add custom scope and decide which test variant will be executed. The Manage Upgrade Tests app is where you find some capabilities to customize the PUT scope and assign data variants.

In some cases, PUTs cannot be executed on behalf of the customer. This could happen if the respective business process is not active, the data picking logic fails to fetch all the required data necessary for the automates, or the execution pre-check is failed. Other reasons could include not maintaining custom logic or custom field changes that have been made to an actual business process in the corresponding PUT automate, not maintaining the execution variant, or if the default test user doesn't have the necessary permissions to run the PUTs.

Triggering Post-Upgrade Tests

After consent has been provided, post-upgrade tests can be triggered from the play icon after a release upgrade. However, a test execution variant needs to be maintained in the Manage Upgrade Tests app in order for the post upgrade tests to run successfully. A series of system checks will be conducted before the tests begin, and as long as no issues are found with the pre-checks, the post-upgrade tests will then be executed for the relevant business processes.

Learn more about managing your post-upgrade tests in the SAP Help Portal.

Try it yourself!

Learn how to trigger post-upgrade tests in this tutorial.

Analyze Automated and Post-Upgrade Test Results

The Analyze Automated Test Results app displays the results of both post-upgrade tests and customer-specific tests that are ready to be executed, or have been executed by the Test Automation Tool with graphical dashboards. You can drill down into the reason why a particular test plan may have failed by identifying the specific step and viewing a screenshot of a potential error message. You can also share links to the dashboard apps via email or save as a tile that displays on your own launchpad.

Learn more about analyzing your automated test results in the SAP Help Portal.

Try it yourself!

Learn about the features of the Analyze Automated Test Results app in this tutorial.

Test Data Refresh service for SAP S/4HANA Cloud Public Edition

Immediately after implementation, the customer's SAP S/4HANA Cloud test and production systems typically have the same sets of data, however over time as the production system is used to run the customer's day-to-day business transactions, much more data is created in production that doesn't exist in test. Many customers prefer to run business process tests against their recent transactional and master data from production, so SAP offers the Test Data Refresh service to transfer application data from the customer's production system back to the test system. Sensitive data is depersonalized to comply with the GDPR (General Data Protection Regulation in the E.U.). This service has an additional fee.

Learn more about the Test Data Refresh Service in the SAP Community.