Skip to main content

Test a Software Component

This tutorial focuses on the Verify tool in Tangram Pro™. Verify allows you to run a software component in a container, and test it against a message sequence model so you can be more confident that your software is doing what it's supposed to do. We made this tool to help streamline the verification of software behavior, in a online collaborative workspace that links a component in a system design to actual software and user-defined test sequences.

You'll learn how to:

  • Build a software component and test environment
  • Define test sequences and constraints
  • Configure and run tests

What software will be tested in this tutorial?

  • We provide example source code in GitHub of a C++ software component one might find on a mission-critial UAV, albeit a simplified version for demonstration purposes, that processes mission commands.
  • This source code utilizes Tangram Pro-generated interface libraries that enable processing messages defined by the OpenUxAS LMCP standard and communicating over a ZeroMQ transport.

What behavior will be tested in the example software component?

The software is expected to handle these two mission sequences:

  • A) When it receives a MissionCommand message, a sequence of messages expected to happen
  • B) When it receives a CameraAction message, a different sequence of messages is expected to happen

Additionally, there are specific values or ranges of values that are expected within these messages. Tangram Pro can help verify those as well!

Step 1: Connect to GitHub​

Your Tangram Pro account can pull source code from GitHub and GitLab repositories to build software executables and container images. For this tutorial, we provide example source code in GitHub.

Prerequisite

A GitHub account is required to complete this tutorial. If you do not have one, please go to https://github.com/ to set one up.

Once you've confirmed your GitHub login, go ahead and connect your Tangram Pro account to it:

  1. In Tangram Pro, go to Account by clicking your avatar image in the upper right corner of the page
  2. Scroll down to Source Code and click Add Hosting Service
  3. Choose GitHub.com
  4. Click the Register an OAuth App link to open GitHub
  5. Follow these instructions to complete the integration

Now you'll be able to link GitHub repositories to components in Tangram Pro. We'll do that in a later step.

Step 2: Create a Project​

To get started create a new project:

  1. Go to Projects and click New Project
  2. Enter a Project Name
  3. Select your account for Owner

Step 3: Add a Transport​

Add a transport (or networking) library to your project. The example software component requires the ZeroMQ library to send and receive messages. Let's add it:

  1. Find Transports on the lefthand side and click the + button
  2. Choose ZeroMQ

Step 4: Add a Component​

The example software is expected to be able to process a specific subset of LMCP formatted messages. Let's add those to a component.

Add a single component:​

  1. Click the Add Component button
  2. Click anywhere on the stage to place it
  3. Double-click on the component's name and change it to "MissionComputer"

Define the component's interface:​

  1. Click the component's ... menu, and select Interface. Here you can choose the Flex Package, and the Messages that will be input or output from the component.
  2. Click into Search Packages and select OpenUxAS::LMCP::v3
  3. Click into Search Messages and select the following:
    • AirVehicleState
    • CameraAction
    • CameraConfiguration
    • CameraState
    • GoToWaypointAction
    • MissionCommand
  4. Change these three messages to be an Output
    • AirVehicleState
    • CameraState
    • CameraConfiguration

Your component interface should look like this:

Step 5: Build the Software​

Now you're ready to build the software component.

Enter Build mode:​

  1. Click Build on the top left
  2. Select the "MissionComputer" component on the left
  1. Click the Code tab
  2. For Host URL choose https://github.com (this is what you setup at the beginning)
  3. For Repository type in the following name tangrampro-verify-examples and select it
  4. The Branch Name will auto-populate with master. Keep that as is.

Choose the Build actions for Tangram Pro to complete:​

  1. Click the Actions tab
  2. Enable Generate Interface Code and choose C++ for the output language.
  3. Keep Compile Code disabled. The compilation will occur inside the container so no need to do that twice.
  4. Enable Containerize
  5. Enable Use Dockerfile Path
  6. Type in the following path: mission_computer/Dockerfile

Start the Build:​

  1. Click the blue Build button. (Note: if the button is disabled, be sure to add the ZeroMQ transport)
  2. Streaming logs will display as it runs in the cloud. This process may take a several minutes

During this process Tangram Pro generates C++ interface code to support the LMCP messages you selected, including a LMCP serializer and deserializer, and the ZeroMQ transport library. After the interface code is generated, a container image is generated based on the Dockerfile. This copies the interface code and the source code into the container image, and compiles them together.

Step 6: Create Test Sequences​

Nice work! Now let's enter Verify mode and start definining our test sequences.

  1. Click Verify on the top left
  2. Select the "MissionComputer" component on the left if it's not already selected
  3. Click the Component Sequences tab

A sequence is an ordered list of messages that will be input to your component-under-test, and messages that are expected to be output by it. You can add multiple sequences to cover as many test scenarios as you need.

Let's start by adding the following "Rescue" mission sequence, which our software component should support.

  • Our component receives a MissionCommand from a control station
  • Our component reports the UAV's location in 3D space via an AirVehicleState message
  • Our component receives a GoToWaypointAction command to fly the UAV to a new location
  • Our component again reports an AirVehicleState to send updated UAV location data

To add this sequence:​

  1. Add a new sequence by clicking the Add Sequence button
  2. Name it "Rescue"
  3. Click the Add Initial Message button
  4. For Interface Message choose MissionCommmand. It will default to Input since we specified that direction in the design.
  5. Keep all other fields as their defaults and click Add and Continue
  6. Add the following messages:
    • AirVehicleState
    • GoToWaypointAction
    • AirVehicleState

Your Rescue sequence should like this:

Now let's add the following "Scan" mission, which our software component should also support.

  • Our component receives a CameraAction from a control station
  • Our component reports the UAV camera's CameraConfiguration message
  • Our component receives another CameraAction command
  • Our component reports the UAV camera's CameraState

To add this sequence:​

  1. Add a new sequence by clicking the Add Sequence button
  2. Name it "Scan"
  3. Click the Add Initial Message button
  4. For Interface Message choose CameraAction
  5. Keep all other fields as their defaults and click Add and Continue
  6. Add the following messages:
    • CameraConfiguration
    • CameraAction
    • CameraState

Your Scan sequence should like this:

Step 7: Define Rescue Constraints​

Each message in a sequence can have constraints. By adding constraints, messages can be called "valid" or "invalid" based on their contents, including how the messages of one field relate to a message that came previously in a sequence.

Let's add field constraints to the messages in the Rescue sequence.

Add a Message Template​

  1. Click the Message Templates button in the top right to access your templates
  2. Add a new one by clicking Add Message Template
  3. For Package choose OpenUxAS::LMCP::v3
  4. For Message choose Waypoint
  5. A default Template Name is created, but let's change the name to be WptItem
  6. Copy and paste the following YAML, then click Save Template and close the window
- name: Number
  valid_values:
    - value: 0
- name: Latitude
  valid_values:
    - range: {min: 19.50139, max: 64.85694}
- name: Longitude
  valid_values:
    - range: {min: -161.75583, max: -68.01197}

This template is defining some valid ranges for the fields Number, Latitude, and Longitude in the Waypoint message struct, which will be input to our component. The latitude and longitude ranges cover continental US.

Add Field Constraints for MissionCommand​

  1. Click Constraints next to MissionCommand
  2. Copy and paste the following YAML, then click Save
- name: FirstWaypoint
  valid_values: 
    - value: 0
- name: WaypointList
  valid_values:
    - indexes: { 0: WptItem }

Since MissionCommand is an input, these constraints are controlling what values that will be input to our component.

Add Field Constraints for GoToWaypointAction​

  1. Click Constraints next to GoToWaypointAction
  2. Copy and paste the following YAML, then click Save
- name: WaypointNumber
  valid_values:
    - value: MissionCommand_1.WaypointList.0.Number

Since GoToWaypointAction is also an input, these constraints are controlling what values that will be input to our component. By referencing MissionCommand_1, you're able to pull in data from the MissionCommand message at the beginning of the sequence.

Add Field Constraints for AirVehicleState​

  1. Click Constraints next to the second AirVehicleState (AirVehicleState_2)
  2. Copy and paste the following YAML, then click Save
- name: Location.Latitude
  valid_values:
    - value: MissionCommand_1.WaypointList.0.Latitude
- name: Location.Longitude
  valid_values:
    - value: MissionCommand_1.WaypointList.0.Longitude

AirVehicleState is an output, so when we run the test, Tangram Pro will check the field values sent by our software component and verify it fits within these constraints.

Step 8: Define Scan Constraints​

Next, let's add constraints to two of the messages in the Scan sequence.

Add Field Constraints for CameraConfiguration​

  1. Click Constraints next to CameraConfiguration
  2. Copy and paste the following YAML, then click Save
- name: MinHorizontalFieldOfView
  valid_values:
    - range: {min: 20, max: 150}
- name: MaxHorizontalFieldOfView
  valid_values:
    - range: {min: MinHorizontalFieldOfView, max: 150}

CameraConfiguration is an output, so when we run the test, Tangram Pro will check the field values sent by our software component and verify it fits within these constraints.

Add Field Constraints for CameraAction​

  1. Click Constraints next to the second CameraAction
  2. Copy and paste the following YAML, then click Save
- name: HorizontalFieldOfView
  valid_values:
    - range: {min: CameraConfiguration_1.MinHorizontalFieldOfView, max: CameraConfiguration_1.MaxHorizontalFieldOfView}

CameraAction is an input, so these constraints are controlling the values that will be input to our component, making sure it fits within the range of what our software component expects. For reference check out the logic in the source code here

Step 9: Build the Test Environment​

Now that we've defined some test sequences, let's build the test environment.

  1. Click the yellow New Test Build Required button

A test build generates a test environment that includes:​

  • A software component that acts as the inverse of the component being tested (e.g. if the component-under-test has MissionCommand defined as an input, the inverse component will have MissionCommand as an output)
  • A transport proxy which your component and the inverse component connect to (ZeroMQ in this case)

Step 10: Run Tests​

Great work! Now let's run these tests.

  1. Click the Setup New Test button. (Note: if the button is not visible, wait for the test build to complete)
  2. The Use Project Build option will be selected since you completed the project build earlier
  3. Keep all other options set as the defaults. For more information about these options see User Guide: Verify
  4. Click Begin Verifier Test

When a test is running, Tangram Pro provides live reporting as it processes and analyzes each sequence, including the contents of the messages being sent. When complete, the test summary is saved for you to view all of those details later.

Note: If a Timeout occurs, try running the test again.

Success! The two missions that our software component was expected to support have been verified!

How to read the results​

Tangram Pro reports a sequence as "Satisfied" if both of these conditions are met:

  • The messages were sent and received in the order specified by sequences
  • The message contents met all specified constraints

A sequence will be "Unsatisfied" if either of the above conditions are not met. It is important to note, a sequence may be "Unsatisfied" if the wait time between subsequent messages exceeds the "Message Timeout" and if the number of retries reaches the "Max Path Retries".

Tangram Pro also provides two real-time log streams.

  • Component Logs: These are the logs output by your software component
  • Debug Logs: These are the logs output by Tangram Pro's test environment

To view previous tests​

  1. Go to View Tests
  2. Click on a test to view its results in more detail

What does an "Unsatisfied" test look like?​

  • Try adding add another message to one of the sequences, and run the test again.
  • Try removing a constraint, and run the test again.

Cheers!​

By completing this tutorial, you've learned how to use the Verify tool in Tangram Pro to test a software component against a message sequence model.

Check out what you accomplished:

  • Linked a component to GitHub repository
  • Built a software component using Tangram Pro-generated interface code and source code from a GitHub repository
  • Built a container image
  • Defined test sequences and message field constraints
  • Verified the two missions are supported by our software component