Reference

Verifier tests your component at runtime, watching the messages flowing through the network. As mentioned in What is Verifier?, Verifier looks at a message model that you define to see if your component sends and receives the messages it should, does so at the correct times, and over the correct network transports.

To assist finding topics you care about, please use the sidebar to jump directly to those topics.

What You'll Need

To test your software with Verifier, you'll need the following:

• Your software component, containerized
  • Messages should be sent and received over one of the available transports.
    • If you're using Verifier in Tangram Pro, you'll need to set up your component's transport connection to match the transport proxy created by Tangram Pro
  • Messages should be serialized using one of the available serializers
• Your message set in Flex form
  • Tangram Pro has built-in Flex packages for a number of message sets; you may want to check and see if yours is already included
• A Tangram Pro project which includes a representation of your component, with its interface & transport defined
  • Sequences defined in the Verify mode of a project for the component you want to test
• A valid Tangram Pro license which includes at least one seat for the Verifier toolkit

To test your software with Verifier in GitLab CI, you'll need to fulfill additional requirements.

Component Software

Verifier runs against containerized components. To prepare an image with your component binary, you can either:

• Use Tangram Pro™ to create a container: Go to the Build mode of your project, link your component to your code repository and complete a containerized build.
• Or provide your own container image: Create a container image that runs your component as the entrypoint command, and push the image to your registry. You can authorize Tangram Pro to pull the image from your registry.

info

To test a component in Tangram Pro™, your component will need to use one of the supported transport types.

For more information on setting your component up for testing, see the how-to guide.

Message Model

A component's "message model" is made up of the sequences a component should be able to successfully complete, which includes:

• The messages the component should send or receive, and the order in which those messages should be sent or received
• The contents of those messages
• The transports with which messages in a sequence are communicated

Sequences

A sequence is an ordered list of messages that your component-under-test will send or receive in a distinct flow. You can add multiple sequences to cover as many test scenarios as you need.

A sequence details:

• The type of messages the component should be able to send and receive
• The order in which the component is expected to be able to handle those messages
• The contents of the message fields

The sequence model includes the ability to define robust and powerful constraints to define the ranges of valid values for message fields. These constraints can be simple static values, ranges of values, or can even be based on the contents of other fields within the same message or from other prior messages in the sequence. Field constraints can include expressions, which allow the user to insert math expressions into their message data validations.

Sequence Messages

See the how-to guide

Each message in a sequence includes these properties:

• Interface Message: The name of the message to send or receive
• Direction: "Input" messages are generated by Tangram Pro's test environment and sent to your software component. "Output" messages are what Tangram Pro expects to receive from your software component.
• Transport: The type of connection, or message broker, to use with your software component to send/receive messages
• Topic: The name of a topic the message should be subscribed to (Default provided)
• Message ID: The id of the message (Default provided)

Once added, a message can have constraints applied to it.

Alternate Message Paths

See the how-to guide

You can define alternate message paths in a sequence, which is like adding an OR statement to the message sequence model.

When the sequence is tested, instead of verifying that the next message in the sequence is only "Message A", an alternate message lets you verify that the next message is either "Message A" or "Message B". Then, if "Message A" was received, Tangram Pro will test the sequence that's defined under "Message A". Or, if "Message B" was received, Tangram Pro will test the sequence that's defined under "Message B".

Constraints

Each message in a sequence can have field constraints. By adding constraints, messages can be called "valid" or "invalid" based on their contents, including how the messages of one field relate to a message that came previously in a sequence.

tip

• Add constraints to Input messages to set field values in the incoming message.
• Add constraints to Output messages to test field values sent by your software.

See the how-to guide to learn how to add constraints to a message in a sequence.

Message Templates

You can create message templates to save the constraints that you need to reuse and apply them to multiple occurrences of a message. Templates allow for reduced yaml duplication as well as a reproducible way to apply constraints to list and complex types.

Constraints YAML

Field constraints are specified as a list of objects in YAML format.

Each field under the fields attribute may have:

• name: Required The name of the field, which should precisely match the name specified in Flex
• valid_values: Optional A list of constraints designating possible valid values for the field
• invalid_values: Optional A list of constraints designating invalid values for the field

All constraints are optional. Each valid or invalid constraint may be one of:

• value: A single value or expression. If the field is a string, this is a regex expression (make sure to escape regex characters like hyphens). If the field is an Enum, this may be either the string name of the Enum value or an integer index of the Enum value
• range: Not available for strings or lists of strings A range of values or expressions
  • min: Minimum allowed value, which is a type-appropriate default if not set
  • max: Maximum allowed value, which is a type-appropriate default if not set
  • min_inclusive: Defaults to true The range includes the minimum value if true, or excludes it if false
  • max_inclusive: Defaults to true The range includes the maximum value if true, or excludes it if false
• string_length: Only for strings Sets the maximum allowable length for the generated value
• list_length: Only for lists Sets the allowable number of values when generated by Verifier
  • min: Minimum number of allowed values
  • max: Maximum number of allowed values
  • fixed: Fixed number of types allowed
• anyOf: Only for structs or lists of structs A list of available message templates (NOT COMPONENT MESSAGES) to specify sub ranges or inherited type to fill out list. See examples here.
• indices: Only for lists A map to constraint the item at a specific index in a list of items, which can be one of a few things:
  • A specific message template (NOT COMPONENT MESSAGE)
  • A specific value (integer, float, string, or expression)
  • A range of values
• is_present: Only for Optional fields Allows specifically constraining the presence of an Optional field
  • maybe: Default Verifier will randomly set the presence of the field
  • true: Optional field will always be present
  • false: Optional field will never be present

Each value specified in a constraint should match the type of the field being constrained. Trying to constrain a float32-typed field with a string would be invalid, and will cause Verifier to warn that the model is invalid.

Example Field Constraints

For a usage example, consider the following Flex messages:

extensible struct InnerThing {
    secret: int32;
}
struct InnerThingDerivedA extends InnerThing {
    longSecret: int64;
}
struct InnerThingDerivedB extends InnerThing {
    shortSecret: int8;
}

message struct ThingA {
    fieldJ: int32;
    fieldK: string;
    fieldL: int32;
}

message struct ThingB {
    fieldX: float64;
    fieldY: int32;
    inner: InnerThing;
}

message struct ThingC {
    fieldX: float64;
    fieldY: int32;
    inner: InnerThing[];
}

Each constraint may include either static values or expressions.

Static Values

Static values are simple values that need no evaluation, and they look as you would expect:**

Constraints for message "ThingA":

- name: fieldJ
  valid_values:
    - value: 10
    - range: {min: 20, max: 40, minInclusive: false}
- name: fieldK
  valid_values:
    - value: "abcdefg"

Constraints for message "ThingB":

- name: fieldX
  valid_values:
    - range: {min: -100.0, max: 100.0}
  invalid_values:
    - range: {min: -1.0, max: 1.0}

Expressions

Expressions allow dynamic reference to other fields within the same message or fields from other, previously received messages. They are mathematical, calculated at runtime, and allow for dynamic determination of values.

Let's assume that the message sequence will look like ThingA -> ThingB in our example below:

# constraints for message "ThingA":
- name: fieldJ
  valid_values:
    - range: {min: 20, max: 40, minInclusive: false}
- name: fieldL
  valid_values:
    - range: {min: fieldJ, max: 80}

# constraints for message "ThingB":
- name: fieldX
  valid_values:
    - range: {min: abs(ThingA.fieldJ) * 2, max: ceil(abs(ThingA.fieldJ) * 3.8)}

There are two more important requirements for expressions:

1. Recursive requirements in constraints are not allowed, if FieldB uses FieldA in an expression, FieldA cannot use FieldB in an expression.
2. Only fields of messages that came previously in a sequence may be referenced. In the above examples, we assume for brevity that a sequence was described such as ThingA -> ThingB

Expression ID Syntax

Fields within the same message can usually be simply addressed by name. However, if referencing a nested field (like in the example below), you'll need to prepend the field name with self in expressions. Fields in other messages need to be prepended with the message ID.

Nested Field Example:

Constraints for message ThingB:

# Notice we don't need `self` in front of the field name in the `name` attribute, because
#  we can only be setting field constraints for the current message.
- name: inner.secret
  valid_values:
    - value: 20
- name: fieldX
  valid_values:
    # However, whenever there's a `.` in an expression, we could be referencing another
    #  message, so we need a `self` to distinguish which message the field is in.
    - value: self.inner.secret

Templates allow for reduced yaml duplication as well as a reproducible way to apply constraints to list and complex types. Let's assume that the message sequence will look like ThingA -> ThingC in our example below:

Constraints for message ThingA:

- name: fieldJ
  valid_values:
    - range: {min: 20, max: 40, minInclusive: false}
- name: fieldL
  valid_values:
    - range: {min: fieldJ, max: 80}

Constraints for message ThingC:

- name: inner
  valid_values:
    - list_length:
      min: 3
      max: 4
    - indices:
      0: ITDA1
      1: ITDB1
    # Where `ITDA1` and `ITDB1` are template names:
    - anyOf: [ITDA1,ITDB1]

There are some important considerations when using message template:

1. Expressions are only allowed with in the template scope, for the example above expressions in ITDA1 could not reference fields within ThingC. This is because there are no guarantees on the fields being there if the template was used in a different location
2. Indices must be within the range of the length, otherwise the length will be the max index + 5. In the example below, the max length of the list would be implicitly set to 7

      - name: inner
        valid_values:
          - indices:
            0: ITDA1
            1: ITDB1

3. Indices start at 0.

      - name: inner
        valid_values:
          - indices:
            0: ITDA1
            1: ITDB1

4. In the case that a list_length is given but no anyOf is provided it will use derived types. In the case below, the list will always have at least 3 elements. This will choose randomly between InnerThingDerivedA and InnerThingDerivedB types or any other derived types available.

      - name: inner
        valid_values:
          - list_length:
            min: 3
            max: 4

Field Constraint Functions

Constraints are backed by the exprtk library. You can find documentation about exprtk's capabilities in their README, but the available functions are copied here for reference:

> [SECTION 01 - CAPABILITIES]
> The  ExprTk expression  evaluator supports  the following  fundamental
> arithmetic operations, functions and processes:
>
>  (00) Types:           Scalar, Vector, String
>
>  (01) Basic operators: +, -, \*, /, %, ^
>
>  (02) Assignment:      :=, +=, -=, \*=, /=, %=
>
>  (03) Equalities &
>       Inequalities:    =, ==, <>, !=, <, <=, >, >=
>
>  (04) Logic operators: and, mand, mor, nand, nor, not, or, shl, shr,
>                        xnor, xor, true, false
>
>  (05) Functions:       abs, avg, ceil, clamp, equal, erf, erfc,  exp,
>                        expm1, floor, frac,  log, log10, log1p,  log2,
>                        logn, max,  min, mul,  ncdf,  not_equal, root,
>                        round, roundn, sgn, sqrt, sum, swap, trunc
>
>  (06) Trigonometry:    acos, acosh, asin, asinh, atan, atanh,  atan2,
>                        cos,  cosh, cot,  csc, sec,  sin, sinc,  sinh,
>                        tan, tanh, hypot, rad2deg, deg2grad,  deg2rad,
>                        grad2deg
>
>  (07) Control
>       structures:      if-then-else, ternary conditional, switch-case,
>                        return-statement
>
>  (08) Loop statements: while, for, repeat-until, break, continue
>
>  (09) String
>       processing:      in, like, ilike, concatenation
>
>  (10) Optimisations:   constant-folding, simple strength reduction and
>                        dead code elimination
>
>  (11) Calculus:        numerical integration and differentiation

Configuration

Clicking on Setup New Test from the Verify mode of a project reveals the configuration settings for a Verifier test.

• Container Selection
  • Use Project Build: This option will allow Verifier to test a container that has been built using Tangram Pro's Build mode
  • Use Container Registry: You may alternatively use a container that you've built externally, as long as you link the container registry so that Tangram Pro can pull the image
• Test Runs Total: Verifier will repeat all tests this many times. If there are five sequences and Max Path Retries is set to 3, Verifier will test all 5 sequences in order with up to 3 retries, and then repeat the entire process four more times.
  • Because Verifier will generate randomized message fields (within your applied constraints) each time it performs a run, this can be a useful way to test a component against a variety of input values in a single Verifier execution
• Max Path Retries: The number of attempts Verifier will make to pass an individual sequence when testing it in a single Run
  • See here for a more detailed explanation on why this is needed.
• Message Timeout: The maximum number of seconds that Verifier will wait to receive a message. If no message is received within this time, Verifier will determine that no message will be received and will record a Timeout event.
• Component Startup Delay: Allows the user to tell Verifier how long a maximum amount of time (in seconds) that a component may take to completely start and be ready for testing
• Sequence Switches: Allows the user to selectively skip testing sequences for a given Verifier test

Transports

Transports are defined and attached to a component's interface in the Design mode of Tangram Pro. However, they're attached to specific sequences in the Verify mode, allowing you to test a component's behavior over multiple different transport mechanisms.

Verifier supports a fixed set of transports for communicating with your component. Each transport type has specific configuration options that need to be set in Tangram Pro's Transport panel while in Design mode.

Transport Connection

Regardless of which transport is used, your component must be able to talk to Verifier. Currently, Verifier supports only the proxy-based transports that you can see listed below. You can configure your component so that it connects to to the proxy in two different ways:

• In Tangram Pro's Design tab, click on your component. Then, click on the Run tab. Look for the Container name listed for the transport your component uses (e.g. zeromq-transport): this is the hostname of the proxy which which your component should use to connect
• Each transport makes two environment variables available to the component (e.g. TANGRAM_TRANSPORT_zeromq_transport_HOSTNAME and TANGRAM_TRANSPORT_zeromq_transport_PORTS), which you can read in your component container to configure its connection.
  • Hostname is a single string name which generally is used in place of an IP address
  • The ports variable may contain a single port, or multiple comma-separated ports, dependent on the proxy

The component hostname & ports should remain stable for the most part, but are subject to change when the project changes or when significant changes occur to Tangram Pro. Therefore, we recommend the environment variable option. These environment variables are listed under each of the transports below.

Verifier currently supports only one transport per component.

ActiveMQ

Connection Environment

• Default hostname: activemq-transport
• Host environment variable name: TANGRAM_TRANSPORT_activemq_transport_HOSTNAME
• Port environment variable name: TANGRAM_TRANSPORT_activemq_transport_PORTS
  • The variable value contains a single port

Configuration in Design Mode

The ActiveMQ transport uses the publish/subscribe pattern and requires the following configuration:

Required Settings:

• Broker URI: The URI for configuring the broker, typically in the format <IP>:<port> with any additional options
  • Example: failover:(tcp://127.0.0.1:61616)?timeout=1000

Optional Settings:

• Auto Acknowledge Broker: When enabled, automatically acknowledges data from the broker
• Convert bytes to text: Converts bytes to text when sending/receiving
• Max Sending Size: Maximum allowed message size for sending (in bytes)
• Max Receiving Size: Maximum allowed message size for receiving (in bytes)
• Startup Wait in ms: Time in milliseconds to wait for network initialization

TLS/SSL Configuration (Optional):

• Keystore Value Path: Path to the keystore file
• Keystore Password: Password for the keystore
• Trust Cert File Path: Path to the trust store file
• Trust Cert Password: Password for the trust store

Kafka

Connection Environment

• Default hostname: kafka-transport
• Host environment variable name: TANGRAM_TRANSPORT_kafka_transport_HOSTNAME
• Port environment variable name: TANGRAM_TRANSPORT_kafka_transport_PORTS
  • The variable value contains a single port

Configuration in Design Mode

The Kafka transport uses the publish/subscribe pattern with the following settings:

Required Settings:

• Broker URL: URL for the broker (format: <IP>:<port>)
• Component Group ID: Group ID for the component

Optional Settings:

• Message Partition: Partition to use for messages
• Max Sending Size: Maximum message size for sending (in bytes)
• Max Receiving Size: Maximum message size for receiving (in bytes)
• Startup Wait in ms: Network initialization wait time (in milliseconds)

MQTT

Connection Environment

• Default hostname: mqtt-transport
• Host environment variable name: TANGRAM_TRANSPORT_mqtt_transport_HOSTNAME
• Port environment variable name: TANGRAM_TRANSPORT_mqtt_transport_PORTS
  • The variable value contains a single port

Configuration in Design Mode

The MQTT transport uses the publish/subscribe pattern and requires configuration for both publishing and subscribing:

Required Settings:

• Pub Socket: Publishing socket configuration
  • IP/Hostname: IP address to connect/bind to
  • Port: Port number
  • Socket Type: Either Bind (server) or Connect (client)
• Sub Socket: Subscribing socket configuration (same fields as pub_socket)

Optional Settings:

• Max Sending Size: Maximum message size for sending (in bytes)
• Max Receiving Size: Maximum message size for receiving (in bytes)
• Startup Wait in ms: Network initialization wait time (in milliseconds)

TLS Configuration (Optional):

• Certificate authority file path: Path to certificate authority certificate files
• PEM-encoded certificate file path: Path to PEM-encoded certificate file
• PEM-encoded key file path: Path to PEM-encoded key file

NATS

Connection Environment

• Default hostname: nats-transport
• Host environment variable name: TANGRAM_TRANSPORT_nats_transport_HOSTNAME
• Port environment variable name: TANGRAM_TRANSPORT_nats_transport_PORTS
  • The variable value contains a single port

Configuration in Design Mode

The NATS transport uses the publish/subscribe pattern with extensive configuration options:

Optional Settings:

• URL: NATS server URL (format: nats://<IP>:<port>)
• Max Sending Size: Maximum message size for sending (in bytes)
• Max Receiving Size: Maximum message size for receiving (in bytes)
• Startup Wait in ms: Network initialization wait time (in milliseconds)

Authentication (Optional):

• NATS Server Username: NATS server username
• NATS Server Password: NATS server password
• NKey for the NATS server: NKey for the NATS server
• NKey seed: NKey seed

TLS Configuration (Optional):

• PEM-encoded key file path: Path to TLS key file
• PEM-encoded certificate file path: Path to TLS certificate file

RabbitMQ

Connection Environment

• Default hostname: rabbitmq-transport
• Host environment variable name: TANGRAM_TRANSPORT_rabbitmq_transport_HOSTNAME
• Port environment variable name: TANGRAM_TRANSPORT_rabbitmq_transport_PORTS
  • The variable value contains a single port

Configuration in Design Mode

The RabbitMQ transport uses the publish/subscribe pattern with the following configuration:

Required Settings:

• URL: Broker URL (format: amqp://<IP>:<port>)
• Virtual Channel Number: Virtual channel number

Optional Settings:

• Max Sending Size: Maximum message size for sending (in bytes)
• Max Receiving Size: Maximum message size for receiving (in bytes)
• Startup Wait in ms: Network initialization wait time (in milliseconds)

SSL Configuration (Optional):

• PEM-formatted certificate: Path to PEM-formatted certificate
• Certificate Authority Cert File: Path to certificate authority cert file
• PEM-formatted secret key: Path to PEM-formatted secret key
• Peer verification: Enable peer verification
• Hostname verification: Enable hostname verification

ZeroMQ

Connection Environment

• Default hostname: zeromq-transport
• Host environment variable name: TANGRAM_TRANSPORT_zeromq_transport_HOSTNAME
• Port environment variable name: TANGRAM_TRANSPORT_zeromq_transport_PORTS
  • The variable value consists of two comma-separated ports: <PUBLISH PORT>,<SUBSCRIBE PORT>, Your component should publish to the publish port and subscribe to the subscribe port

Configuration in Design Mode

The ZeroMQ transport uses the publish/subscribe pattern and requires configuration for both publishing and subscribing:

Required Settings:

• External Proxy: Set to true if using an external proxy
• Pub Socket: Publishing socket configuration
  • IP/Hostname: IP address to connect/bind to
  • Port: Port number
  • Socket Type: Either Bind (server) or Connect (client)
• Sub Socket: Subscribing socket configuration (same fields as pub_socket)

Optional Settings:

• Max Sending Size: Maximum message size for sending (in bytes)
• Max Receiving Size: Maximum message size for receiving (in bytes)
• Startup Wait in ms: Network initialization wait time (in milliseconds)

Curve Authentication (Optional):

• Public Key File: Path to public key file
• Secret Key File: Path to secret key file
• Server IP: Curve authentication server IP
• Server Key: Auth server public key

Serializers

Verifier supports a broad range of serializers, matching the serializers supported by our C++ libraries when generating a CSI:

• CSV
• Direct
• JSON
• LMCP
• MAVLINK
• OMS v2.X (XML)
• ROS
• STANAG 4586
• XML

Verifier currently supports only one serializer per component. You can set the serializer used by a component for all messages it sends and receives using the message configuration panel. Note that the selected serializer will be used for all messages the component sends and receives, across all sequences.