Skip to main content

Reference

Verifier tests your component at runtime, watching the messages flowing through the network. As mentioned in What is Verifier?, Verifier looks at a message model that you define to see if your component sends and receives the messages it should, does so at the correct times, and over the correct network transports.

To assist finding topics you care about, please use the sidebar to jump directly to those topics.

What You'll Need​

To test your software with Verifier, you'll need the following:

  • Your software component, containerized
  • Your message set in Flex
    • Tangram Pro has built-in Flex packages for a number of message sets; you may want to check and see if yours is already included
  • Your component's interface model, specified in Flex protocols
  • A Tangram Pro project using your Flex package (containing messages & protocols) as a Source of Truth
  • A valid Tangram Pro license which includes at least one seat for the Verifier toolkit

To test your software with Verifier in GitLab CI, you'll need to fulfill additional requirements.

Component Software​

Verifier runs against containerized components. To prepare an image with your component binary, you can :

info

To test a component in Tangram Pro™, your component will need to use one of the supported transport types.

For more information on setting your component up for testing, see the how-to guide.

Message Model​

A component's "message model" is made up of the Flex protocols a component should be able to successfully complete, which includes:

  • The messages the component should send or receive, and the order in which those messages should be sent or received
  • The contents of those messages
  • The transports which messages in a protocol are send and received over

Protocols​

A protocol is a Flex-based ordering of messages that your component-under-test will send or receive in a distinct flow. You can add multiple protocols for your component to cover as many use cases as you need to test.

A protocol contains:

  • The type of messages the component should be able to send and receive
  • The order in which the component is expected to be able to handle those messages
  • The contents of the message fields

Protocols are written in the Flex programming language, which allows you to write regular Flex expressions alongside the send and recv statements that define the message sequence. As a result, you can define powerful constraints which define the ranges of valid values for message fields. These constraints can be simple static values, ranges of values, or can even be based on the contents of other fields within the same message or from other prior messages in the protocol. Field constraints are defined as expressions, which allow the user to perform math in their message data validations.

For more information about protocols, visit the Flex protocols page.

Configuration​

Clicking on Setup New Test from the Verify mode of a project reveals the configuration settings for a Verifier test.

Verifier Config

  • Container Selection
    • Use Container Registry: You may use a container that you've built externally, as long as you link the container registry so that Tangram Pro can pull the image
  • Test Runs Total: Verifier will repeat all tests this many times. If there are five protocols and Max Path Retries is set to 3, Verifier will test all 5 protocols in order with up to 3 retries, and then repeat the entire process four more times.
    • Because Verifier will generate randomized message fields (within your applied constraints) each time it performs a run, this can be a useful way to test a component against a variety of input values in a single Verifier execution
  • Max Path Retries: The number of attempts Verifier will make to pass an individual protocol when testing it in a single Run
    • See here for a more detailed explanation on why this is needed.
  • Message Timeout: The maximum number of seconds that Verifier will wait to receive a message. If no message is received within this time, Verifier will determine that no message will be received and will record a Timeout event.
  • Component Startup Delay: Allows the user to tell Verifier how long a maximum amount of time (in seconds) that a component may take to completely start and be ready for testing

Transports​

Transports are defined and attached to a component's interface in the Design mode of Tangram Pro. They're also attached to the Verifier test in the Verify panel, allowing you to test a component's behavior over multiple different transport mechanisms.

Verifier supports a fixed set of transports for communicating with your component. Each transport type has specific configuration options that need to be set in Tangram Pro's Transport panel while in Design mode.

warning

Transport details specified in Tangram Pro will only affect Verifier in CI -- they are not used by Verifier when running in Tangram Pro!

This is because the transports running on the Tangram Pro infrastructure need to adhere to rules of that environment, but when Verifier runs in your CI the environment is determined by you!

Transport Connection​

Regardless of which transport is used, your component must be able to talk to Verifier. Currently, Verifier supports only the proxy-based transports that you can see listed below. You can configure your component so that it connects to to the proxy in two different ways:

  • In Tangram Pro's Design tab, click on your component. Then, click on the Run tab. Look for the Container name listed for the transport your component uses (e.g. zeromq-transport): this is the hostname of the proxy which which your component should use to connect
  • Each transport makes two environment variables available to the component (e.g. TANGRAM_TRANSPORT_zeromq_transport_HOSTNAME and TANGRAM_TRANSPORT_zeromq_transport_PORTS), which you can read in your component container to configure its connection.
    • Hostname is a single string name which generally is used in place of an IP address
    • The ports variable may contain a single port, or multiple comma-separated ports, dependent on the proxy

The component hostname & ports should remain stable for the most part, but are subject to change when the project changes or when significant changes occur to Tangram Pro. Therefore, we recommend the environment variable option. These environment variables are listed under each of the transports below.

Verifier currently supports only one transport per component.

ActiveMQ​

Connection Environment​

  • Default hostname: activemq-transport
  • Host environment variable name: TANGRAM_TRANSPORT_activemq_transport_HOSTNAME
  • Port environment variable name: TANGRAM_TRANSPORT_activemq_transport_PORTS
    • The variable value contains a single port

Configuration in Design Mode​

The ActiveMQ transport uses the publish/subscribe pattern and requires the following configuration:

Required Settings:

  • Broker URI: The URI for configuring the broker, typically in the format <IP>:<port> with any additional options
    • Example: failover:(tcp://127.0.0.1:61616)?timeout=1000

Optional Settings:

  • Auto Acknowledge Broker: When enabled, automatically acknowledges data from the broker
  • Convert bytes to text: Converts bytes to text when sending/receiving
  • Max Sending Size: Maximum allowed message size for sending (in bytes)
  • Max Receiving Size: Maximum allowed message size for receiving (in bytes)
  • Startup Wait in ms: Time in milliseconds to wait for network initialization

TLS/SSL Configuration (Optional):

  • Keystore Value Path: Path to the keystore file
  • Keystore Password: Password for the keystore
  • Trust Cert File Path: Path to the trust store file
  • Trust Cert Password: Password for the trust store

Kafka​

Connection Environment​

  • Default hostname: kafka-transport
  • Host environment variable name: TANGRAM_TRANSPORT_kafka_transport_HOSTNAME
  • Port environment variable name: TANGRAM_TRANSPORT_kafka_transport_PORTS
    • The variable value contains a single port

Configuration in Design Mode​

The Kafka transport uses the publish/subscribe pattern with the following settings:

Required Settings:

  • Broker URL: URL for the broker (format: <IP>:<port>)
  • Component Group ID: Group ID for the component

Optional Settings:

  • Message Partition: Partition to use for messages
  • Max Sending Size: Maximum message size for sending (in bytes)
  • Max Receiving Size: Maximum message size for receiving (in bytes)
  • Startup Wait in ms: Network initialization wait time (in milliseconds)

MQTT​

Connection Environment​

  • Default hostname: mqtt-transport
  • Host environment variable name: TANGRAM_TRANSPORT_mqtt_transport_HOSTNAME
  • Port environment variable name: TANGRAM_TRANSPORT_mqtt_transport_PORTS
    • The variable value contains a single port

Configuration in Design Mode​

The MQTT transport uses the publish/subscribe pattern and requires configuration for both publishing and subscribing:

Required Settings:

  • Pub Socket: Publishing socket configuration
    • IP/Hostname: IP address to connect/bind to
    • Port: Port number
    • Socket Type: Either Bind (server) or Connect (client)
  • Sub Socket: Subscribing socket configuration (same fields as pub_socket)

Optional Settings:

  • Max Sending Size: Maximum message size for sending (in bytes)
  • Max Receiving Size: Maximum message size for receiving (in bytes)
  • Startup Wait in ms: Network initialization wait time (in milliseconds)

TLS Configuration (Optional):

  • Certificate authority file path: Path to certificate authority certificate files
  • PEM-encoded certificate file path: Path to PEM-encoded certificate file
  • PEM-encoded key file path: Path to PEM-encoded key file

NATS​

Connection Environment​

  • Default hostname: nats-transport
  • Host environment variable name: TANGRAM_TRANSPORT_nats_transport_HOSTNAME
  • Port environment variable name: TANGRAM_TRANSPORT_nats_transport_PORTS
    • The variable value contains a single port

Configuration in Design Mode​

The NATS transport uses the publish/subscribe pattern with extensive configuration options:

Optional Settings:

  • URL: NATS server URL (format: nats://<IP>:<port>)
  • Max Sending Size: Maximum message size for sending (in bytes)
  • Max Receiving Size: Maximum message size for receiving (in bytes)
  • Startup Wait in ms: Network initialization wait time (in milliseconds)

Authentication (Optional):

  • NATS Server Username: NATS server username
  • NATS Server Password: NATS server password
  • NKey for the NATS server: NKey for the NATS server
  • NKey seed: NKey seed

TLS Configuration (Optional):

  • PEM-encoded key file path: Path to TLS key file
  • PEM-encoded certificate file path: Path to TLS certificate file

RabbitMQ​

Connection Environment​

  • Default hostname: rabbitmq-transport
  • Host environment variable name: TANGRAM_TRANSPORT_rabbitmq_transport_HOSTNAME
  • Port environment variable name: TANGRAM_TRANSPORT_rabbitmq_transport_PORTS
    • The variable value contains a single port

Configuration in Design Mode​

The RabbitMQ transport uses the publish/subscribe pattern with the following configuration:

Required Settings:

  • URL: Broker URL (format: amqp://<IP>:<port>)
  • Virtual Channel Number: Virtual channel number

Optional Settings:

  • Max Sending Size: Maximum message size for sending (in bytes)
  • Max Receiving Size: Maximum message size for receiving (in bytes)
  • Startup Wait in ms: Network initialization wait time (in milliseconds)

SSL Configuration (Optional):

  • PEM-formatted certificate: Path to PEM-formatted certificate
  • Certificate Authority Cert File: Path to certificate authority cert file
  • PEM-formatted secret key: Path to PEM-formatted secret key
  • Peer verification: Enable peer verification
  • Hostname verification: Enable hostname verification

ZeroMQ​

Connection Environment​

  • Default hostname: zeromq-transport
  • Host environment variable name: TANGRAM_TRANSPORT_zeromq_transport_HOSTNAME
  • Port environment variable name: TANGRAM_TRANSPORT_zeromq_transport_PORTS
    • The variable value consists of two comma-separated ports: <PUBLISH PORT>,<SUBSCRIBE PORT>, Your component should publish to the publish port and subscribe to the subscribe port

Configuration in Design Mode​

The ZeroMQ transport uses the publish/subscribe pattern and requires configuration for both publishing and subscribing:

Required Settings:

  • External Proxy: Set to true if components speak to a central ZeroMQ proxy rather than speaking direclty to each other
    • If this is true, you should set both the Pub Socket and the Sub Socket to be the Connect type
    • ZeroMQ doesn't have an official proxy executable, but there is an official proxy API that we've used to create a simple proxy that can run in Tangram Pro. This allows you to use the External Proxy setting when running Verifier in Tangram Pro, but you'll need to have your own proxy or use ours if you want to use this setting in Verifier in CI.
  • Pub Socket: Publishing socket configuration
    • IP/Hostname: IP address to connect/bind to
    • Port: Port number
    • Socket Type: Either Bind (server) or Connect (client)
  • Sub Socket: Subscribing socket configuration (same fields as pub_socket)

Optional Settings:

  • Max Sending Size: Maximum message size for sending (in bytes)
  • Max Receiving Size: Maximum message size for receiving (in bytes)
  • Startup Wait in ms: Network initialization wait time (in milliseconds)

Curve Authentication (Optional):

  • Public Key File: Path to public key file
  • Secret Key File: Path to secret key file
  • Server IP: Curve authentication server IP
  • Server Key: Auth server public key

Serializers​

Verifier supports a broad range of serializers, including:

  • CSV
  • Direct
  • JSON
  • LMCP
  • MAVLINK
  • OMS v2.X (XML)
  • ROS
  • STANAG 4586
  • WOSA
  • XML

Verifier currently supports only one serializer per component. You can set the serializer used by a component for all messages it sends and receives using the message configuration panel. Note that the selected serializer will be used for all messages the component sends and receives, across all sequences.