Assertions

Updated 3 weeks ago by Rahul Lahiri

The response to each API test request is compared against a reference response at the end of a test. You can configure assertions for each JSON element in the response to determine how the element should be evaluated, and how result mismatches should be treated. This article describes the available assertions.

Structure of assertions

The properties for each element in the JSON, and whether to compare the values, are characterized by the following properties for each item:

  • Presence
  • Data type
  • Transformation
  • Comparison type

Property

Values

Description

Presence

  • Required
  • Optional

Specifies whether the item must be present in the response, or the item is optional. A required item must be present in the response. Otherwise it will be treated as an error. A missing optional item generates a warning but not an error.

Default rule: Required

Data Type

  • Default
  • String
  • Integer
  • Float
  • List [Array]
  • Set [Array]
  • Unstructured array
  • Object

When the type is set to 'Default', no type checking is done on the item. Otherwise, the item is checked to conform to the specified type.

Default rule: Default

Transformation

  • Default
  • Regex
  • Round
  • Floor
  • Ceil

Data transformation rules that are applied to the item before the comparison is performed. The transformation rules are applied to both columns.

Default rule: Default

Comparison Type

  • Equal
  • Ignore

When set to 'Ignore', the values for that item are not compared.

Default rule: Equal

Default assertions

If no assertions are specified when a test is run, then the system uses default rules. The default rule requires that the responses from the test must exactly match the response stored in the test suite. Any deviations, either in the structure of the JSON, or in the values of the fields, are evaluated as test failures.

Inheritance

Within the JSON structure, child elements inherit the Presence and Comparison Type rules from the parent elements. The inherited rules can be overridden by adding local rules for any element.

Applying assertions

Let us look at some examples.

Presence: Required vs Optional

In the example above, the property 'language' appears in the reference column, but not in the test results on the right-hand side. The red icon indicates that this condition is an error according to the current assertions.

Let us assume that the 'language' information is not critical to this specific test, and missing the property in the response should not trigger an error. To change the assertion, hover the mouse over whitespace to the left of the item. The following icon will appear:

Click on the '+' icon. A pop-up dialog will appear.

Click on the Update Rule button. This will open the following dialog:

The path item shows the JSON path for which you are setting the rule. Here it shows that the rule is being set for the language element, and the full path to the element is /body/0/book_info/details/language.

Path: /body/0/book_info/details/language: The /0 in the path indicates that this is the first item in a JSON array.

Change Presence to Optional, and Comparison Type to Ignore. Then click Apply to make the change. Once the change is made, you will see a change pending reminder icon next to the item indicating that a change has been made to rules for that item, but the changes have not been saved yet.

If you want to remove the changes made, doubleclick the icon. The icon will disappear, and the associated changes will be removed.

Missing data items

If a required item is missing in the test response, it is indicated as an error.

If an optional item is missing in the test response, it is indicated as a notication rather than an error.

Data type

The data type is set to Default in most cases unless set manually to a specific type, and no data type checking is performed. You can change this setting if you want an explicit check on the data type for that field. The data type options are shown below:

For example, if you want to ensure that a specific data element is an integer, you should set its Data Type to Int for that item.

The one exception to the default setting above is JSON arrays. We set the data type for arrays to List [array] by default, which means the array should be an ordered list of elements.

While the JSON array specifications specify that an array must be an ordered set, we have found that in practice arrays can be treated either as an ordered set of elements, or as an unordered set of items. So we have added support for comparing arrays either as an ordered list of items, or as a set.

For example, in the example below, the arrays contain the same values, but the order is different.

If you do not care that the items in the array appear in a particular order, but that all the expected items do appear in the result, then set the Data Type to Set [Array]. For this array type, we try to align the elements in the array regardless of the order in which they appear in the arrays. An error would be flagged only if the items in the test array do not match the reference array.

The rule must be set for the array, i.e. on the line that contains that array name, and not for any of the individual line items.

Transformation

Transformations are applied to an element in the JSON before the comparison step. The same transformations are applied to both the reference and the test results. Currently available transformations are:

Data Type

Transformations

Float

Round, Floor, Ceil

String

Regex

Comparison type

The values of the specific item from the reference and the test result are compared based on the settings. If a transformation is applied, then the transformed values are compared. For items that are not relevant to the test, the comparison can be turned off by configuring the setting to Ignore.

The output of the comparison determines whether the test succeeded or not. If the configured comparison rules trigger an error outcome, then the test is marked as a fail.

Applying the changes

When the rules are changed, they are not applied to the existing test results. The test suite must be saved with the updated rules, and the test must be re-run to see the effects of the rule changes.


How did we do?