jq Expressions

Each workflow instance is associated with a data model. A data model consists of a JSON object regardless of whether the workflow file contains YAML or JSON. The initial content of the JSON object depends on how the workflow is started. If the workflow is created using the Cloud Event, then the workflow content is taken from the data property. However, if the workflow is started through an HTTP POST request, then the workflow content is taken from the request body.

The workflow expressions in the Serverless Workflow specification are used to interact with the data model. The supported expression languages include JsonPath and jq. jq expression language is the default language. However, you can change the expression language to JsonPath using the expressionLang property.

This document describes the usage of jq expressions in functions, switch state conditions, action function arguments, data filtering, and event publishing.

JQ expression might be tricky to master, for non-trivial cases, it is recommended to use helper tools like JQ Play to validate the expression before including it in the workflow file.

Example of jq expression in functions

Expressions can be used in functions of type expression to manipulate the workflow model. As with any other function, the result of the expression evaluation will be merged into the model.

For example, in the serverless-workflow-expression-quarkus example application, a max function adds to the model two properties: max, containing the maximum value of x coordinate in the numbers array (which is a workflow input parameter) and min, containing the minimum value of y coordinate

Example conditions in serverless-workflow-expression-quarkus
  {
      "name": "max",
      "type": "expression",
      "operation": "{max: .numbers | max_by(.x), min: .numbers | min_by(.y)}"
    }

Example of jq expressions in switch conditions

The conditions occurring in a switch state enable the workflow designer to select the path that workflow follows based on the data model content.

A condition in a switch state is an expression, which returns a boolean value when evaluated against the data model. If a condition associated with a state transition returns true, then the workflow must follow that transition.

For example, in the serverless-workflow-greeting-quarkus example application, a message is displayed depending on the selected language, that is English or Spanish.

If the value of the language property is English, the constant literal injected on the message property is Hello from, otherwise the constant value injected on the message property is Saludos desde….

switch condition

The switch state in the serverless-workflow-greeting-quarkus example application contains the following conditions, which in turn contains two jq expressions returning a boolean.

Example conditions in serverless-workflow-greeting-quarkus
"dataConditions": [
        {
          "condition": "${ .language == \"English\" }",
          "transition": "GreetInEnglish"
        },
        {
          "condition": "${ .language == \"Spanish\" }",
          "transition": "GreetInSpanish"
        }
      ]

The Serverless Workflow specification requires all the expressions to be embedded within ${… }. However, SonataFlow figures out whether or not a string is an expression. Therefore, you can save characters and skip ${ in the beginning and } in the end. In case of portability, you must embed the expressions within ${… }.

Example of jq expressions in function arguments

In the Serverless Workflow specification, you can define workflow functions, which can be invoked several times by the workflow states. Each workflow function call might contain different arguments, which are specified using the function arguments.

For example, you can see the temperature conversion function definition in serverless-workflow-temperature-conversion example application. The temperature conversion function performs OpenAPI invocations to convert Fahrenheit to Celsius. For more information about OpenAPI, see Orchestrating the OpenAPI services.

Following is the subtraction function in serverless-workflow-temperature-conversion example application:

Example subtraction function in serverless-workflow-temperature-conversion
"functions": [
{
  "name": "subtraction",
  "operation": "specs/subtraction.yaml#doOperation"
}]

The arguments in subtraction function are expressed as a JSON object, and the property values of the JSON object are either a string containing an expression or a JSON data type, such as string, number, or boolean.

Example arguments in subtraction function
"functionRef":
{
  "refName": "subtraction",
  "arguments":
  {
    "leftElement": ".fahrenheit",
    "rightElement": ".subtractValue"
  }
}

In the previous example, the left number is equal to the fahrenheit property (an input number that invokes the workflow), and the right number is equal to the subtractValue property (a constant number that is injected to the workflow model by SetConstants state). Once the expression evaluation is resolved for all properties that contain an expression, the resulting object is passed in the OpenAPI request. Based on the OpenAPI definition, the properties in the JSON object are used as body, path, query, or header of the upcoming REST invocation.

Following is an example of function arguments defined as string that contains an expression, returning a JSON object:

Example function arguments defined as string
"functionRef": {
  "refName": "subtraction",
  "arguments": "{leftElement: .fahrenheit, rightElement : .subtractValue}"
 }

In the previous example, the result of the expression evaluation is the same JSON object than in the first case, which is passed as arguments of the OpenAPI request.

Example of jq expressions in data filtering

The Serverless Workflow specification defines the following filtering mechanisms to select which information must be part of the workflow data model:

  • Action data filters: Select the part of the action result that is merged into the data model, which overrides the properties that share the name with the selected action result.

  • Event data filters: Similar to the action data filters, but apply to the events instead of actions.

  • State data filters: Define the workflow model to the JSON object, which is returned by the expression and discards an existing property.

    State and Action data filter example

    You can see serverless-workflow-expression-quarkus example application, in which actions and state data filters are used.

    expression diagram

    Following is an expression function in serverless-workflow-expression-quarkus example application:

    Example expression function in serverless-workflow-expression-quarkus
    "functions": [
    {
      "name": "max",
      "type": "expression",
      "operation": "{max: .numbers | max_by(.x), min: .numbers | min_by(.y)}"
    }
]

    In the previous example, an array of complex numbers (x is real coordinate and y is imaginary coordinate) is accepted and an expression function is defined to calculate the maximum value of x and minimum value of y for the numbers array.

    Also, the serverless-workflow-expression-quarkus example application contains an action data filter defined inside squareState action and a state data filter defined inside finish state. The action data filter selects the maximum value of x to be merged to the workflow model, and the state data filter defines the maximum value as the entire workflow model that is returned as the workflow response.

    The previous example expression also contains a max function of type expression and an operation property containing a string of jq expression. This jq expression returns a JSON object, in which the max property is the maximum value of the x coordinate and the min property is the minimum value of the y coordinate.

    Following is an action data filter in serverless-workflow-expression-quarkus example application:

    Example action data filter in serverless-workflow-expression-quarkus
    "actions": [
        {
          "name": "maxAction",
          "functionRef": {
            "refName": "max"
          },
          "actionDataFilter": {
             "results" : ".max.x",
             "toStateData" : ".number"
          }
        }
 ]

    In case the previous example of action data filter is not added in the serverless-workflow-expression-quarkus, then the entire JSON object returned by the function is merged into the workflow model. The previous action data filter contains the following properties:

    • results, selecting the attribute from the data returned by the action.

    • toStateData, indicating the name of the target property inside the workflow model. If the target property does not exist, then a target property is added.

    Therefore, after executing the action, the workflow model consists of a number property, containing the maximum value of x and the original numbers array. After that, the workflow transitions to the finish state.

    Example state data filter in serverless-workflow-expression-quarkus
    "name": "finish",
"type": "operation",
"stateDataFilter": {
   "input": "{result: .number}"
}

    The original numbers array should not be returned as a result of the workflow execution, therefore the final stage consists of a state data filter defining the content of the output model. The output model should contain a result property and the value of result property should be the maximum number that is stored by the previous state in the number property.

    In the previous example, the workflow model is changed by the input property of the filter, which means that the output model is updated before the state is executed. As a final result, the output model consists of a result property, containing the maximum value of x.

    Event data filter example

    You can find an example of event data filtering in the serverless-workflow-callback-quarkus example application.

    Example event filter
      "eventRef": "waitEvent",
  "eventDataFilter": {
     "data": ".result",
     "toStateData": ".move"
   }

    The previous example of the event filter copies the content of CloudEvent data result field into the workflow model move field.

Example of jq expressions in event publishing

When publishing a Cloud Event, you can select the data that is being published using a jq expression that generates a JSON object. Note that in yaml double quotes are required to allow using {} characters.

Example data expression returning an object
transition:
      nextState: WaitForSaveTransformationCompletionEvent
      produceEvents:
        - eventRef: saveTransformationEvent
          data: "{gitRepo:.repositoryURL|sub(\"http(s)?://\";\"ssh://\"), branch: .targetBranch, token: .token, workspaceId: .workspaceId, projectId: .projectId, transformId: .transformId, workflowCallerId: $WORKFLOW.instanceId}"

In the previous example, a CloudEvent was published when the state transitioned. Its data field looks like

Example of generated event
data={"gitRepo":"ssh://bitbucket.org/m2k-test","branch":"aaaaaaasssss","token":null,"workspaceId":"b93980cb-3943-4223-9441-8694c098eeb9","projectId":"9b305fe3-d441-48ce-b01b-d314e86e14ec","transformId":"723dce89-c25c-4c7b-9ef3-842de92e6fe6","workflowCallerId":"7ddb5193-bedc-4942-a857-596b31f377ed"}

ForEach state

ForEach iteratiomParam should be accessed as a variable, using a $ prefix, not as a JSON property, since the loop variable is not part of the workflow model the expression is evaluated against. Therefore, instead of accessing it like a JSON property (with a . prefix), the loop variable should be referenced with a $ prefix

For instance, this ForEach specification example

 "states": [
  {
      "name":"SendConfirmState",
      "type":"foreach",
      "inputCollection": "${ [.orders[] | select(.completed == true)] }",
      "iterationParam": "completedorder",
      "outputCollection": "${ .confirmationresults }",
      "actions":[
      {
       "functionRef": {
         "refName": "sendConfirmationFunction",
         "arguments": {
           "orderNumber": "${ .completedorder.orderNumber }",
           "email": "${ .completedorder.email }"
         }
       }
      }],
      "end": true
  }]

should be modified to

 "states": [
  {
      "name":"SendConfirmState",
      "type":"foreach",
      "inputCollection": "${ [.orders[] | select(.completed == true)] }",
      "iterationParam": "completedorder",
      "outputCollection": "${ .confirmationresults }",
      "actions":[
      {
       "functionRef": {
         "refName": "sendConfirmationFunction",
         "arguments": {
           "orderNumber": "${ $completedorder.orderNumber }",
           "email": "${ $completedorder.email }"
         }
       }
      }],
      "end": true
  }]

Workflow secrets, constants and context

As per specification, you can use Workflow Constants and Workflow Secrets whenever an expression is accepted. In SonataFlow you can use $SECRET to access any configuration property, not just sensitive ones. So, assuming you have added to your application.properties a line with the myname=john property, the following function will append the string my name is john to the message variable

{
      "name": "secretMessage",
      "type": "expression",
      "operation": ".message |= \"my name is \"+$SECRET.my_name"
}

$SECRET always returns a value of type string. If you want to use it in a comparison with a value that is not a string, you must perform the conversion explicitly. In the next example, the retries property is a number, so we need to convert the property value accordingly by calling the tonumber built-in jq function.

{
  "condition": ".retries > ($SECRET._max_retries|tonumber)"
}

Besides constants and secrets, you might access contextual information of the running workflow by using the $WORKFLOW reserved word. SonataFlow supports the following contextual keys:

  • id: The id of the running workflow definition

  • name: The name of the running workflow definition

  • instanceId: The id of the running workflow instance

  • headers: Optional map containing the headers, if any, of the invocation that started the running workflow instance

  • prevActionResult: In a foreach state using multiple actions per loop cycle, give access to the result of the previous action. See example

  • identity: Quarkus security identity

Therefore, the following function, for a serverless workflow definition whose id is expressionTest, will append the string worklow id is expressionTest to the message variable

{
      "name": "contextMessage",
      "type": "expression",
      "operation": ".message |= \"workflow id is \"+$WORKFLOW.id"
}

Customizing workflow context

In addition to the predefined keys mentioned previously, you can add your own keys to workflow context using Java Service Loader mechanism, by providing an implementation of class KogitoProcessContextResolverExtension

This feature was used to add quarkus security identity support, you can check source code as reference here.

Additional resources

Found an issue?

If you find an issue or any misleading information, please feel free to report it here. We really appreciate it!