Task Chaining with qlik-cli

Overview

Reloading data into Qlik applications is a necessary component of the BI lifecycle. In this tutorial, you are going to learn how to use the reload API to create tasks, loop to catch status changes in a reload, and how to think about setting up task chaining using a script.

Prerequisites

qlik-cli version 1.0.0 or greater - Text editor for manipulating PowerShell or Bash scripts

Reload using CLI basics

You can access reloads using qlik-cli through the qlik reload command. Running the command with the --help flag lists the different sub-commands and flags available.

Create a reload task

To use the create command, first obtain the app metadata of the application using the item command.

$appName = #the name of the app to start reloading (e.g. cli-app-1)
$app = qlik item ls --resourceType app --name $appName | ConvertFrom-Json

Note: Qlik Sense Apps in Qlik Cloud have an itemId and a resourceId. The resourceId is often referred to as the appId and is a GUID. The itemId is the unique ID of the app in the items collection.

Issue the create command referencing the resourceId property from the app metadata object.

$reload = qlik reload create --appId $app[0].resourceId | ConvertFrom-Json
Write-Host $reload

The response returns all of the relevant information related to the newly created reload entity. If you add a -q to the end of the CLI command, only the ID for the reload task is returned. This can be useful when chaining multiple commands together.

{
  "appId": "dae9a0b6-ef7b-4cf5-8153-126ef9aef3b0",
  "creationTime": "2020-06-29T13:32:03.833Z",
  "id": "5ef9ed53a45bbd0001116aa9",
  "status": "CREATED",
  "tenantId": "TohbNbUjAGrVXhrvVujU73hzgDQ6DxnG",
  "type": "hub",
  "userId": "BewQZxFax8r47sLrGUmfy4K8YKzYAkjA"
}

‎

Check reload status

Check the status of the reload you created by using the get command and passing the item ID. The item ID is the value in the response from the create command.

$itemId = $reload.id
$reloadStatus = qlik reload get  | ConvertFrom-Json

Because the reload API is RESTful, the commands in the CLI are request/response in nature. The system doesn’t notify you when the reload reaches a completed state. To evaluate the status of reloads, put the get command in a loop.

function loopReloadStatus {
    param([String] $itemId)
    do {
        $reloadStatus = qlik reload get $itemId | ConvertFrom-Json
        Start-Sleep -Seconds 1.5
    } until ($reloadStatus.status -eq 'SUCCEEDED' -or $reloadStatus.status -eq 'FAILED')
    return $reloadStatus.status
}
loopReloadStatus $itemId

As long as the status of the reload isn’t SUCCEEDED or FAILED, the loop keeps running.

Task chaining example

The ability to execute a reload upon the completion status of a preceding reload is a desired capability in the Qlik platform. Task chaining, as this process is commonly called, needs to be able to accept an instruction set to know what to do based on the outcome of a previously run reload. qlik-cli isn’t opinionated with regard to the instruction set because there aren’t specific task chaining commands available in the tool. That said, functions like the loopReloadStatus example and a mechanism to facilitate the next step in an instruction set use the standard reload CLI commands, giving you the flexibility to design an instruction set that works for your requirements.

Make a task chain instruction set

In this example, the instruction set is a JSON array with simple logic. Each entry in the array has a name for the task, the name of the Qlik app to run the reload command, the array item (by number) to go to next upon a SUCCEEDED reload outcome, and the array item (by number) to go to next upon a FAILED outcome. A -1 value in the succeed or failed properties instructs the process to end.

    {
        "name": "task1",
        "run": "cli-app-0",
        "SUCCEEDED": 1,
        "FAILED": -1
    },

Check reload success or failure

The Check reload status section covers the loopReloadStatus function that observes the reload task status before going to the next step.

Run the task chain

To run the task chain in a logical fashion, employ another function called runTaskChainItem that accepts two arguments: one passing the instruction set, the $taskObject and another passing the specific item to run, the $task.

function runTaskChainItem {
param([System.Array] $taskObject, [System.Object] $task)
}

The function gets the app name to reload and calls qlik item ls to retrieve the application properties.

    $appName = $task.run
    $app = qlik item ls --resourceType app --name $appName | ConvertFrom-Json

With the app properties returned, run qlik reload create passing the resourceId from the app properties to start the reload task.

    $reload = qlik reload create --appId $app[0].resourceId | ConvertFrom-Json

Once created, the loopReloadStatus function is run passing in the reload ID from the previous command. The function loops until a success or failure status prints in the response from the tenant.

    $reloadResult = loopReloadStatus -itemId $reload.id

The $reloadResult variable contains SUCCEEDED or FAILED for a value. The next command evaluates the value to return from a matching property name.

    Write-Host "Reload for $($task.name) completed in $($EndMs-$StartMs) milliseconds with the result $reloadResult"

Based on the $result variable value, the function either runs the corresponding task item in the instruction set or finishes running.

    if($task."$result" -ne -1)
    {
        runTaskChainItem -taskObject $taskObject -task $taskObject[$task."$result"]
    } else {
        Write-Host 'Task chain execution complete'
    }

Now that the functions and instruction set are written, call the runTaskChainItem function.

$taskObject = $taskChain | ConvertFrom-Json
runTaskChainItem -taskObject $taskObject -task $taskObject[0]

Summary

Reloading apps is useful to obtain the latest data for a Qlik app. But for large deployments, reloads are multi-layered and often generate datasets as part of a downstream app reload. That’s why chaining reloads together is so useful. Using qlik-cli, you can combine reloads and tasks with operational workflows from other applications to increase efficiency and automate insight to action.

Complete scripts

#reload-task-chaining provides two functions for checking reload status and
#looping through an instruction set to direct reload events through a chain.

#This is a sample instruction set identifying where to go in the array if the
#named reload succeeds or fails. -1 means done.
$taskChain = '[
    {
        "name": "task1",
        "run": "cli-app-0",
        "SUCCEEDED": 1,
        "FAILED": -1
    },
    {
        "name": "task2",
        "run": "cli-app-1",
        "SUCCEEDED": 2,
        "FAILED": 3
    },
    {
        "name": "task3",
        "run": "cli-app-2",
        "SUCCEEDED": 3,
        "FAILED": -1
    },
    {
        "name": "task4",
        "run": "cli-app-3",
        "SUCCEEDED": -1,
        "FAILED": -1
    }
]'

#calls the reload command with a specific item id to check status.
function loopReloadStatus {
    param([String] $itemId)
    do {
        $reloadStatus = qlik reload get $itemId | ConvertFrom-Json
        Start-Sleep -Seconds 1.5
    } until ($reloadStatus.status -eq 'SUCCEEDED' -or $reloadStatus.status -eq 'FAILED')
    return $reloadStatus.status
}

#Given an array and a task object, executes the reload, calls loopReloadStatus,
#and returns the result to determine next action in chain.
function runTaskChainItem {
param([System.Array] $taskObject, [System.Object] $task)

    Write-Host "Running $($task.name)"
    $appName = $task.run
    $app = qlik item ls --resourceType app --name $appName | ConvertFrom-Json

    $StartMs = (Get-Date).Millisecond | Out-String
    
    $reload = qlik reload create --appId $app[0].resourceId | ConvertFrom-Json
    $reloadResult = loopReloadStatus -itemId $reload.id

    $EndMs = (Get-Date).Millisecond | Out-String
    
    Write-Host "Reload for $($task.name) completed in $($EndMs-$StartMs) milliseconds with the result $reloadResult"

    $result = $($task | Get-Member -MemberType *Property).Name -eq $reloadResult
    
    if($task."$result" -ne -1)
    {
        runTaskChainItem -taskObject $taskObject -task $taskObject[$task."$result"]
    } else {
        Write-Host 'Task chain execution complete'
    }

}

#kick off the script when executed
$taskObject = $taskChain | ConvertFrom-Json
runTaskChainItem -taskObject $taskObject -task $taskObject[0]
loopReloadStatus $itemId

#!/bin/bash

#reload-task-chaining provides two functions for checking reload status and
#looping through an instruction set to direct reload events through a chain.

#This is a sample instruction set identifying where to go in the array if the
#named reload succeeds or fails. -1 means done.

JSON='[
    {
        "name": "task1",
        "run": "cli-app-7",
        "SUCCEEDED": 1,
        "FAILED": -1
    },
    {
        "name": "task2",
        "run": "cli-app-8",
        "SUCCEEDED": 2,
        "FAILED": 3
    },
    {
        "name": "task3",
        "run": "cli-app-9",
        "SUCCEEDED": 3,
        "FAILED": -1
    },
    {
        "name": "task4",
        "run": "cli-app-10",
        "SUCCEEDED": -1,
        "FAILED": -1
    }
]'

#Given an array and a task object, executes the reload, calls loopReloadStatus,
#and returns the result to determine next action in chain.
runTaskChainItem(){
  #pass the full array into the function, and the first task item to execute
  taskObject=$1
  task=$2

  appName=$(echo ${task} | jq -r '.run')
  echo "Running ${appName}"

  appId=$(qlik item ls --resourceType app --name ${appName} | jq -r '.[0].resourceId')

  SECONDS=0
  
  #create the reload
  reload=$(qlik reload create --appId ${appId} | jq -r '.id')
  reloadStatus=

  until  [ "$reloadStatus" = "SUCCEEDED" ] || [ "$reloadStatus" = "FAILED" ]; do
      reloadStatus=$(qlik reload get ${reload} | jq -r '.status')
      echo "Current reload status: ${reloadStatus}"
      sleep 1.5
  done

  # #run the loopReloadStatus function
  # reloadResult=$(loopReloadStatus ${reload})

  duration=$SECONDS
  echo "Reload for $appName $reloadResult in $(($duration / 60)) minutes and $(($duration % 60)) seconds."

  result=$(echo $task | jq -r --arg v "$reloadStatus" '.[$v]')

  if [ $((result)) -ne -1 ]
  then
    taskToRun=$(echo $taskObject | jq --arg v $result '.[$v|tonumber]')
    runTaskChainItem "$taskObject" "$taskToRun"
  else
    echo "Task chain execution complete"
  fi
}

runTaskChainItem "$(jq -r '.' <<< ${JSON})" "$(jq -r '.[0]' <<< ${JSON})"

Overview
Prerequisites
Reload using CLI basics
Create a reload task
Check reload status
Task chaining example
Summary
Complete scripts