Workflows

Orchestrate complex chaos experiments with serial and parallel task execution

Overview

Workflows in Chaos Mesh enable you to orchestrate complex chaos engineering scenarios by combining multiple chaos experiments and tasks. Workflows support serial execution, parallel execution, conditional branching, and custom tasks, providing powerful capabilities for testing system resilience.

Workflow Structure

A Workflow consists of an entry point and a collection of templates. Each template defines a specific step in the workflow execution.

Basic Workflow Structure

apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
  name: example-workflow
spec:
  entry: the-entry  # Starting template name
  templates:
    - name: the-entry
      templateType: Serial
      children:
        - step-1
        - step-2

Template Types

Serial Templates

Serial templates execute child templates sequentially, one after another. Each step must complete before the next begins.

Serial Execution

apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
  name: try-workflow-serial
spec:
  entry: the-entry
  templates:
    - name: the-entry
      templateType: Serial
      deadline: 240s
      children:
        - workflow-stress-chaos
        - prefix-suspending
        - workflow-network-chaos
        - suffix-suspending
        - workflow-pod-chaos
    
    - name: prefix-suspending
      templateType: Suspend
      deadline: 10s
    
    - name: workflow-network-chaos
      templateType: NetworkChaos
      deadline: 20s
      networkChaos:
        direction: to
        action: delay
        mode: all
        selector:
          labelSelectors:
            "app": "hello-kubernetes"
        delay:
          latency: "90ms"
          correlation: "25"
          jitter: "90ms"
    
    - name: workflow-stress-chaos
      templateType: StressChaos
      deadline: 20s
      stressChaos:
        mode: one
        selector:
          labelSelectors:
            "app": "hello-kubernetes"
        stressors:
          cpu:
            workers: 1
            load: 20
            options: ["--cpu 1", "--timeout 600"]
    
    - name: suffix-suspending
      templateType: Suspend
      deadline: 5s
    
    - name: workflow-pod-chaos
      templateType: Schedule
      deadline: 40s
      schedule:
        schedule: "@every 2s"
        concurrencyPolicy: Allow
        type: PodChaos
        podChaos:
          action: pod-kill
          mode: one
          selector:
            labelSelectors:
              "app": "hello-kubernetes"

Parallel Templates

Parallel templates execute all child templates simultaneously, allowing you to inject multiple chaos experiments at once.

Parallel Execution

apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
  name: try-workflow-parallel
spec:
  entry: the-entry
  templates:
    - name: the-entry
      templateType: Parallel
      deadline: 240s
      children:
        - workflow-stress-chaos
        - workflow-network-chaos
        - workflow-pod-chaos
    
    - name: workflow-network-chaos
      templateType: NetworkChaos
      deadline: 20s
      networkChaos:
        direction: to
        action: delay
        mode: all
        selector:
          labelSelectors:
            "app": "hello-kubernetes"
        delay:
          latency: "90ms"
          correlation: "25"
          jitter: "90ms"
    
    - name: workflow-pod-chaos
      templateType: Schedule
      deadline: 40s
      schedule:
        schedule: "@every 2s"
        concurrencyPolicy: Allow
        type: PodChaos
        podChaos:
          action: pod-kill
          mode: one
          selector:
            labelSelectors:
              "app": "hello-kubernetes"
    
    - name: workflow-stress-chaos
      templateType: StressChaos
      deadline: 20s
      stressChaos:
        mode: one
        selector:
          labelSelectors:
            "app": "hello-kubernetes"
        stressors:
          cpu:
            workers: 1
            load: 20
            options: ["--cpu 1", "--timeout 600"]

Suspend Templates

Suspend templates pause workflow execution for a specified duration, useful for adding delays between chaos experiments.

Suspend Template

templates:
  - name: pause-for-observation
    templateType: Suspend
    deadline: 30s

Task Templates

Task templates allow you to run custom Kubernetes containers as part of your workflow, enabling custom validation, setup, or teardown logic.

Custom Task

templates:
  - name: custom-validation
    templateType: Task
    deadline: 60s
    task:
      container:
        name: validation
        image: my-validation:latest
        command: ["./validate.sh"]
      volumes:
        - name: config
          configMap:
            name: validation-config

Conditional Branches

Conditional branches in Task templates allow dynamic workflow paths based on expression evaluation.

Conditional Branching

templates:
  - name: decision-task
    templateType: Task
    task:
      container:
        name: decision
        image: decision-maker:latest
    conditionalBranches:
      - target: high-load-path
        expression: "context.result == 'high'"
      - target: low-load-path
        expression: "context.result == 'low'"
      - target: default-path
        # Empty expression always evaluates to true

Workflow Status

status · WorkflowStatus

Represents the current state of the workflow execution.

Show properties

entryNode · string

Name of the entry node created for this workflow.

startTime · metav1.Time

Time when workflow execution started.

endTime · metav1.Time

Time when workflow execution completed.

conditions · []WorkflowCondition

Latest observations of workflow state.

Condition types:

• Accomplished: Workflow has completed all steps
• Scheduled: Workflow has been scheduled for execution

Workflow Management

Creating a Workflow

kubectl

kubectl apply -f workflow.yaml

Viewing Workflow Status

Check Status

kubectl get workflow my-workflow -n chaos-mesh -o yaml

Aborting a Workflow

You can abort a running workflow by adding an annotation:

Abort Workflow

kubectl annotate workflow my-workflow \
  workflow.chaos-mesh.org/abort=true -n chaos-mesh

Deleting a Workflow

Delete Workflow

kubectl delete workflow my-workflow -n chaos-mesh

Best Practices

Set Appropriate Deadlines

Always specify deadlines for templates to prevent indefinite execution. The deadline should account for the expected duration plus a safety margin.

Use Suspend for Observation

Add Suspend templates between chaos experiments to allow time for system observation and metric collection.

Combine Serial and Parallel

Nest Serial and Parallel templates to create complex execution patterns. For example, run multiple chaos types in parallel within each step of a serial workflow.

Include Status Checks

Incorporate StatusCheck templates to validate system health during workflow execution. See Status Checks for details.

Common Template Types

All chaos types can be used as workflow templates:

• PodChaos - Pod lifecycle faults
• NetworkChaos - Network delay, loss, corruption
• StressChaos - CPU and memory stress
• IOChaos - I/O delays and failures
• TimeChaos - Time skew simulation
• HTTPChaos - HTTP request/response manipulation
• DNSChaos - DNS resolution faults
• KernelChaos - Kernel-level faults
• JVMChaos - JVM application faults
• AWSChaos, GCPChaos, AzureChaos - Cloud provider faults

Next Steps

Scheduling

Schedule workflows and experiments to run on a recurring basis

Status Checks

Validate experiment outcomes and system health

Dashboard

Monitor and manage workflows through the web UI

Monitoring

Track workflow metrics with Prometheus