Scheduling

Schedule chaos experiments to run automatically on a recurring basis

Overview

Chaos Mesh Schedules enable you to run chaos experiments automatically at specified times or intervals using cron syntax. This is useful for continuous chaos engineering, periodic resilience testing, and automated fault injection.

Schedule Structure

A Schedule wraps any chaos experiment type and executes it according to a defined schedule.

Basic Schedule

apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: schedule-pod-kill-example
spec:
  schedule: "@every 5s"
  type: "PodChaos"
  historyLimit: 5
  concurrencyPolicy: Forbid
  podChaos:
    action: "pod-kill"
    mode: one
    selector:
      labelSelectors:
        "app.kubernetes.io/component": "tikv"

Schedule Specification

schedule · string · required

Cron expression or predefined schedule defining when the chaos experiment runs.

Examples:

• "@every 5s" - Every 5 seconds
• "@every 1m" - Every minute
• "@every 1h" - Every hour
• "@hourly" - Same as 0 * * * *
• "@daily" - Same as 0 0 * * *
• "*/5 * * * *" - Every 5 minutes
• "0 9 * * 1-5" - 9 AM weekdays

type · string · required

The type of chaos experiment to schedule. Must be one of:

• PodChaos
• NetworkChaos
• StressChaos
• IOChaos
• TimeChaos
• HTTPChaos
• DNSChaos
• KernelChaos
• JVMChaos
• AWSChaos
• GCPChaos
• AzureChaos
• BlockChaos
• PhysicalMachineChaos

startingDeadlineSeconds · int64

Optional deadline in seconds for starting the job if it misses scheduled time for any reason. Missed jobs executions will be counted as failed ones.

Must be greater than 0.

concurrencyPolicy · ConcurrencyPolicy · default: "Forbid"

Specifies how to handle concurrent executions of the scheduled chaos experiment.

• Forbid (default): Do not allow concurrent runs. Skip new run if previous is still active.
• Allow: Allow concurrent runs. Multiple instances of the experiment can run simultaneously.

historyLimit · int · default: "3"

Number of completed chaos experiment records to retain for this schedule. Must be at least 1.

This helps manage resource usage by cleaning up old experiment objects.

Cron Syntax

Chaos Mesh supports the standard cron format with optional seconds field:

┌───────────── second (0 - 59, optional)
│ ┌───────────── minute (0 - 59)
│ │ ┌───────────── hour (0 - 23)
│ │ │ ┌───────────── day of month (1 - 31)
│ │ │ │ ┌───────────── month (1 - 12)
│ │ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday)
│ │ │ │ │ │
│ │ │ │ │ │
* * * * * *

Special Strings

@every <duration>

Run at the specified interval. Duration can use units: s, m, h.

Examples: @every 30s, @every 5m, @every 2h

@hourly

Run once an hour at the beginning of the hour. Equivalent to 0 * * * *.

@daily

Run once a day at midnight. Equivalent to 0 0 * * *.

@weekly

Run once a week at midnight on Sunday. Equivalent to 0 0 * * 0.

@monthly

Run once a month at midnight on the first day. Equivalent to 0 0 1 * *.

@yearly

Run once a year at midnight on January 1st. Equivalent to 0 0 1 1 *.

Schedule Examples

Network Chaos Every 5 Minutes

NetworkChaos Schedule

apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: network-delay-schedule
spec:
  schedule: "*/5 * * * *"
  type: NetworkChaos
  concurrencyPolicy: Forbid
  historyLimit: 10
  networkChaos:
    action: delay
    mode: all
    selector:
      namespaces:
        - production
      labelSelectors:
        "app": "api-server"
    delay:
      latency: "100ms"
      jitter: "50ms"
    duration: "2m"

Pod Chaos During Business Hours

Business Hours Schedule

apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: business-hours-pod-chaos
spec:
  schedule: "0 9-17 * * 1-5"  # 9 AM - 5 PM, Monday-Friday
  type: PodChaos
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  podChaos:
    action: pod-failure
    mode: fixed
    value: "1"
    selector:
      namespaces:
        - staging
      labelSelectors:
        "environment": "test"
    duration: "5m"

Stress Chaos Every 30 Seconds

High-Frequency Stress

apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: frequent-stress-test
spec:
  schedule: "@every 30s"
  type: StressChaos
  concurrencyPolicy: Allow  # Allow overlapping executions
  historyLimit: 20
  stressChaos:
    mode: one
    selector:
      namespaces:
        - load-test
    stressors:
      memory:
        workers: 4
        size: "256MB"
    duration: "20s"

Daily Database Chaos

Daily Schedule

apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
  name: daily-io-chaos
spec:
  schedule: "0 2 * * *"  # 2 AM daily
  type: IOChaos
  concurrencyPolicy: Forbid
  ioChaos:
    action: latency
    mode: all
    selector:
      namespaces:
        - database
      labelSelectors:
        "app": "postgres"
    volumePath: /var/lib/postgresql/data
    delay: "100ms"
    duration: "10m"

Schedule Status

status · ScheduleStatus

Current state of the schedule.

Show properties

active · []corev1.ObjectReference

List of currently running chaos experiments created by this schedule.

lastScheduleTime · metav1.Time

Time when the schedule last created a chaos experiment.

Managing Schedules

Creating a Schedule

kubectl

kubectl apply -f schedule.yaml

Viewing Schedule Status

Get Schedule Details

kubectl get schedule my-schedule -n chaos-mesh -o yaml

Pausing a Schedule

You can pause a schedule without deleting it by adding a pause annotation:

Pause Schedule

kubectl annotate schedule my-schedule \
  chaos-mesh.org/pause=true -n chaos-mesh

Deleting a Schedule

Delete Schedule

kubectl delete schedule my-schedule -n chaos-mesh

Deleting a schedule does not automatically delete chaos experiments that are currently running. Those will complete based on their duration settings.

Schedules in Workflows

You can embed scheduled chaos within workflow templates:

Schedule in Workflow

templates:
  - name: scheduled-pod-chaos
    templateType: Schedule
    deadline: 60s
    schedule:
      schedule: "@every 5s"
      concurrencyPolicy: Allow
      type: PodChaos
      podChaos:
        action: pod-kill
        mode: one
        selector:
          labelSelectors:
            "app": "web"

Workflows cannot be scheduled. The Schedule CRD only supports chaos experiment types, not the Workflow type itself.

Best Practices

Choose Appropriate Intervals

Balance testing frequency with system stability. Start with longer intervals and decrease gradually as confidence grows.

Set History Limits

Configure historyLimit to prevent accumulation of old experiment objects that consume cluster resources.

Use Forbid for Safety

Default to concurrencyPolicy: Forbid to prevent overlapping chaos experiments that could compound failures.

Consider Time Zones

Cron schedules use the timezone of the Chaos Mesh controller. Ensure your schedule accounts for the controller’s timezone.

Monitor Schedule Health

Track schedule execution through the dashboard or Prometheus metrics to detect missed executions or failures.

Next Steps

Workflows

Combine scheduled chaos with complex workflow orchestration

Status Checks

Add automated validation to scheduled experiments

Monitoring

Track schedule metrics with Prometheus

Dashboard

View and manage schedules through the web UI