IOChaos

Inject I/O failures and delays into file system operations

IOChaos allows you to simulate I/O failures, delays, and attribute modifications to test how your application handles file system issues and storage failures.

Supported Actions

IOChaos supports four types of I/O chaos:

• latency: Add delay to I/O operations
• fault: Return error codes from I/O operations
• attrOverride: Override file attributes (permissions, timestamps, size, etc.)
• mistake: Inject incorrect data into read/write operations

Configuration

Basic Example

apiVersion: chaos-mesh.org/v1alpha1
kind: IOChaos
metadata:
  name: io-delay-example
spec:
  action: latency
  mode: one
  selector:
    labelSelectors:
      app: etcd
  volumePath: /var/run/etcd
  path: /var/run/etcd/**/*
  delay: "10ms"
  percent: 10
  duration: "400s"

Spec Fields

action · string · required

The I/O chaos action to perform. Must be one of:

• latency: Add delay to I/O operations
• fault: Return errno from I/O operations
• attrOverride: Override file attributes
• mistake: Inject incorrect data

selector · PodSelectorSpec · required

Specifies the target pods for the chaos experiment. See PodChaos documentation for selector details.

mode · string · required

Selection mode: one, all, fixed, fixed-percent, or random-max-percent

volumePath · string · required

The mount path of the volume to inject I/O chaos. Must be a root of a mount path.

path · string

The path of files for injecting I/O chaos action. Supports glob patterns (e.g., /var/run/etcd/**/*)

percent · integer · default: "100"

Percentage of I/O operations to affect (0-100)

duration · string

Duration of the chaos action. Format: “300ms”, “1.5h”, “2h45m”

containerNames · string[]

List of container names to inject chaos into. If not set, the first container will be injected.

Action-Specific Fields

delay · string

Amount of delay to add (required when action is latency). Format: “300ms”, “10ms”, “1s”. Valid units: ns, us, ms, s, m, h.

errno · uint32

Error code to return from I/O operations (required when action is fault). See Linux error codes

attr · AttrOverrideSpec

File attributes to override (required when action is attrOverride)

Show AttrOverrideSpec fields

ino · uint64

Inode number

size · uint64

File size in bytes

blocks · uint64

Number of blocks

atime · Timespec

Access time

Show Timespec fields

sec · int64

Seconds since epoch

nsec · int64

Nanoseconds

mtime · Timespec

Modification time

ctime · Timespec

Change time

kind · FileType

File type: namedPipe, charDevice, blockDevice, directory, regularFile, symlink, socket

perm · uint16

File permissions (octal, e.g., 72 for 0110)

nlink · uint32

Number of hard links

uid · uint32

User ID

gid · uint32

Group ID

rdev · uint32

Device ID (if special file)

mistake · MistakeSpec

Mistake injection configuration (required when action is mistake)

Show MistakeSpec fields

filling · string

Type of data to fill: zero or random

maxOccurrences · int64

Maximum number of wrong data segments (minimum: 1)

maxLength · int64

Maximum length of each wrong data segment in bytes (minimum: 1)

methods · IoMethod[]

I/O methods to inject chaos into. If not specified, all methods are affected. Available methods:

lookup, forget, getattr, setattr, readlink, mknod, mkdir, unlink, rmdir, symlink, rename, link, open, read, write, flush, release, fsync, opendir, readdir, releasedir, fsyncdir, statfs, setxattr, getxattr, listxattr, removexattr, access, create, getlk, setlk, bmap

remoteCluster · string

Remote cluster where chaos will be deployed

Examples

I/O Latency

Add 10ms delay to 10% of I/O operations:

apiVersion: chaos-mesh.org/v1alpha1
kind: IOChaos
metadata:
  name: io-delay-example
spec:
  action: latency
  mode: one
  selector:
    labelSelectors:
      app: etcd
  volumePath: /var/run/etcd
  path: /var/run/etcd/**/*
  delay: "10ms"
  percent: 10
  duration: "400s"

I/O Errors

Return errno 5 (EIO - I/O error) for 50% of I/O operations:

apiVersion: chaos-mesh.org/v1alpha1
kind: IOChaos
metadata:
  name: io-errno-example
spec:
  action: fault
  mode: one
  selector:
    labelSelectors:
      app: etcd
  volumePath: /var/run/etcd
  path: /var/run/etcd/**/*
  errno: 5
  percent: 50
  duration: "400s"

Attribute Override

Override file permissions:

apiVersion: chaos-mesh.org/v1alpha1
kind: IOChaos
metadata:
  name: io-attr-example
spec:
  action: attrOverride
  mode: one
  selector:
    labelSelectors:
      app: etcd
  volumePath: /var/run/etcd
  path: /var/run/etcd/**/*
  attr:
    perm: 72
  percent: 10
  duration: "400s"

Use Cases

Testing Disk Latency

Use the latency action to simulate slow storage devices or network-attached storage delays.

Simulating Disk Failures

Use the fault action with appropriate error codes to test how your application handles I/O errors, disk failures, or quota exceeded scenarios.

Permission Testing

Use attrOverride to modify file permissions and test your application’s error handling for permission denied scenarios.

Data Corruption Testing

Use the mistake action to inject random or zero data to test data validation and corruption detection mechanisms.

Best Practices

1. Volume Path Selection: Ensure volumePath points to a mount point root, not a subdirectory
2. Path Patterns: Use glob patterns to target specific files or directories (e.g., *.db for database files)
3. Percentage Control: Start with low percentages (10-20%) to avoid completely blocking I/O
4. Method Filtering: Use the methods field to target specific I/O operations (e.g., only read or write)
5. Error Code Selection: Choose appropriate errno values that match real-world failure scenarios:
   • errno 5 (EIO): I/O error
   • errno 28 (ENOSPC): No space left on device
   • errno 13 (EACCES): Permission denied
6. Monitor Application: Watch application logs and metrics to understand I/O chaos impact
7. Container Selection: Specify containerNames when working with multi-container pods

Notes

• IOChaos is implemented using FUSE (Filesystem in Userspace) and requires the volume to be remounted
• The chaos daemon intercepts I/O operations at the file system level
• Path supports glob patterns for flexible file matching
• All file times in attrOverride use Unix timestamp format
• File permissions in perm should be specified as octal values (e.g., 72 decimal = 0110 octal)
• The mistake action can help test data validation and checksumming mechanisms
• IOChaos requires the target pod to use a volume mount (not hostPath or emptyDir in some cases)