Monitoring long-running processes#

One of the most powerful features of chalk exec is its ability to generate periodic “heartbeat” reports that continue for as long as your application runs.

Setting Up Heartbeat Monitoring#✦

Let’s first create a configuration file that enables heartbeat reporting:

Terminal

# Create a heartbeat configuration file
$ cat << EOF > heartbeat_config.c4m
# Enable heartbeat and set interval to 10 seconds
exec.heartbeat.run: true
exec.heartbeat.rate: <<10 seconds>>

# Define what to include in heartbeat reports
report_template heartbeat_report {
  key._PROCESS_PID.use                        = true
  key._PROCESS_STATE.use                      = true
  key._PROCESS_CWD.use                        = true
  key._TIMESTAMP.use                          = true
  key._OP_TCP_SOCKET_INFO.use                 = true
  key._DATETIME.use                           = true
}

# Use this template for heartbeat operations
outconf.heartbeat.report_template: "heartbeat_report"
EOF

Now, let’s create a long-running program to monitor:

Terminal

# Create a script that runs for a while
$ cat << EOF > long-running.sh
#!/bin/bash
echo "Starting long-running process..."
count=1
while [[ \$count -le 5 ]]; do
  echo "Iteration \$count"
  sleep 15
  ((count=count+1))
done
echo "Process complete."
EOF

$ chmod +x long-running.sh

Load our heartbeat configuration and run the program:

Terminal

# Load the heartbeat configuration
$ chalk load heartbeat_config.c4m

# Run with exec and heartbeat enabled
$ chalk exec -- ./long-running.sh

All reports are by default logged to ~/.local/chalk/chalk.log.

If you tail -f that log file, you’ll see a new entry for the heartbeat report generated every 10 seconds as specified in the configuration, providing regular snapshots of your application’s state as it runs.

Customizing Exec Reports#✦

You can customize what data gets collected and how it’s reported by configuring Chalk’s reporting templates. Let’s explore a few examples:

Customizing Output Location#✦

Let’s create a configuration that sends exec reports to a specific file:

Terminal

# Create a file output configuration
$ cat << EOF > file-output_config.c4m
# Define a sink for file output
sink_config exec_file_output {
  sink:     "file"
  enabled:  true
  filename: "./exec-reports.log"
}

# Subscribe our new sink to the report topic
subscribe("report", "exec_file_output")
EOF

# Load the configuration
$ chalk load file-output_config.c4m

# Run a command with the new configuration
$ chalk exec -- ls -la

Now check the contents of exec-reports.log to see the exec report that was written to the file.