pipe_io – Pipe IO Wrapper

Last updated March 2020

The pipe_io module provides facilities for writing content via an arbitrary program. Momentum uses an I/O abstraction layer for writing logs. With this module, you can post-process or transform logs.

pipe_io should not be used for post-processing the paniclog on Momentum nodes.

Configuration

This module is loaded automatically as required and does not need to be explicitly included. The configuration is in two parts: module-wide error logging and options for a specific pipe.

71.54.1.1. Module Options

error_file: If the invoked program writes to STDERR, log it to this file. If this is not set, nothing will be logged.
error_mode: When creating error_file, use this file mode. Defaults to 0666.
output_file: If the invoked program writes to STDOUT, log it to this file. If this is not set, nothing will be logged.
output_mode: When creating output_file, use this file mode. Defaults to 0666.
timeout: When closing the pipe, timeout specifies how many seconds pipe_io should wait for the program to exit normally. If the program does not exit normally, pipe_io will terminate it.

71.54.1.2. URI Parameters

The URI specifies the path to the program that should be connected to the read end of the pipe; for example, pipe:///usr/bin/myscript. The URI parameters specify the environment and arguments for the program, and how to handle errors.

argN: The Nth command-line parameter.
clear_env: If 1, remove all variables from the environment. This can be used with envN to specify a complete environment for your program. By default, the program will inherit the environment of the parent process.
envN: Insert a variable into the environment. The value is of the form name=value.
reinvoke: If 1, then one attempt will be made to restart the program if it exits unexpectedly. This is the default. Otherwise no attempt will be made to restart the program.
shutdown_on_failure: If 1, shutdown if the program cannot be started, restarted, or exits unexpectedly. This is the default behavior One attempt will be made to restart the program (see reinvoke) -- the shutdown will occur if that fails.

If 0, no shutdown will occur, but data will be lost silently.

Post-processing of Clustered Logs

The following example shows the use of pipe_io in eccluster.conf for post-processing of logs on the cluster manager.

pipe_io {
  output_file = "/var/log/eccluster/pipe_io_output.log"
  output_mode = 0664
  error_file = "/var/log/eccluster/pipe_io_error.log"
  error_mode = 0664
  timeout = 5
}

Logs {
  # arg1 = path, last component is the log name
  # arg2 = log name
  # arg3 = node name
  logfile = "pipe:///opt/msys/ecelerity/bin/myscript?arg1=/var/log/eccluster/%Y/%m/%d/%{l}&arg2=%{l}&arg3=%{n}"
  file_mode = "0640"
  dir_mode = "0755"

  # Reopen logs every ten minutes, if there's no activity.
  # The pipe reader is expected to manage its own file handles
  # intelligently, so there is less need for eccmgr to age them out.
  max_idle = "600"
}

Other Considerations

Pipe readers should not do intensive post-processing, or any processing that will introduce significant latency (e.g., database access).
Pipe readers should be aware that the write end of the pipe is closed when logs are re-opened or when the instance shuts down.
To use this module on Linux, add export ECELERITY_MEMORY=NOREPLACE to /opt/msys/ecelerity/etc/environment.

pipe_io – Pipe IO Wrapper

Configuration#

Post-processing of Clustered Logs#

Other Considerations#

See Also#

Configuration

Post-processing of Clustered Logs

Other Considerations

See Also