PowerMTA Pipe Accounting to External Scripts: Complete 2026 Operator Guide

Why pipe accounting matters

PowerMTA accounting records are the authoritative data on what happened to every message: delivered, bounced, deferred, complained. By default these records go to a CSV file on disk that a downstream process reads periodically. Pipe accounting offers an alternative: instead of, or alongside, the file, PowerMTA streams each record to an external script in real time. The record reaches the consumer the moment PowerMTA generates it.

This guide exists because pipe accounting is powerful for real-time use cases but has a reliability consideration that operators must understand before using it: a pipe has no persistence, so a consumer failure can lose data. Used well, with the right protections, pipe accounting enables real-time analytics, instant alerting, and live dashboards. Used naively, it silently loses accounting records. The structure of this guide: the difference between pipe and file accounting, the acct-file pipe configuration, the record format the pipe carries, building a robust consumer script, the consumer process lifecycle and what happens when it dies, buffering and backpressure, real-time ingestion patterns, the strong recommendation to run pipe and file together, and the diagnostic workflow when pipe accounting stops flowing.

Pipe accounting versus file accounting

The two accounting delivery methods differ in fundamental ways:

Aspect	File accounting	Pipe accounting
Destination	CSV file on disk	External program via pipe
Latency	File rotation or polling interval	Real time, immediate
Persistence	Records durable on disk	No persistence in the pipe itself
Failure risk	Low, records survive on disk	Consumer failure can lose records
Best for	Durable record, periodic analytics	Real-time processing, alerting, dashboards
Complexity	Simple, well-understood	Requires a robust consumer

File accounting writes records to a CSV file. The records sit durably on disk; a downstream process reads the file periodically, typically after rotation or on a polling interval. The latency is minutes, and the records survive any downstream problem because they are on disk.

Pipe accounting streams records to a consumer program in real time. PowerMTA launches the program and feeds records to its standard input as they are generated. The latency is effectively zero, but the pipe has no persistence: a record that the consumer fails to process is gone.

The trade-off is real time versus durability. The good news is that an operator does not have to choose: both can be configured at once, which is the recommended pattern for most production deployments.

The acct-file pipe configuration

Pipe accounting is configured through the acct-file directive, the same directive used for file accounting, but pointed at a program instead of a file path.

File accounting looks like:

<acct-file /var/log/pmta/accounting.csv>
    records d,b,t,rb,f
    max-size 500M
</acct-file>

Pipe accounting points acct-file at a program, typically with a pipe prefix or a syntax that tells PowerMTA the target is a program to execute rather than a file. The exact syntax follows the PowerMTA documentation for the version in use, but the structure is: PowerMTA launches the named program and feeds accounting records to its standard input.

<acct-file |/opt/pmta/scripts/acct-consumer.py>
    records d,b,t,rb,f
</acct-file>

The leading pipe character indicates the target is a program. PowerMTA executes /opt/pmta/scripts/acct-consumer.py and streams records to its standard input.

The records directive works the same as for file accounting: it specifies which record types are included. Common selections include d (delivered), b (bounced), t (transient failure), rb (received bounce), f (feedback loop complaint). Selecting only the record types the consumer needs reduces the volume the pipe carries.

An operator can configure multiple acct-file directives, which is how pipe and file accounting run together: one acct-file pointing at the durable CSV, another pointing at the pipe consumer. Both receive the same records.

The record format the pipe carries

The records the pipe carries are the same PowerMTA accounting records that go to the CSV file: comma-separated values, one record per line, with a defined set of fields per record type.

The accounting record types:

Type	Meaning
d	Delivered, message accepted by the receiver
b	Bounced, permanent failure
t	Transient failure, will be retried
rb	Received bounce
r	Received message
f	Feedback loop complaint
tq	Queued
rs	Resent

Each record carries fields relevant to its type: the timestamp, the record type, the recipient, the recipient domain, the source IP, the VMTA, the job identifier, the DSN status code, the diagnostic text, and others depending on type. The consumer parses these fields the same way a CSV reader would, splitting on commas and mapping to the field names.

The first line PowerMTA sends down the pipe is typically a header row naming the fields, the same header that appears at the top of the CSV file. A robust consumer reads the header to learn the field order rather than assuming a fixed layout, because the field set can vary with the record types selected and the PowerMTA version.

The consumer should handle the records as a stream: read a line, parse it, process it, repeat. Records arrive continuously as PowerMTA generates them, so the consumer is a long-running process reading from standard input indefinitely.

Building a robust pipe consumer

The pipe consumer is the program PowerMTA feeds records to. Building it robustly is the key to reliable pipe accounting.

A basic consumer skeleton:

#!/usr/bin/env python3
import sys
import csv

def process_record(record):
    # Insert into database, push to queue,
    # update metrics, trigger alerts, etc.
    pass

def main():
    reader = csv.reader(sys.stdin)
    header = None
    for row in reader:
        try:
            if header is None:
                header = row
                continue
            record = dict(zip(header, row))
            process_record(record)
        except Exception as e:
            # Never let one bad record kill the consumer
            sys.stderr.write(f"record error: {e}\n")
            continue

if __name__ == '__main__':
    main()

The robustness principles this skeleton illustrates:

Never let one bad record kill the process. The per-record try/except is essential. A malformed record, an unexpected field, a parsing error, none of these should crash the consumer. The consumer logs the error and moves to the next record. A consumer that crashes on a bad record loses every record after it until restarted.

Read the header for field mapping. The consumer reads the first line as the header and maps subsequent rows to field names, rather than assuming a fixed field order. This makes the consumer resilient to field set changes.

Keep processing fast. The process_record function should be fast. If it does slow work, that work should be deferred (queued for a separate worker, batched) rather than done inline, because a slow consumer creates backpressure.

Handle exceptions everywhere. Any operation that can fail, a database insert, a network call, a queue push, must be wrapped so its failure does not crash the consumer.

Flush and exit cleanly on EOF. When standard input closes (PowerMTA stopped feeding records), the consumer should flush any buffered work and exit cleanly.

The consumer process lifecycle

Understanding what happens to the consumer process is the single most important reliability consideration for pipe accounting.

PowerMTA launches the consumer when accounting starts and feeds it records. The consumer runs as long as PowerMTA is feeding it. The reliability question is what happens when the consumer stops, whether it crashes, hangs, or exits.

If the consumer exits or crashes. PowerMTA may restart it, but records generated in the gap between the consumer dying and being restarted can be lost. The pipe has no persistence; a record handed to a dead pipe is gone. This is the fundamental difference from file accounting, where records sit durably on disk regardless of any downstream problem.

If the consumer hangs. A consumer that stops reading from standard input but does not exit creates backpressure. PowerMTA's records back up, and depending on configuration this can affect PowerMTA's operation. A hung consumer is in some ways worse than a crashed one, because it does not trigger a restart.

If the consumer is slow. A consumer that reads records but processes them slower than PowerMTA generates them also creates backpressure as the buffer fills.

The protections against these failure modes:

• Write the consumer defensively so it does not crash on bad input.
• Keep the consumer fast so it does not create backpressure.
• Add a watchdog that detects a hung or dead consumer and alerts.
• Run file accounting alongside the pipe so the file can backfill any gap.

Buffering and backpressure

Backpressure is what happens when the consumer cannot keep pace with PowerMTA's record generation rate.

PowerMTA generates accounting records as fast as it processes messages. At high volume this is a high rate. If the consumer processes records slower than that rate, the records buffer. The buffer is finite; when it fills, backpressure propagates back toward PowerMTA.

The mitigations:

Make the consumer fast. The most direct mitigation. The consumer's per-record work must be fast enough to keep pace. If the natural per-record work is slow, decouple it.

Decouple slow work. The consumer that reads from the pipe should do minimal work: parse the record and hand it to something else. The something else, a message queue, an in-memory buffer drained by worker threads, a batch accumulator, does the slow work asynchronously. This keeps the pipe-reading path fast regardless of how slow the eventual processing is.

#!/usr/bin/env python3
import sys, csv, queue, threading

record_queue = queue.Queue(maxsize=100000)

def worker():
    batch = []
    while True:
        record = record_queue.get()
        batch.append(record)
        if len(batch) >= 1000:
            # Slow work (DB insert) happens here, off the pipe path
            flush_batch(batch)
            batch = []

def main():
    threading.Thread(target=worker, daemon=True).start()
    reader = csv.reader(sys.stdin)
    header = None
    for row in reader:
        try:
            if header is None:
                header = row
                continue
            record_queue.put(dict(zip(header, row)))
        except Exception as e:
            sys.stderr.write(f"error: {e}\n")

if __name__ == '__main__':
    main()

This pattern keeps the pipe-reading loop doing only fast work (parse, enqueue) while a worker thread does the slow batch processing. The internal queue absorbs short bursts. The pipe-reading path stays fast enough to avoid backpressure even when the eventual processing is slow.

Batch the slow work. Database inserts in particular are far faster batched than one at a time. The worker accumulates records and inserts them in batches, which dramatically increases throughput.

Real-time ingestion patterns

The common use of pipe accounting is real-time ingestion into an analytics system.

ClickHouse ingestion. ClickHouse is well-suited to PowerMTA accounting data because of its analytical query performance. The pipe consumer batches records and inserts them into a ClickHouse table. ClickHouse's batch insert performance is excellent, so the batch-accumulate pattern works well: the consumer accumulates a batch of records and inserts them, and ClickHouse handles the high insert rate comfortably.

Postgres ingestion. Postgres works for moderate volumes. The consumer batches inserts using the COPY mechanism or multi-row inserts for performance. At very high volume Postgres insert performance can become the bottleneck, which is part of why ClickHouse is often preferred for high-volume accounting analytics.

Message queue. The consumer can push records to a message queue (Kafka, RabbitMQ, Redis streams), decoupling the pipe entirely from the eventual storage. This is the most robust pattern: the consumer's only job is pipe-to-queue, the queue provides durability, and separate consumers drain the queue into storage. The message queue absorbs backpressure and provides the persistence the pipe lacks.

Metrics and alerting. The consumer can update real-time metrics (counters for delivered, bounced, complained per minute) and trigger immediate alerts on specific events (a feedback loop complaint, a particular bounce pattern). This is where the real-time nature of pipe accounting pays off: an alert can fire the instant the triggering record is generated.

Live dashboard feed. The consumer can push records or derived metrics to a live dashboard, giving operators a real-time view of delivery activity that file-based accounting, with its minutes of latency, cannot provide.

Running pipe and file together

The strong recommendation for most production deployments is to run pipe accounting and file accounting together.

The configuration is two acct-file directives:

# Durable file record
<acct-file /var/log/pmta/accounting.csv>
    records d,b,t,rb,f
    max-size 500M
    move-to /var/log/pmta/archive
</acct-file>

# Real-time pipe feed
<acct-file |/opt/pmta/scripts/acct-consumer.py>
    records d,b,t,rb,f
</acct-file>

Both directives receive the same records. The file provides the durable, authoritative record: it survives any pipe consumer failure, it persists on disk, and it can be used to backfill any gap. The pipe provides the real-time feed: live dashboards, immediate alerting, real-time ingestion.

This combination gives the best of both. The real-time capability comes from the pipe. The durability and reliability come from the file. If the pipe consumer ever fails, no accounting data is lost permanently, because the file has it; the operator can replay the file for the failure window to backfill the real-time system.

Using only the pipe, with no file, is appropriate only when occasional accounting data loss is genuinely acceptable. For most operators, accounting data is the authoritative record of what happened to their mail, used for billing, compliance, deliverability analysis, and customer reporting, and losing it is not acceptable. For those operators, the file is mandatory and the pipe is the real-time addition.

When pipe accounting stops flowing

When the pipe consumer is not receiving records, the diagnostic procedure:

Step 1: confirm the consumer process is running. Check whether the consumer process exists. PowerMTA launches it, so it should be running as a child of the PowerMTA process. If it is not running, it crashed or exited.

ps aux | grep acct-consumer

Step 2: check the consumer's error output. The consumer should log errors to standard error or a log file. Review it for crash traces, parsing errors, or exceptions that indicate why the consumer failed.

Step 3: check whether records are being generated at all. Confirm PowerMTA is generating accounting records by checking the file accounting output (if running both). If the file is also empty, the problem is upstream of accounting, not the pipe.

Step 4: check for backpressure symptoms. Is the consumer slow rather than dead? If PowerMTA shows signs of accounting backpressure, the consumer may be alive but not keeping pace. Profile the consumer's per-record processing time.

Step 5: check the consumer for a hang. A consumer that is running but not processing, perhaps blocked on a database call or a network operation, hangs the pipe. Check whether the consumer is making progress (incrementing a counter, logging activity) or stuck.

Step 6: verify the acct-file pipe configuration. Confirm the acct-file directive correctly names the consumer program and the program is executable. A configuration error or a non-executable script means PowerMTA cannot launch the consumer.

Step 7: backfill from the file. If pipe accounting was down for a period and file accounting was running alongside, use the file to backfill the real-time system for the gap. This is the payoff of running both: the file makes the pipe failure recoverable.

Step 8: fix and harden. Fix the immediate cause (the crash, the slowness, the hang), and harden the consumer so the same failure does not recur, better exception handling, the decoupled-worker pattern for backpressure, a watchdog for hangs.

PowerMTA pipe accounting streams accounting records to an external consumer in real time, enabling live dashboards, immediate alerting, and real-time ingestion that file accounting's minutes-of-latency cannot match. The configuration is straightforward: point an acct-file directive at a program. The reliability discipline is what matters: the pipe has no persistence, so the consumer must be written defensively to survive bad records, must be fast enough to avoid backpressure (using the decoupled-worker pattern when the real work is slow), and should be watched for hangs. The strong recommendation for production is to run pipe and file accounting together, gaining the pipe's real-time capability and the file's durability, so that a consumer failure is recoverable rather than a permanent data gap. Operators who treat the pipe consumer as a robust, defensively-written, monitored component, backed by file accounting, get reliable real-time accounting; operators who treat it as a simple script and skip the file discover the pipe's lack of persistence at the worst possible moment.