PowerMTA Logging and Accounting Log: Complete Operational Guide

PowerMTA's accounting log is the most operationally valuable data source in email infrastructure management. Every delivery attempt — success, deferral, or permanent failure — generates a structured accounting record with the SMTP response, the sending configuration, and the delivery timing that together answer every deliverability diagnostic question the infrastructure team needs to address. This guide covers the complete accounting log configuration, the fields that matter most, and the ETL pipeline architecture that converts raw log data into actionable operational intelligence.

Accounting Log Overview: What It Captures

The PowerMTA accounting log records a structured event for every significant delivery state transition: message injection (I record), delivery attempt result (D record — which includes the SMTP response code and full response text for every attempted connection), bounce event (B record — for permanent failures), and FBL complaint (F record — when PowerMTA's FBL processor identifies an ARF complaint). Together, these record types provide a complete audit trail of every message from injection through final disposition.

The accounting log is fundamentally different from system logs (which record MTA operational events like process starts and configuration loads) and mail logs (which record SMTP session-level events in a human-readable format). The accounting log is structured, machine-readable, and designed for programmatic processing — it is the data source for ETL pipelines, operational dashboards, and automated alerting systems. The mail log is for human debugging; the accounting log is for systematic monitoring.

Each D (delivery attempt) record in the accounting log contains the information needed to classify the delivery event precisely: the recipient address (or a hash of it for privacy), the recipient domain, the VMTA used for this delivery, the source IP address, the destination MX server, the SMTP response code, and the full SMTP response text. This combination — code plus text — enables bounce classification patterns that distinguish address errors from policy rejections from infrastructure failures, as documented in the bounce classification framework note.

Configuring the Accounting Log in PowerMTA

The accounting log is configured in the PowerMTA configuration file (/etc/pmta/config) within an <acct-file> block. The block specifies the log file path, the log format (field list), the trigger events to log, and the log rotation settings:

<acct-file /var/log/pmta/acct.csv>
  records d,b,f                           # delivery, bounce, FBL records
  record-type d                           # detailed delivery attempt records
  max-size 500m                           # rotate when file reaches 500MB
  delete-after 90d                        # delete rotated files after 90 days

  # Field specification — order matters, matches CSV column order
  fields timeLogged,type,vmta,vmtaPool,
         orig,rcpt,dlvSourceIp,
         dlvDestIp,dlvType,
         dlvRc,dlvStatusMessage,
         dlvStatus,
         msgSize,msgId,
         dlvTime,qTime
</acct-file>

The records directive specifies which event types generate accounting records. d (delivery attempt) is the most important — it records every SMTP connection attempt with the server's response. b (bounce) records permanent failures separately (redundant with d records for permanent failures, but useful for bounce-specific processing). f (FBL) records feedback loop complaints processed by PowerMTA's built-in FBL processor.

Log rotation is critical for production deployments. A high-volume programme generating 2M messages/day will produce approximately 200-500MB of accounting log data daily depending on field selection. The max-size directive rotates the active log file when it reaches the specified size, creating a new file with a timestamp suffix. The delete-after directive removes rotated files after the specified retention period. For compliance with GDPR data minimisation requirements, consider pseudonymising the recipient address field after the operational retention period (90 days) by hashing the address before archiving.

Essential Fields: What to Log and Why

PowerMTA supports 30+ accounting log fields. Logging all fields for every event produces very large log files with much data that is rarely queried. The recommended field selection balances operational coverage with storage efficiency:

For programmes that need to correlate accounting log events with campaign-level data from the sending application, adding a custom header to each message (X-Campaign-ID, X-Customer-ID) and logging that header value in the accounting log is the mechanism that enables per-campaign and per-customer delivery analytics. PowerMTA supports logging custom header values via the macro field type in the accounting log field specification.

Log Formats: CSV vs Pipe-Delimited vs Custom

PowerMTA supports multiple accounting log output formats. The format choice affects how the ETL pipeline parses and processes the data:

CSV (comma-separated): The default format. Each field value is separated by commas; string values containing commas are quoted. CSV is the most universally compatible format for ETL tools (Filebeat, Logstash, pandas, and virtually every data processing library support CSV natively). The limitation: the SMTP response text field (dlvStatusMessage) frequently contains commas, requiring careful CSV parsing that handles quoted strings correctly.

Pipe-delimited: Uses the pipe character (|) as field separator. SMTP response text rarely contains pipes, making pipe-delimited format more reliable for robust ETL parsing without complex quote-handling logic. Recommended for production deployments where accounting log data is processed programmatically at high throughput.

Custom format with record-type differentiation: PowerMTA allows different field specifications per record type — D records can log more fields than I or B records. For monitoring-optimised logging, D records include the full field set (all 14 fields listed above), while B records include a minimal set (timeLogged, rcpt, dlvStatusMessage) for fast bounce suppression lookup without the delivery-specific fields that are not relevant for bounce records.

Building the ETL Pipeline

The accounting log ETL pipeline reads new records from the accounting log file in near-real-time and writes them to a queryable database optimised for the monitoring and analytics queries that operational visibility requires. The pipeline has three components: the log reader, the transformer, and the loader.

Log reader: Filebeat (part of the Elastic Stack) is the most common choice for tailing PowerMTA accounting log files and forwarding records to a downstream queue. Alternatively, a custom Python or Go script using the inotify interface (Linux) can tail the log file and forward new records to a Redis stream or Kafka topic. The key requirement: the reader must handle log rotation correctly — when PowerMTA rotates the log file (creating a new file), the reader must detect the rotation and continue reading from the new file without missing records from the rotation window.

Transformer: Parses the pipe-delimited or CSV record, validates field values, maps the record type to the correct processing path (D records → delivery events table, B records → bounce events table, F records → complaint events table), and applies any normalisation (domain extraction from recipient address, IP normalisation, timestamp standardisation). In Python:

import csv
from datetime import datetime

def parse_acct_record(line):
    fields = line.strip().split('|')
    if len(fields) < 14:
        return None
    return {
        'time': datetime.fromisoformat(fields[0]),
        'type': fields[1],
        'vmta': fields[2],
        'sender': fields[3],
        'recipient': fields[4],
        'source_ip': fields[5],
        'dest_domain': fields[6].split('.')[-2:],  # extract domain
        'smtp_code': int(fields[7]) if fields[7].isdigit() else None,
        'smtp_text': fields[8],
        'status': fields[9],
        'msg_size': int(fields[10]) if fields[10].isdigit() else None,
        'msg_id': fields[11],
        'dlv_time_ms': int(fields[12]) if fields[12].isdigit() else None,
        'queue_time_s': int(fields[13]) if fields[13].isdigit() else None,
    }

Loader: Writes the parsed and transformed records to the target database. PostgreSQL with a partitioned table (partitioned by date) handles the append-heavy write pattern efficiently and supports the aggregation queries that monitoring requires. ClickHouse (a columnar database optimised for analytical queries) provides significantly faster query performance for high-cardinality aggregations (per-ISP deferral rate across 10M+ records) at the cost of more complex operational management. For most programmes below 10M messages/day, PostgreSQL with proper indexing is sufficient; above 10M messages/day, ClickHouse's query performance advantage becomes material.

Key Operational Queries from the Accounting Log

The following SQL queries, run against the accounting log database, provide the core operational metrics that delivery monitoring requires. These queries assume the PostgreSQL schema derived from the field set above.

-- Per-ISP deferral rate (last 60 minutes)
SELECT dest_domain,
       COUNT(*) FILTER (WHERE smtp_code BETWEEN 400 AND 499) AS deferred,
       COUNT(*) AS total,
       ROUND(100.0 * COUNT(*) FILTER (WHERE smtp_code BETWEEN 400 AND 499) / COUNT(*), 2) AS deferral_rate_pct
FROM delivery_events
WHERE time > NOW() - INTERVAL '60 minutes'
  AND type = 'd'
GROUP BY dest_domain
HAVING COUNT(*) > 100
ORDER BY deferral_rate_pct DESC;

-- Per-campaign hard bounce rate
SELECT msg_id_prefix,  -- first 32 chars of msg_id = campaign identifier
       COUNT(*) FILTER (WHERE smtp_code >= 500) AS hard_bounces,
       COUNT(*) AS total_attempts,
       ROUND(100.0 * COUNT(*) FILTER (WHERE smtp_code >= 500) / COUNT(*), 3) AS bounce_rate_pct
FROM delivery_events
WHERE time > CURRENT_DATE - INTERVAL '7 days'
GROUP BY msg_id_prefix
ORDER BY bounce_rate_pct DESC;

-- Per-IP delivery rate (last 24 hours)
SELECT source_ip,
       COUNT(*) FILTER (WHERE smtp_code = 250) AS delivered,
       COUNT(*) AS total,
       ROUND(100.0 * COUNT(*) FILTER (WHERE smtp_code = 250) / COUNT(*), 2) AS delivery_rate_pct
FROM delivery_events
WHERE time > NOW() - INTERVAL '24 hours'
  AND type = 'd'
GROUP BY source_ip
ORDER BY delivery_rate_pct ASC;

Log Retention and Storage Management

Accounting log data volume at scale is significant. A programme delivering 2M messages/day with 14 logged fields per event generates approximately 150-300MB of raw log data per day. Over 90 days: 13-27GB of raw logs. After ETL processing into PostgreSQL, the structured data is typically 20-40% smaller than the raw log (due to numeric storage of codes vs string storage in CSV). Plan for 15-25GB of database storage per 90 days of data at 2M messages/day.

The retention strategy: keep full-resolution data (all 14 fields, per-event) for 90 days in the operational database. After 90 days, archive to compressed cold storage (S3, Azure Blob, or equivalent) and delete from the operational database. For compliance purposes, if the recipient address field is stored in plain text in the operational database, pseudonymise (SHA-256 hash) the recipient address in the archived data before cold storage to comply with GDPR data minimisation requirements. The hashed recipient address retains the ability to identify duplicate records and correlate with suppression database entries without storing the personally identifiable email address beyond the operational retention period.

Troubleshooting Delivery Problems with Log Data

The accounting log is the primary diagnostic tool for delivery problem investigation. When inbox placement falls, a reputation event occurs, or an ISP begins generating unusual response codes, the accounting log provides the evidence needed to identify the cause precisely and apply the correct intervention.

The diagnostic workflow for an unexplained deliverability problem:

The accounting log makes the difference between a deliverability investigation that takes 6 hours and one that takes 45 minutes. With per-event data and full SMTP response text, every question the investigation needs to answer has a precise, queryable answer. Without per-event data — on shared relay infrastructure or on MTA deployments without accounting log ETL — the same investigation requires cross-referencing multiple incomplete data sources and often concludes without a definitive root cause. Build the accounting log pipeline; build the queries; and deliverability investigations become evidence-based diagnostic exercises rather than educated guesswork.

The accounting log is the operational conscience of the email infrastructure — it records everything that happened, in the order it happened, with the precision needed to understand why. Configure it completely; process it continuously; query it systematically; and the infrastructure will never produce a delivery problem that the accounting log does not reveal within minutes of it beginning. That visibility is the foundation of proactive deliverability management and the difference between an email programme that discovers problems through stakeholder complaints and one that resolves them before any stakeholder notices.