Logging Architecture for Email Infrastructure

• October 2022
• Engineering Memo · External Release

Every email delivery event generates a log entry — in the MTA's accounting log, in the bounce processing daemon's database, in the FBL complaint processor's records, in the authentication verification service's output. The logging architecture that captures, stores, and organises these events is the foundation of all operational monitoring, deliverability diagnostics, and compliance documentation for an email infrastructure deployment.

This note documents the logging requirements for high-volume email infrastructure, the schema design that makes logs queryable for operational purposes, the retention policies that balance compliance requirements against storage costs, and the specific queries that convert raw log data into operational intelligence.

The PowerMTA Accounting Log: The Primary Data Source

PowerMTA's accounting log is the most data-dense single log source in a high-volume email infrastructure. Every SMTP session — every connection attempt, every message delivery, every deferral, every bounce — generates a tab-separated record in the accounting log. At 1 million messages per day, the accounting log produces approximately 50–100 MB of raw data per day, containing the complete SMTP-layer history of every delivery attempt in the environment.

The accounting log record for a delivered message includes: timestamp, virtual MTA, source IP, destination domain, recipient address (hashed for privacy, full for operational needs), SMTP response code, SMTP response message text, message size, DKIM signing status, queue entry time, delivery time, and campaign/batch identifier if configured. This record set contains everything needed to answer the most common operational questions: which campaigns delivered, to which ISPs, with what response codes, and when.

The accounting log is a sequential append-only file by default. For operational use beyond simple tail-and-grep investigation, it must be parsed into a queryable format — a database (PostgreSQL, MySQL, or TimescaleDB for time-series workloads) that allows per-ISP delivery rate queries, per-campaign bounce rate calculations, per-hour volume tracking, and SMTP response code analysis. The pipeline that reads new accounting log entries (every 30–60 seconds), parses each record, and inserts it into the operational database is the logging infrastructure investment that makes the accounting log operationally useful rather than just technically complete.

The Operational Database Schema

The core table in the operational database is the delivery events table, which stores one record per SMTP delivery attempt. The minimal schema for operational use:

CREATE TABLE delivery_events (
  id              BIGSERIAL PRIMARY KEY,
  event_ts        TIMESTAMPTZ NOT NULL,
  vmta            VARCHAR(50),           -- PowerMTA virtual MTA name
  source_ip       INET,                  -- Sending IP address
  dest_domain     VARCHAR(255),          -- Destination domain (e.g. gmail.com)
  recipient_hash  VARCHAR(64),           -- SHA-256 of recipient address
  campaign_id     VARCHAR(100),          -- Campaign identifier from X-header
  result          VARCHAR(20),           -- 'delivered', 'bounced', 'deferred'
  smtp_code       SMALLINT,              -- e.g. 250, 421, 550
  smtp_message    TEXT,                  -- Full SMTP response text
  queue_time_s    INTEGER,               -- Seconds in queue before delivery attempt
  attempt_num     SMALLINT               -- Which retry attempt (1 = first)
);
CREATE INDEX ON delivery_events (event_ts DESC);
CREATE INDEX ON delivery_events (dest_domain, event_ts DESC);
CREATE INDEX ON delivery_events (campaign_id, event_ts DESC);
CREATE INDEX ON delivery_events (source_ip, event_ts DESC);

The indexes are designed for the most common query patterns: time-range queries for recent activity (event_ts), per-ISP delivery rate analysis (dest_domain + event_ts), per-campaign performance analysis (campaign_id + event_ts), and per-IP reputation investigation (source_ip + event_ts). All queries should specify a time range to avoid full-table scans on a table that grows by millions of rows per day.

TimescaleDB (a PostgreSQL extension optimised for time-series workloads) is particularly well-suited for the delivery events table, as it automatically partitions data by time and provides time-series query optimisations that standard PostgreSQL lacks. For environments with 1M+ daily delivery events, TimescaleDB's hypertable partitioning reduces query times by an order of magnitude compared to unpartitioned PostgreSQL for time-range queries covering days or weeks of data.

Operational Queries: Converting Logs to Intelligence

The operational database is only valuable when operators know which queries to run against it. The five most important operational queries:

Per-ISP delivery rate (last 24 hours):

SELECT dest_domain,
       COUNT(*) FILTER (WHERE result = 'delivered') AS delivered,
       COUNT(*) AS total_attempts,
       ROUND(100.0 * COUNT(*) FILTER (WHERE result = 'delivered') / COUNT(*), 2) AS delivery_rate_pct,
       ROUND(100.0 * COUNT(*) FILTER (WHERE smtp_code BETWEEN 400 AND 499) / COUNT(*), 2) AS deferral_rate_pct
FROM delivery_events
WHERE event_ts > NOW() - INTERVAL '24 hours'
GROUP BY dest_domain
ORDER BY total_attempts DESC
LIMIT 20;

SMTP attempts-to-delivery ratio (retry pressure indicator, per IP):

SELECT source_ip,
       COUNT(*) AS total_attempts,
       COUNT(*) FILTER (WHERE result = 'delivered' AND attempt_num = 1) AS first_attempt_deliveries,
       COUNT(*) FILTER (WHERE result = 'delivered') AS total_deliveries,
       ROUND(1.0 * COUNT(*) / NULLIF(COUNT(*) FILTER (WHERE result = 'delivered'), 0), 2) AS attempts_per_delivery
FROM delivery_events
WHERE event_ts > NOW() - INTERVAL '24 hours'
GROUP BY source_ip
ORDER BY attempts_per_delivery DESC;

These two queries — per-ISP delivery rate and per-IP attempts ratio — are the primary daily monitoring queries that reveal the most operationally significant patterns in the accounting log data. The first identifies which ISPs are experiencing elevated deferral rates. The second identifies which IPs are generating high retry overhead. Both run in seconds on a properly indexed TimescaleDB table and produce the data needed to drive daily monitoring decisions.

Log Retention Policy

Email infrastructure logs must be retained for multiple purposes with different retention requirements. The primary retention requirements:

Operational monitoring and troubleshooting: 90 days of full delivery event detail in the operational database. This covers the time horizon needed to diagnose reputation trends, investigate campaign-level performance, and respond to ISP postmaster inquiries about specific sending events. After 90 days, aggregate summaries (daily delivery rate per ISP, daily bounce rate, daily complaint rate) can replace full event-level detail for trend analysis.

Compliance documentation (GDPR / CAN-SPAM): Evidence of delivery to a specific recipient address may be required to respond to legal inquiries or regulatory complaints. The compliance log needs to retain the recipient address (or pseudonymised reference), the delivery timestamp, the message identifier, and the delivery status for at least 24 months. This compliance log can be a separate, more compact table than the full operational events table, containing only the fields required for compliance documentation rather than the full SMTP-layer detail needed for operational diagnosis.

Bounce processing history: The suppression list database should retain hard bounce records permanently — or until a specific re-validation event confirms the address is valid again. Soft bounce history should be retained for 12 months to support the soft-to-hard bounce promotion logic. The bounce processing database serves both operational purposes (preventing re-sends to bounced addresses) and compliance purposes (documenting why an address is not receiving email).

FBL complaint records: Complaint records should be retained for 24 months as part of the programme's compliance documentation — evidence that complaints were received and processed (resulting in suppression) is relevant to both regulatory inquiries and ISP postmaster communications. The complaint record minimum fields: recipient address (or hash), complaint timestamp, campaign identifier, and suppression action taken.

The logging architecture is the invisible infrastructure of email operations. It does not deliver messages, manage reputation, or configure ISP relationships — but it enables all of these functions to be performed with the data precision that distinguishes reactive guesswork from evidence-based operational management. The investment in a well-designed logging architecture — the parsing pipeline, the operational database schema, the retention policy, and the standard query library — is a one-time engineering investment that produces operational value for every campaign, every monitoring review, and every deliverability investigation that follows. Without it, the infrastructure operates largely blind; with it, every operational decision is grounded in the specific, queryable evidence of what the infrastructure is actually doing.

Logging for Multi-Virtual-MTA Environments

PowerMTA generates a single accounting log file per instance, but a production deployment typically runs multiple virtual MTAs — one per sending IP or per traffic type — and may run multiple PowerMTA instances if the environment spans multiple servers. The logging architecture must aggregate all virtual MTA accounting log data into a single queryable database that allows per-virtual-MTA queries (which pool is generating elevated deferral rates?) while also providing aggregate views across all pools (what is the total campaign delivery rate?).

The accounting log parser script must add the server identifier and virtual MTA name as fields in the database record, in addition to the fields extracted from the accounting log line itself. The vmta column in the schema above serves this purpose — it allows filtering all delivery events from a specific virtual MTA (and by extension, from the specific IP that virtual MTA represents) while the source_ip column provides the finer-grained per-IP breakdown within a virtual MTA that handles multiple IPs.

For multi-server environments where multiple PowerMTA instances generate separate accounting log files, the ETL pipeline must collect log files from all servers into a central processing location before parsing and database insertion. This collection can be implemented with rsync (pulling log files from remote servers to the database server every 30 seconds), with a centralized log forwarding agent (sending new log entries to the central database in real time), or with a message queue (each server publishes new log entries, the database consumer processes the queue). The rsync approach is simplest to implement and produces 30-second data latency, which is adequate for all but real-time monitoring use cases.

Integrating FBL and Bounce Data with Delivery Events

The full operational picture requires correlating the accounting log delivery events with the FBL complaint records and the bounce processing records. A delivery event that resulted in delivery, followed by an FBL complaint from the same recipient, tells a more complete story than either record alone — the message was delivered but the recipient found it unwanted. A delivery event that resulted in a hard bounce can be linked to the bounce processing record that shows when the address was suppressed and whether it was on a list that should have been validated before sending.

The correlation key is the message identifier — the Message-ID header or the internal campaign/message ID that the sending application assigns to each message and records in the PowerMTA accounting log via X-headers. If the FBL complaint ARF record includes the original Message-ID (which it typically does), the complaint can be linked to the specific delivery event record that shows when the message was delivered, from which IP, to which ISP domain, in which campaign. This linkage is what converts FBL data from a suppression signal into a diagnostic signal — which specific campaigns and list segments are generating complaints, at what rate, and with what delivery characteristics.

Implementing this cross-record correlation requires that the Message-ID be included in the accounting log record (which it is by default in PowerMTA), that the FBL complaint processor extract the Message-ID from the ARF record and store it in the complaint record, and that the database schema includes a foreign key or indexed join between the complaint_records table and the delivery_events table via the message_id field. Once this linkage is in place, the standard complaint attribution query — which campaign and list segment generated each complaint — runs in seconds rather than requiring manual log cross-referencing.

Logging for Compliance: Evidence of Delivery and Consent

In regulatory contexts — GDPR erasure requests, CAN-SPAM compliance inquiries, spam complaint investigations by ISP postmaster teams — the logging infrastructure provides the evidentiary basis for the programme's compliance documentation. A data subject who requests erasure needs evidence that their data has been removed from the active list and added to the suppression list. A regulatory inquiry about an alleged spam campaign needs evidence that the messages were sent to addresses that had given valid consent and were not re-sent after unsubscribing.

The compliance log — a simplified version of the delivery events table containing only the fields required for compliance documentation — serves this purpose. The minimum compliance log fields: recipient identifier (the email address, or a cryptographic hash of it if the address has been erased but the send history must be preserved), the delivery timestamp, the campaign identifier, the consent source identifier (which acquisition event produced this address), and the suppression status with the date of any suppression event. This record set allows responding to the most common compliance inquiries without requiring access to the full operational database.

The tension between GDPR's right to erasure and the operational need to retain delivery records is resolved by separating the personal data (the email address itself) from the operational records. When a recipient exercises the right to erasure, the email address is deleted from the active list and from the compliance log, but the delivery records that reference that address via a cryptographic hash are retained. The hash cannot be reverse-engineered to recover the original address, satisfying the erasure requirement, while the delivery record and its operational context (campaign, delivery timestamp, result) are preserved for audit purposes. This hash-based approach satisfies both the erasure requirement and the audit documentation requirement that compliance programmes maintain.

The logging architecture is ultimately the memory of the email infrastructure — the complete, queryable record of what the infrastructure has done, when, to whom, and with what result. Building this memory correctly, from the initial schema design through the ETL pipeline implementation, the retention policy configuration, and the standard query library, converts raw log data into institutional knowledge that makes every subsequent operational decision faster, more precise, and more evidence-grounded. The investment in logging architecture is the investment in the operational intelligence that separates professionally managed email infrastructure from infrastructure that operates by feel rather than by evidence.

Practical Implementation: Getting the Pipeline Running

For operators setting up the accounting log pipeline for the first time, a minimal working implementation can be achieved in a day using Python, PostgreSQL, and cron. The pipeline script reads new lines from the accounting log file (tracking the file position between runs), parses each tab-separated record into a Python dictionary, and inserts the record into the delivery_events table using a batch INSERT with conflict handling for duplicate records. The cron job runs the script every 30 seconds, providing near-real-time data freshness for monitoring purposes.

The critical implementation detail: the accounting log file path and format must be verified against the PowerMTA configuration before the pipeline is deployed. PowerMTA's accounting-dir and accounting-format configuration options determine the file location and field layout. The field order in the accounting-format configuration must match the field parsing logic in the ETL script exactly — a mismatch produces incorrect data in the database without necessarily generating an error, which is harder to detect than an explicit parse failure.

For environments with high delivery volumes (1M+ messages per day), batch INSERTs of 1,000 records per transaction are more efficient than individual record INSERTs. The ETL script should accumulate records in memory and flush them to the database in batches, rather than committing each record immediately. The 30-second run interval and 1,000-record batch size produce a database write pattern of approximately 1 batch per 30-second interval at 1M messages per day, which is well within the write capacity of a standard PostgreSQL instance.

The monitoring layer sits on top of the operational database: a dashboard tool (Grafana, Metabase, or a custom web application) that runs the standard operational queries on a schedule and displays the results as time-series charts. The per-ISP delivery rate chart, the per-IP retry pressure chart, and the campaign-level performance chart are the three views that provide the daily situational awareness the monitoring practice requires. Building these dashboards on top of the properly indexed operational database takes 1–2 days and produces monitoring infrastructure that serves the programme for years.

The logging architecture is not glamorous work — it is the plumbing of email infrastructure operations, invisible when it works and critically important when it doesn't. Programmes that build it correctly from the beginning have the diagnostic capability to answer any operational question quickly and precisely. Programmes that skip it or implement it partially discover its absence most acutely during deliverability incidents, when they most need the specific evidence that good logging provides and find themselves limited to the aggregate metrics that poor logging leaves behind. Build the logging infrastructure first; it enables everything else to be managed effectively.

Alerting on Log-Derived Metrics

The logging pipeline produces the data that powers alerting — the automated detection of threshold crossings that require human attention. Effective alerting converts the operational database from a passive record into an active notification system that brings anomalies to the operator's attention without requiring constant manual review. The alert implementation: a monitoring script that runs every 15 minutes, queries the operational database for the key metrics, compares them against configured thresholds, and sends alerts (email, Slack, PagerDuty) when thresholds are exceeded.

The five alert conditions that every email infrastructure logging implementation should include: (1) per-ISP deferral rate above 15% in the past 30 minutes -- indicates developing throttle pressure requiring investigation; (2) per-IP retry-pressure ratio above 1.8 in the past 60 minutes -- indicates retry overhead consuming significant throughput; (3) campaign bounce rate above 0.5% at campaign completion -- indicates list quality issue requiring investigation before next send; (4) FBL complaint rate above 0.05% in the past 24 hours -- indicates complaint-generating campaign requiring root cause identification; (5) delivery event count drops to zero for more than 5 minutes during an active send window -- indicates possible infrastructure failure requiring immediate investigation.

Each alert should include the specific metric value that triggered it, the threshold it crossed, and the specific query or data source that produced the value. Alert messages that say only "delivery rate alert" without the specific metric value, the affected ISP, and the time window require the operator to re-query the database before understanding what to investigate -- adding unnecessary delay at the moment when fast response is most valuable. Alert messages that include this context allow the operator to assess the situation from the alert alone and respond immediately with the appropriate investigation or intervention.

The combination of the logging pipeline (capturing events), the operational database (making events queryable), the dashboard (providing ongoing visibility), and the alerting system (notifying when anomalies occur) constitutes the complete observability stack for email infrastructure. Each component is simpler to build and maintain than it might appear, and together they transform the infrastructure from a black box into a transparent, monitored system where every operational question has a data-driven answer and every anomaly is detected within minutes rather than hours or days. This observability is not a luxury for large-scale operations -- it is the operational foundation that makes professional email infrastructure management possible at any scale.