Contents
Why structured monitoring matters
PowerMTA monitoring is the difference between catching a deliverability problem at the early-signal stage and discovering it after it has become a full incident. A throttling event, a reputation decline, a queue accumulation, a blacklist listing: each of these gives warning signs before it becomes a visible failure, and structured monitoring is what catches those signs. An operator without structured monitoring learns about problems from customer complaints, which means the problem has already been affecting deliverability for some time.
This guide exists because PowerMTA monitoring is frequently ad hoc: an operator checks the web monitor occasionally, runs pmta show queues when something feels wrong, and otherwise hopes for the best. That approach misses the early signals and turns preventable problems into incidents. The structured alternative organizes monitoring into tiers by urgency: things that need continuous automated alerting, things that need daily human review, things that need weekly trend analysis. The structure of this guide follows that organization: the leading-versus-lagging indicator distinction that underpins effective monitoring, the continuous tier with alerting, the daily checklist, the weekly review, alerting thresholds, Prometheus and Grafana integration, the external reputation sources, alert hygiene to keep alerts actionable, and the operational rhythm that ties it all together.
Leading versus lagging indicators
The most important conceptual distinction in monitoring is between leading and lagging indicators.
Leading indicators predict problems before they fully materialize. They give the operator a chance to act before a problem becomes an incident.
Lagging indicators confirm problems that have already happened. They tell the operator what occurred and how big it was, but by the time they fire the damage is done.
| Leading indicator | What it predicts |
|---|---|
| Rising complaint rate in FBL data | An approaching throttling event |
| Increasing deferral rate from one ISP | That ISP starting to throttle |
| Declining reputation score trend | Deliverability degradation ahead |
| Queue depth beginning to climb | A developing delivery problem |
| Lagging indicator | What it confirms |
|---|---|
| Sharp delivery rate drop | Throttling has already happened |
| Blacklist listing | Reputation damage already done |
| Red SNDS filter result | Filtering already occurring |
| Bounce rate spike | Bad addresses already bounced |
Effective monitoring watches both. Leading indicators give the chance to prevent incidents; lagging indicators confirm and measure them. A monitoring setup built only on lagging indicators is purely reactive, always cleaning up after the fact. Adding leading indicators makes some incidents preventable: a complaint rate climbing toward 0.3 percent, caught early, can be addressed before it triggers the throttling that the lagging delivery-rate drop would confirm.
Continuous monitoring with alerting
The continuous tier covers conditions that develop too fast, and at hours too unpredictable, for human checking. These need automated alerting.
| What | Why continuous | Alert on |
|---|---|---|
| Service health | An unexpected stop is an outage | PowerMTA not running |
| Queue depth | Accumulation signals delivery failure | Depth exceeds threshold sustained |
| Delivery rate | A sharp drop signals a real problem | Rate drops 40-50% from baseline |
| Disk space | A full disk halts PowerMTA | 80% utilization on spool/log partitions |
Service health. Whether PowerMTA is running at all. An unexpected stop or crash is a service outage and must trigger an immediate alert. Monitor the process and the service status.
Queue depth. Total and per-destination queue depth. Deep queues that do not drain signal a delivery problem, and the continuous alert catches accumulation early.
Delivery rate. The rate of successful deliveries. A sudden sharp drop indicates a throttling event, reputation problem, or infrastructure issue, and the continuous alert catches it within minutes rather than hours.
Disk space. The partitions holding the spool and the logs. A full disk stops PowerMTA from writing accounting records and can halt processing entirely; the continuous alert at 80 percent utilization gives time to act before the disk fills.
Each of these four represents a condition that, undetected, leads to an outage or significant deliverability loss within hours. That is what makes them the continuous tier: they are too fast and too consequential for the daily-review tier.
The daily checklist
The daily tier covers things important enough to check every day but not requiring sub-hour detection. These are the operator's morning routine.
Backoff queues. Check pmta show queues for any queues in backoff mode. A queue in backoff means PowerMTA detected throttling from that destination. Identify which destinations are in backoff and whether the backoff is expected or new.
pmta show queues | grep BACKOFFBounce patterns. Review the day's bounce activity in the accounting log. Look for unusual patterns: a spike in a particular bounce code, a cluster of 5.7.x codes indicating a sender-side problem, an increase in hard bounces indicating list quality decline.
Complaint rates. Check the feedback loop data, Gmail Postmaster Tools, Yahoo CFL, Microsoft JMRP, for the complaint rate. A complaint rate trending up toward the 0.3 percent threshold is a leading indicator worth catching daily.
Reputation signals. Glance at Google Postmaster Tools and Microsoft SNDS for the reputation indicators. Daily review catches a reputation decline while it is still a trend rather than an incident.
Overall queue health. A quick pmta show queues to confirm queues are processing normally and depth is at the expected steady-state level.
Disk space. Even with continuous alerting at 80 percent, a daily glance at disk usage on the spool and log partitions confirms there is no slow leak the threshold has not yet caught.
The daily checklist takes a few minutes when nothing is wrong and is the routine that catches developing problems while they are still small.
The weekly review
The weekly tier covers trend analysis that only makes sense over a longer window.
IP reputation trends. Review the week's reputation data per IP in Google Postmaster Tools and Microsoft SNDS. A single day's reputation reading is noisy; the weekly trend reveals whether reputation is stable, improving, or declining. A slow decline invisible day to day becomes clear over a week.
Deferral patterns by ISP. Analyze the week's deferral activity grouped by receiving ISP. Which ISPs are deferring most? Is the deferral rate to a particular ISP trending up? This per-ISP weekly view catches a developing problem with one receiver that the daily view might not isolate.
Accounting log hygiene. Verify the accounting log rotation is working, the archive directory is being consumed by downstream ingestion, and there is no accumulation. Confirm log retention is as intended.
Throttle and pattern-list effectiveness. Review whether the domain block throttling and smtp-pattern-list configuration are still well-matched to ISP behavior. ISPs change; a throttle setting that was right six months ago may need adjustment.
Bounce rate trend. The weekly bounce rate trend reveals list quality direction. A slowly rising hard bounce rate indicates list decay that the daily view does not surface.
Capacity review. Check whether queue depth, memory usage, disk usage, and CPU are trending toward limits. The weekly capacity view catches the need to scale before it becomes urgent.
Alerting thresholds
Alerting thresholds should fire early enough to act but not so sensitively that they produce noise.
| Metric | Threshold | Reasoning |
|---|---|---|
| Service health | Any unexpected stop | An outage, no threshold needed |
| Queue depth (single destination) | Over 5000 sustained 15 min | Indicates accumulation |
| Queue depth (total) | Multiple of normal steady-state | Calibrated to deployment |
| Delivery rate drop | 40-50% below recent baseline | Real problem, not fluctuation |
| Disk utilization | 80% on spool/log partitions | Time to act before full |
| Complaint rate | 0.2% warning, 0.3% critical | 0.3% is Gmail/Yahoo threshold |
| Backoff queue | In backoff over 30 min with depth | Persistent throttling |
| Memory usage | Approaching physical RAM, swap active | OOM risk |
The general principle: thresholds should be calibrated to the specific deployment's normal behavior. A queue depth of 3000 is normal for one deployment and alarming for another. Set thresholds with enough margin that normal fluctuation does not trigger them, and tune them over time as the operator learns which alerts were actionable.
The queue depth and total-volume thresholds in particular must be calibrated per deployment, because they depend entirely on the deployment's normal volume. A high-volume deployment's normal queue depth would be an emergency for a low-volume one.
Prometheus and Grafana integration
Production PowerMTA monitoring benefits from Prometheus and Grafana, which provide historical trending, dashboards, and flexible alerting.
The architecture: a PowerMTA exporter scrapes the web monitor's JSON feed and exposes Prometheus metrics, Prometheus scrapes the exporter, Grafana visualizes the metrics, AlertManager fires alerts on threshold rules.
The exporter pattern, parsing the PowerMTA queue JSON:
#!/usr/bin/env python3
import json, time, requests
from prometheus_client import start_http_server, Gauge
queue_depth = Gauge('pmta_queue_depth', 'Messages queued',
['domain', 'vmta', 'status'])
delivery_rate = Gauge('pmta_delivery_rate', 'Rate per minute',
['domain', 'vmta'])
def collect():
data = requests.get('http://localhost:8080/api/queues').json()
for q in data.get('queues', []):
queue_depth.labels(q['domain'], q['vmta'],
q['status']).set(q['recip'])
delivery_rate.labels(q['domain'],
q['vmta']).set(q['rate'])
if __name__ == '__main__':
start_http_server(9100)
while True:
collect()
time.sleep(30)The Prometheus alerting rules implement the thresholds from the table above: a service-down rule firing when the exporter is unreachable for 2 minutes, a queue-depth rule firing when any queue exceeds 5000 for 15 minutes, a backoff-persistent rule firing when a queue stays in backoff with depth for 30 minutes, and a disk rule firing at 80 percent utilization. Each rule carries an annotation that names the affected domain and the measured value so the operator opening the alert immediately knows what and where.
The Grafana dashboards should present the data in the tiered structure: a top-level health panel showing service status and total queue depth, a per-destination panel showing queue depth and delivery rate by ISP, a trend panel showing the week-over-week reputation and bounce rate movement. Dashboards organized by the continuous, daily, and weekly tiers mirror the monitoring rhythm and make the relevant view available for each context.
External reputation sources
Some of the most important monitoring data does not come from PowerMTA itself but from the mailbox providers and reputation services.
| Source | What it provides | Review cadence |
|---|---|---|
| Google Postmaster Tools | Gmail domain/IP reputation, spam rate, authentication | Daily glance, weekly trend |
| Microsoft SNDS | Outlook IP reputation, filter result, complaint data | Daily glance, weekly trend |
| Yahoo Sender Hub | Yahoo reputation signals and guidance | Weekly |
| Blacklist monitors | Spamhaus and other RBL listing status | Continuous (automated check) |
| Feedback loops | Complaint reports per recipient | Continuous processing, daily rate review |
Google Postmaster Tools shows Gmail's view of domain and IP reputation, the spam rate Gmail users generate, and authentication pass rates. The spam rate is a leading indicator; a rising spam rate predicts Gmail throttling.
Microsoft SNDS shows the Outlook reputation picture, the SmartScreen filter result color, and complaint data. The filter result trending toward yellow or red is the signal to act.
Blacklist monitoring should be automated: a continuous check of the PowerMTA outbound IPs against the major RBLs, alerting on any new listing. A blacklist listing is a lagging indicator but catching it fast minimizes the damage window.
The external sources are essential because they show the receiver's view, which PowerMTA's own accounting cannot. PowerMTA accounting shows what receivers told PowerMTA at SMTP time; the postmaster tools show the fuller reputation picture including user behavior signals that never appear in an SMTP response.
Alert hygiene and false positives
An alerting system is only useful if its alerts are trusted, and alerts are trusted only if they are mostly actionable. An alert that fires constantly for non-problems gets ignored, and an ignored alert is worse than no alert because it creates false confidence.
The practices that keep alerts actionable:
Calibrate thresholds to the actual deployment. A threshold copied from a guide without adjustment to the deployment's normal behavior will either fire constantly or never fire. Watch the metrics for a period, learn the normal range, set thresholds with margin above normal.
Use sustained-duration conditions. Alerting on an instantaneous threshold crossing produces noise from momentary spikes. Alerting on a threshold sustained for a duration (15 minutes for queue depth, 30 minutes for backoff) filters out the transient spikes that are not real problems.
Review every alert. After each alert, ask whether it was actionable. If it was not, the threshold or condition needs adjustment. Over time this review converges the alerting on genuinely actionable conditions.
Tier the alert severity. Not every alert is a page. Service-down is a page; a queue-depth warning might be a notification that can wait for business hours. Matching alert severity to genuine urgency prevents alert fatigue.
Suppress known-benign patterns. If a particular destination predictably enters backoff at a particular time for a documented reason, the alerting can suppress that known pattern while still alerting on novel backoff.
The failure mode of alerting is not missing an alert; it is an alerting system so noisy that the operator has learned to ignore it. When that happens, the genuine alert arrives, gets dismissed along with the noise, and the incident proceeds undetected, while everyone believes monitoring is in place. Alert hygiene, calibrating thresholds, using sustained-duration conditions, reviewing and tuning, is not optional polish; it is what makes the alerting system actually function. A smaller set of trusted alerts beats a large set of ignored ones.
The operational rhythm
The monitoring tiers combine into an operational rhythm:
Continuously, automated: the alerting watches service health, queue depth, delivery rate, disk space, blacklist status, and feedback loop processing. No human attention needed unless an alert fires.
Every day, a few minutes: the operator runs the daily checklist, backoff queues, bounce patterns, complaint rates, reputation signals, queue health, disk space. This catches developing problems while small.
Every week, a longer review: the operator does the weekly trend analysis, IP reputation trends, deferral patterns by ISP, accounting hygiene, throttle effectiveness, bounce rate trend, capacity. This catches slow drifts and the need to scale.
After every alert: the operator handles the alert and reviews whether it was actionable, tuning the alerting.
After every incident: the operator does a brief post-incident review, what was the leading indicator that could have caught it earlier, and adds or adjusts monitoring so the next occurrence is caught sooner.
This rhythm is not heavy. The continuous tier is automated. The daily check is minutes. The weekly review is perhaps half an hour. The cost is small and the payoff is large: most deliverability problems are caught at the early-signal stage rather than discovered as full incidents.
An operator we worked with had solid continuous alerting, service health, queue depth, delivery rate, all monitored with good thresholds. But their monitoring was almost entirely lagging indicators, and they had no daily or weekly review of complaint rates. One quarter, a series of campaigns to an aging list segment slowly pushed the Gmail complaint rate upward. None of the continuous alerts fired, because the complaint rate climbing is a leading indicator and their continuous tier only watched lagging ones. The complaint rate crept from 0.05 percent to 0.15 percent to 0.28 percent over about six weeks, and then crossed 0.3 percent, at which point Gmail began throttling, the delivery rate dropped sharply, and finally the lagging delivery-rate alert fired. By then the reputation damage was done and recovery took weeks. We added two things: a daily complaint-rate check from Google Postmaster Tools, and a weekly reputation trend review. The very next quarter, a similar list-segment issue started pushing the complaint rate up, the daily check caught it at 0.16 percent, the operator scrubbed the offending segment, and the complaint rate fell back to baseline without ever crossing the threshold. No throttling, no incident. The lesson: continuous alerting on lagging indicators is necessary but not sufficient. The daily and weekly review of leading indicators is what turns monitoring from incident confirmation into incident prevention.
PowerMTA operational monitoring works when it is structured into tiers: continuous automated alerting for the fast-moving outage-class conditions, a daily human checklist for the developing problems, a weekly trend review for the slow drifts. The leading-versus-lagging distinction is the key insight, because a monitoring setup that watches only lagging indicators can only confirm incidents, while one that also watches leading indicators can prevent them. The external reputation sources, Google Postmaster Tools, Microsoft SNDS, the feedback loops, provide the receiver-side view that PowerMTA accounting cannot. Alert hygiene keeps the whole system trustworthy. The operational rhythm that results is not burdensome, a few automated systems, a few minutes daily, a half hour weekly, and it converts deliverability management from reactive incident response into proactive early intervention. Operators who build this structured monitoring catch their problems early; operators who monitor ad hoc discover their problems from customer complaints.