MailWizz A/B Testing — Setting Up Subject Line and Content Tests

MailWizz's A/B testing functionality enables controlled comparisons between campaign variants — most commonly subject line variations, but also sender name, content, and send time variations. Proper A/B testing produces actionable data for improving both engagement and deliverability.

Creating an A/B Test Campaign

# Backend → Campaigns → Create New
# Campaign Type: A/B Split Test
# Configure:
# - Test group size: percentage of list for initial test (e.g., 30%)
# - Winner criteria: open rate, click rate, or manual selection
# - Winner send time: when to send winning variant to remaining subscribers

Subject Line A/B Test Configuration

• Variant A and B should test one variable at a time: length, question vs statement, personalization presence
• Keep all other variables identical: same from name, same content, same send time
• Minimum list size for statistically meaningful results: 2,000+ subscribers per variant
• Test duration: allow at least 4 hours of open tracking before declaring a winner

Using A/B Results for Deliverability Improvement

Subject line tests that reveal a significant open rate difference often also reveal inbox placement differences — the subject line with higher open rates is typically also landing in inbox more consistently. Subject lines with spam-filter-triggering content will show depressed open rates that correlate with spam placement, not recipient disinterest.

Automating Winner Selection

# MailWizz can automatically send the winner based on:
# 1. Open rate after N hours
# 2. Click rate after N hours
# 3. Manual selection by operator

# Automatic winner selection timing:
# Set 'auto send winner after' to 4-6 hours from test start

Troubleshooting Common Issues

Production MailWizz deployments encounter predictable issues at predictable stages. Understanding the diagnostic workflow for the most common problems in this configuration area saves time and prevents the escalating complexity that comes from applying fixes to a misdiagnosed problem. The diagnostic approach is always the same: identify the symptom precisely (not just "it's not working"), isolate the layer where the failure occurs (MailWizz application, delivery server connection, DNS, ISP rejection), and fix at the correct layer.

Systematic Diagnosis Approach

Check MailWizz logs first (available in Backend → Misc → Application Logs), then check the delivery server SMTP logs, then check the PowerMTA accounting log. Most issues surface in one of these three places. A problem that does not appear in any of these logs is almost always a configuration issue — the system is not attempting what you expect it to attempt.

# MailWizz diagnostic log locations:
# Application logs: Backend → Misc → Application Logs
# Delivery logs: Backend → Campaigns → [Campaign] → Delivery Logs
# Bounce logs: Backend → Bounce Servers → [Server] → Logs

# Server-side logs:
# MailWizz application: /path/to/mailwizz/apps/common/runtime/application.log
# PowerMTA delivery: /var/log/pmta/pmta.log
# PowerMTA accounting: /var/log/pmta/accounting.csv

Performance Optimization for Production Scale

MailWizz performance at scale depends on three infrastructure layers: the web application server (PHP/nginx or Apache), the database (MySQL — query optimization is critical at high subscriber counts), and the delivery infrastructure (PowerMTA connection pool sizing). Performance problems in any of these layers manifest as slow campaign sends, delayed processing, or timeouts that appear unrelated to the specific configuration area being managed.

The most common performance constraint in production MailWizz environments is MySQL query efficiency. As subscriber lists grow beyond 500,000 records, unoptimized database queries for segmentation, bounce processing configuration, and campaign statistics become significant bottlenecks. Ensure that subscriber tables have appropriate indexes on email, status, date_added, and any custom field columns used for segmentation.

# MySQL optimization for large MailWizz installations
# Check slow query log:
SHOW VARIABLES LIKE 'slow_query_log%';
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 1;  # Log queries over 1 second

# Key indexes to verify exist:
SHOW INDEX FROM mailwizz_lists_subscribers;
# Should have indexes on: email, status, date_added, list_id

# Add missing index if needed:
ALTER TABLE mailwizz_lists_subscribers 
  ADD INDEX idx_email_status (email, status);
  
# Campaign sends table — index on campaign_id + subscriber_id:
ALTER TABLE mailwizz_campaigns_tracking_opens
  ADD INDEX idx_campaign_sub (campaign_id, subscriber_id);

Security Considerations

MailWizz installations handling production sending volumes are valuable targets. Key security practices: use HTTPS for all MailWizz access (including tracking and unsubscribe links), restrict Backend access to authorized IP ranges via web server configuration, rotate API keys periodically and revoke unused keys, maintain regular database backups (automated, offsite), and ensure PHP and MailWizz are kept current with security patches.

The tracking domain (used for open and click tracking) requires special attention: it must have a valid SSL certificate (Let's Encrypt is acceptable), and its DNS records must point exclusively to your MailWizz server. A compromised tracking domain can redirect recipients to malicious sites or reveal subscriber click data to third parties.

Every configuration decision in MailWizz has a corresponding deliverability implication — because MailWizz is the layer that determines what gets sent, to whom, and when, while PowerMTA determines how it gets delivered. Operators who understand both layers produce better outcomes than those who treat either layer as a black box. The configuration coverage in this reference is designed to bridge that understanding.

For MailWizz deployments on managed infrastructure operated by the Cloud Server for Email team, the configuration tuning, monitoring, and incident response described here is handled operationally as part of the service. Contact infrastructure@cloudserverforemail.com for a technical assessment of your current MailWizz configuration.

Statistical Significance in MailWizz A/B Tests

Running an A/B test without sufficient list size produces results that are statistically meaningless — the observed open rate difference could be random variation rather than a genuine performance difference between variants. For subject line A/B tests in MailWizz, aim for at least 2,000 recipients per variant before interpreting results as actionable.

MailWizz does not calculate statistical significance natively — you need to evaluate this externally. A simple rule of thumb: if the open rate difference between variants is less than 2 percentage points on a sample of under 5,000, the result should not be treated as definitive. Test with larger samples or repeat the test in the next campaign before acting on the results.

Advanced A/B Variables Beyond Subject Line

Most MailWizz A/B testing focuses on subject lines, but three other variables consistently produce actionable results:

• From name: Testing 'Company Name' versus 'First Name at Company' or 'First Name (Company)' often reveals significant open rate differences, especially for newer lists that don't yet recognize the brand
• Send time: MailWizz supports send time optimization within A/B tests — test morning vs afternoon delivery and segment results by time zone
• Preheader text: The preview text visible in inbox views alongside the subject line. Testing different preheader approaches (continuation of subject, complementary information, standalone hook) affects open rates independently of the subject line
• Content format: HTML rich template versus plain text variant — some audiences show significantly higher engagement with plain text for certain message types

Deliverability Implications of A/B Testing Results

A/B test results carry deliverability information beyond engagement metrics. When variant A outperforms variant B in open rate by a significant margin, and the test sample was randomly allocated, the higher open rate indicates better inbox placement — not just better subject line appeal. Variant A is reaching more inboxes, not just being opened more often by the same inbox recipients.

The reverse is also diagnostic: if both variants show low open rates despite a significant sample, the issue is inbox placement rather than subject line performance. Neither subject line is reaching enough inboxes to produce meaningful engagement. This suggests an underlying deliverability issue that A/B testing subject lines cannot fix.

Configuring Winner Selection Logic

# MailWizz A/B test winner configuration:
# Backend → Campaigns → Create → A/B Split Test

# Recommended settings:
# Test group size: 20-30% of list (10-15% per variant)
# Winner criteria: Open rate (most reliable signal)
# Auto-send winner after: 4-8 hours (allow time for opens to accumulate)
# Winner send to: Remaining 70-80% of list

# For click-through rate as winner criteria:
# Increase test period to 12-24 hours (clicks lag opens)
# Minimum sample: 5,000+ per variant for CTR comparison

# Manual winner selection (when to use):
# - When auto-selection timing is awkward (e.g., auto-send at 3am)
# - When margin is very small and you want to review both results
# - When testing a completely new message format/approach

Building a Test History for Long-Term Optimization

Individual A/B tests have limited value in isolation. Their value multiplies when accumulated into a test history that reveals consistent patterns about your specific audience's preferences. Maintain a spreadsheet of all A/B tests with: date, list segment, test variable, variant descriptions, winner, margin of victory, and any observable list-quality or seasonal factors.

After 20-30 tests, patterns emerge: does personalization consistently outperform generic subjects? Do shorter subjects consistently outperform longer ones? Are there specific subject line structures (question format, urgency framing, number-led) that consistently produce higher open rates? This accumulated knowledge produces more reliable uplifts per test than starting from scratch each time.