← RFC Reference

How Spam Filters Work

The filtering pipeline that every email passes through — from connection-time checks to machine-learning classifiers — and how major providers decide what reaches the inbox.

The Filtering Pipeline

Spam filtering is not a single check. It is a multi-stage pipeline that evaluates a message at every phase of the SMTP transaction and after delivery. Each stage can reject, defer, or flag a message. The stages roughly follow this order:

1. Connection-time checks — IP reputation, blocklists, rate limiting
2. Envelope checks — Sender verification, recipient validation
3. Authentication checks — SPF, DKIM, DMARC evaluation
4. Header analysis — Structural validation, consistency checks
5. Content analysis — Body scanning, URL checking, attachment inspection
6. Reputation scoring — Sender reputation weighted against all signals
7. Machine learning classification — Bayesian and neural network models
8. Post-delivery signals — Engagement, user actions, complaint feedback

Modern spam filters at providers like Gmail and Outlook run most of these in parallel, producing a composite score that determines inbox placement. But understanding them as a pipeline helps explain how each layer contributes.

Stage 1: Connection-Time Checks

Before a single byte of email content is transmitted, the receiving server evaluates the connecting IP address.

• Blocklist queries: The server checks the IP against DNS-based blocklists (DNSBLs) like Spamhaus SBL/XBL, Barracuda BRBL, and SpamCop. A listing on Spamhaus can cause immediate rejection with a 550 response.
• IP reputation cache: Large providers maintain their own internal reputation databases. An IP with a history of sending spam at that provider may be rejected or throttled regardless of external blocklist status.
• Reverse DNS (FCrDNS): The server checks whether the connecting IP has a valid PTR record and whether that PTR record resolves back to the same IP (Forward-Confirmed reverse DNS). Servers without valid rDNS are often rejected outright.
• Rate limiting: Unusual volume from an IP triggers throttling. The server responds with 421 (try again later) to slow down the sender.

Connection-time checks are the most cost-effective filter. Rejecting at connection saves the server from processing the entire message.

Stage 2: Envelope Checks

During the SMTP envelope phase (MAIL FROM and RCPT TO), additional checks run:

• Sender existence: Some servers perform a callback verification, connecting to the sender's MX to check whether the MAIL FROM address actually exists. This catches forged bounce addresses.
• Recipient validation: Non-existent recipients are rejected immediately (550 5.1.1 User unknown). High rates of invalid recipients from a single sender trigger throttling or blocking.
• Greylisting: The server temporarily rejects (450) the first delivery attempt from an unknown sender/IP/recipient combination. Legitimate servers retry after a few minutes; many spam tools do not.

Stage 3: Authentication Checks

Once the message content arrives, the server evaluates email authentication:

• SPF: Does the sending IP match the domain's published SPF record?
• DKIM: Is the cryptographic signature valid? Does the signing domain match the From: header?
• DMARC: Does either SPF or DKIM pass with alignment to the From: domain? What policy did the domain publish?

Authentication results are recorded in the Authentication-Results header:

Authentication is a prerequisite, not a guarantee. Passing SPF, DKIM, and DMARC does not mean your message reaches the inbox. Spammers can set up valid authentication too. But failing authentication is a strong negative signal that will almost certainly route your message to spam or rejection.

Stage 4: Header Analysis

Spam filters inspect message headers for anomalies:

• From:/Reply-To: mismatch: Different domains in From: and Reply-To: can indicate phishing.
• Missing or malformed headers: A missing Date: or Message-ID: header suggests the message was generated by crude spam software rather than a legitimate mail client.
• Received: chain analysis: The sequence of Received: headers should tell a coherent story of how the message traveled. Forged Received: headers or impossible timestamps are red flags.
• Excessive recipients: A To: header with hundreds of addresses, or a Bcc-heavy send pattern, is characteristic of bulk unsolicited mail.
• Header injection: Newlines or unusual characters in header values can indicate attempted header injection attacks.

Stage 5: Content Analysis

Content analysis examines the message body, HTML structure, and attachments.

Text and HTML analysis

• Keyword and phrase scoring: Certain phrases ("act now," "limited time," "click here") contribute to a spam score. No single phrase triggers filtering — it is the accumulation of multiple signals.
• HTML-to-text ratio: An email that is entirely images with almost no text is suspicious. So is an email with a tiny amount of visible text and a large block of hidden text.
• Hidden text: White text on a white background, zero-pixel fonts, or CSS display:none content is a classic spam technique that filters specifically detect.
• Image-only emails: Messages consisting solely of one large image with no text were historically used to evade text-based filters. Modern filters flag this pattern.
• Obfuscation: Using character substitution ("fr33," "v1agra"), Unicode lookalikes, or Base64-encoded content to hide spam keywords. Filters decode and normalize content before analysis.

URL and link analysis

• URL blocklists: Links are checked against URIBL, SURBL, Google Safe Browsing, and provider-specific databases. A single link to a known-bad domain can cause the entire message to be flagged.
• URL shorteners: Shortened URLs (bit.ly, tinyurl) are resolved to their final destination and checked. Excessive use of URL shorteners is itself a negative signal.
• Mismatched link text: An anchor tag that says "www.bank.com" but links to "evil.example.com" is a phishing signal.
• Too many links: An email with dozens of links to different domains suggests a spam or affiliate marketing message.
• Newly registered domains: Links to domains registered within the last few days are suspicious.

Attachment analysis

• Executable files: .exe, .scr, .bat, and similar executable attachments are almost always blocked or quarantined.
• Password-protected archives: .zip files with passwords prevent scanning and are treated with suspicion.
• Macro-enabled documents: .docm, .xlsm files are common malware vectors.
• File type mismatches: A file with a .pdf extension but executable content in its binary headers is flagged.

Stage 6: Reputation Scoring

All of the above signals feed into a reputation model. This is where IP and domain reputation have their greatest impact.

Reputation acts as a multiplier. A sender with excellent reputation gets the benefit of the doubt — borderline content is delivered to the inbox. A sender with poor reputation gets no benefit of the doubt — even clean content may be filtered. This is why reputation is often more important than content.

Providers weigh signals differently:

• Gmail places heavy emphasis on domain reputation and user engagement. Google Postmaster Tools categorizes domain reputation into four levels: High, Medium, Low, and Bad.
• Outlook.com weights IP reputation heavily and relies on Sender Reputation Data (SRD) from a panel of voter users who rate messages as junk or not-junk.
• Yahoo uses a combination of IP and domain reputation with significant weight on complaint rates from their feedback loop program.

Stage 7: Machine Learning Classification

Modern spam filters use machine learning models trained on billions of messages.

Bayesian filtering

The foundational technique. A Bayesian filter calculates the probability that a message is spam based on the frequency of its words (tokens) in known-spam versus known-ham corpora. If the word "invoice" appears in 80% of ham and 5% of spam, it is a strong ham signal. If "unsubscribe" appears alongside "Congratulations! You won!" the combined probability shifts toward spam.

Bayesian filters are adaptive — they learn from new messages. When a user marks a message as spam, the filter updates its probability tables. This per-user learning is why the same message might be filtered as spam for one user and delivered to the inbox for another.

Neural network models

Major providers now use deep learning models that go far beyond individual word frequencies. These models evaluate:

• Semantic meaning of the message (not just keywords)
• Structural patterns in HTML
• Temporal patterns (sending time, frequency, bursts)
• Relationship between sender and recipient (have they exchanged mail before?)
• Similarity to known spam campaigns (cluster analysis)

Google's spam filters, for example, process over 99.9% of spam before it reaches any inbox, while maintaining a false-positive rate below 0.05%. This is only possible with large-scale machine learning.

Stage 8: Post-Delivery Signals

Filtering does not stop when the message hits the inbox. Post-delivery signals continuously refine placement:

• "Report Spam" clicks: The most direct negative signal. If many recipients report messages from a sender as spam, future messages from that sender are more likely to be filtered for all recipients.
• "Not Spam" / rescue from junk: Moving a message from spam to inbox is a positive signal that tells the filter it made a mistake.
• Read/open behavior: Messages that are consistently opened and read signal value. Messages that are deleted without reading signal the opposite. Gmail uses this heavily.
• Reply behavior: Replying to a message is a very strong positive signal — you do not reply to spam.
• Contact list: If the sender is in the recipient's address book, the message is almost always delivered to the inbox.
• Dwell time: How long a recipient spends reading a message before moving on.

Engagement-based filtering creates a feedback loop: if your early messages to a new subscriber are not opened, future messages are more likely to be filtered. This is why IP warming advice always says to start with your most engaged recipients.

How Major Providers Differ

Gmail

Gmail's filtering is the most sophisticated and the most engagement-driven. Key characteristics:

• Domain reputation weighs more than IP reputation.
• Engagement signals (opens, replies, spam reports) heavily influence inbox placement.
• Gmail categorizes some mail into tabs (Primary, Promotions, Social, Updates) which is separate from spam filtering but affects visibility.
• Since February 2024, Gmail requires bulk senders (5,000+ messages/day to Gmail) to authenticate with SPF, DKIM, and DMARC, provide one-click unsubscribe, and maintain spam complaint rates below 0.3%.

Outlook.com / Microsoft 365

• IP reputation is heavily weighted. Microsoft maintains a large internal IP reputation database.
• The Sender Reputation Data (SRD) panel — real users who vote on whether messages are wanted — feeds directly into filtering decisions.
• Exchange Online Protection (EOP) uses multiple layers including connection filtering, policy filtering, and content filtering.
• Microsoft's SmartScreen filter analyzes message characteristics against a model trained on known spam and phishing.

Yahoo / AOL

• Complaint rates from Yahoo's feedback loop are a primary signal.
• Yahoo was an early adopter of DMARC p=reject, which it enforces strictly.
• Yahoo joined Gmail in requiring bulk sender authentication and one-click unsubscribe in 2024.

Spam Filter Testing and Debugging

When your messages land in spam, you need a systematic approach to diagnose the cause.

Reading filter headers

Most spam filters add headers to the message that reveal their verdict. Send a test message to yourself and inspect the raw headers:

These headers tell you exactly which tests were applied and what their results were. The Authentication-Results header is standardized; the spam-score headers are filter-specific.

Seed testing

Send test messages to accounts at multiple providers (Gmail, Outlook, Yahoo, corporate servers) and check whether they land in the inbox or spam. Do this before every major campaign or infrastructure change. Several third-party services automate this with panels of test addresses across dozens of providers.

Isolating the variable

If a message lands in spam, change one variable at a time to identify the trigger:

• Send the same content from a different domain — if it delivers, the problem is reputation, not content.
• Send different content from the same domain — if it delivers, the problem is content-specific.
• Send to the same provider from a different IP — if it delivers, the problem is IP reputation or blocklisting.
• Remove all links and send again — if it delivers, one of your URLs is blocklisted.

What Can Go Wrong

Legitimate email filtered as spam

Your transactional emails (password resets, order confirmations) land in spam because your marketing emails on the same domain tanked your domain reputation. The fix: consider separating transactional and marketing email onto different subdomains so reputation damage from marketing does not affect critical transactional delivery.

Content triggers on legitimate content

Your invoice email contains the word "payment" plus an attachment plus a link — all legitimate, but the combination scores high. The fix: ensure strong authentication and reputation so that content signals are evaluated in the context of a trusted sender.

Engagement death spiral

You send to a large list of inactive subscribers. Few open your email. The low engagement rate causes providers to move subsequent messages to spam. Even fewer people see them. Open rates drop further. More messages go to spam. The fix: regularly prune inactive subscribers and use re-engagement campaigns before they become disengaged.

URL blocklisting

A domain linked in your emails gets blocklisted (perhaps your tracking domain, or a shared link shortener). Every email containing that link is now flagged. The fix: use your own domain for tracking links, monitor link reputation, and avoid shared URL shorteners in email.

Key Takeaways

• Spam filtering is multi-layered. No single check determines inbox placement. It is the aggregate of connection, authentication, content, reputation, and engagement signals.
• Authentication is necessary but not sufficient. Passing SPF/DKIM/DMARC does not guarantee inbox delivery, but failing them nearly guarantees spam or rejection.
• Reputation trumps content. A trusted sender with borderline content gets delivered. An untrusted sender with clean content gets filtered.
• Engagement is the new frontier. Gmail especially uses opens, replies, and spam reports as primary filtering signals. Sending to people who want your email is the most effective deliverability strategy.
• Each provider is different. Gmail is engagement-driven, Outlook is IP-driven, Yahoo is complaint-driven. Optimize for each.
• Separate your mail streams. Use different subdomains for transactional and marketing email to isolate reputation.
• Monitor and adapt. Spam filtering evolves constantly. What worked last year may not work today. Use Postmaster Tools and feedback loops to stay informed.

Further Reading

Related RFCs