Mark Minasi's Windows Networking Tech Page
Issue #51 October 2005

To subscribe, visit http://www.minasi.com/nwsreg.htm. To unsubscribe, link to http://www.minasi.com/unsubs.htm. To change e-mail address, switch between HTML or text format, etc., link to http://www.minasi.com/edit-newsletter-record.htm.  Visit the Archives at http://www.minasi.com/archive.htm.  Please do NOT reply to this mail; for comments, please link to www.minasi.com/gethelp.  Document copyright 2005 Mark Minasi.

What's Inside

  • News
    • SMTP CD Set At a Great Price
    • LAST Active Directory and Security Classes in DC In Two Weeks
  • Tech Section
    • We Now Have a Newsletter RSS Feed
    •  ISO Recorder:  A Great Free CD/ISO Tool
    • Group Policy Management Console Won't Run On Windows x64
    • Restart W32Time After Changing the Time Server
    • Understanding Sender Policy Framework -- So "AOL Doesn't Reject Your E-Mail"
  • Conferences
  • Bring a Seminar to Your Site

News

This month brings an announcement -- I finally got around to doing an RSS feed.  You'll also hear about some bad news on x64 and Group Policy Management Console, some troubles with the Windows Time Service and a lengthy article about anti-spam techniques followed by a how-to article on what I'm finding is an interesting contender as an anti-spam, anti-phishing tool, something called Sender Policy Framework or SPF.  Plenty to get to, but first, a word from our sponsor...

Times Running Out!  The ONLY Active Directory and Security Classes Planned So Far Are In DC -- In Just Two Weeks!

I'm trying to stay off the road so I can start working on Server R2 and Vista books, but a fair number of folks have kindly asked for another seminar... so here goes.  The only public seminar I will do for the rest of the year (and probably a good bit of the next) will be in Washington, DC at the Marriott near George Washington University this November 14-17.  I'll be running the Active Directory class and the Security class.  You can find the Security class (which is now twice the size and three times as many PowerPoint slides, all packed into two enlightening days) outline is at www.minasi.com/secoutln.htm, the Active Directory class outline is at www.minasi.com/2003outln.htm, and you can find the seminar registration page at www.minasi.com/pubsems.htm.

Tech Section

We Now Have a Newsletter RSS Feed

Some of you've asked for it, so now I've set up a simple RSS feed on my Web site.  Just visit the www.minasi.com home page and you'll see the little XML icon.  You'll see updates about once a month, whenever a new newsletter comes out.  For those who've never played with RSS, it's a simple way to find out when new things appear on the Web site.

ISO Recorder:  A Great Free CD/ISO Tool

I use XP and while it's nice that Microsoft included some support for CD writing, I find the limited version of Roxio shipped on XP to often be worse than nothing.  Now and then, I either get a CD that I'd like to convert to a .ISO file for archival and backup, or I may want to take a .ISO file and burn it to CD.  Of course, the built-in CD writing support in XP does none of that.

That's why I'm so glad that Alex Feinman wrote "ISO Recorder."  Visit http://isorecorder.alexfeinman.com/isorecorder.htm and download his ISO Recorder v2 and you'll see why.  The application, which runs on XP, 2003 or even the Vista beta, lets you create ISOs from existing CDs, or burn .ISO files to a CD.  He's implemented an x64 version, as well as a Vista version that uses Vista's DVD writing abilities to let you create ISOs and burn ISOs from/to DVDs.  (Not protected ones, of course.)  He's also got a command-line CD writer utility that I've not played with, but it might just be the answer for all of those readers of Mastering Windows XP Professional who write to ask me how to use Windows Backup to back up to a CD.  (You can't, not without extra software.)

Alex has done the community a great service.  Give his nifty program a try and drop him a line to tell him how you like it.  (And if you like it a lot, consider throwing him a few bucks at his "Paypal Donate" link.)

Group Policy Management Console Won't Run On Windows x64

I absolutely love the AMD64 chips.  They're reasonably priced and quite fast -- so fast that I'm less enamored of their 64-bitness than I am of their speed.  I moved from a Mobile Pentium 1.7 GHz-based laptop to a Turion 2 GHz system recently and am amazed at the difference in speed.  The Turion-based system (Acer's Ferrari 4000 laptop, a very nice portable) has a clock rate only slightly faster than the Pentium-based system's, but runs benchmarks 50 to 100 percent faster.  (And that not's just a difference that you only see in benchmarks; it seemed significantly faster from the first time I set it up.)  Add the nice side-effects of 64-bit, like the built-in anti-worm protection evinced by the "no execute" page flags and, again, appealing processor prices when compared to Intel's offerings and you've got a nice combination.  So one of my recent hardware purchases -- it was time to update the network hardware, as for some reason the old 600 MHz DCs won't run Server 2003 -- was an AMD64-based system.

Now, the AMD64 processors will run 32-bit operating systems without a hitch, so I could have installed regular old Windows Server 2003 Standard Edition with SP1, but I figured I'd boldly go where few have gone and all that sort of thing and install Server 2003 x64.  It installed fine and as far as I can see runs pretty much identically to 32-bit Server, except for a few different entries in Task Manager, or, better yet, Mark Russinovich's special 64-bit version of Process Explorer.

So I was stumped by x64's reaction to Group Policy Management Console.

I mean, GPMC is a "given" around my network.  Set up a DC, you gotta put GPMC on it, right?  Well, you can surely do that, but it won't work on Server x64.  Right-click on a policy and try to edit it, and GPMC will tell you that it can't find "gpedit.msc," although it's there.  Group Policy expert Darren Mar-Elia tells me that the problem is that GPMC is built atop the .NET programming platform and apparently the 64-bit implementation of .NET 1.1 is a bit different (heck, it's probably 32 bits different, y'know?) from the 32-bit version, sufficiently so to break some .NET apps... and GPMC is apparently one of them.  And, I'm told, there is no fix, nor is one coming any time soon.

All's not lost, though.  You can always run GPMC on a 32-bit workstation or server and do your work remotely, that seems to work.  But it's quite discouraging to see this sort of thing, particularly as Microsoft's official stance about 64-bit hardware is that 64-bit is the standard for server hardware, and that 32-bit server hardware is considered "legacy."

Restart W32Time After Changing the Time Server

Most of the time, Server 2003 behaves as 2000 Server did.  But now and then something small trips me up.

I needed to change the time server for a forest's root domain PDC emulator.  This is usually the first DC that you've ever installed on any domain in the forest, and it's the only system that doesn't have a system elsewhere in the network that it uses to automatically synchronize its time.  It's the top of the time synchronization hierarchy in Active Directory, and needs to be told where to go to synchronize its clock so that all other systems in your AD have time that is consistent and correct.  (In AD, consistent is more important than correct; every member server or workstation must have a time setting that is within five minutes of the time setting on their domain controller or authentications don't work.)

As you may know, you set the time synchronization on the domain controller holding the PDC emulator FSMO role in your forest's root domain by opening a command prompt at that system and typing

net time /setsntp:name-or-IP-address-of-external-time-server

Then press Enter.  The information goes into the Registry and survives a reboot, so you needn't type it in every day (a common question).  So, for example, if some kind soul ran a publicly-available time server at time.bigfirm.com, then I'd tell my system to synchronize to it by typing

net time /setsntp:time.bigfirm.com

None of that's a surprise to anyone who's worked on AD for a while, or to anyone who's read my Server books.  But I found that after typing in the NET TIME command to the client's DC, the time didn't start synchronizing as I'd expected it to. After a bit I thought, "well, let's restart the Windows Time Service:"

net stop w32time & net start w32time

I then forced the server to try to resynchronize its clock with this command:

w32tm /resync

And all was well.  I'm surprised that I've never stumbled on that before, but I wanted to pass it along to you.  Bottom line: when you do a NET TIME /SETSNTP: on a 2003 system, stop and start the Windows Time Service.

Understanding Sender Policy Framework -- So "AOL Doesn't Reject Your E-Mail"

Have you heard the latest Internet end-of-the-world rumor?  Seems that if you run your own e-mail server then you absolutely must implement something called the Sender Policy Framework ("SPF") because if you don't, then pretty soon (the exact date always varies or is left vague) you won't be able to send mail to AOL, Yahoo!, Hotmail, and, well, just about anyone.  Panicked yet?  Good -- then someone can sell you something, right?  Arrgh.

Okay, the truth is that while none of those folks have said that they will reject all e-mail from people who haven't done SPF, it is true that SPF is becoming one way for networks to help reduce spam or, more precisely, to reduce e-mail address forging, wherein someone sends you an e-mail falsely claiming to be from someone.  And it is true that AOL has been attempting to filter spam by creating a "whitelist" of domains that it trusts to send it spam-free e-mail, and that to be on that list AOL wants you to do SPF.  You do not, however, have to be on that whitelist to get your mail delivered to AOL, but it's not a bad idea, as AOL uses some kind of quirky rules of thumb when deciding whether or not to deliver a piece of mail to an AOL member, and so right at this very moment AOL's mail servers are refusing perfectly legitimate e-mails that "smell funny."  Having SPF configured should make your e-mail seem a bit more aromatically pleasing to AOL's servers.

What's that you say, you don't care about mailing to AOL?  I suspect that whatever AOL ends up doing, other big ISPs will do as well.  I don't think that SPF will ever become an absolute necessity, but I do think that as time goes by we'll see SPF as part of how spam filters score a particular message's probability of "spamness."  And besides, SPF's really easy to do once someone explains it, and in this article I hope to make it extremely easy to understand.  But first, a little background.  (If you're already savvy about anti-spam stuff but want to know about SPF, just skip down to "Sender Policy Framework / Sender ID:  Making Forgeries Less Likely.")

The State of Anti-Spam Tools

E-mail's great... except for spam, with the spyware, phishing, viruses and other garbage that often accompanies it.  Everyone hates spam, but apparently mere hate isn't going to change anything, as it continues to mount in our mailboxes unabated.  Now, part of the reason that spam irritates me stems from the fact while once enjoyed what e-mail could do for me, now I spend precious time each day sifting through hundreds of spam messages a day, looking for the 50 or so legitimate e-mails.  But that source of annoyance is nothing compared to how much spam makes life tough for people running e-mail servers, or ISPs.  Now, most of the world doesn't have a spam-to-nonspam ratio as bad as mine (a few dozen spam to each good message), but imagine the effect of even a 2:1 ratio.  That means that the enterprise setting up this e-mail server must budget hardware and network services three times the size of their actual needs, must spend three times what they ought to have to just to accommodate the spam.  Ugh.  For ISPs it can be worse, as they've got worry about their own members sending spam in addition to having to deal with unwanted e-mails to their members' mailboxes.

What'll fix it?  Legislation?  Probably not, given the international nature of the Internet.  The US's CAN-SPAM act, which took effect in the beginning of 2004, sure hasn't slowed down the flood of junk e-mail into my mailbox, and I'll bet it hasn't done much for yours either.  Bad guys just host their stuff on a server in another country, and the threat of penalties disappears.  There are systems with personal whitelists or "friends and family" lists, but that stuff doesn't scale well and, besides, such things usually only kick in once the spam-ee's SMTP server has already accepted the message and wasted hard disk space temporarily storing it before the whitelist/friends & family list filter gets around to checking it.

Spam filters that learn spam-like behavior, so-called "Bayesian" filters, can be somewhat effective as an anti-spam tool, but do we really want to address an Internet-clogging problem like spam by having to buy and run a server whose sole job is to reject e-mail from people who send us mail that we don't want and never asked for?  I sure don't; I'd rather kill the unwanted mail at the source, before it gets a chance to chew up bandwidth, CPU cycles, disk space and, often, my time as I pick through my mailbox looking for the odd legitimate message.  I admit that I may have a greater personal antipathy to the need for anti-spam softare because I run a 100 percent opt-in newsletter -- the one that you're reading -- and am terribly frustrated by the fact that probably a quarter to a half of my subscribers don't get my e-mailed notice that a new newsletter's available because of those spam filters.  (Believe me, that fact was a contributing factor in deciding to set up RSS.)  Don't misunderstand me, I fully sympathize with the admins who put the spam filters in, it'd just be nice if they weren't necessary.  It'd be nice if it were a bit harder for spammers to maintain what is essentially a 24/7 denial of service attack on our e-mail servers and on the Internet itself.

How Spammers Get Away With Spamming

Why can't we easily stop spamming?  Mostly because of the trusting nature of Internet protocols.  To understand why, let's look at a circa-1995 spamming.  (We'll get to the present day in no time, I promise -- unfortunately almost nothing's changed in ten years.)

Suppose I'm a spammer and I intend for your in-box to be my helpless victim.  I'd prefer that you not be able to find out who I am or what SMTP server I'm using.  That's because if you can easily determine who I am or where I'm working from then you can probably either quickly get me shut down, or, if you were of a more vigilante bent, you might try to launch some kind of denial of service attack against me.  And heck, if you did that, then how would I perform the vital service of selling Viagra, offering great mortgage rates, informing people that they've won a contest that they never entered, or proffer pictures of naked women?

Suppose my SMTP server is called " "Spammer.bigisp.net" and yours is "Receiver.unwittingvictim.com"  Here's how my server sends yours the junk e-mail.

Spammer to Receiver:  "hi there, Receiver.  My name is goodserver.bigfirm.com."

Hold on -- this machine's name isn't goodserver.bigfirm.com, it's spammer.bigisp.net."  True, but under standard SMTP, Receiver does not try to verify that.  There might even be a machine named goodserver.bigfirm.com out on the Internet, but it'll never know that Spammer's claiming to be it.  So, returning to our SMTP conversation...

Receiver responds, "hello back; I can accept the following commands," and lists its capabilities.  It does that because over the years SMTP has changed, and so the receiving SMTP might not support all of the latest bells and whistles.  So when the sending SMTP server says hello, which is an SMTP statement all in itself -- either HELO for a really old or simple SMTP server, or EHLO ("enhanced HELO") for a smarter one -- then the receiving SMTP server informs the sending SMTP server of its strengths and weaknesses when the conversation starts out.

Spammer responds, "I've got some mail here from trustedperson@bigfirm.com." 

Hold on again -- how can we verify this?  We can't under standard SMTP.  Back in the early 90's, I used to teach classes about TCP/IP and SMTP and loved to demonstrate how I could connect to the White House e-mail server and send myself e-mail from the President.  (You can't do that any more, as I'm sure you've guessed.)

Anyway, Receiver does nothing to verify this under standard SMTP, and just says "sounds good, tell me more."

Spammer says, "this mail is destined for someuser@example.com."

Receiver could in theory say "do we have a user named 'someuser?'"  Or it might even say "do I have any business even accepting e-mail for the example.com domain?  But it doesn't, and says "great, let's see it."

Spammer sends along the message, and then disconnects.  The session's over.

Notice that Spammer asserted three things:

  • Its fully-qualified domain name (FQDN) "goodserver.bigfirm.com."
  • The e-mail address of the person that Spammer claims that it is sending the mail for.  This address is called (because of the corresponding SMTP command) the "MAIL FROM" address, and in this case it's "trustedperson@bigfirm.com."
  • The e-mail address of the desired receiver "someuser@example.com."

And note once again that none of those are checked by our 1995 SMTP server.  Consider what a fraudulent individual might do:

  • Someone could, as I did, get onto an organization's e-mail server and send bogus e-mail from someone in that organization to someone anywhere on the Internet.
  • Someone could, as many spammers still do, run an SMTP server that contacts your organization offering mail to you... with a return address (the MAIL FROM, recall) of... you!
  • Someone run an SMTP server that sends out objectionable or criminal material from a server with some given actual FQDN, while claiming that its FQDN is anything else, including the FQDN of one of your e-mail servers.  This could potentially affect your organization's reputation, or at least net you a lot of e-mail and phone calls from some angry people.

So what did we do about it?  Not much, unfortunately.  But around the middle of the 90s, we fixed one thing.

1995's Improvement:  Closing Relays

In the mid-90s, Internet spam started getting totally out of hand, and that led many folks to look at that last unverified part -- the receiver address.  After all, if you're the SMTP server for bigfirm.com and I try to send you a piece of mail for velma@bigfirm.com, then you ought to be able to say right on the spot "hey, I'm sorry, but we don't have a Velma here."

That seems reasonable at first glance; why not just build SMTP servers to reject invalid receiver addresses before the mail's even accepted?  It's perfectly good thinking, except for one thing -- spammers really want to discover the e-mail addresses on your e-mail servers.  So if I were a spammer and wanted to find out the legitimate e-mail addresses at bigfirm.com, all I'd have to to would be to generate a list of a few thousand possible user names -- sue, joe, joesmith, janedoe, etc -- and try to send bigfirm.com a piece of e-mail for each of those names.  I'd then note which ones the bigfirm.com SMTP server accepted, and now I've got a bunch of valid new e-mail addresses that I can inundate with spam, sell to others, or the like. 

In fact, there is a standard SMTP command, VRFY, that lets you ask an SMTP server to verify whether or not a particular e-mail address is valid, and for a time spammers used it to test and harvest names, but most e-mail servers at this point have either redefined the command -- Microsoft's SMTP server will just say "yup, that's a properly-formatted e-mail address, but I have no idea whether or not I've got someone here by that name" -- or disabled it altogether.  Of course, that's not stopped the spammers.  Now they once again send out e-mails to random addresses, but they embed in the e-mail body a image link back to their servers.  Click on the e-mail, your e-mail client program automatically grabs the image -- the request contains your e-mail account's name in the image request -- and the spammer's server now knows that it's got a "live one."  That's why recent versions of Outlook and Outlook Express do not show images in e-mails unless you tell them to.  This spammer practice of tricking you into revealing yourself to their servers is called "beaconing."

Validating the recipient's entire e-mail address, then, won't help much.  But what about the domain of the would-be recipient's e-mail address?  Ah, that could be fruitful.  The spammer wanted to send e-mail so someuser@example.com.  But let's suppose that Receiver, our spamming victim, wasn't the e-mail server for example.com; what happens then?

E-mail Relaying Explained

In the early days of the Internet, where everyone trusted others and spammers were still making a living trying to sell parcels of swamp land to old folks and running telemarketing shops, then our Receiver would just say "gosh, I don't handle e-mail for example.com, but heck, it'll just take me a minute to look up example.com's MX record in DNS and then I'll just re-send the e-mail to example.com."  In other words, Receiver's just being a nice guy by assuming some of another SMTP server's burdens.  That sounds good, decent, and reasonable, and that's pretty much how every SMTP server that I know of ran back in 1995.  Oh, by the way, I just said that Receiver "re-sends" e-mail that it got from one SMTP server to another, but the official e-mail phrase is "relaying."  When an SMTP server "relays," that means that it accepts e-mail for domains that it does not serve.  Receiver does not serve example.com as its SMTP server, so accepting e-mail for example.com is relaying on Receiver's part.  (More on this in a moment.)

But, of course, the spammers figured out how to ruin that too.  Suppose a spammer wants to send about a million unwanted e-mails.  He doesn't, however, want the address of his computer to be obvious.  So instead of telling his SMTP server to send that million e-mails directly to their proper destinations -- send spam intended to me directly to my SMTP server, spam intended for you directly to your SMTP server, and so on -- he chooses some victim running a big, fast SMTP server on a really fast Internet connection, and sends all million e-mails to that SMTP server.  In effect, the spammer's SMTP server is dropping all of that mail on the victim's SMTP server, saying "please look up all of the relevant MX records for all of these recipients' domains, and then waste a lot of time trying to contact those SMTP servers and sending this junk e-mail."  This, then, was the thanks that an e-mail administrator got for being helpful and letting his or her SMTP server relay: wasted CPU time on the server, wasted bandwidth, and let's not forget the angry people who receive the spam, look at the SMTP headers and see that they got this spam from the e-mail administrator's SMTP server.  As they say, no good deed goes unpunished.

To combat this, most e-mail administrators adjusted their SMTP server's behavior vis-a-vis relaying.  Now, relaying is a bad word in e-mail circles, but not all relaying is bad.  You see, there is good relaying and bad relaying.  To understand why, remember the definition of relaying.  Most SMTP servers serve a particular domain or set of domains.  For example, my SMTP server receives e-mail for e-mail addresses in the minasi.com and mmco.com domains; send an e-mail to some address there and it will, if all goes, well, end up knocking or port 25 of my SMTP server, and at that point it's reached its final destination.  But that SMTP server has another job, as well:  it sends out e-mail from me.  So let's say that I want to send e-mail to someone@example.com.  Outlook prepares the message for me and contacts my SMTP server, saying "please send this to someone@example.com."  As you'd expect, my e-mail server takes the mail and sends it along.

But hey, wait -- that's relaying!  By accepting a piece of mail destined for someone@example.com, my SMTP server accepted e-mail for a domain that it doesn't serve.  But there's nothing wrong with that; you'd be pretty annoyed if it worked any other way.  Any time that you ask your local SMTP server to send e-mail to someone else in your company, then there's no relaying, because that SMTP server is supposed to serve your company's e-mail domains.  But if you want your e-mail server to send a message to someone outside of your organization, then you are asking it to relay.  That's good relaying.  In other words,

  • If you want your SMTP server to accept Internet e-mail for a domain other than you own, that's good relaying.
  • If your SMTP server accepts e-mail from someone outside of your organization for re-sending to yet another domain outside of your organization, then spammers can exploit your SMTP server, and that's bad relaying, and you are said to have an "open relay."

So, as I said before, most e-mail administrators have learned to reconfigure their servers so that those servers won't relay for the whole world, and instead will only relay for internal users.  But it didn't happen overnight...

Open Relay and Spammer Lists

Anyone who ran an SMTP server program in the early 90's can testify that closing an SMTP server's open relays wasn't very simple in those days.  Worse yet, many e-mail admins just plain didn't understand what an open relay was.  As a result, they didn't understand that they were dooming their systems and their bandwidth to exploit by spammers, and in the process making the Internet less usable for the rest of us.

In response, several private groups decided to create a sort of vigilante effort against people running servers with open relays.  They built publicly-available online databases of IP addresses of SMTP servers that the database maintainers believe have open relays.  The idea is that you and I can then configure our SMTP servers so that every time someone tries to send us an e-mail using an SMTP server whose IP address is listed in those databases, then our SMTP servers reject the message and perhaps send back an e-mail explaining that the message was rejected and why.  Why do this?  Two reasons.  First, if an unprotected SMTP server has become a favorite jumping-off point for spammers, then the chances are good that the incoming mail is spam, and won't be missed if we reject the communication.  Second, the vigilantes wanted to get the attention of the administrators of these misconfigured SMTP servers, and one way to do that would be for the users of that SMTP server to notice that lots of their legitimate e-mail was not being delivered.  This is supposed to cause users in companies whose mail servers have open relays to angrily storm the e-mail administrator's office, demanding that he or she fix this problem.

This approach has annoyed some people because while getting onto an open relay database is easy -- you just need someone to rat you out to the open relay people -- then getting off can take a while because, as the folks who run the various open relay databases are prompt to point out, they're running a free service, and the only reason why they're inconveniencing people is because some e-mail admins have found their databases useful.  In at least one case, ISPs started blocking those IP addresses on their routers, meaning that no Internet traffic at all could pass from the organization with the open relaying SMTP server to that ISP.  That was a bit of overkill, as a bit of configuration ignorance on the e-mail administrator's part meant that that the users of that ISP could not communicate with the organization at all -- couldn't send them mail, couldn't visit their Web site, etc.  That's bad, but what's worse is that after the e-mail admins fixed their problem and were no longer running open relays, the ISPs who blocked them at their routers took months or years to remove the blocks.  As a result, the former open relayers had stepped into the light, but still couldn't communicate with clients in the blocking ISP's networks.  Some former open relayers had to leave their ISP -- and their tainted IP addresses -- and go to another ISP.  Furthermore, in my opinion it's all well and good to say "hey, I'm just doing this for free, nothing is my responsibility," but in fact one database, Osirusoft, left the field after repeated denial of service attacks by spammers.  That's a shame, but their response in late August 2003 was to modify their list of spammers to "the whole world" so as to convince everyone who was using their database to stop using that database, as they took their football and went home.  You kind of have to wonder how many legitimate e-mails went lost because the Osirusoft database operator decided to stop the service after spending years convincing the Internet community to trust it.

Blacklists and Whitelists

The success of open relay databases led to "blacklisting databases" of known spammers.  Places like Spamhaus and Spamcop run free anti-spam databases used by thousands and thousands of e-mail admins.

While you have to admire the work that these folks do for no pay, I think they'd be the first people to say that they hope that a day will come when they're obsolete.  So what more can be done to detect and reduce spam?

In addition to blacklists, we've got "whitelists," which essentially say to the world, "I will reject all e-mail except for this list of people that I trust."  That probably works for some folks, but it surely doesn't work for me, as I have no idea who's read one of my books, seen a presentation that I've done, or read something that I've written on the Web.  That said, there are some very clever approaches to whitelist technologies when used in tandem with Bayesian spam filters.  For example, some Bayesian filters watch who's in your Contacts list and who you send mail to, automatically adding those folks to your "don't bother checking to see if this is spam" whitelist -- a smart feature.

2005 Antispam

So far we've got five approaches to reducing spam -- three options we've discussed, and two for-pay options we haven't but that I'm sure you've already heard of:

  • Close our open relays and use open relay databases to chivvy others into closing their open relays
  • Use spam blacklists to reject e-mail from systems in the "known spammers" databases
  • Use some kind of whitelist either to control entirely what gets in, or to modify a Bayesian filter's behavior
  • Install servers in our network that employ technologies that let them make decent guesses at what's spam and what's not
  • Install servers in our network or rent access to commercially-maintained databases of spam and spammers

What more can we do?  Short of hiring an "e-mail butler" with the intelligence to be able to filter the wheat from the chaff, there's not a lot more that will examine the message and make a decision based on that.  The next step seems to be to examine the sender more closely, to ask, "do I trust this guy?"  We're doing some of that now with open relay databases and spam blacklists, but spammers can move quickly, more quickly than these databases can respond.

If we could fairly reliably identify the senders, then we could stop the spammers.  But how, given that they can move around so quickly?  Well, they can move around so quickly because it's so simple to fire up a computer on a random IP address and start spewing spam by lying about who the sender is.

So one answer would be to change that, to say to your SMTP server, "if you can't positively verify that this sender is who it says that it is, then reject its mail."  How would that work?  Here are a few approaches based on the information that an SMTP server gets when contacted by another SMTP server, in roughly increasing order of severity in terms of how much the Internet would have to change for it to happen:

  • Look for systems that are not configured according to RFC standards, such as those lacking reverse-lookup (PTR) records, and reject their communications.  (Discussed below.)
  • Verify that the fully qualified domain name specified in the initial "hello" command matches the IP address of the server.  (Discussed below.)
  • Go back to the domain that the sending server claims to be sending mail from, and ask that domain if that machine is authorized to send mail for it.  (Discussed below.)
  • Demand and require that a machine sending e-mail from a given domain present a digital certificate identifying it as an authorized sender of e-mail from that domain.  The technology for this exists, but I think that digital certs have to become cheaper before this will work.  Sure, I can afford to shell out a few hundred bucks a year for Web SSL certs, but I make money from my Web site.  Saying that the entry fee to running your own SMTP server is a few annual C-notes to VeriSign, Thawte, Network Solutions or the like would be, at least in my opinion, a bad thing -- while I dislike the current anonymity that is part of the warp and woof of the Internet, I love its egalitarian nature.  Saying "you've got to make VeriSign richer or you can't run a mail server" troubles me a bit.
  • Before accepting mail from a given domain, check with a trusted third-party that the domain has done something to establish a reputation as a non-spammer.  This is an example of what anti-spammers call a "reputation"-based rule, and it's still very fuzzy how it'd work.  The first firm that I know of that supports this is the "Bonded Sender" program at www.bondedsender.com.  The basic idea seems to be that you want to send e-mail and be recognized as a good guy, so you post a bond with that "trusted third-party" as well as furnish some identification information.  If too many people complain that you've spammed them, then there's some process whereby they figure out whether you truly are a spammer or not.  They then offer a list of trustworthy senders.  An interesting idea and perhaps where we'll go one day, although as far as I can see it'll cost you at least a grand a year to maintain your "white hat emailer" status, and that might be a bit rich for many small firms' blood.

I'm not so sure about the last two, although I could imagine things will get so bad that we'll get there.  But let's look at the first one, then take up the next three in the SPF section.

Requiring Reverse DNS Lookup:  Checking PTR records

Many SMTP servers have an option that looks something like "reject incoming messages with invalid PTR records" or the like.  (On MailEnable, which I'm using now, the option is "Require PTR DNS entry from unauthenticated connections," but I've seen many variations on mail servers over the years.)  What does this mean?  Can it help you repel spam messages?

First, a bit of review and background.  According to RFC 1912, every system visible on the Internet should have an A ("Address") record, so for example someone who wants to connect to web2.minasi.com can ask DNS, "what's the IP address corresponding to web2.minasi.com?" so that DNS can respond, "70.168.214.165."  Some DNS zone should also have, according to that same RFC, a PTR record which would let someone say "hey, what host name is associated with IP address 70.168.214.165?" and get in return the answer "web2.minasi.com."  Most of us are good about the A records; we're not always so careful about the PTRs.  But it's likely that a spammer who's thrown up an SMTP server for a few hours of junk mail spewing hasn't had the time or inclination to bother with PTR records, and we might be able to use that to our advantage.

Again, in the perfect world -- or at least the RFC-compliant world -- then you should be able to start with a host name like web2.minasi.com, look up its A record to get an IP address of 70.168.214.165, look up that IP address's PTR record and get back web2.minasi.com.  But accomplishing this A-leads-to-PTR-leads-to-identical-A equivalence might be tough for a lot of folks, because many of us control our forward lookup DNS domains, but don't control the reverse lookup domains; rather, the ISP controls the reverse lookup domains.  So it's likely that a PTR lookup for web2.minasi.com's IP address would yield not "web2.minasi.com" but instead something more like "wsip-70-168-214-165.hr.hr.cox.net."

If you see oddball PTR results like that, then it's likely that the IP address in question is controlled by a big ISP that's probably a big DSL and/or cable provider.  In my case, the folks who provide my IP addresses are Cox Cable.  Cox knows that RFC 1912 compliance wants to see both forward and backward lookups for any given host, but they also know that while their customers can create their own forward DNS zones, they can't create reverse DNS zones, as Cox controls them.  So Cox worked around it -- as many large providers do -- by creating an "hr.hr.cox.net" subdomain, and then creating an A record for each of their IP addresses.  They could then create reverse lookup domains and point the PTR records for each IP address to that address's corresponding hr.hr.cox.net record. 

How does this affect what an SMTP server would do?  Over the years, people have sometimes tried to introduce an element of accountability into Internet communications by saying to any incoming system, "I see your IP address; I will then look to see if there is a PTR record that corresponds to that IP address."  And that's all that this setting does.  So, for example, suppose you've got an SMTP server with one of these "I need a valid PTR or we won't talk" settings, which you've enabled, and I try to send you an e-mail.  The communication could look like this:

  • My SMTP server to yours:  "Hi, I'm an SMTP server named web2.minasi.com that you can see has an address of 70.168.214.165.  Can we talk?"
  • Your SMTP server:  "Hmmm... let's query DNS for a PTR record for 165.214.168.70.in-addr.arpa... ah, I see that it's a system named "wsip-70-168-214-165.hr.hr.cox.net" ... well, that doesn't match the name that you sent me, but what the heck, at least you've got some PTR record, so let's talk.

(In case you're wondering, I don't know of any e-mail servers that have a rule that says, "I'll take the incoming IP address and look up its PTR record, then take the FQDN that came from that lookup and compare it to the name that you claimed when you said hello," although certain SPF settings do that, as you'll see later.)

Suppose you decided to require that all SMTP servers that tried to talk to your SMTP server have a PTR record of some kind.  What would be the pros and cons?  The benefit would be that you'd stop some spam.  In my experience, 100 percent of the incoming e-mail that I've seen (in the past week, anyway, when I turned the feature on to test it) is spam -- I know this from looking at my SMTP logs.  The downside?  There's a chance that you'll register a "false positive" -- in other words, you'll reject legitimate e-mail from people who either don't know how to configure PTR records or who are served by an ISP who refuses to provide PTR records.

Again, why would this work as an anti-spam tool?  Because it assumes that spammers tend to run poorly-configured servers.  Is that a good assumption?  It's not a terrible one, as spamming is becoming more and more criminalized around the world and so therefore spammers will have to move around a lot and won't want to invest the time to create servers that follow RFCs.  But it definitely has the side-effect of raising the bar a bit for someone who wants to set up his or her own SMTP server.  For example, people who lack a static IP address may not be able to set up an SMTP server and have anyone accept mail from it.  That troubles me a bit, but the fact is that inasmuch as we don't have a "reputation-based" way of finding and filtering spam producers, we end up using some degree of RFC compliance as a proxy for reputation.  But admittedly, it is just a stopgap; we need some better means. 

Well, then, short of creating a certificate- and reputation-based e-mail infrastructure that significantly raises the cost of entry for people wanting to run an e-mail server, what can we do?  Some people have an answer -- SPF.  Which brings me to the next article.

Sender Policy Framework / Sender ID:  Making Forgeries Less Likely

The latest class of proposals to make the Internet a somewhat more trustworthy place, and to make opening e-mails a bit less scary, is a pair of ideas called Sender Policy Framework (SPF) and Sender ID.  These two proposals are from different groups and there's a ton of politics between the two, but I'll skip that for this discussion and focus on SPF because if you're configured for SPF, then you're also configured for Sender ID.

The goal of SPF is simple:  when you get an e-mail from someone at some domain, then SPF tells you whether or not you can feel fairly certain that this e-mail did indeed come from that person.  Before SPF, you don't know if a return address for a given piece of mail is trustworthy or forged; SPF tries to stamp out return address forgery.  You've seen forged return addresses if you've ever seen

  • An email addressed to you ... from you
  • A fake message from some network support address at Microsoft with an enclosed "security patch" that is, of course, a virus
  • An e-mail from PayPal, your bank, your brokerage firm, or the like with a dire message exhorting you click somewhere to update your account or it'll be closed; of course you are actually directed to some server that will, once you type in your account name and password, steal your money
  • A virus sent from friend A, who never sent the virus because in actuality friend B opened a virus and the virus found friend A's e-mail address in friend B's Contacts list and decided to send viruses to everyone on friend B's Contacts list... but with a forged return address from friend A

This is the kind of thing that SPF and Sender ID want to stop.  So, to exact, SPF isn't an anti-spam tool, it's an anti-return address forging tool.  But a lot of spammer, viruses, and phishing schemes employ forged return addresses.  Yes, even in an all-SPF world, spammers could still spam you... just so long as they specify their real server and domain names.  Or so long as they forged addresses from a domain that didn't bother to implement the most basic SPF, as we'll see.

Here's another way to think about SPF.  Consider how you'd send me a piece of e-mail now.  You know that my e-mail address is help@minasi.com.  But what server accepts e-mail for minasi.com?  As you probably know, your SMTP server queries my DNS server for "Mail Exchanger" or MX records.  The minasi.com DNS server responds, "the server that receives e-mail for minasi.com is web2.minasi.com."  Now your SMTP server knows where to send e-mail to me.  Notice that you went to my DNS domain to get that information, so unless someone's monkeyed with DNS then you feel pretty sure that this is indeed the server that I want you to use to send me e-mail.

SPF does the reverse.  Where MX records identify the systems that I've authorized to receive e-mail for minasi.com, SPF adds new DNS records that identify which servers send e-mail for minasi.com.  So now when your SMTP server receives a piece of e-mail purporting to be from minasi.com, then your SMTP server can query the minasi.com DNS server and say, "which systems should I expect to be receiving e-mail from you on?"  The DNS server will respond (in my case) web2.minasi.com, the same system that receives e-mail.  If your SMTP server sees that it's being offered a message from somebody@minasi.com but the server offering it is a system other than web2.minasi.com, then your SMTP server can safely discard that message.  And, once again, notice that when you receive an e-mail from someone who claims to be from minasi.com and you want to double-check which SMTP servers to accept mail from, then you come to the minasi.com DNS zone to get those answers.  I control what servers my domain accepts mail from, no one else.  No certificates needed, no seals of approval from others, no wondering how to get an ISP to add a PTR record.

How SPF Works I: Overview

To see how SPF works, let's review what we know so far about an SMTP message transfer:  what information it offers, and what we can do with that information to detect deception, which may point to unwanted e-mail.  Let's assume that this a spammer working from a machine named pc42.whocares.com on IP address 8.8.8.8 wants to send an e-mail to joe@bigfirm.com.  The spammer needs to use a return address -- not that he's intending on accepting responses! -- so he picks a random name to forge for as a return address, jane@megacorp.com.  Let's even say that the spammer is smart enough to check around and see that Megacorp, the place where Jane works, has a mail server called mail.megacorp.com. 

  • pc42.whocares.com, the spammer's SMTP server, looks up the MX records in bigfirm.com and finds the name and IP address of Bigfirm's e-mail server, as would any SMTP server.  pc42.whocares.com then says to Bigfirm's SMTP server, "EHLO mail.megacorp.com."  This is, as we've seen, a correctly-formatted greeting from one SMTP server to another.  And yes, it also contains a big fat lie: the sender's FQDN.
  • Bigfirm's SMTP server responds to the spammer's SMTP server and awaits further communication.
  • Next, the spammer's SMTP server says, "I'd like to send some e-mail from jane@megacorp.com.  Another big fat lie:  the MAIL FROM address.

At this point, Bigfirm's SMTP server has three pieces of information about this communication, which might or might not be spam.  First, Bigfirm's SMTP server knows the IP address of the would-be sender, and it's pretty sure that the address is fairly reliable.  Yes, it might be the IP address of a NAT router or the like, but that's not important; the important thing, you see, is that the spammer almost certainly can't make Bigfirm's SMTP server think that the spammer's SMTP server, pc42.whocares.com, has a valid IP address from Megacorp's network.  Bigfirm's SMTP server also knows the FQDN that the would-be sender's SMTP server claims to have, obtained from the EHLO/HELO greeting.  Finally, Bigfirm's SMTP server also has "MAIL FROM," the e-mail address that the sender claims is the source of the e-mail.

Next, Bigfirm's SMTP server takes that claimed MAIL FROM address and extracts that address's domain.  As the purported MAIL FROM address claims to be from megacorp.com, Bigfirm's SMTP server now knows that

  • There's data coming from a server with 8.8.8.8
  • The system claims to have an FQDN of mail.megacorp.com
  • The system claims that it's got an e-mail from megacorp.com

Here's where the SPF stuff comes in.  Assuming that Bigfirm's SMTP server is equipped with SPF-aware code, it contacts the megacorp.com DNS server and says, "what rules should I use to identify the servers are authorized to send mail from megacorp.com?"  (I'll show you how to do that a bit later.)  Such rules are called "mechanisms" and they're basically just patterns to match.  Here are some examples of what the Bigfirm SMTP server might get back from megacorp.com's DNS server:

  • "Take the IP address of whatever server's contacting you to send mail for us and do a PTR query on it.  If the resulting host name is in the megacorp.com domain or the mailservers.megacorp.com domain, then the server's okay."
  • "If the IP address of whatever server's contacting you is in XYZ range, then it's okay."
  • "If the IP address of whatever server's contacting you matches the A record of the megacorp.com domain, then it's okay."
  • "Look up the PTR record on whatever server's contacting you.  If it matches the FQDN that it's claiming, then it's okay."

Using this information, the Bigfirm SMTP server passes this information to its SPF filter code, asking whether the SMTP server from 8.8.8.8 is probably an authorized sender of e-mail for megacorp.com.  SPF answers this question with one of several score values:

  • None means that the Bigfirm SMTP server looked for SPF information in the megacorp.com domain and there wasn't any.  This is the most common answer nowadays, as most people don't do SPF.  (But SPF will help you even if all you want is to block email to you forging your e-mail address as its return address, as you'll see.)
  • Neutral means that megacorp.com included SPF information in their DNS zone, but it just said "we don't have a comment on this."  The effect is like "none."
  • Pass means that the SPF information in the megacorp.com domain allowed Bigfirm's SMTP server to positively identify the server at 8.8.8.8 as an authorized sender of e-mail for megacorp.com.
  • Fail means that the SPF information in the megacorp.com domain allowed Bigfirm's SMTP server to conclude that the server at 8.8.8.8 is explicitly not an authorized sender of e-mail for megacorp.com.
  • Soft fail is a state wherein the server can be identified as one that the domain's owner knows, but that the domain's owner discourages people receiving SMTP traffic from.  You might use it when in transition from a world where anyone in the organization could run their own SMTP server to one where IT controlled which servers were allowed to offer SMTP services.  In the transition period your SPF information might tell others "these are our official e-mail servers -- rate them 'pass' -- but if you see a server from inside our range of IP addresses that isn't on that list, then don't fail it, 'soft fail' it."  How would that be helpful?  We might imagine that Bigfirm's SMTP server might treat a soft fail as "let the mail pass, but send a message to the administrator at Megacorp that he's got a rogue server at XYZ IP address."

Based on the SPF return of "none," "neutral," "pass," "fail," or "soft fail," Bigfirm's SMTP server may delete the message, put it in a specific folder, mark it in some way, or whatever -- that's entirely up to the software implementer.

How SPF Works II: the Two Tests

I gave you a few examples of how SPF would score an e-mail, but let's get more specific.  SPF implementations can score messages based on two things:  the MAIL FROM domain, and the FQDN offered on the EHLO/HELO command.  More specifically, according to a SPF techie on the official SPF forum responding to a recent question:

  • All SPF checkers will check "Mail From" in creating an SPF score
  • Some, however, will always check both "Mail From" and the FQDN in the EHLO/HELO; some will stop at "Mail From"
  • Almost all SPF checkers will, if offered a message with a null "Mail From" value, go check the FQDN in EHLO/HELO
  • No SPF checkers check EHLO/HELO, but not "Mail From" values

(Hey, it's an evolving system.)

Sidebar:  What You'd Need To Do To Use SPF

That's a quick overview and I'm about to show you more of the nitty-gritty of SPF, but let's step back for a minute.  You're a busy person.  How much work is this going to entail?  And what good will it do?

There are two parts to SPF.  First, by putting SPF information in your domain's DNS zone, you advertise to SMTP servers equipped with SPF filters who can and can't send e-mail from your domain.  As I've said, large ISPs are starting to implement this, so there are a large number of e-mail users who will never get an e-mail forged in the name of any of your domain accounts.  Since installed SPF, I've stopped getting the near-daily bogus "your Paypal account will be terminated" e-mails.  (Why my dopey bank, BB&T, hasn't taken five minutes to do this is beyond me.)  I

The second part is to install some SPF-aware code for your SMTP server.  There is a free SPF checker that you can put on Exchange's IIS module, and you'd be surprised how many other systems support SPF.  The chances are good that it won't cost you much more than a download and a few minutes' configuration.  (For a list of SPF support, look at http://www.spf.pobox.com/downloads.html.)  The immediate benefit will be reduced e-mail with forged return addresses from AOL, Hotmail and a growing number of other big providers.  And, as I've suggested already, if nothing else you'll never get another e-mail from yourself, at least none that you didn't actually send!

But even if you only publish your SPF record in your DNS, then you make it a little bit harder for bad guys.  If you're a bank, financial institution of any kind, or for that matter any firm with a large consumer population who's tech-aware, then please consider at least publishing SPF records.  Won't cost you a dime.

So let's see how...

Creating SPF Records In DNS

I've said that SPF-aware SMTP servers can "score" an incoming message based on some rules stated in a DNS record.  Those rules are called "mechanisms" by SPFers for some reason.  You state the rules for determining if a given server is authorized using an "SPF record" that you create and store in a DNS record.  SPF records are just a line of text that looks like (for example)

v=spf1  +all

I'll show you how to construct your own SPF records, but first I want to show you how to install an SPF record on your DNS domain.  (By the way, the SPF record that I've typed above says "I authorize any system on the entire Internet to send mail for me," so you won't be using this for long.  On the other hand, though, if you don't have any SPF records currently, then that's about what you're offering now anyway.)

There isn't any such thing as a SPF type of DNS record, although some people would like to see such a thing exist.  Instead, we'll wrap the SPF records for our domains inside a DNS record type that you may never have used before -- a TXT record.  Its job is to let you create a record in your DNS zone and put any kind of text in it that you like. 

You can create a DNS TXT record in Windows 2000 or 2003-based DNS servers by right-clicking your zone, then choosing "Other New Records...," then "Text (TXT)" and "Create Record..."  That offers you a dialog box labeled "New Resource Record" and two fields, "Record name (uses parent domain if left blank") and "Text:."  The "Record name" field acts as it always does in DNS, to describe the thing on the left-hand-side of the DNS record.  (For RFC fans, it's called the "owner" in RFC 1034.  So, for example, when you create an A record that looks like "myhost A 10.10.10.10," then "myhost.whateveryourdomainiscalled.com" is the owner -- remember that any of those things that you put on the left-hand-side of a DNS record implicitly includes the domain's name.)  As you want to create an SPF record for your entire domain, just leave the "Record name..." field blank, as, again, blank means "this domain."  In the text field, type 

v=spf1 +all

And click OK.  Look at your DNS zone, and you'll now see something like

(same as parent folder)    Text (TXT)    v=spf1 +all

Sometimes you may want to create SPF records that apply not to the whole domain but instead to a particular host.  In that case, fill in the host's name in the "Record name..." field.

Well, congratulations, now you've got an SPF record for your domain.  But it's not a very good one, as I said, so let's see how to make it better.

How SPF Works III: Mechanisms

SPF rules have two parts.  The first part says "this is an SPF version 1.0 record."  It looks like

v=spf1

It's just "boilerplate," a "must have" set of characters up front.  Following that, you can have a series of pattern-matching rules that SPF uses to separate the "no good" servers, the "good" servers, and the "I don't care" servers.  Those pattern-matching rules are called "mechanisms."

The basic mechanisms are a, mx, ptr -- which all emply DNS records of the same name, as you've probably guessed -- ip4, which lets you specify ranges of IP addresses, and "all" which matches everything.  (There are others, like "ip6," but I'm just trying to get you started here.)  Once SPF has found a matching mechanism, you tell it whether this means "if this matches, then fail," "if this matches, then pass," "if this passes, then soft fail," or "if this matches, then consider this neutral" by prefixing it with a -, +, ~, or ?, respectively.  If you do not prefix a mechanism with anything, then SPF assumes that you meant "+," or pass.  For example, to use the absolutely simplest mechanism, "all:"

  • "v=spf1 -all" would mean "always match this mechanism" -- remember, "all" matches everything -- and now that you've matched, return a "fail" result.
  • "v=spf1 +all" would mean "always match" and the "+" would mean "now that you've matched, return a 'pass' result."
  • "v=spf1 all" would have the same effect, as the lack of a prefix assumes that prefix "+."

Each mechanism uses different algorithms for matching, and I'll explain them very soon.  But here's a point that's not so obvious until we remember that each mechanism can do two different kinds of matches.  Why is this?  Because, recall, SPF filters built into SMTP servers may do two tests -- first, they may try to match the MAIL FROM domain to something in your domain and, second, they may try to match the FQDN on the EHLO/HELO command to something in your domain.  And recall that what one piece of SPF-evaluating code doesn't necessarily match what another vendor's might do;.

So, for example, if our SPF-aware Bigfirm SMTP server were approached by a system from address 8.8.8.8 offering a server name of "mail.megacorp.com" in its EHLO/HELO command, then Bigfirm's SMTP server would first query megacorp.com's DNS zone for a TXT record.  It would then examine the TXT record and, assuming that megacorp.com had a lame SPF record like the one we're using now ("v=spf1 +all") then Bigfirm's SMTP server would ask itself one or two questions, depending on how its designer built it.

First, it would ask, "does the revealed IP address of this server, 8.8.8.8, match anything in the SPF record's mechanisms?"  Sure it does, because "all" matches everything.  Bingo on the first question.  Some SPF filters would stop here, but some would go further and ask the second question:  "does the FQDN claimed in the EHLO/HELO match anything in the SPF record's mechanisms?"  Again, it would because "all" matches everything.  Bingo on the second, and we've got a definite "pass" on this entry.

Here's a rundown on the other mechanisms.  Sometimes they do different things when applied to the MAIL FROM test than when applied to the EHLO/HELO test, as you'll see.

The SPF "a" mechanism

Including the "a" mechanism says  to SPF,

  • Do a DNS query for an A record that matches this.
  • If it's there, then you'll get an IP address returned.
  • Compare that IP address to the IP address of the would-be SMTP sending server.
  • If they match, great, then this "a" mechanism matches.

But what does it match to?  In the case of a MAIL FROM test, it's looking for an A record for the domain itself.  If the minasi.com domain had an "a" mechanism in its SPF record, that would instruct any SPF code to look for an A record for "minasi.com," rather than any host inside minasi.com.  The IP address that returns from that query, assuming that there's an A record for minasi.com, must then match the would-be SMTP server's address.

If testing the EHLO/HELO, the test is a little different.  SPF looks for an A record matching the FQDN claimed in the EHLO/HELO message, and compares the IP address the A query returns to the actual IP address of the SMTP server that's trying to send the message.

But what if your SMTP servers live in a different DNS domain, or if you just want to name a particular mail server?  (Most of us don't have a mail server with an A record for the whole domain.)  Then add a colon to the "a," followed by a server name.  For example, "a:serv1.abc.com" would mean "the system named 'serv1.abc.com' can send mail for my domain."  And you can add as many of a particular kind of mechanism as you like.  For example, this SPF record for minasi.com:

v=spf1 +a +a:po.otherfirm.com -all

Would say "accept mail claiming to be from my domain from either a system with an A record of 'minasi.com,' or accept it from a system with an A record of po.otherfirm.com."  SPF scans mechanisms left-to-right and stops as soon as it gets a "pass."  If neither the +a or +a:po.otherfirm.com patterns match, then the last pattern, "-all," matches -- because "all" always matches -- and causes SPF to score the message as a "fail."  Thus, if the original "+a" matched, then SPF would stop looking, satisfied, and return a "pass."

The SPF "ptr" mechanism

On the MAIL FROM test, a PTR mechanism starts from the IP address of the sending SMTP server and does a PTR lookup on that address.  If the result (which is a fully qualified domain name like "mail.mychurch.com") has a DNS domain that matches the domain that the SMTP is trying to send mail from, SPF scores the message a "pass."

On the EHLO/HELO test, SPF looks up the PTR record for the IP address of the sending SMTP server.  The result must match the FQDN claimed in the EHLO/HELO line. 

For example, if my mail server were called po.minasi.com, it were at IP address 8.8.8.8, and there was a PTR record allowing you to query 8.8.8.8 and get back "po.minasi.com," then I could use this SPF record

v=spf1 ptr -all

The MAIL FROM test would first try to find a PTR record for 8.8.8.8.  If that returns a FQDN of anythingatall.minasi.com, then there's a match.

The EHLO/HELO test would first do a PTR lookup on 8.8.8.8, and then compare the FQDN returned to the FQDN claimed in the EHLO/HELO command.  They must be equal for the mechanism to match.  And remember, the ptr mechanism needn't have a + in front of it because if there's nothing at the front of a mechanism, that means "pass."

The SPF "mx" mechanism

The mx mechanism is a bit more complex.  First, it takes the domain of the purported sender's e-mail address.  Then it looks up all of the MX records in the domain, up to ten MX records.  (More could be used by a spammer to slow down the SPF to a crawl by installing hundreds of bogus MX records.)  Then SPF looks up the A record associated with each of those MX records,  It then examines all of the IP addresses found from all of those A record queries.  If there are any matches to the IP address of the incoming SMTP server, then SPF scores the message "pass."

It does the same test whether it's a MAIL FROM test, or an EHLO/HELO test.  You can also indicate that e-mail servers from another domain may send e-mail for your domain by adding a colon to the end of the mx mechanism, and then name the domain, as in

v=spf1 mx mx:acme.com -all

This would instruct an SPF filter to search for MX records that lead to A records which produce IP addresses first in your own domain, and then, if that failed, the acme.com domain.

The SPF "ip4" mechanism

This is the simplest mechanism besides "all" -- it lets you just specify a range of IP addresses.  If the incoming server uses one of those addresses, the message passes. This test doesn't have to examine the claimed FQDN on the EHLO/HELO command at all, and it behaves identically whether it's doing a MAIL FROM or EHLO/HELO test.  You specify IP ranges using the CIDR "slash" notation.  For example, to tell SPF to allow any system in the C network starting at 200.14.22.0, use

v=spf1 ip4:200.14.22.0/24

Remember you can have as many ip4 mechanisms as you like.  Particular IP addresses do not need a slash.  For example, to specify that there the only two machines, 50.10.2.3 and 50.10.4.2 that can send e-mail from a given domain, we could create this SPF record:

v=spf1 ip4:50.10.2.3 ip4:50.10.4.2 -all

Testing Your SPF Record

Once you think you've got it set up right, try it out.  There's a Web page with some really useful SPF testing tools at http://www.kitterman.com/spf/validate.html.

SPF's not a perfect answer, and it's clearly an interim one.  But despite the common use of the phrase "Internet years," the Internet's become a place used by so many people that it's becoming pretty difficult to change it in short time -- so interim solutions aren't a bad idea, in my opinion.  But I'd love to hear your opinion!

Conferences

Join me at ...

TechTarget's Web Class on SP2 and SP1.  As they say it...

"Windows School is in session with Mark Minasi Mark Minasi dissects Windows XP SP2 and Windows Server 2003 SP1 in five, 15-minute webcast cram sessions. You can even download a worksheet to follow along with the lesson you're hearing. Find out the good and bad about XP SP2 and Windows 2003 SP1 in Mark's inimitable style. Topics cover: Data execution protection, stack changes, de-anonymizing XP, IE and more."
http://searchwin2000.techtarget.com/general/0,295582,sid1_gci1084934,00.html?offer=minasi

Windows Connections Fall 2005 (San Diego):   I'm not sure how California managed to snag three of the four best IT shows this year (Spring Connections in San Francisco, TechMentor in San Jose, Fall Connections in San Diego) but if you're a fan of the Golden State's weather then 2005's the conference-going year for you!  Information on Connections at www.winconnections.com.  Our program chair Amy Eisenberg's trying to out-do herself so it'll be a great show.  I'm doing my SMTP talk, the MSDE/SSX talk based on my earlier newsletter, how to fix broken Active Directories, the Windows Logons talk, and a keynote on the state of Windows.

Bring Mark to your site to teach

I'm keeping busy doing Active Directory and Security seminars and writing, but I've still got time to visit your firm.  In just two days, I'll make your current NT techies into 2000, XP, Active Directory and 2003 experts.  (And better yet they won't have to sit through any Redmondian propaganda.)  To join the large educational, pharmaceutical, agricultural, aerospace, utility, banking, government, telecommunication, law enforcement, publishing, transportation, and other organizations that I've assisted, either take a peek at the course outlines at www.minasi.com/presentations.htm, mail our assistant Jean Snead at Assistant@Minasi.com, or call her at (757) 426-1431 (only between noon-5 Eastern time, weekdays, please).

Until Next Month...

Have a quiet and safe month. 

Please share this newsletter; I'd like very much to expand this periodical into a useful source of NT/2000/2003/XP information.  Please forward it to any associates who might find it helpful, and accept my thanks.  We are now at over 40,000 subscribers and I hope to use this to get information to every single Mastering 2003, XP, NT and 2000 Server reader. Thanks for letting me visit with you, and take care.  Many, many thanks to the readers who have mailed me to offer suggestions, errata, and those kind reviews.  As always, I'm at http://www.minasi.com/gethelp and please join us at the Forum with technical questions at www.minasi.com/forum. 

To subscribe, visit http://www.minasi.com/nwsreg.htm. To change e-mail, format, etc., link to http://www.minasi.com/edit-newsletter-record.htm.  To unsubscribe, link to http://www.minasi.com/unsubs.htm. Visit the Archives at http://www.minasi.com/archive.htm. Please do NOT reply to this mail; for comments, please link to http://www.minasi.com/gethelp.

All contents copyright 2005 Mark Minasi. You are encouraged to quote this material, SO LONG as you include this entire document; thanks.