Relay Restrictions and SMTP service, IISOLE.dll reference
This is one of *those* posts I refer to from time to time. There have been a few posts on http://Forums.iis.net regarding how to manage relay restrictions in SMTP service. I've used the IISOLE.dll against an IIS 6 server with no issues. I've not tested on IIS 7, so mileage may vary. Make sure to install the IIS 6.0 compat modules and test on a non-production box. If nothing else, download a trial version and create an instance using vmware, hyper-V or Virtual PC.
http://forums.iis.net/p/1148477/1865759.aspx
On a side note for SMTP questions. I'd recommend checking the SMTP / NNTP newsgroup host by Microsoft. This is an active and targeted source of information for the SMTP service.
Hope this helps.
Cheers,
Steve