The smtp options in the vcenter gui are severely lacking compared to any other email enabled appliance or software these days. Since the vcenter appliance basically uses sendmail for sending alerts (gah!), the options in the gui could easily include authentication options, but alas they don't. So, if you are having problems receiving vcenter alerts because of what is being described in this thread, you aren't alone. This is typically the case when you using a bogus TLD like .lan or .local on your vcenter appliance.
Duck67 above makes a nice suggestion, but I will warn that if you are going to modify your /etc/sendmail.cf on your vcenter appliance, you should also modify /etc/sysconfig/mail so that /etc/sendmail.cf is no longer generated on reboot, else a reboot of vcenter could wipe out the changes suggested by duck67.
/etc/sysconfig/mail:
MAIL_CREATE_CONFIG="yes" <--- change to "no"
A better approach might be to also update /etc/sysconfig/sendmail, and regenerate the /etc/sendmail.cf using information in the link provided below (make a copy of /etc/sendmail.cf before replacing it of course).
Then use the /etc/mail/genericstable and add a line like:
root@vcenterhost.mycompany.local vcenter@mycompany.com
then run from the vcenter root command prompt:
makemap hash genericstable < genericstable
service sendmail restart
Basically when my vcenter host with the hostname vcenter51.company.local sends email now, it sends as vcenter@company.com, prior to my changes it would only send as root@company.com or root@vcenter51.company.local, neither of which are conducive.
ref: Unofficial SUSEFAQ - Customizing sendmail.cf via SuSEconfig