There are many reasons for using TLS, but mainly to prevent man-in-the-middle attacks while transferring data on a public network (Internet). The system supports the STARTTLS command (and SSL without STARTTLS, often referred to as port 465). Your SMTP listeners certificate may just as well be self-signed as authority-signed (trusted CA) since most e-mail systems use opportunistic TLS (may encrypt if possible) but doesn't validate the certificate or issuer (since it wouldn't really add additional security since the TLS was optional to begin with).
However, when connecting to a remote SMTP server you may choose to also verify the remote servers certificate (as done with any of the TLS settings including the verify option). The remote servers certificate must then be added on the web administration's Configuration > Email engine -> Certificates and keys page (in PEM format and include the BEGIN CERTIFICATE section). You should not use verify option for transports delivering to lookup-mx types of transports (since that would require you to add all CA and self-signed ever used in the entire world, which would be a highly unpractical and time consuming task).
PKI with optional/may (only warnings):
X.509: /C=US/ST=California/L=Mountain View/O=Google Inc/CN=mx.google.com issued by /C=US/O=Google Inc/CN=Google Internet Authority G2 X.509 error: unable to get local issuer certificate (20) Connection is now using TLS
X.509: /CN=mail.halon.se issued by /CN=mail.halon.se X.509 error: self signed certificate (18) Connection is now using TLS
PKI with verify (bad, error):
X.509: /C=US/ST=California/L=Mountain View/O=Google Inc/CN=mx.google.com issued by /C=US/O=Google Inc/CN=Google Internet Authority G2 X.509 error: unable to get local issuer certificate (20) Connection could not be secured using TLS (aborting)
X.509: /CN=mail.halon.se issued by /CN=mail.halon.se X.509 error: self signed certificate (18) Connection could not be secured using TLS (aborting)
PKI with verify (ok after installing the Equifax Secure Certificate Authority certificate 578d5c04.0):
X.509: /C=US/ST=California/L=Mountain View/O=Google Inc/CN=mx.google.com issued by /C=US/O=Google Inc/CN=Google Internet Authority G2 Connection is now using TLS
PKI with verify (ok after installing the our self-signed public key):
X.509: /CN=mail.halon.se issued by /CN=mail.halon.se Connection is now using TLS
When announcing TLS, you may choose to use either a self-signed or authority-signed certificate, both are equally good. These should be installed on the System > PKI page (in PEM format, with both the BEGIN CERTIFICATE and BEGIN RSA PRIVATE KEY part) and selected to be used per mail listener. The same applies to the web administration and other services.
Convert between PKCS#8 and PKCS#1
With OpenSSL 1.0.0 and later, you need to convert the private.key from PKCS#8 to PKCS#1 before importing. You can tell if the private.key starts with BEGIN RSA PRIVATE KEY (PKCS#1) or BEGIN PRIVATE KEY (PKCS#8).
openssl rsa -in private.key -out private2.key
Convert between PKCS#12 and PEM
To convert a PKCS#12 file into a PEM file you can use the following OpenSSL command
openssl pkcs12 -in filename.pk2 -out filename.pem -nodes
Generate self-signed certificates
You can generate a self-signed certificate using the OpenSSL command found in both Mac OS X and Linux/BSD flavours
openssl req -x509 -nodes -days 1460 -newkey rsa:2048 -sha256 -keyout private.key -out certificate.crt
and paste the contents of both those files into the text area when creating a new PKI certificate.
Install a public key using OpenSSL
Go to System > PKI and create a new PKI of type "X.509" without private key, containing the public key (in PEM format). You may fetch a certificate using [OpenSSL]. That however, is not a secure way to obtain a certificate if you already is affected by a man-in-the-middle attack. Instead you should extract the certificate from the original certificate installed on the server, but that is not covered in this documentation.
openssl s_client -starttls smtp -connect smtp.google.com:25 -showcerts
And for HTTPS services
openssl s_client -connect gmail.com:443 -showcerts
Use the first -----BEGIN CERTIFICATE----- section.