From Roaring Penguin
If port 25 connection on your server are only intermittently available you may be under attack by a SYN flood.
When it happens, does the command:
netstat -tuna | grep :25 | grep SYN_RECV
show a lot of connections?
If so, it's a "SYN flood attack" - see the page at: