As I've mentioned, I handle hundreds of messages per day for myself and my clients. I have separated my email from bulk email effectively now, but still found my computer sluggish.
The problem seemed to be around Apple Mail. I'm new to IMAP so I decided to dig deeper. I initially thought the issue was with Rules, as I have SpamSieve and many dozens of rules to deal with bulk email (things I might want to read but don't want in my InBox).
The problems turns out to be something else altogether:
Smart Mailboxes. Every time you get new mail your Smart Mailboxes folders update their unread counts: "Updating Smart Mailbox Unread Counts" is the message you will see in Mail's Activity Monitor.