I think you’re right. Was playing with the plugin on a sandbox site I’ve got. And at some point, I believe I turned on Role Manager to give authors the moderate comments permission.

seems like your plugin could be modified rather easily to do a check on a logged in user.
Something like…
IF post-author-id == logged-in-userid THEN display moderation tools