Mostly you get what you give.
We've woken up on a Sunday to find ourselves in the middle of a fishing match, one peg having been moved a tad to place it at the stern. I exchanged a few pleasantries with that fisherman and when it was time for us to move off I expressed regret that I was going to have to disturb his swim. He shrugged and was fine about it. I moved off as gently as I could and proceeded down the middle of the channel at tickover. In general, on canals, boats don't particularly scare fish but the wash from the prop will decimate a fisherman's carefully placed ground bait.
Just like most boaters, most fishermen appreciate a nod and a bit of courtesy. Yes, there are grumpy fishermen. And probably about the same proportion of grumpy boaters (many of whom seem to be on this forum).
You get what you give. Try slowing down, nodding and smiling next time!