If you meet an isolated fisherman there is a good chance he'll indicate which way he'd like you to go, but in the absence of a signal you may be able to see which side they are fishing and therefore go the other way. Most of the time if they are using a long pole they wil be fishing on the offside and you'll be able to see that as you approach so you should aim for the nearside. If they're using a short rod they could be either side, so it's best to aim for the mddle unless directed otherwise.
It can be very difficult when you meet a line of fishermen in a competition, some fishing nearside and some offside - and probably one or two in the middle as well - all waving at you to go one side or the other to their own benefit and possibly trying also to get you to upset their competitors. You definitely can't please all of them, but going down the centre will usually be the best option.
Do remember that while slowing down is usually the best thing to do, on a windy day you must keep going fast enough to ensure that you don't get blown across to the side. If you run aground next to an angler, squashing his keepnet as you do so, and then churn up the bottom while you struggle to get going again, you're not going to have made him very happy. In fact the first issue of the waterways code for boaters, on the advice of the angling associations, specifically advised against slowing down for this very reason. And on certain waterways (the Chesterfield in particular) you'll get frequent requests to speed up and churn up the mud a bit so the fish can't see so well - but this is definitely on only a minority of waters.