That's the trick. You don't need to deliberately dawdle, but rushing - or worse allowing yourself to be rushed - is for idiots. Once you get into a smooth rhythm it tends to speed up, but doing all the steps of the process carefully and deliberately is far more important than doing it quickly.
It's not actually that difficult if you can climb ladders, but as Arthur says above sometimes it's just easier to not bother and line the boat in or out of a drained lock than it is to keep climbing up or down. Where you choose to do it matters as well - on a 5' deep lock it's fairly trivial, on a 15' lock it feels harder!