I'd take my chances in gale force winds on the ocean over a sunny Sunday on The Harbour. At least waves and rocks are more predictable than power boats! Its usually not too bad Saturday Mornings - especially if its rainy or overcast and can be almost deserted in Winter.
We usually hit Sydney Heads by about 8.30 and depending on the group either go for coffee at Manly on the inside or round to Shelly Beach on the outside.
Thinking about a sight like this 238 years ago sent a shiver up my spine