The only budget digs on the bay, depending on which room you get, JJ's can actually be quite nice, with spacious wooden bungalows decked out with good mattresses, flat-screen TV and split-system air-con. It's on the innermost point of the bay – you'll need to walk up and over the hill or take a water taxi to reach the action.
Look online and you can often get rooms for as low as US$50 – a steal for the area.