The old French governor's residence, Villa Sili (1936), which was for many years almost on the verge of collapse, has been recently renovated and is a fun place to explore. Look out for the horse statues in the garden. There are no set opening hours but the guardian will, as if by magic, appear and open up for you.
The Case à Palabres shares the same compound.