There's actually a lot to look at as you walk around the island. If you're in a car or bus you tend to miss the small things. The lighthouse is at the entrance to the harbor so you can't miss that.
The fort is up on a hill and unless you go there you won't even know it exists. I walked up but didn't go in.