For history & culture, São Jorge Castle (Castelo de São Jorge) delivers.
The walls are mostly reconstructed and the 'castle' itself is light on interior, but you don't come for the masonry — you come to walk the battlements over the whole tumbling-red city to the river. The catch: it's pricey for what's there, the climb up is steep, and the view-terrace gets shoulder-to-shoulder at sunset.