In the early period of the Portuguese rule they were not in the least interested in taking possession of Trincomalee. But after the arrival of the Dutch in the east coast they got alarmed and took control of the two ports of Trincomalee and Baticaloa. In 1622 the Portuguese destroyed the temple as well, Later the Dutch taken the control of the fort.

In 1795 British taken over the fort and named, FORT FEDRICK after the then Commander in chief the Duke of York