The original Fraunces Tavern wasn't built until 1719, 55 years after the English takeover.
Nothing is left from the Dutch days. Nothing of the wall, nor Fort Amsterdam. Just the original street paths.
Having said that - I should mention that there were Dutch remains found on Governors Island in the 1990's: charred wood posts.
Probably the remains of a wind-powered sawmill built by the Dutch West India Company in 1625-26 because In 1648, the company ordered the then-ruined sawmill dismantled or burned to recover its iron, which was scarce in the colony.