SpoilerWhy would he have returned in the first place? He hated it, the entire reason he ran in to Dragon is because he decided to leave the island as a pirate first. Even if he didn't have amnesia, he wouldn't have gone back there for anything short of visiting Dadan.