This is an easy slip-of-the-fingers to make, because when said aloud, “latter” and “ladder” frequently sound pretty similar.
But “ladder” is for the thing with rungs you climb to get to a high place. “Latter” is the much less common word you use in the phrase “the former and the latter” (meaning the one prior and the one second).
I tried but failed to come up with a helpful mnemonic for this: anyone have suggestions?