One factors into every other natural number. Two is two ones, three is three ones, and so on. There cannot be any higher digits without one's unitary role in their sum. If three did not include three ones, it would not be three. If three were actually not three ones, but three "near-one" digits, then three would not equal three. As such, one is a transcendental for all natural numbers. The nature of all higher natural numbers transcendentally requires the existence of one. Now, does the analytic ubiquity of one in higher numbers negate the real substance of those numbers? Not at all. For three is truly three and not one. And so on.
My point is allegorical. Critical idealism argues that, since our cognitive structure figures into every perception we have, we have no objective perceptions. Our perceptions are always filtered and distorted by the role our own cognitive categories play in perception. We don't see the world as it really is: we see the-world-as-we-see-it. This leads some idealists to deny there is any such thing as the world as it is apart from our perception of it. For the very means by which we perceive the world also (transcendentally) generate the contours of the world as we think it is. The sky is not really blue, we are told; a blue sky is merely an effect in our brains based on the range of light frequency our eyes are evolved to perceive. Dogs see no blue sky since their eyes are different perceptual filters for the world. This line of reasoning, however, strikes me as naive as saying there isn't really an objective thing called 3, since 1 is a transcendental condition for its existence. Three needs one to be three but one cannot make three be three. Likewise, the perceived world needs perceivers to be perceived but the perceivers cannot make the world perceivable in the way it actually exists. Just as one is a necessary but not sufficient transcendental condition for natural numerality, so formally ordered cognition is a necessary but not sufficient factor for there being a perceivable world.