The only numbers we really can use are the natural ones
I am a little confused by this. Are there no circles? Are there no triangles?
I think your assumption that mathematics is about numbers is an incorrect one. A Boolean algebra, for example, has no numbers and you would be hard pressed to argue that it is not mathematics. I can constructs valid algebraic fields using only the abstract notion of an identity, a relation, and an operation, none of which are really required to be any sort of number. I can spend hours doing deductive geometric proofs in which the essence of cardinality only sullies the purity and beauty of the form.
If you're limiting yourself to a formal system of numerical arithmetic you will still find there is going to be an infinite number of linear transformations from
N to
N. In fact, there is an interesting result that states that if you do manage to enumerate all the relations
R:N->N, there are an infinite number more that exist that you can't enumerate within your formal system and have it remain consistent and closed. If someone sells your a calculator that gives you and answer, you have been duped.