The worst "performance hit" you could have is that the JVM has to load the .class file into memory the first time you reference it (wheter its a static field, or you initialize an object of that class, etc) and run any static initializer blocks you have in it.
But that operation is done only once per class and only when you do something with it, JVM won't load classes that you don't touch.
And as Diego said, its all memory accesses, as costly as any other memory access. ie, if you have two objects of different classes, and you access an int field of one object and an int field of the other object, access cost will be probably the same since they're memory accesses (or if you're lucky, the thing is in CPU cache and it will be much faster, it isn't something you can control precisely in Java).
So don't try to cram stuff into a single class because you think it will run faster.