How to determine witch tile does the mouse point/click onto. I mean is there some calculation? Do i look only at one part of the tile or what.
Here is the type of isometric representaion that i use:

[attachment=8664:Isometric.png]

Do i look only at the upper part of the cube or the lower or what? If so what is the calculation since it is not a rectangle?

Depending upon what environment you're coding in it may be easier or harder. For example, it's almost trivial in Unity, but more challenging if you're coding from scratch.

What you effectively want to do is cast a ray from the camera, through a pixel position, and get the first tile that it hits, correct? Each tile has up to three visible faces, and each face is a simple diamond. Determining if a point is in a diamond is pretty easy.

Determining if the top face is clicked on is particularly easy because you'll notice that the top faces of the grey cubes match perfectly with the grid of the top faces of the green cubes. If you number your grid with the x axis going up and to the right, z axis going up and to the left, and y going straight up, a hit on a top face diamond at (x, y, z) could also be a hit on a cube at (x-1, y+1, z-1), or (x-2, y+2, z-2), etc.

Beyond that, I don't have time to do the maths right now. Good luck!

EDIT: I didn't notice that you said Java. Not sure what the API is like for 3D.

A simplistic way to do this may just be to use your own mouse, start it at grid postion [0][0] and just scroll threw your array with mouse movements, This maybe useless for you though depending on what you need it to do

This article seems like a good starting point for isometric mathematics and your particular problem, and it is not language specific.