Why do you need the objects coordinates for that? Use selection, and as names, you load the index where you find the point in the array. That way you can directly use the name in the hit record as index into both the vertex and color array to identify the selected point.

For example, for the 42:nd point, push the name 41 (0-based indices, you know), and when you get the hit record back, change the entry with offset 41 in the color array to white.

Thanks Brother Bob I'll try that one out.