I want to allow the user to ... apply a color to all points on the texture map within a specified radius from the selected point ... the triangles on the texture map may be discontiguous.

It sounds like you want a colored shape to appear on the textured mesh, whatever the tex coords for the underlying face vertices (triangles) may be. Is that correct?

I did that years ago, and don't have the code, unfortunately. However, I remember the principle I used to do it. You'll need to determine which faces on the mesh are affected by the shape. { Maybe for each face, determine if one or more vertices are within the specified distance.} Then, for each face, calculate** the barycentric coordinates of several (maybe many) points within or on the edge of the shape projected onto the mesh. Apply those barycentric coords to the texture coordinates for each vertex (rather than the vertex position) to determine the area on the texture to color.

** That is, determine the world coordinates for a point where the shape intersects the triangle. Transform those world coordinates into barycentric coordinates for the triangle. Use those barycentric coordinates to transform the texture coordinates of the vertices - that gives you (just) one set of tex coords for the texture that need to be set to the desired color.

The principle is to transform the portion of the shape in world coordinates which "overlap" the triangle to the texture coordinates defined by the triangle vertices.

EDIT: Don't know if this pix helps. The lines are supposed to indicate transformations between world coordinates and texture coordinates.

[attachment=26519:stenciling.png]

So the problem basically breaks down like this?

As a basic description, that's correct. N.B., I'm assuming you're working in 3D! As a result, the individual steps, of course, can get complex. As mentioned above, I'm having to go by memory and a bit of analysis.

You mentioned that triangles that are adjacent in the world may not have contiguous tex coords on the texture. As mentioned, that means you have to handle each triangle individually (which you apparently realize). And, by the way, the process you're asking about is a form of stenciling - if you want to google for "texture stenciling" and similar, you can learn a bit more about it. So, for the sake of this discussion, I'll call your "shape" a stencil.

1) and 2) depend on how you want to project the brush area. That is, one or more of the triangles may not be normal (perpendicular) to the raycast. Visualize shining a cylinder of light (the light is at a particular position) in a particular direction onto a sloped surface. The projection of the cylinder onto the surface will be an ellipse, not a circle. Whatever your stencil, it may be "stretched" when you project it onto a surface that's not normal to the "light" direction.

If you want the stencil to appear as if it were applied to a flat surface, you can transform the triangle (in world space) such that the face normal is parallel (well, anti-parallel) to the stencil direction. You have to be careful when you do that, as you want to rotate the individual triangles in world space in a manner that keeps them contiguous.

[attachment=26526:flat_stencil.png]

EDIT2: Sorry. I inadvertently swapped b and c in the right-hand part of the pix. I.e., on the right-hand side, b should be c; c should be b.

My hand drawing sucks, but, in something like the above, if your raycast intersects face a, the projection of (for instance) a cylinder onto faces b and c will be stretched. You can rotate faces b and c in world coordinates about the common edges so the faces are normal to the stencil direction. Then, when you project the stencil onto those triangles, it may appear more to your liking.

EDIT: Mathematically, you can rotate the stencil application to the normal for each triangle, but, at the moment, the subtleties of determining the center of that rotation escapes me. I'm still working on my first post of coffee.

A friend brought to my attention that stenciling can be done by first generating a UV atlas - a set of data that maps 3D coordinates to 2D. If you want to investigate that further, there are algorithms and tools for doing that, with the Least Squares Conformal Mapping algorithm apparently being one of the more complex, but perhaps the most flexible. I've not gone into that myself (except for my "lost" code and the method described above) so I can't help you much with the implementation.

You don't mention how you generate your mesh tex coords, but once a uv map has been generated, it can be implemented in a shader to add or overlay a second texture on the first.