Look up 'convex hull'.

This is like puting a rubber band around all points and excluding interior points.

It would return a point / edge sequence like 5,3,4,2 excluding point 1.

Finaly you can triangulate this to a surface simply by connecting all edges with center of mass.

For 2D case this is quite easy to implement e.g. by sorting points by angle.

For 3D i'd use a library like https://code.google.com/archive/p/juliohull/

Could bu used to easily precalculate a low polygon aprroximization of the ship for faster processing.

Although a convex hull would not solve the problem you specifically mentioned. It does a damn good job at collecting all the external points. You can then use a trinagalization algorithm to compute triangles, which you can then create an area from for your buoyancy problem.

When calculating volume of a triangle mesh a good way is to sum tetrahedron volumes given a reference point. This reference point does *not* need to be inside of the volume itself, as long as you sum *signed* volumes. For an explanation I recommend 2013 GDC lecture "Interacting with 3D Geometry" by Stan Melax. He explains signed volume summing quite well.

We can place the reference point on the water plane so that all triangles on the water plane contribute no volume to the summation. This lets you ignore these triangles when summing volume. I also have slides about the zero-volume triangle trick (which I learned from Catto's Game Programming Gems chapter, I think it was Gems 6?), be sure to check out slide 15 in particular: http://www.randygaul.net/wp-content/uploads/2014/02/RigidBodies_WaterSurface.pdf

Although a convex hull would not solve the problem you specifically mentioned. It does a damn good job at collecting all the external points. You can then use a trinagalization algorithm to compute triangles, which you can then create an area from for your buoyancy problem.

Yeah alvaro thats the fact but, i am not quite sure how to find a pressure of a submerged triangle at all,

So how i am supposed to find boundary points?

Not sure what you mean, but i assume you finaly only need to know the volume under the water plane?

An easy, fast enough solution would be this:

Use the convex hull lib i suggested to precompute a hull of maximum 30 vertices, should be accurate enough.

At runtime, build a list of vertices from this hull that are below the water plane.

Test all edges of the hull with the waterplane and add intersecting points to the list to handle points exactly at the water plane.

Use the lib again th create another convex hull and its volume from the list.

I know it's a big waste to compute a hull every frame when most of this data is already known,

but for a single ship it's surely fast enough, and you can continue to develop simulation, see how it works and optimize later.

Note: The convex hull algorithm comes from the Newton Game Dynamics engine, which also handles buoyancy out of the box.