I checked he Text3d demo, debugging it with IDA Pro, in function DrawText it calls ScriptPlace/ScriptShape and ExtTextOut, I guess it uses these function to calculate each pixel of text string. But the real game never uses these function, what function could the game possibly use to do the calculation?
It's unlikely that it uses any D3D function for this. The work will be done on the CPU, so what it will do is something like use the string to calculate a list of positions and texture coords, then lock the vertex buffer, memcpy the list across, then unlock.
Exactly, what API(s) could be used to do the calculation, except the ScriptPlace/ScriptShape/ExtTextOut.
It's likely that no APIs are used to do the calculation. Instead it's just regular arithmetic operators: +, -, *, /, and some array lookups, followed by a memcpy.
For example, one way of drawing text might be to store the characters in a texture with 16 rows and 16 columns. That gives enough for 256 characters, so we're assuming a standard extended ASCII character set, but the principle will be the same for other character sets. The integer equivalent of each character (space = 32, A = 65, B = 66, etc) is used to look up an array which supplies the texture coords to use for that character (this can also be done with math). This and the positions are then copied out to an array of structs that looks something like:
The array is sized large enough for the string being drawn. When we reach the end of the string the vertex buffer is locked, the array is memcpyed in, the vertex buffer is unlocked, a draw call is issued.
In other words, no API is needed for this; it's not an API problem, it's a "do it yourself in C/C++" problem.