Is your question focused more on how to use D3DVERTEXELEMENT9 and CreateVertexDeclaration() or is it focused more on how to describe the "half" data types in a vertex declaration?

You need to make an array of D3DVERTEXELEMENT9's, one element for each item in your vertex input structure, plus one extra for the "end" marker.
D3DVERTEXELEMENT9 endMarker = D3DDECL_END();
D3DVERTEXELEMENT9 elements[10];
//todo - fill in elements[0] - elements[8]
elements[9] = endMarker;
IDirect3DVertexDeclaration9* result = 0;
HRESULT hr = device->CreateVertexDeclaration(elements, &result);
//todo - check hr for error/success

For each element, you set the Usage and UsageIndex to match the HLSL semantic, e.g.
//element for "half2 texCoord3 : TEXCOORD2;"
D3DVERTEXELEMENT9 element;
element.Usage = D3DDECLUSAGE_TEXCOORD;
element.UsageIndex = 2;
However, the rest of the settings aren't based on your HLSL code; these settings are based on your model data format (i.e. how your vertex data is laid out):
element.Method = D3DDECLMETHOD_DEFAULT; //pretty much always use default method...
element.Stream = ...;
element.Offset = ...;
element.Type = ...;To fill these out, you'll have to tell us how your vertex data is stored in memory.

One small additional question is when I pass "vertcol"s not the material colors for xoliulshader.fx, would the scene look exactly the same as 3dsmax?

You've been asking lots of questions about this xoliulshader thing, but you've got to assume that we don't know what xoliulshader is... Can you give us some context?

Hmmm, I had a look at the xoliulshader link you post. I think calling it a "viewport" shader is strange, it makes more sense for me to think of it as a scene shader. Anyway it looks like its basically a super shader for 3DS Max/Maya and probably RenderMonkey too. It will require your art assets and your rendering pipeline to work in very specific ways to achieve the same visual effects you see in 3DStudio Max however.

I wasn't able to download it, I didn't want to install WinRAR on my office computer. I don't understand your concern about the lack of a material parameter for the shader. Vertex colors can supply the same behavior and produce better results. You don't need textures to see the effects of either a material color or a vertex color. The material color is going to be the same for all pixels of a primitive, whereas the vertex colors could allow better fading between different colors. However, you will need to supply accurate vertex color data in your vertex stream for that effect. If you want to imitate a material color, simply set the vertex color for every vertex to the same value as the material color.

You need the declaration, either in the form of a VertexDeclaration or an FVF because otherwise the framework cannot map your input stream source to the vertex shader in the FX file correctly.

Yes you should specify the vertcol in your .X file (if that is how you are filling in the vertex buffer). As I mention above, you could just use the same value over and over again to emulate a material color.