Jump to content
  • Advertisement
Sign in to follow this  
Followers 0
  • entries
    17
  • comments
    4
  • views
    32995

About this blog

My journal as a broadcast graphics engineer.

Entries in this blog

 

Practical Example of needing decoupling...

Some people are probably thinking "Matty, your crazy. Coupling
is how programs do things; there is no way around it!" Well...
all I have to say is its time to put on your big girl panties
and grasp the larger picture.

All software projects grow until they are abandoned. The more
flexible the project--the better the chances it can adapt
to new requirements. The better the coupling [eg, as minimal
as possible, and when it exists; use a balance of signature
and type coupling. Avoid logical coupling whenever possible.] the
more flexible the software.

Lets say that you have a killer game. It started off as a small project, but ended
up growing very quickly. Lets imagine the worst case scenario: the project
is in visual studio 2003. There is no seperation between the renderer and the MFC UI:
they are on seperate threads [hopefully], but in the same monolythic app. GL is the renderer,
so there are glXXX calls all over the renderer. There is no shader support, its
all GL FFP.

Lets say the first task you have to do is to add DX9 support. What could we do?
Lets follow the rule, that only simple classes should have external coupling.

Rule 1: NEVER MAKE DIRECT RENDERING API CALLS!
Rule 2: When in doubt, refer to rule 1!

GL is C; however, don't treat it as such. If you are stuck in C then
make your own vtable... but always make a virtual base class.

Make a file called RenderAPIInterface.h:

typedef enum
{
DEPTH_TEST = 0
// etc... all potential renderstates
} RenderAPIEnum;

typedef enum
{
DX9 = 0,
GL_FPP = 1
// etc... all supported renderers [GL+GLSL, GL+CG, DX+HLSL, DX11, etc.]
} RenderersEnum;

class CRenderAPIInterface
{
public:
virtual bool SetState(RenderAPIEnum RS, DWORD Value) {return false;}
// ...
};

class CRendererBuilder
{
static CRenderAPIInterface* CreateRenderer(RenderersEnum DesiredRenderer);
}


Now, this system is supposed to target both DX and GL.
We will need implementations for that.

Make a file DX9APIInterface.h:

#include "RenderAPIInterface.h"

class CDX9APIInterface : CRenderAPIInterface
{
public:
CDX9APIInterface();
~CDX9APIInterface();
virtual bool SetState(RenderAPIEnum RS, DWORD value);
// ...
};


In the code for the DX9 cpp
DX9APIInterface.cpp:

#include "DX9APIInterface.h"
#include "D3D9.h"

LPDIRECT3DDEVICE9 gDevice = NULL; // assuming a single global device, I'll get into who to
// encapsulate this later.


CDX9APIInterface::CDX9APIInterface()
{
// init gDevice
}

CDX9APIInterface::~CDX9APIInterface()
{
// teardown gDevice
}

bool CDX9APIInterface::SetState(RenderAPIEnum Value, DWORD value)
{
switch (Value)
{
case DEPTH_TEST:
{
gDevice->SetRenderState(D3DRS_ZENABLE, value);
break;
}

default:
{
return false;
}
}

return true;
}


Now we need to wrap fixed function GL.
Make a file GLFFPAPIInterface.h:


#include "RenderAPIInterface.h"

class CGLFFPAPIInterface : CRenderAPIInterface
{
public:
CGLFFPAPIInterface();
~CGLFFPAPIInterface();
virtual bool SetState(RenderAPIEnum RS, DWORD value);
// ...
};


Now we need GLFFPAPIInterface.cpp:

#include "GLFFPAPIInterface.h"
#include "g/GL.h"
#include "gl/GLU.h"

CGLFFPAPIInterface::CDX9RenderState()
{
// init GL
}

CGLFFPAPIInterface::~CDX9RenderState()
{
// teardown GL
}

bool CGLFFPAPIInterface::SetState(RenderAPIEnum Value, DWORD value)
{
switch (Value)
{
case DEPTH_TEST:
{
::glEnable(GL_DEPTH_TEST);
break;
}

default:
{
return false;
}
}

return true;
}


The final file we need at this point is CRendererBuilder.cpp

#include "GLFFPAPIInterface.h"
#include "DX9APIInterface.h"

CRenderAPIInterface* CRendererBuilder::CreateRenderer(RenderersEnum DesiredRenderer)
{
switch (DesiredRenderer)
{
case DX9: return (CRenderAPIInterface*)new CDX9RenderState();
case GL_FPP: return (CRenderAPIInterface*)new CGLFFPAPIInterface();
}

return NULL;
}


Okay, there is a lot of typing there... But conceptually it is really simple.
You want to have a renderer that can run in a few modes. In this case DX9 and GL.

Yes, this is a heavy handed method... you will have to copy a lot of
both sets of headers into your own system. Its not the most glamourous work, but is also well worth it.

We are going to use ambigious type coupling to isolate the rendering API.

Stub out all of the set/get/create features in the base class. In the derived classes
map the calls into their api specific calls.

The only places where your application should talk directly DX9/GL are in these 3 CPP's.
If the rest of the system talks to the renderer via its base class -- then you can
drop renderer's in and out to your hearts content. -- you have completely uncoupled
the core of the system from the rendering API!

[NOTE: you WILL need to also have texture and potentially mesh cpp's are bound to the API]

Lets say you want to add new "middleware" but keep the base GL or DX9 support.
Rendering middleware... What do I mean by that? Well, graphics APIs are essentially bunk
today. DX and OpenGL talk to the same physical graphics card hardware. Before hardware T&L
and shaders you could really argue which was "better." Now, all the do is hand up values
across the pcie bus.

Straight DX and GL use fixed function middleware. You upload the values for per-vertex lighting.
The card then does its magic [even though this is simulated by shaders on newer cards...]

Historically, the way I've encapsulated the change to shaders has been to add more
APIInterface's like the ones above. You could add another 2:


class CDX9HLSLAPIInterface : CDX9APIInterface

and

class CGLGLSLAPIInterface : CGLFFPAPIInterface


Now, in each of these new base classes... simply extend your
functionality to call the shaders.

After that, extend your builder to:


CRenderAPIInterface* CRendererBuilder::CreateRenderer(RenderersEnum DesiredRenderer)
{
switch (DesiredRenderer)
{
case DX9: return (CRenderAPIInterface*)new CDX9RenderState();
case GL_FPP: return (CRenderAPIInterface*)new CGLFFPAPIInterface();
case DX9_HLSL:return (CRenderAPIInterface*)new CDX9HLSLAPIInterface();
case DX9_GLSL:return (CRenderAPIInterface*)new CGLGLSLAPIInterface();
}

return NULL;
}


Here are some thought exercises:
* How could we make this better/safer with smart pointers?

* How do smart pointers effect coupling?

This decoupling example uses ambiguous type coupling:
* What would be needed to do signature coupling?
* What what signature coupling get us? What would it cost us?

* How would we recast this in the other modern languages?
[Particularly C# -- what is the immediate danger?]

* Say later we go cross platform. How else does this type of
coupling/encapsulation help?

The answers might surprise you...

More later,
Matty
Mathucub

Mathucub

0
 

Coupling

It is impossible to write any non-trivial software without having to deal with
some variant of coupling. Even simple applications need to be coupled against
standard OS or Framework functionality to achieve many tasks.

Coupling is the action of having two independent entities where; for one to
function the other must be present.

The term entity is purposfully nebulous. An application having a compile time
dependency on an external SDK is a form of coupling. So is one class inheriting
members from a base class. So is making an RPC call to some remote host to
perform an action. So is having an interface change trigger a JIT when the
system is exposed to a new assembly.

Coupling is usually contrasted with cohesion. Low coupling often correlates
with high cohesion, and vice versa. ... Low coupling is often a sign of a well
structured computer system and a good design, and when combined with high
cohesion, supports the general goals of high readability and maintainability.

There are two main families of coupling.
Static Coupling
Dynamic Coupling

These are obviously compile time vs. runtime combinations.
Each family of coupling has corresponding species memebers:
Logical Coupling
Type Coupling
Signature Coupling

With logical coupling, individual classess and components have direct
connections into the inner working of other classes or components. This style
of coupling is notorious for breaking the rules of encapsulation and eventually
bringing development to a halt. With classes relying on the internal state of
each other; you cannot predict where a change will ripple to. Fixing one bug
or adding a feature often introduces another. Most changes, even insignificant
ones, causes a total rebuild and regression test.

Type coupling is the intermediate sophistication of coupling. Rules
of encapsulation are followed and the compiler is used to detect errors.
Operations are wrapped in objects such that the compiler's static typing will
prevent mixing the wrong objects and operations. Much of the features of this
type of coupling come from properly applying OO techniques. Components should
be properly abstracted and direct in purpose. If the objects are not properly
purposed--their interactions can form the same style of rippling bugs that are
due to coupled logic.

Type coupling itself can be divided in two: Ambigious and Unambigious type
coupling. With unambiguous type coupling; one passess and accesses a class
itself. Any changes to that class will cause a ripple. Ambigious access lets
you work with the class without knowing exactly what it is. Ambigious coupling
is most commonly achieved in two ways: First, handing around a reference to a
base class [with virtual methods] instead of the final product. Second, relying
on a variant style container class that envelopes the specific class [or his base...]

These are common when using COM; with the first case being the IUnknown
interface himself. All classes must derive from him, so you can always
downcast to that least common denominator. The second being COM's VARIANT type.
A variable that will envelope most data formats himself.

Signature coupling is considered the most sophisticated form of
coupling. Methods are rarely called directly [using using a SOA, delegate
wrappers, or lamda's.] Most parameters will be passed as combinations of basic
types and enums. Testing becomes easier since if one can guarentee that
the inputs of an entire code segment are the same: the outputs will be good.
One must only test the output of the feeder code.

There are some serious issues with full signature coupling. First, is the
obvious overhead of having to repackage data and marshall calls. Even if in
the same thread, repeatedly copying strings can be time consuming. Second,
the power of full signature coupling is that you do NO binding until
runtime. Without having proper wiring diagrams and documentation, this is
not possible.

The general rule with coupling: only the simplest objects should have couplings.
[Eg. Have a simple object that functions as the base class do the binding and
do little else. From that class you can derive children that make use of
the class's properties.]
Mathucub

Mathucub

0
 

System Design -- Finite State Machines

Hello fellow developers! I'm working a project where I can share info with
the public again so I figured I would restart my developer journal.

I've been tasked with re-architecting a large project. The project grew
'organically' and has evolved into a large tangled mess. At this point it is
impossible to fix anything or add any new functionality; without causing ripples
across the entire system.

More often than not these ripples cause bugs.
Due to the design of the system, unit testing is not possible. The testing
process has decayed into a QA engineer throwing whatever playout combos
that they can think of at the system. Typically just a list of the last bugs
that were marked fixed.

No point in laying blame; I'm going to put on the engineering armor and slay
the bad design dragon.

I'm not going to get into too much detail about the current project
architecture. Lets just leave it at there being state machines at the low level
that control the visual elements. Above is a gateway into the Abyss.

The state machine architecture needs to be propagated through the rest
of the system so that its always clear what each component is doing.

So, What is a finite state machine?

When I was taught about state machines, the example used was that of
a cola vending machine:



In this example, the vending machine charges .55 for a soda... [Not a very
friendly vending machine, It does not provide change, and doesn't give you an
option of what soda you want.]

You can add any number of Quarters, Dimes, and Nickels to it. The goal is to
have the final state reach .55, so that it dispenses your drink and resets.

The possible states for the machine are each .05 cent up to .55. The granularity
of the .05 comes from the fact that it is legal to buy a cola with just Nickels.

As you add money, you can jump from any lower state to a higher one, just trace
the lines associated with the Quarters, Dimes, and Nickels as input.

Example: you could add a Dime to be at state .10; then a Quarter to jump to
0.35; then a Nickel to jump to .40; etc...

The next state only depends on the current state and an input.

Depending on the project, you may have any number of state machines running at
the same time.

The same could apply to the vending machine: if there is a LCD display showing
advertisements then it could be doing its state transitions as you were adding
money. Same to be said if the refrigeration were controlled via state.

In my project; I have a state machine that is controlling commercials, another
that toggles between sponsor logos, network logos, and the clock, another that
controls interruption overlays, and another that controls headers.

All of these systems are designed so that they that only need to know the
minimal state of each other system to do their job.

Using state machines makes postmortem debugging easier as well.
If you get a crash dump, and it has your state variables--then you
can backtrace to figure out what it would have taken to get the system into
such a state.

Okay, back to work.
Next entry will be on the evils of coupling, and how to start fixing it.





Mathucub

Mathucub

1
 

Debugging the Dreaded 1935 During MSI Install

This is one of the most annoying issues when coping with installers.
At least its better than "your boned, good luck finding it;" however,

Error 1935. An error occured during the installation of assembly component
{} HRESULT:

is pretty frustrating.

Luckily, for once, it actually put an error in the event log.

Syntax error in manifest or policy file
"C:\windows\winsxs\installtemp\\Assembly1.dll.manifest on line 2


Anyway, the WinSxS HRESULTs are pretty hard to find. Anything SxS is--if you
do a google search with adding a TON of details: the only hits you are going
to get are "Can I delete winsxs?," "Flight simulator didn't install,"
"Fix 1935 errors now!", etc.

Depending on which version of visual studio you are using: these will not
be defined in a header...

Here are the Fusion/SXS codes that I can find:
Return type ?????????&nbsp Return code?? Description of error
TYPE_E_DLLFUNCTIONNOTFOUND 0x8002802F Function not defined in specified DLL
ERROR_ACCESS_DENIED 0x7FF8FFFB Access is denied
COR_E_MODULE_HASH_CHECK_FAILED 0x80131039 The check of the module's hash failed
FUSION_E_REF_DEF_MISMATCH 0x80131040 The located assembly's manifest definition does not match the assembly reference
FUSION_E_INVALID_PRIVATE_ASM_LOCATION 0x80131041 The private assembly was located outside the app-base directory
FUSION_E_ASM_MODULE_MISSING 0x80131042 A module specified in the manifest was not found
FUSION_E_UNEXPECTED_MODULE_FOUND 0x80131043 Modules which are not in the manifest were streamed in
FUSION_E_PRIVATE_ASM_DISALLOWED 0x80131044 A strongly-named assembly is required
FUSION_E_SIGNATURE_CHECK_FAILED 0x80131045 The check of the signature failed
FUSION_E_DATABASE_ERROR 0x80131046 An unexpected error was encountered in the Assembly Cache database
FUSION_E_INVALID_NAME 0x80131047 The given assembly name or code-base is invalid
FUSION_E_CODE_DOWNLOAD_DISABLED 0x80131048 HTTP download of assemblies has been disabled for this app-domain
FUSION_E_UNINSTALL_DISALLOWED 0x80131049 Uninstall of given assembly is not allowed
FUSION_E_NGEN_DEPENDENCY_NOT_FOUND 0x80131050 One of the native image dependencies cannot be found
FUSION_E_NGEN_INDEX_CORRUPTED 0x80131051 ngen index corrupted
ERROR_SXS_SECTION_NOT_FOUND 0x800736B0 The requested section was not present in the activation context.
ERROR_SXS_CANT_GEN_ACTCTX 0x800736B1 The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log for more detail.
ERROR_SXS_INVALID_ACTCTXDATA_FORMAT 0x800736B2 The application binding data format is invalid.
ERROR_SXS_ASSEMBLY_NOT_FOUND 0x800736B3 The referenced assembly is not installed on your system.
ERROR_SXS_MANIFEST_FORMAT_ERROR 0x800736B4 The manifest file does not begin with the required tag and format information.
ERROR_SXS_MANIFEST_PARSE_ERROR 0x800736B5 The manifest file contains one or more syntax errors.
ERROR_SXS_ACTIVATION_CONTEXT_DISABLED 0x800736B6 The application attempted to activate a disabled activation context.
ERROR_SXS_KEY_NOT_FOUND 0x800736B7 The requested lookup key was not found in any active activation context.
ERROR_SXS_VERSION_CONFLICT 0x800736B8 A component version required by the application conflicts with another component version already active.
ERROR_SXS_WRONG_SECTION_TYPE 0x800736B9 The type requested activation context section does not match the query API used.
ERROR_SXS_THREAD_QUERIES_DISABLED 0x800736BA Lack of system resources has required isolated activation to be disabled for the current thread of execution.
ERROR_SXS_PROCESS_DEFAULT_ALREADY_SET 0x800736BB An attempt to set the process default activation context failed because the process default activation context was already set.
ERROR_SXS_UNKNOWN_ENCODING_GROUP 0x800736BC The encoding group identifier specified is not recognized.
ERROR_SXS_UNKNOWN_ENCODING 0x800736BD The encoding requested is not recognized.
ERROR_SXS_INVALID_XML_NAMESPACE_URI 0x800736BE The manifest contains a reference to an invalid URI.
ERROR_SXS_ROOT_MANIFEST_DEPENDENCY_NOT_INSTALLED 0x800736BF The application manifest contains a reference to a dependent assembly which is not installed
ERROR_SXS_LEAF_MANIFEST_DEPENDENCY_NOT_INSTALLED 0x800736C0 The manifest for an assembly used by the application has a reference to a dependent assembly which is not installed
ERROR_SXS_INVALID_ASSEMBLY_IDENTITY_ATTRIBUTE 0x800736C1 The manifest contains an attribute for the assembly identity which is not valid.
ERROR_SXS_MANIFEST_MISSING_REQUIRED_DEFAULT_NAMESPACE 0x800736C2 The manifest is missing the required default namespace specification on the assembly element.
ERROR_SXS_MANIFEST_INVALID_REQUIRED_DEFAULT_NAMESPACE 0x800736C3 The manifest has a default namespace specified on the assembly element but its value is not "urn:schemas-microsoft-com:asm.v1".
ERROR_SXS_PRIVATE_MANIFEST_CROSS_PATH_WITH_REPARSE_POINT 0x800736C4 The private manifest probed has crossed reparse-point-associated path
ERROR_SXS_DUPLICATE_DLL_NAME 0x800736C5 Two or more components referenced directly or indirectly by the application manifest have files by the same name.
ERROR_SXS_DUPLICATE_WINDOWCLASS_NAME 0x800736C6 Two or more components referenced directly or indirectly by the application manifest have window classes with the same name.
ERROR_SXS_DUPLICATE_CLSID 0x800736C7 Two or more components referenced directly or indirectly by the application manifest have the same COM server CLSIDs.
ERROR_SXS_DUPLICATE_IID 0x800736C8 Two or more components referenced directly or indirectly by the application manifest have proxies for the same COM interface IIDs.
ERROR_SXS_DUPLICATE_TLBID 0x800736C9 Two or more components referenced directly or indirectly by the application manifest have the same COM type library TLBIDs.
ERROR_SXS_DUPLICATE_PROGID 0x800736CA Two or more components referenced directly or indirectly by the application manifest have the same COM ProgIDs.
ERROR_SXS_DUPLICATE_ASSEMBLY_NAME 0x800736CB Two or more components referenced directly or indirectly by the application manifest are different versions of the same component which is not permitted.
ERROR_SXS_FILE_HASH_MISMATCH 0x800736CC A component's file does not match the verification information present in the component manifest.
ERROR_SXS_POLICY_PARSE_ERROR 0x800736CD The policy manifest contains one or more syntax errors.
ERROR_SXS_XML_E_MISSINGQUOTE 0x800736CE Manifest Parse Error : A string literal was expected, but no opening quote character was found.
ERROR_SXS_XML_E_COMMENTSYNTAX 0x800736CF Manifest Parse Error : Incorrect syntax was used in a comment.
ERROR_SXS_XML_E_BADSTARTNAMECHAR 0x800736D0 Manifest Parse Error : A name was started with an invalid character.
ERROR_SXS_XML_E_BADNAMECHAR 0x800736D1 Manifest Parse Error : A name contained an invalid character.
ERROR_SXS_XML_E_BADCHARINSTRING 0x800736D2 Manifest Parse Error : A string literal contained an invalid character.
ERROR_SXS_XML_E_XMLDECLSYNTAX 0x800736D3 Manifest Parse Error : Invalid syntax for an xml declaration.
ERROR_SXS_XML_E_BADCHARDATA 0x800736D4 Manifest Parse Error : An Invalid character was found in text content.
ERROR_SXS_XML_E_MISSINGWHITESPACE 0x800736D5 Manifest Parse Error : Required white space was missing.
ERROR_SXS_XML_E_EXPECTINGTAGEND 0x800736D6 Manifest Parse Error : The character '>' was expected.
ERROR_SXS_XML_E_MISSINGSEMICOLON 0x800736D7 Manifest Parse Error : A semi colon character was expected.
ERROR_SXS_XML_E_UNBALANCEDPAREN 0x800736D8 Manifest Parse Error : Unbalanced parentheses.
ERROR_SXS_XML_E_INTERNALERROR 0x800736D9 Manifest Parse Error : Internal error.
ERROR_SXS_XML_E_UNEXPECTED_WHITESPACE 0x800736DA Manifest Parse Error : Whitespace is not allowed at this location.
ERROR_SXS_XML_E_INCOMPLETE_ENCODING 0x800736DB Manifest Parse Error : End of file reached in invalid state for current encoding.
ERROR_SXS_XML_E_MISSING_PAREN 0x800736DC Manifest Parse Error : Missing parenthesis.
ERROR_SXS_XML_E_EXPECTINGCLOSEQUOTE 0x800736DD Manifest Parse Error : A single or double closing quote character (\' or \") is missing.
ERROR_SXS_XML_E_MULTIPLE_COLONS 0x800736DE Manifest Parse Error : Multiple colons are not allowed in a name.
ERROR_SXS_XML_E_INVALID_DECIMAL 0x800736DF Manifest Parse Error : Invalid character for decimal digit.
ERROR_SXS_XML_E_INVALID_HEXIDECIMAL 0x800736E0 Manifest Parse Error : Invalid character for hexidecimal digit.
ERROR_SXS_XML_E_INVALID_UNICODE 0x800736E1 Manifest Parse Error : Invalid unicode character value for this platform.
ERROR_SXS_XML_E_WHITESPACEORQUESTIONMARK 0x800736E2 Manifest Parse Error : Expecting whitespace or '?'.
ERROR_SXS_XML_E_UNEXPECTEDENDTAG 0x800736E3 Manifest Parse Error : End tag was not expected at this location.
ERROR_SXS_XML_E_UNCLOSEDTAG 0x800736E4 Manifest Parse Error : The following tags were not closed: %1.
ERROR_SXS_XML_E_DUPLICATEATTRIBUTE 0x800736E5 Manifest Parse Error : Duplicate attribute.
ERROR_SXS_XML_E_MULTIPLEROOTS 0x800736E6 Manifest Parse Error : Only one top level element is allowed in an XML document.
ERROR_SXS_XML_E_INVALIDATROOTLEVEL 0x800736E7 Manifest Parse Error : Invalid at the top level of the document.
ERROR_SXS_XML_E_BADXMLDECL 0x800736E8 Manifest Parse Error : Invalid xml declaration.
ERROR_SXS_XML_E_MISSINGROOT 0x800736E9 Manifest Parse Error : XML document must have a top level element.
ERROR_SXS_XML_E_UNEXPECTEDEOF 0x800736EA Manifest Parse Error : Unexpected end of file.
ERROR_SXS_XML_E_BADPEREFINSUBSET 0x800736EB Manifest Parse Error : Parameter entities cannot be used inside markup declarations in an internal subset.
ERROR_SXS_XML_E_UNCLOSEDSTARTTAG 0x800736EC Manifest Parse Error : Element was not closed.
ERROR_SXS_XML_E_UNCLOSEDENDTAG 0x800736ED Manifest Parse Error : End element was missing the character '>'.
ERROR_SXS_XML_E_UNCLOSEDSTRING 0x800736EE Manifest Parse Error : A string literal was not closed.
ERROR_SXS_XML_E_UNCLOSEDCOMMENT 0x800736EF Manifest Parse Error : A comment was not closed.
ERROR_SXS_XML_E_UNCLOSEDDECL 0x800736F0 Manifest Parse Error : A declaration was not closed.
ERROR_SXS_XML_E_UNCLOSEDCDATA 0x800736F1 Manifest Parse Error : A CDATA section was not closed.
ERROR_SXS_XML_E_RESERVEDNAMESPACE 0x800736F2 Manifest Parse Error : The namespace prefix is not allowed to start with the reserved string "xml".
ERROR_SXS_XML_E_INVALIDENCODING 0x800736F3 Manifest Parse Error : System does not support the specified encoding.
ERROR_SXS_XML_E_INVALIDSWITCH 0x800736F4 Manifest Parse Error : Switch from current encoding to specified encoding not supported.
ERROR_SXS_XML_E_BADXMLCASE 0x800736F5 Manifest Parse Error : The name 'xml' is reserved and must be lower case.
ERROR_SXS_XML_E_INVALID_STANDALONE 0x800736F6 Manifest Parse Error : The standalone attribute must have the value 'yes' or 'no'.
ERROR_SXS_XML_E_UNEXPECTED_STANDALONE 0x800736F7 Manifest Parse Error : The standalone attribute cannot be used in external entities.
ERROR_SXS_XML_E_INVALID_VERSION 0x800736F8 Manifest Parse Error : Invalid version number.
ERROR_SXS_XML_E_MISSINGEQUALS 0x800736F9 Manifest Parse Error : Missing equals sign between attribute and attribute value.
ERROR_SXS_PROTECTION_RECOVERY_FAILED 0x800736FA Assembly Protection Error : Unable to recover the specified assembly.
ERROR_SXS_PROTECTION_PUBLIC_KEY_TOO_SHORT 0x800736FB Assembly Protection Error : The public key for an assembly was too short to be allowed.
ERROR_SXS_PROTECTION_CATALOG_NOT_VALID 0x800736FC Assembly Protection Error : The catalog for an assembly is not valid, or does not match the assembly's manifest.
ERROR_SXS_UNTRANSLATABLE_HRESULT 0x800736FD An HRESULT could not be translated to a corresponding Win32 error code.
ERROR_SXS_PROTECTION_CATALOG_FILE_MISSING 0x800736FE Assembly Protection Error : The catalog for an assembly is missing.
ERROR_SXS_MISSING_ASSEMBLY_IDENTITY_ATTRIBUTE 0x800736FF The supplied assembly identity is missing one or more attributes which must be present in this context.
ERROR_SXS_INVALID_ASSEMBLY_IDENTITY_ATTRIBUTE_NAME 0x80073700 The supplied assembly identity has one or more attribute names that contain characters not permitted in XML names.
ERROR_SXS_ASSEMBLY_MISSING 0x80073701 The referenced assembly could not be found.
ERROR_SXS_CORRUPT_ACTIVATION_STACK 0x80073702 The activation context activation stack for the running thread of execution is corrupt.
ERROR_SXS_CORRUPTION 0x80073703 The application isolation metadata for this process or thread has become corrupt.
ERROR_SXS_EARLY_DEACTIVATION 0x80073704 The activation context being deactivated is not the most recently activated one.
ERROR_SXS_INVALID_DEACTIVATION 0x80073705 The activation context being deactivated is not active for the current thread of execution.
ERROR_SXS_MULTIPLE_DEACTIVATION 0x80073706 The activation context being deactivated has already been deactivated.
ERROR_SXS_PROCESS_TERMINATION_REQUESTED 0x80073707 A component used by the isolation facility has requested to terminate the process.
ERROR_SXS_RELEASE_ACTIVATION_CONTEXT 0x80073708 A kernel mode component is releasing a reference on an activation context.
ERROR_SXS_SYSTEM_DEFAULT_ACTIVATION_CONTEXT_EMPTY 0x80073709 The activation context of system default assembly could not be generated.
ERROR_SXS_INVALID_IDENTITY_ATTRIBUTE_VALUE 0x8007370A The value of an attribute in an identity is not within the legal range.
ERROR_SXS_INVALID_IDENTITY_ATTRIBUTE_NAME 0x8007370B The name of an attribute in an identity is not within the legal range.
ERROR_SXS_IDENTITY_DUPLICATE_ATTRIBUTE 0x8007370C An identity contains two definitions for the same attribute.
ERROR_SXS_IDENTITY_PARSE_ERROR 0x8007370D The identity string is malformed. This may be due to a trailing comma, more than two unnamed attributes, missing attribute name or missing attribute value.
ERROR_MALFORMED_SUBSTITUTION_STRING 0x8007370E A string containing localized substitutable content was malformed. Either a dollar sign ($) was follwed by something other than a left parenthesis or another dollar sign or an substitution's right parenthesis was not found.
ERROR_SXS_INCORRECT_PUBLIC_KEY_TOKEN 0x8007370F The public key token does not correspond to the public key specified.
ERROR_UNMAPPED_SUBSTITUTION_STRING 0x80073710 A substitution string had no mapping.
ERROR_SXS_ASSEMBLY_NOT_LOCKED 0x80073711 The component must be locked before making the request.
ERROR_SXS_COMPONENT_STORE_CORRUPT 0x80073712 The component store has been corrupted.
ERROR_ADVANCED_INSTALLER_FAILED 0x80073713 An advanced installer failed during setup or servicing.
ERROR_XML_ENCODING_MISMATCH 0x80073714 The character encoding in the XML declaration did not match the encoding used in the document.
ERROR_SXS_MANIFEST_IDENTITY_SAME_BUT_CONTENTS_DIFFERENT 0x80073715 The identities of the manifests are identical but their contents are different.
ERROR_SXS_IDENTITIES_DIFFERENT 0x80073716 The component identities are different.
ERROR_SXS_ASSEMBLY_IS_NOT_A_DEPLOYMENT 0x80073717 The assembly is not a deployment.
ERROR_SXS_FILE_NOT_PART_OF_ASSEMBLY 0x80073718 The file is not a part of the assembly.
ERROR_SXS_MANIFEST_TOO_BIG 0x80073719 The size of the manifest exceeds the maximum allowed.
ERROR_SXS_SETTING_NOT_REGISTERED 0x8007371A The setting is not registered.
Mathucub

Mathucub

0
 

Installing To \Windows\WinSxS

If you've been following my manifest journal entries, then at this point you
will have everything you need to use winsxs/fusion--save one major problem: to
place files into \windows\winsxs you have to be running from a .msi installer.

The the days of xcopy and regsvr32 are officially over.

I've found three ways to do a winsxs intall: one relies on InstallShield
totally, the other mixes both InstallShield and WIX, and the last uses
just WIX.

[If you are starting a new installer, just use WIX by itself...]

What you need to understand that we are converting your library into an
assembly. There are .net assemblies that go in the global assemblies cache, and
win32 assemblies that go in the native assemblies cache.


Method 1: Just InstallShield:
Step 1:
Open your project [or create a new one...] in any recent version of
Installshield.

Step 2:
Use the file/folder assistant to add your .dll
[A side effect of this is that installshield will create a component for you.
You can make a component and populate it yourself if your an advanced user.]

Step 3:
You component will already have your dll, now you need to add your security
cabinet and manifest.

Step 4:
In the Advance Settings of the component, right-click on the Assembly node and
select "New .Win32 Assembly".

Step 5:
After selecting the Win32 Assembly node, examine the properties. You'll see that
your .MANIFEST file has automatically been selected as the Manifest file.

Step 6:
Set the Name, Type, and Version properties. These values must be copied exactly
as they appear in the assembly manifest. If your assembly contains other
properties, you may add them by clicking on the last row of the property list
for your assembly. [eg, publicKeyToken, processorArchitecture, Language, etc.]

This is all fine and dandy if you use InstallShield manually.
Every time you cut a release and you change the assembly's version you are
going to have to come back into this prompt and enter the new version
information.

If you use an automated build process with InstallShield's COM Automation
Interface: this won't work.

Face it, InstallShield sucks these days. For the cost, other than adding win7
support--the new version doesn't seem to be worth it. The vast majority of what
it does is simply a wrapper on top of the existing windows installer api. It
does add "InstallScript"--but its so easy to write c# [or even c++] custom
actions that do the same thing. Your paying for the GUI, and the trialware
extensions if you use them.

WIX is free, well documented, and quite elegant.

I've got too much stuff in my InstallShield projects to move over
to WIX at the moment so I'm stuck at least partially with InstallShield.

[If you need to know more about windows installer/msi's: I strongly recommend
the book "The definitive guide to Windows Installer" by "Phil Wilson." With
this book and WIX [Windows Installer XML] you can do anything that you can
with InstallShield without the overhead.]

The issue is InstallShield's COM/Automation interface does not provide a
mechanism to update the values of the win32 assembly [or anything off the
advanced tab.] The nussiance is that they don't even NEED this information--the
windows installer architecture will extract it manually.

The the solution is to use WIX to make a merge module: and reference
the merge module in InstallShield!

Method 2: InstallShield + WIX
Step 1:
Go through your installer and remove all your win32 assemblies.

Step 2:
Create a new WIX merge module project.

Step 3:
Populate your wxs with data, locate this:





After that add:


Source="\Assembly1\Release\Assembly1.dll" KeyPath="yes" Vital="yes" />
Source="\Assembly1\Release\Assembly1.dll.manifest" Vital="yes" />
Source="\Assembly1\Release\Assembly1.dll.cat" Vital="yes" />



Add one of these for each of your assemblies. The guid is in registry format
and is used to identify the component for install/uninstall, so use visual
studio to generate a new one for each of your compoenents.

When you compile your XML--it will produce a merge module that you can copy
to C:\Program Files\Common Files\Merge Modules and it will be available
to InstallShield.

NOTE: You can use this same syntax if you make a pure WIX installer .msi
to install into \windows\winsxs.

You should be good to go!

NOTE: Remember, on Vista/7 or higher: to install to \windows\winsxs
your assembly must be signed with a 2048bit cert. Otherwise it will fail
with a cryptical HRESULT. Look at my prior entry about how to obtain
a proper cert from Verisign.
Mathucub

Mathucub

0
 

strtok not threadsafe on windows

Anyone, like me, who started off in the land of "C" or before
stl really got standardized probably uses, or has legacy code that has strtok
for tokenization.

On Unix/Mac platforms--strtok has been replaced by strsep.
To me, this is more for convenience: strsep is smart enough to know
to skip double delimiters.

Eg, strtok("--Hi!-Dash-Seperated", "-") = failure, 2 delimeters back to back.
strsep("--Hi!-Dash-Seperated", "-") = "Hi!"

Thats all great and wonderful, except if you are developing on windows,
strsep is not available in the msvc runtime.

The ANSI definition for strtok is a bit vague as it does not reference
what it ought do when it comes to threads and processes:


char * strtok ( char * str, const char * delimiters );
Split string into tokens

A sequence of calls to this function split str into tokens, which are sequences
of contiguous characters separated by any of the characters that are part of
delimiters.

On a first call, the function expects a C string as argument for str, whose
first character is used as the starting location to scan for tokens. In
subsequent calls, the function expects a null pointer and uses the position
right after the end of last token as the new starting location for scanning.

To determine the beginning and the end of a token, the function first scans
from the starting location for the first character not contained in delimiters
(which becomes the beginning of the token). And then scans starting from this
beginning of the token for the first character contained in delimiters, which
becomes the end of the token.

This end of the token is automatically replaced by a null-character by the
function, and the beginning of the token is returned by the function.

Once the terminating null character of str has been found in a call to strtok,
all subsequent calls to this function with a null pointer as the first
argument return a null pointer.


Seems okay, right? What does MSDN say about threads and strtok?
Almost all of the above plus this little note:


Note:
Each function uses a thread-local static variable for parsing the string into
tokens. Therefore, multiple threads can simultaneously call these functions
without undesirable effects. However, within a single thread, interleaving
calls to one of these functions is highly likely to produce data corruption and
inaccurate results. When parsing different strings, finish parsing one string
before starting to parse the next. Also, be aware of the potential for danger
when calling one of these functions from within a loop where another function
is called. If the other function ends up using one of these functions, an
interleaved sequence of calls will result, triggering data corruption.


Well, that looks good as well. Looks like I can use it and not worry about
thread interactions.

But low and behold, what does the C Runtime say?


Using the statically linked CRT implies that any state information saved by the
C runtime library will be local to that instance of the CRT. For example, if
you use strtok, _strtok_l, wcstok, _wcstok_l, _mbstok, _mbstok_l when using a
statically linked CRT, the position of the strtok parser is unrelated to the
strtok state used in code in the same process (but in a different DLL or EXE)
that is linked to another instance of the static CRT. In contrast, the
dynamically linked CRT shares state for all code within a process that is
dynamically linked to the CRT. This concern does not apply if you use the new
more secure versions of these functions; for example, strtok_s does not have
this problem.

Wow, so its only thread safe if you static link to the c runtime.
To make matters worse: If you create your COM connections in proc, that is
what they are--in process. So you and all your threads + all the threads
from any libraries you are using that are dynamically linked to the C
runtime have the chance of a collision.

Also--if you throw the /clr switch to use C++/CLI code: you are required
to dynamic link. The static option is mutually exclusive, so you can't
do a quick fix and just switch to static.

Looks like the only solution on windows is to switch to strtok_s--luckily
it works almost the same way as strtok. Almost... it still takes some tweaking
to get the exact functionality.

Good luck, and beware. If you are multi-threaded then this code
has the potential to throw an exception, and since there is no handler:
cause a crash.


...
lFileLength = File->GetLength();

Data = new TCHAR[lFileLength+1];
File->Read(Data, lFileLength);

Data[lFileLength] = NULL;

File->Close();

char *buffer;

// start our tokenization
buffer = strtok(Data, "\n\r");
...


Literally, after this call went on the stack and started to execute:
We had a background thread strtok which changed the value of
its static internal state pointer. When this went to execute, it was at
the end of the other call's buffer and tried to read past it.

If its not at the end of your address space, then the call could simply fail,
returning NULL, or get a garbled state where you are now tokenizing the wrong
buffer.

If you are at the end of the buffer, and the next address is not owned
by you: it will throw an exception.
Mathucub

Mathucub

1
 

Setting up Subversion on OSX

Revision control software is one of the most important tools a developer will ever use.

If you on windows and developing alone, microsoft source safe (VSS) will often be all you need to use.
VSS uses the SMB protocol for transferring files and as such, is not suitable for access across networks.
There are plugins, such as source anywhere and source offsite, that push VSS traffic through a valid
protocol. These packages are nice, but not cheap.

Team foundation server could also do the job, but it more power than I need.

I need to use dedicated machine for my repository. Only spare I have
at the moment is a G4 Mac Mini--so running source safe or tfs under windows is obviously not going to happen.
As long as only a few developers need simultaneous access, hosting a repository doesn't take much horse
power, so almost any old machine will work.


Step 1: Get Subversion. I recommend using MacPorts. After MacPorts is installed, open a terminal window
and type:

$ sudo port install subversion


This will download and compile the current version of Subversion.

Step 2: Make the file structure for your repository.

mkdir -p /Library/Subversion/Repository


Step 3: Make your repository.

cd /Library/Subversion/Repository
svnadmin create myRepo


Step 4: Change repository settings.

cd myRepo/conf
vi svnserve.conf
vi passwd


When you edit 'svnserve.conf', make the contents of the file look like this:

[general]
anon-access = none
auth-access = write
password-db = passwd
realm = myRepo on my Dedicated Server


This will only allow registered users to have read/write access.

Add your individual users to your passwd file.

[users]
joe =
matty =


Step 5: Update your acl's

sudo chown -R root:admin /Library/Subversion/Repository
sudo chmod -R ug+rwX,o= /Library/Subversion/Repository


Step 6: Turn on the server daemon

svnserve -d -r /Library/Subversion/Repository


Now you are ready to do your initial import.
Point your client to IP and provide one of the username/password combinations from your
passwrds file and you are set.
Mathucub

Mathucub

0
 

Dealing with a side effect of sxs.

I've you've been following my posts at this point you are almost
ready to install as many versions of your application on a system as you want.

Ready for another speed bump?
WinSxS is really only appropriate for compiled code:
You will have to change your distribution to place files in a versioned
location and you will have to change the application to read and write
registry keys from a versioned hive path.

In out case we distribute precompiled and encrypted shaders. The interfaces
for these can, and has, changed frequently. There is no guarantee that shaders
from version 1.0 will work with 2.0. We decided to move the shaders to program
files.

If you are running a 32bit app on a 64bit windows: program files will instead
be \program files (x86). Installing to a spanish language machine?--It will be
\Archivos de Programas. It is never safe to assume that it will just be
"C:\Program Files."

To make it worse, if you are using InstallShield and letting the user install
custom locations you can't do any inferring about where the app is installed.

You will have to place the install directory in the registry.
Your hive/registry key needs to look something like this:
Computer\HKEY_CURRENT_USER\Software\MyCompany\MyProduct 1.0\...

You're going to need a MSI custom Action. WiX is really awesome
and helps you make them for C#. My installer is an older c++ dll,
so you will have a to cross-reference for the new call [or PInvoke
into msi.dll]

You need link against msi.lib and have a function like:

UINT __stdcall MSIMyAppInstall(MSIHANDLE hInstall)


Inside your function, you can call "MsiGetProperty" with your handle
and the provided hInstall. It lets you query your MSI properties--
in this case we want the last token of our "INSTALLDIR":
"\Program Files\MyCompany\MyProduct 1.2.3.4"

Write your "INSTALLDIR" into that registry key and you app(s)
will be able to find their support files no matter where the user
(or os) decided to hide them.

In a future post I'll show how to use the InstallShield automation
interface to automatically update your target, versioned, "[INSTALLDIR]".
Mathucub

Mathucub

0
 

Starting to pull the understanding of Manifests together.

If you are writing an application in c++ application in visual studio you will
typically be linking against the "microsoft c runtime."

If you look in your windows\system32 folder you will see a few of these.

...
Directory of C:\Windows\System32

07/13/2009 09:15 PM 149,019 crtdll.dll
07/13/2009 09:15 PM 690,688 msvcrt.dll
07/13/2009 09:15 PM 253,952 msvcrt20.dll
07/13/2009 09:07 PM 60,928 msvcrt40.dll
...


This looks innocent enough. All of these files are old though.
Where are the new ones? Why did the move?

The where is easy: \windows\winsxs

...
Directory of C:\Windows\winsxs\x86_microsoft-windows-msvcrt_31bf3856ad364e35_6.
1.7600.16385_none_d12b8c440039b31e

07/13/2009 09:15 PM 690,688 msvcrt.dll
...


Conceptually the why is necessary: one of the major vulnerabilities
of earlier versions of windows was "dll hell." If a given application
installs a library to \windows\system32 and another overwrites it
there is now a knot where two different apps collide in their library
dependencies.

The old trick was to try to update to the newest version of the dll
and hope that it worked for both. Unfortunately, that was the best
idea in an impossible situation. If an interface changed between
two library versions then you are simply boned.

MS Project Fusion to the rescue! Windows XPSP2 introduced manifests
the ActCtx API and the \windows\winsxs directory. Now, applications should never
install anything dlls to \windows\system32. Everything should be versioned and
go in \windows\winsxs.

Winsxs is a like a database with multiple keys. For an assembly
to be installed to winsxs it must meet the following requirements:

1) The assembly must have an a "SxS Manifest"
a) The manifest must have an "Assembly Identity"
*) The identity must have a name.
*) The identity must have a version.
*) The identity must have a type.
*) The identity must have a publicKeyToken.
*) The identity must have a processorArchitecture
*) The identity may have a language [defaults to en]
*) The identity may have a culture [defualts to neutral]

b) The manifest must have entries for each file in the assembly
*) The must have a hash
*) The must have a for every COM class
*) The must have a for every COM interface.

2) The assembly must have a security cabinet.
3) The assembly must be installed into WinSxS with a MSI installer.
4) All exe's, dll's, and the security cabinet must be signed with
a trusted 2048bit certificate.
5) The assembly may have an optional policy to route versions.


Things to note here: Authenticode certificates are 1024bit only and will not
work. You will need to extract your public key token from your code signing
certificate. To make a SxSManifest from C++ you need to feed mt.exe your tlb
and rgs files. To make a SxSManifest from an ocx or a dll you don't have the
source too: you will need to use "regsvr42" to probe the interfaces. I've gone
over how to do all these in my last two weeks entries,
review them for more information.


As difficult as it is to make a WinSxS install: the "Assembly Identity" shows
the true power of it. I've already been spouting about Reg Free COM and
installing multiple simultaneous versions. But look, we also can target 32bit or
64bit by just changing the manifest entry. We can have 32bit and 64bit installed
on the same machine at the same time with the same GUIDS! The client application
just needs a different dependency manifest to select one versus another.
I've not had to use them, but these same manifest tricks allow you to install
multiple language and cultures.


Implications on dependencies:
Manifests go hand in hand with static imports. If you are a windows developer
you have probably used depends.exe. If you have used it recently, you may
have noticed that there are yellow bangs beside some of your import dlls.



Look at that! It claims that it can't find the C runtime! The reason for this
should be obvious: our application uses a manifest to find the C runtime.
Manifests are scripts to program the ActCtx API -- which only exists at
runtime. Depends literally has no way of knowing how to find that dll
since you can edit your manifest to target any version you want.

Like I said a long time ago, If you are using visual stdudio 2005 or higher
you have been using the ActCtx API all along without knowing it. In you look
in your build directory you will notice that the linker automatically produced
a file called MyApp.intermediate.manifest. Depending on what standard libraries
you linked to, it will look something like this:































[With the default compiler settings with file will get embedded as an RT_MAINFEST
at CREATEPROCESS_MANIFEST_RESOURCE_ID for an exe or ISOLATIONAWARE_MANIFEST_RESOURCE_ID for a dll.]

That last entry has our C runtime link!
Linking to that lib causes a static import to exist in our application but
it relays on the ActCtx at runtime to do the mapping!

For our custom dll's we need to be able to advertise entries like that so
that client applications can find us in winsxs. Unfortunately, there is no
tool in visual studio 2005 to do so.

In an earlier post, I had you start moving your winsxs build process
to the pre and post build steps. You need a batch file that is something like
this:

MakeDLLIntoAssembly [RTMBroker Directory] [Component Directory] [Component name] [Component Filename] [Component TLB] [Component Version] [Component PublicKey] [CertName]"


Lets look at my script:

@echo off
rem begin MakeDLLIntoAssembly.bat

if "%1" == "" goto error
cd %2
attrib -r %4

echo Preparing Files for %3 RTM Distribution.

rem Build the required Assembly Identity, and extract Reg Free COM info From the TLB and RGS
%1\bin\mt /nologo -rgs:RTM.rgs -tlb:%5 -dll:%4 -identity:"MyCompanyName.%3, processorArchitecture=x86, version=%6, type=win32, publicKeyToken=%7" -out:%4.sxs.manifest

rem The vs2005 linker will have produced the intermediate manfiest, merge it with the one we created to make a complete one [gives us c runtime, etc]
%1\bin\mt /nologo -manifest %4.sxs.manifest %4.intermediate.manifest -out:%4.manifest
%1\bin\mt /nologo -manifest %4.manifest -outputresource:%4;2

rem Sign our output with the 2048 bit cert
%1\bin\signcode -spc %1\Signing\ssl-cert.crt -v %1\Signing\ssl-cert.pvk -t http://timestamp.verisign.com/scripts/timstamp.dll -n "Company Name" -i "http://" %4

rem Now that we signed it, the file hash will have changed. Re-hash for security to validate.
%1\bin\mt /nologo -manifest %4.manifest -hashupdate -out:%4.manifest

rem Make the security cabinet
%1\bin\mt /nologo -manifest %4.manifest -makecdfs
%1\bin\makecat %4.manifest.cdf

rem Sign the cabinet with the 2048bit cert
%1\bin\signcode -spc %1\Signing\ssl-cert.crt -v %1\Signing\ssl-cert.pvk -t http://timestamp.verisign.com/scripts/timstamp.dll -n "Company Name" -i "http://" %4 %4.cat

rem intermediate file.
del %4.sxs.manifest

rem applications have to be able to reference us as a dependency
rem mt sadly lacks this option directly, so we have to fake it
echo Making Dependeny Manifest
echo ^ > %4.dependentassembly.manifest
echo ^ >> %4.dependentassembly.manifest
echo ^ >> %4.dependentassembly.manifest
echo ^ >> %4.dependentassembly.manifest
echo ^ >> %4.dependentassembly.manifest
echo ^^> >> %4.dependentassembly.manifest
echo ^^> >> %4.dependentassembly.manifest
echo ^^> >> %4.dependentassembly.manifest

:error
echo Usage MakeDLLIntoAssembly [RTMBroker Directory] [Component Directory] [Component name] [Component Filename] [Component TLB] [Component Version] [Component PublicKey] [CertName]"

:end


If you read the comments in the code: it will give you all the required pieces for a sxs install. I omitted the code for embedding the interop dll's. I'll
come back to that in a future entry.

One of the important tricks here is the "dependency manifest." Lets look
at the output of what it makes:











Its a just a finger print...
Now to use it when you build your application or game, the linker will give you
the standard intermediate like the one above.

If you add:

mt -manifest mygame.intermediate.manifest MyRenderingEngine.dependency.manifest (-out:MyGame.exe.manifest or -outputresource:mygame.exe;2)"


To your post build step, the manifest tool with merge the two manifest snippets
into one and give you a finalized manifest that looks like:



































Now the ActCtx API knows how to find us if we are in winsxs!

If you have a series of libraries: MyRenderer.dll, MyAIEngine.dll,
MyNetworkEngine.dll, etc. You will just need to loop over the mt call
and keep concatenating them together. [The manifest tool only accepts two
manifests at a time, you will have to use an intermediate file.]

After these steps you may still have to tweak your manifest a little more.
If you are running on vista/7 and you need the application to run as
administrator you have to add:



We still have to go over a few topics to finalize this discussion: "local"
assembly manifests, using an MSI to install into WinSXS, and finalizing our
making our interop dll's use reg free com.
Mathucub

Mathucub

0
 

An adventure debugging a winsxs error.

[Bear with my, I'm journaling this as I debug... we will find the solution
together, since I won't have time to do a post edit on this entry.]

As powerful as WinSxS is, its not without its pitfalls. After adding a new
component to our install, I was hit with the application failing to start,
and this in the event log on XP:


Syntax error in manifest or policy file "C:\WINDOWS\WinSxS\Manifests\x86_MyCompany.MyLibrary_d032e2f1f5e7d70c_7.8.9.2_x-ww_f2eac902.Manifest" on line 2.


Syntax error? I haven't seen one of these since when regsvr42 made a bad file.
It had duplicate IID's. Well, mt.exe can validate syntax, lets check him:


C:\WINDOWS\WinSxS\Manifests>mt -manifest x86_MyCompany.MyLibrary_d032e2f1f5e7d70c
_7.8.9.2_x-ww_f2eac902.Manifest -validate_manifest
Microsoft (R) Manifest Tool version 5.2.3790.2075
Copyright (c) Microsoft Corporation 2005.
All rights reserved.
Parsing of manifest MyCompany.MyLib_d032e2f1f5e7d70c_7.8.9.2_x-ww_f2eac90
2.Manifest successful.


It validated... Well, first: its impossible to debug WinSxS under XP.
They added tools under vista/7 and added event log info for untangling these kinds of problems. Note: mt.exe only validates that the manifest is properly
formatted xml--not the contents.

I next ran my installer on Vista/7 so we can use sxstrace.


C:\>sxstrace
WinSxs Tracing Utility.
Usage: SxsTrace [Options]
Options:
Trace -logfile:FileName [-nostop]
Enabling tracing for sxs.
Tracing log is saved to FileName.
If -nostop is specified, will not prompt to stop tracing.
Parse -logfile:FileName -outfile:ParsedFile [-filter:AppName]
Translate the raw trace file into a human readable format and save the re
sult to ParsedFile.
Use -filter option to filter the output.
Stoptrace
Stop the trace if it is not stopped before.
Example: SxsTrace Trace -logfile:SxsTrace.etl
SxsTrace Parse -logfile:SxsTrace.etl -outfile:SxsTrace.txt


Fire up SxSTrace in Trace mode, and then start your application.
After it crashes, stop the trace and parse it.


INFO: Parsing Manifest File C:\Windows\WinSxS\manifests\x86_mycompany.mylibrary_d032e2f1f5e7d70c_7.8.9.2_none_f4218a7198e4129a.manifest.
INFO: Manifest Definition Identity is MyCompany.MyLibrary,processorArchitecture="x86",publicKeyToken="d032e2f1f5e7d70c",type="win32",version="7.8.9.2".
INFO: Reference: Microsoft.VC80.CRT,processorArchitecture="x86",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="8.0.50727.762"
INFO: Reference: Microsoft.VC80.MFC,processorArchitecture="x86",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="8.0.50727.762"
INFO: Reference: MyCompany.SomeOtherLib,processorArchitecture="x86",publicKeyToken="d032e2f1f5e7d70c",type="win32",version="7.8.9.2"
ERROR: Activation Context generation failed.
End Activation Context Generation.


ActCtx Generation defiantly failed with MyLib. Looks like it failed at SomeOtherLib though... (?)

Next (back on XP), load the MyLibrary.dll.manifest [in
\windows\winsxs\manifests] into notepad++, change the language to XML and
pretty print it with line breaks.

This time the syntax error came on line 73 of the manifest. The MSI
autorepair will restore the manifest, and require a reboot... at least
we got another data point.

Line 73:




Nothing looks too odd about that. The iid is unique in the file. I wonder
if it is not unique in the sum of all manifests?

Lets look at the other manifest that failed for that iid:




Ahhh... Look: two iids that are the same but in different manifests.
No wonder it failed. I'm surprised it even worked in COM/regsvr32.

I've changed the offending iid. Still not running. Lets redo the line breaking of the manifest to see if the line number of the error changed.

Ahhh... same problem different interface.

Manifest 1:




Manifest2:




This children is why you never copy/paste IDL without changing the interface GUIDs!
Mathucub

Mathucub

0
 

Lets talk about COM, InterOp, and Manifests

Manifests are loaded two ways: by the OS loader, and manually. They can
either be embedded or the can be on the file system. For MFC Dll's MS
decided that it would manually be managed by the MFC state source. (See my
entry from last week if you missed it.)

It can be argued MFC and COM are legacy, but they aren't going away any time
soon. Back when .NET was just a baby and COM was the ruler MS had to define
a way for the two problem domains to talk to each other so that .NET could
be adopted.

Just as .NET has revolutionized the world of coding by having multiple
languages talk to a common runtime; thus eliminating the short comings of
specific languages and providing a common type system to all assemblies. Back
in the day, COM did the same thing by allowing types to be shared by dll's and applications.

It didn't fix the issue that some languages simply lacked functionality
(such as VB not having true OO and having to reach backwards over your shoulder
to access win32 calls) it did make it so that as long as your compiler
produced COM compatible types, you could mix and match any language at runtime
by communicating through the COM interfaces of either IUnknown or IDispatch.

[When using C++ COM we typically write in an intermediate language called IDL.
COM is so old that when IDL is compiled: it will give you a header that
contains C++ classes and raw old school c with manual implementation of
vtables for you interfaces.]

When a COM project is compiled it will produce something called a type library.
These are "tlb" files that contain the prototypes for all interfaces, structs,
enumerations, etc that the COM library supports. Each of these items should
have an associated globally unique identifier (GUID). GUID's are 128bit numbers
that are most likely unique in the universe. When a COM library is
registered these GUIDs are placed into the registry so that other COM
libraries or hosting containers can find them.

The whole point of Reg Free COM get around this by placing all the registry
information into a manifest file. Otherwise when version 2.0 of GUID xyz is
placed in the registry, it would overwrite version 1.0 of GUID xyz -- since
they have the same unique fingerprint.

If each time you release a new version of your COM library you changed your
GUIDs then there would be no competition and you could install multiple
versions; however, when you instantiate a COM interface you call
CoCreateInstance(GUID, ...). If you change your GUIDs with every release, old
client code will not be able to talk to your new versions.

It can be said that ActiveX controls are COM controls that have a UI element.
(I'm glossing over a lot by saying that but it makes this discussion stay
on scope.) Most people will have encountered ActiveX controls as IE plugins.
Windows UI common controls and well as UI controls for visual basic 6.0 are in
this same family of ActiveX conrols. [Although they are not necessarily
ActiveX, they work the exact same way.] These controls are no good on their own
and must be hosted in an application. What good would a button control be if it
were not on the UI to click?

Taking all this into account, for .NET to work with COM it must:
1) Understand COM variable types
2) Instance and call COM Interfaces
3) Be able to host an ActiveX control

Note: for case 1 the InterOp layer must be convert types. Eg, COM uses
a type of string called a BASIC String (BSTR) that is like a pascal string
that is allocated by ::SysAllocString. .NET uses System::String, InterOp
will do something called "Marshalling" to make one work with the other.
In COM there is a catch all type called a VARIANT that will hold strings,
numbers, interfaces, almost anything. Marshaling a VARIANT can be tricky.

This is where you need to specifically understand how an application or control
can run. I have already talked about executing a standard program: the manifest
file will be loaded and used for the entire app. If your COM is either Reg Free
or requires a specific runtime then it will need its own manifest. As I said
earlier it is the dll's responsibility to manage its own manifest thus managing
its own Activation Context (ActCtx).

How does the relate to the COM InterOp requirements above? Well, we need
to look at how the 3 requirements are satisfied. The first is with an interop
assembly. [I call these tl-InterOp due to the way they are constructed.]
To create one, compile your COM dll and run this:


tlbimp.exe myLibrary.dll


This will produce a new dll that has all your COM types. If you add a reference
in your project to it all the types will magically appear. Lets disassemble
this file and see what is inside it:


namespace VTXComputeLib
{
[ComImport, Guid("11D77BD1-500B-477E-91FE-724116A7F9DE"), TypeLibType((short) 0x10c0)]
public interface IVxPanorama

[ComImport, CoClass(typeof(VxPanoramaClass)), Guid("11D77BD1-500B-477E-91FE-724116A7F9DE")]
public interface VxPanorama : IVxPanorama

[ComImport, ClassInterface((short) 0), Guid("10BA6F3D-BB38-4264-BE4D-62A5F5A2D059"), TypeLibType((short) 2)]
public class VxPanoramaClass : IVxPanorama, VxPanorama
}



Okay, pretty clear what is going on here. This is just a list of the interfaces
in my COM library with their GUIDS and type info. So this dll simply maps the type GUIDs in the registry to my file.

The second assembly is the requirement that .NET be able to call COM code.
We have already partially gotten this requirement from the tlbimp step. It will
give us the ability to make the calls; however, .NET is a garbage collected
language. COM is reference counted. This can cause major problems because .NET
won't necessarily tear COM components down in the correct order when they are
done. This can be done by adding "using System.Runtime.InteropServices;" to your .NET code.

There is a lot of power in the InteropServices. You can hand .NET interfaces
back to COM, you can do manual interface releases, etc. Look at the namespace
on msdn. I can't go into all the power of this guy.

Now for the final requirement (and the reason we have to break out the custom
code). Our .NET UI application has to be able to host our ActiveX control.
Remember that we are running without our control being registered. When
are the two times this happens? 1) At runtime--we can handle this by adding a
simply adding the reference to our library into our application's manifest.










Here is the issue--how do you add controls to a form? In visual studio!
For our application to find our control we modified its manifest. That
obviously is not going to work for visual studio. If this were an MFC dll, then
then we would already be done. We would just embed the manifest above as a
RT_MANIFEST and MFC would switch contexts on its own.

We used tlbimp to get our tl-InterOp assembly. To get an ActiveX one
[ax-InterOp] we use aximp.exe. If you are working on a project that will run
with its components registered: you only need to call "aximp.exe myLib.dll."

It will give you a precompiled assembly with your fittings into .NET.
Luckily, aximp has a command line option "/source" that will give you c# code.
Make a new project and add the source to it. Aximp will give you something that
looks like:

namespace AxMyControlLib
{
public class AxMyScene : System.Windows.Forms.AxHost
...

public AxMyScene() :
base("77dbba22-ecfc-4c1f-90d2-b081a61c5ebd")
{

}

protected override object CreateInstanceCore(Guid clsid)
{
RetObject = base.CreateInstanceCore(clsid);
}

// ...
}


First things first, aximp defaults to giving you code that you cannot add
to visual studio's toolbox palette. You need this if your clients are going
to start a new project. [If your library were registered they could add
it to the toolbox by browsing to it, now they need to browse to your
axInterop dll.]

change:

public class AxMyScene : System.Windows.Forms.AxHost


into:

[System.ComponentModel.DesignTimeVisibleAttribute(true)]
[System.ComponentModel.ToolboxItemAttribute(true)]
public class AxMyScene : System.Windows.Forms.AxHost


Now your clients will love you because they can start new projects!

Looking at the ax-Interop source, you will notice that there is no ActCtx code!
MS made it a convention that dll's should self manage if they have
embedded manifests. This is not a free process though: It looks like the guys
that wrote the AxImp tool didn't get the memo from the VC++ team.

The next step is converting the MFC code I listed last week into C#.


public class CActivationContextState
{
////////////////////////////////////////////////////////////
// PInvoke
[DllImport("Kernel32.dll", SetLastError = true)]
private extern static IntPtr CreateActCtx(ref ACTCTX actctx);

[DllImport("Kernel32.dll", SetLastError = true)]
private extern static bool ActivateActCtx(IntPtr hActCtx, out uint lpCookie);

[DllImport("Kernel32.dll", SetLastError = true)]
private extern static bool DeactivateActCtx(uint dwFlags, uint lpCookie);

[DllImport("Kernel32.dll", SetLastError = true)]
private extern static bool ReleaseActCtx(IntPtr hActCtx);

[DllImport("kernel32.dll")]
static extern uint FormatMessage(uint dwFlags, IntPtr lpSource,
uint dwMessageId, uint dwLanguageId, [Out] StringBuilder lpBuffer,
uint nSize, IntPtr Arguments);

[DllImport("Kernel32.dll", SetLastError = true)]
static extern uint FormatMessage(uint dwFlags, IntPtr lpSource,
uint dwMessageId, uint dwLanguageId, ref IntPtr lpBuffer,
uint nSize, IntPtr pArguments);

[DllImport("Kernel32.dll", SetLastError = true)]
static extern uint FormatMessage(uint dwFlags, IntPtr lpSource,
uint dwMessageId, uint dwLanguageId, ref IntPtr lpBuffer,
uint nSize, string[] Arguments);

[DllImport("kernel32.dll", SetLastError = true)]
static extern IntPtr LocalFree(IntPtr hMem);

[StructLayout(LayoutKind.Sequential)]
private struct ACTCTX
{
public int cbSize;
public uint dwFlags;
public string lpSource;
public ushort wProcessorArchitecture;
public ushort wLangId;
public string lpAssemblyDirectory;
public UInt16 lpResourceName;
public string lpApplicationName;
}

private const uint ACTCTX_FLAG_PROCESSOR_ARCHITECTURE_VALID = 0x001;
private const uint ACTCTX_FLAG_LANGID_VALID = 0x002;
private const uint ACTCTX_FLAG_ASSEMBLY_DIRECTORY_VALID = 0x004;
private const uint ACTCTX_FLAG_RESOURCE_NAME_VALID = 0x008;
private const uint ACTCTX_FLAG_SET_PROCESS_DEFAULT = 0x010;
private const uint ACTCTX_FLAG_APPLICATION_NAME_VALID = 0x020;
private const uint ACTCTX_FLAG_HMODULE_VALID = 0x080;

private const UInt16 RT_MANIFEST = 24;
private const UInt16 CREATEPROCESS_MANIFEST_RESOURCE_ID = 1;
private const UInt16 ISOLATIONAWARE_MANIFEST_RESOURCE_ID = 2;
private const UInt16 ISOLATIONAWARE_NOSTATICIMPORT_MANIFEST_RESOURCE_ID = 3;

private const uint FORMAT_MESSAGE_ALLOCATE_BUFFER = 0x00000100;
private const uint FORMAT_MESSAGE_IGNORE_INSERTS = 0x00000200;
private const uint FORMAT_MESSAGE_FROM_SYSTEM = 0x00001000;
////////////////////////////////////////////////////////////


private uint cookie = 0;
private static ACTCTX actCtx;
private static IntPtr hActCtx = IntPtr.Zero;
private static bool contextCreationSucceeded = false;

public bool EnterActCtx()
{
if (!contextCreationSucceeded || cookie != 0)
return false;

return ActivateActCtx(hActCtx, out cookie);
}

public bool ExitActCtx()
{
if (cookie == 0)
return false;

bool bRet = false;

try
{
bRet = DeactivateActCtx(0, cookie);
cookie = 0;
}
catch (Exception Ex)
{
System.Diagnostics.Debug.Print(Ex.ToString());
PrintError(Marshal.GetLastWin32Error());
}

return bRet;
}

public CActivationContextState()
{
if (EnsureActivateContextCreated())
{
if (!EnterActCtx())
{
// Be sure cookie always zero if activation failed
cookie = 0;
}
}
}

~CActivationContextState()
{
ExitActCtx();

if (contextCreationSucceeded)
{
ReleaseActCtx(hActCtx);
}
}

private void PrintError(int Error)
{
IntPtr lpMsgBuf = IntPtr.Zero;

uint dwChars = FormatMessage(
FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
IntPtr.Zero,
(uint)Error,
0, // Default language
ref lpMsgBuf,
0,
IntPtr.Zero);

if (dwChars == 0)
{
// Handle the error.
int le = Marshal.GetLastWin32Error();
}

string sRet = Marshal.PtrToStringAnsi(lpMsgBuf);

// Free the buffer.
lpMsgBuf = LocalFree(lpMsgBuf);

System.Diagnostics.Debug.Print(sRet);
}

private bool EnsureActivateContextCreated()
{
try
{
string rgchFullModulePath = null;
rgchFullModulePath = System.Reflection.Assembly.GetExecutingAssembly().Location;

actCtx = new ACTCTX();
actCtx.cbSize = Marshal.SizeOf(typeof(ACTCTX));
actCtx.dwFlags = ACTCTX_FLAG_RESOURCE_NAME_VALID;
actCtx.lpSource = rgchFullModulePath;
actCtx.lpResourceName = ISOLATIONAWARE_MANIFEST_RESOURCE_ID;

hActCtx = CreateActCtx(ref actCtx);
contextCreationSucceeded = (hActCtx != new IntPtr(-1));

if (!contextCreationSucceeded)
{
PrintError(Marshal.GetLastWin32Error());

actCtx.lpResourceName = ISOLATIONAWARE_NOSTATICIMPORT_MANIFEST_RESOURCE_ID;
hActCtx = CreateActCtx(ref actCtx);
contextCreationSucceeded = (hActCtx != new IntPtr(-1));
}
if (!contextCreationSucceeded)
{
PrintError(Marshal.GetLastWin32Error());

actCtx.lpResourceName = CREATEPROCESS_MANIFEST_RESOURCE_ID;
hActCtx = CreateActCtx(ref actCtx);
contextCreationSucceeded = (hActCtx != new IntPtr(-1));
}
}
catch (Exception Ex)
{
System.Diagnostics.Debug.Print(Ex.ToString());
}

return contextCreationSucceeded;
}
};


C# is restrictive about when it lets you call your base class's constructor.
It has to be declared with your class's constructor and called right before it.
That is problematic for us because we need to switch to our Reg Free context
before our base class is called.

Luckily, initializers run before constructors, so when the class is created
it will automatically hop to the correct context.

Now your code should look like this:

[System.ComponentModel.DesignTimeVisibleAttribute(true)]
[System.ComponentModel.ToolboxItemAttribute(true)]
public class AxMyScene : System.Windows.Forms.AxHost
{
// this will enter the actctx
private CActivationContextState ActCtxState = new CActivationContextState();

private System.Windows.Forms.AxHost.ConnectionPointCookie cookie;

public AxMyScene() :
base("77dbba22-ecfc-4c1f-90d2-b081a61c5ebd")
{
ActCtxState.ExitActCtx();
}

protected override object CreateInstanceCore(Guid clsid)
{
Object RetObject = null;

try
{
ActCtxState.EnterActCtx();
RetObject = base.CreateInstanceCore(clsid);
}
catch (Exception Ex1)
{

}
finally
{
try
{
ActCtxState.ExitActCtx();
}
catch (Exception Ex)
{

}
}

return RetObject;
}

// ...
}


So, when we are created, we load our context and switch to it. That makes us
hit our library without being registered. When someone tries to create us, we
switch contexts to our registration free one. Remember, you must always push
and pop your ActCtx or you will pollute the calling application. This is why
the constructor calls exit.

Depending on how you've coded your ActiveX control, you may have
to add the Enter/Exit code to more methods. Try your control out in visual
studio and you will quickly get exceptions if something is wrong.

To debug this, you will need to start two copies of visual studio.
Attach the first's debugger to the second. In the second add your
control to your toolbox, and from the toolbox to the form.
You should be good to go!

Now you can have as many versions as you want and let them fully interoperate
with managed code.

As always, pvt me if you have any specific questions.
Mathucub

Mathucub

0
 

Signing with a Verisign 2048 bit Cert

This entry is a little out of order for my manifest discussion; however,
I've finally mastered this monstrosity and its a good time to share the
the wealth. To do a winsxs install on Vista or 7 MS requires you to
sign with a trusted 2048bit cert.

This is a MAJOR problem, and is probably the reason that when you do
a "dir \windows\winsxs" you will typically only see MS products.
(Well, and mine now!) Who are the trusted certificate authorities? (ca's)
Thawte and Verisign among others. Now the problems begin. As of writing this,
Thawte doesn't off ANY 2048 certs let a lone code signing ones.

Verisign does offer certs of that strength; however, we are in a brave
new ground. They only offer two forms of code signing certs: MS authenticode
at 1024 bit only, and Java Signing at whatever strength you request in
your certificate sign request (csr). [I'm not sure what their limit is, but
can definitely do 2048 bit.]

Here is the rub: Verisign has no idea how to issue a certificate for use
with MS Signing tools (ignoring authenticode) that is anything but 1024! We are
approaching the realm of true black magic since we cannot use the Java signing
for what we need, not even to generate the request since their tool can't
override the bit depth to 2048 bit in a way we can use.

There are a couple challenges here: first getting the
correct certificate. Verisign only really support Java certs for Java use. Kinda
retarded: its just a public key/private key pair. In my opinion they should not
make things as obfuscated as they do. They ought provide the key pair in
whatever format the client requests, especially since they are not cheap.

The next challenge is converting what they give you to a useful format. Then
finally, we have to do the signing. To do this you will have to d'l a couple
tools: pvk.exe and signcode.exe. The signtool.exe that comes with newer
versions of visual studio will work--but only from the wizard mode. The command
line doesn't work. If you are doing an automated build, signcode is only
way to go.

Here is the step by step:
[You will need OpenSSL 9.8 or higher (for the -nocerts option), otherwise
when you try to extract your PFX you will be hosed. You have to be able to
extract just the key without its associated match since Verisign's format
doesn't provide it in any useful way [eg, its keyed for Java]]

1. Make your key (this is a pem file format)
openssl genrsa -out private.key 2048

2. Make your csr
openssl req -new -nodes -key private.key -out codesign.csr

3. Verify your key/csr pair
[if this doesn't work your openssl distribution is torqued]
openssl req -in codesign.csr -noout -verify -key private.key

4. Deliver CSR to Verisign [You need a Java Object Signing (Class 3)]
(wait for them to do your background check and email you your cert)

5. Save the cert that Verisign emails you to "ssl-cert.crt" (this is a spc file format)

6. Export your PFX
[http://www.mail-archive.com/openssl-users@openssl.org/msg35041.html]
openssl pkcs12 -export -in ssl-cert.crt -inkey private.key -nocerts -out cert-export.pfx

7. Do the intermediate conversions to get the valid format for signcode
openssl pkcs12 -in cert-export.pfx -nocerts -nodes -out userkey.pem
pvk -in userkey.pem -topvk -nocrypt -out ssl-cert.pvk

[... note: you need to PERMANENTLY keep your private.key, codesign.csr,
ssl-cert.crt, and ssl-cert.pvk. You will need to recycle the csr
to keep your keys the same next year when your cert expires. The other
files are consumed by signcode...]

8. Extract your public key token to use in your manifests
[copy out the second publicKeyToken="****************"]
pktextract.exe ssl-cert.crt

9. Now to do the signing, we have to use the older signcode, not signtool!
signcode -spc ssl-cert.crt -v ssl-cert.pvk -t http://timestamp.verisign.com/scripts/timstamp.dll -n "ProgramName" -i "Associated Website"

You now have a binary signed with 2048 security. MS is forcing all the ca's
to move to 2048 at some point. Before you do all these steps contact your ca
and specifically ask if they can give you a 2048 bit authenticode cert. As
of this date, they will not even think what you are asking for is possible...


EDIT Oct 2010: Please Read this: I got this email from Verisign:

***********************************************************************
2048-bit root upgrade completed for SSL
***********************************************************************

Dear customer,

On October 10, 2010, at 0600 PDT, VeriSign Identity and Authentication Security,
now a Division of Symantec, completed the upgrade to the 2048-bit VeriSign
Class 3 Public Primary Root Certification Authority-G5?, creating a stronger,
chained CA hierarchy for all our SSL and Code Signing Certificates.

All new SSL and Code Signing certificates are NOW being issued from inter-
mediate CAs under this 2048-bit root.
Mathucub

Mathucub

0
 

Generating SxS Manifests

The visual studio ide provides limited support for Reg Free COM generation but
its not really a complete solution. I decided to bail on using it. Unfortunately
this moves a lot of code to the prebuild, postbuild, and external batch files.


First things first... You have to understand that a Reg Free COM manifest must
contain ALL the information that would normally go in the registry. This comes
from ALL the "rgs" files in your project and the project's type library. [I'm
not sure what you will do if you have more than one typelib being produced by
your project. I would probably collapse them all into a single idl/tlb.]


The manifest tool only allows you to provide a single "rgs" and "tlb" on his
command line. Its easy enough to merge the rgs files in a prebuild step [cmd /c
type *.rgs > RTM.rgs] That is where the easy ends...


A little fore-note, the reason this gets messy is because of the power we are
going to leverage. We are doing what I call a Release to Marketing (RTM) build.
In this build style we will be placing versioned dlls into \windows\winsxs. You
can only place dlls there via a .msi installer. Adding that step would make
debugging a mess. The release build can run as usual with the dlls simply
registered. The RTM process does the necessary conversion steps and is only done
when a release is being cut.


For this trick to work, we will write a batch file that causes all the steps
necessary for a sxs install. Believe me there are many...


Here is the post-build step:


"$(SolutionDir)RTMBroker\bin\mt" /nologo -manifest "$(TargetPath).manifest" -outputresource:"$(TargetPath)";2

echo call "$(SolutionDir)RTMBroker\MakeDLLIntoAssembly" "$(SolutionDir)RTMBroker" "$(TargetDir)" $(TargetName) "$(TargetFileName)" VTX.tlb 3.0.0.91 fbf3841e0e84639c AJTSystems >> "$(SolutionDir)RTMBroker\FusionizeBuild.bat"

echo del /F "$(TargetPath)" >> "$(SolutionDir)RTMBroker\DeleteBuildProducts.bat"


The first line embeds the manifest so we can run normally.
The second gives all the info needed to my batch file. The the third makes it
so I can delete all my final build products without rebuilding. (This
is needed for back to back builds since you can't sign an assembly twice.)


This entry is for generating manifests, so I'm only going to dive into that
part of the batch file today. The code to do this is as follows:



%1\bin\mt /nologo -rgs:..\RTM.rgs -tlb:%5 -dll:%4 -identity:"%3, processorArchitecture=x86, version=%6, type=win32, publicKeyToken=%7" -out:%4.sxs.manifest


Remember, we are handing around parameters to a batch file, so you need to
understand the % notations: they are the parameters we wrote to
FusionizeBuild.bat in order from [1 to n].

%1: where our RTM files are located
%3: our assembly name
%4: our dll name
%5: our tlb
%6: our version
%7: publicKeyToken

Also remember this is for a sxs install. For a real "localized" install [where
the dll runs from the same directory as the exe] you can do without the
publicKeyToken and the name needs to be the same as the assembly of the
executable. If you are using control's that are based on IDispatch then you wil
l probably need to do them as localized. I've not gotten IDispatch to work when
the assembly is in \windows\winsxs.


There we go... you now have a manifest. At this point you are probably thinking:
"Matty, what is the point of that? I can put all that information into the
visual studio ide. Isn't that easier?"


Well, technically, yes you could, especially if you are doing a localized build
with no .net interop. We are targeting sxs and also have to make interop dlls
work. That adds a big chunk of other requirements, so you will have to hand all
the same information to the batch file anyway. I prefer to have the whole
process in one pile vs having to edit one file then hunt and peck in the ide.


This is all using c++. If you have a vb ocx you are in for a new set of problems
to make the manifest. The MSDN documentation pretty much says that you should
hand write you manifest. (Ha, some of these things are 1000 lines+) There is a
GPL tool called regsvr42 that will probe a com dll or ocx and make a capable
manifest. Its got a few bugs in it though [no support to hand in all the parts
of the assembly identity, "version" is strncmp'ed against an index to small, and
it can write out duplicate iid entries that break the rules of the manifest
such that windows complains about the syntax and exits]


You can download it over on "code project". I'm going to post the fixed version
of it on my gamedev space when I get time. If you need it sooner just private
message me.
Mathucub

Mathucub

0
 

Manifest Embedding and Activation

First, you cannot understand manifests without understanding the Activation
Context (ActCtx) API. I'll go into more details later, but the ActCtx API
subverts ::LoadLibrary, ::CoCreateInstance and a couple other windows functions
that deal with loading libraries and resources.

This allows a module to use resources that are in \Windows\WinSxS or in the
execution directory and not have them be registered in the system registry. It
allows for any number of versions of a library to be present on a particular
system. (Eg, you can have msvcrt.dll for versions 6, 2005, 2008, 2010 and have
the same filename for each, just a different location in WinSxS. No more DLL
Hell!) For Reg Free COM, the manifest will have all the same information as
what would normally be in the registry. They are very similar to "plists" on OS
X.


Manifests can either be stored on the file system beside their module as
myprogram.exe.manifest or mylibrary.dll.manifest or embedded as a RT_MANIFEST
resource. Be very careful here: on Windows XP, if a program had an RT_MANIFEST,
and someone placed a myprogram.exe.manifest on the file system--the file would
take priority and override the resource based one. On Vista/7 they changed this
behavior--the resource will always win.


There are 3 values that the RT_MANIFEST resource can be embedded as. Here are
their general uses:


Resource ID 1 (CREATEPROCESS_MANIFEST_RESOURCE_ID) is activated implicitly
when the .EXE is executed. The activation context blankets all DLLs implictly
or explicitly loaded by the .EXE. (Unless they self manage, loaded DLLs inherit
this context.)


Resource ID 2 (ISOLATIONAWARE_MANIFEST_RESOURCE_ID) intercepts static imports
of the DLLs listed in the manifest for this module only. Also, if the module is
built as a USRDLL with MFC version 7 or later it activates the context for
dynamic use when execution control reaches any entry point in the DLL, via
AFX_MANAGE_STATE(AfxGetStaticModuleState()).


Resource ID 3 (ISOLATIONAWARE_NOSTATICIMPORT_MANIFEST_RESOURCE_ID) is never
used by the Windows loader. It is used only in explicit user code that wants to
manually load and establish an activation context. In particular it is used by
the #define intercepts declared by setting ISOLATION_AWARE_ENABLED=1 before
including windows.h. For the macro implementation of ISOLATION_AWARE_ENABLED
see WinBase.inl, WinUser.inl, Commctrl.inl, Comdlg.inl, and Prsht.inl. The
intercept macros wrap the XP-theme-aware APIs such as CreateWindowEx() and
LoadLibrary() in order to activate the V6 context, which is loaded from resource
ID 3 - WinBase.inl WinbaseIsolationAwarePrivatetRgzlnPgpg actCtx.lpResourceName
= (LPCWSTR)(ULONG_PTR)3. The context is wrapped around each API call such that
the context is activated upon each API call and deactivated upon each API
return.


Now that we know where these values can be embedded--when are they used?
Well, there are two major ways. The windows executable loader will find these
and make an ActCtx when a module is loaded. This is where it gets interesting
though: they also need to be manually managed at runtime. At runtime, ActCtx's
have to be pushed and popped when crossing module boundaries that target
different library versions.


Say my library is loaded and I have a manifest on
ISOLATIONAWARE_MANIFEST_RESOURCE_ID (2); after my module is loaded the calling
application has ever right (and should) restore his own ActCtx, otherwise I've
polluted all his calls.


MFC can take care of this automatically. Lets look at his code to understand
what he is doing. Later on, we will have to take the same code and convert it
to C# so that AxHost InterOp Dlls can gain MFC's capabilities such that we can
use InterOp Dlls that reference unregistered DLLs in visual studio's design
window.

In afxstate.cpp:

void AFX_MODULE_STATE::CreateActivationContext()
{
_AfxInitContextAPI();
HMODULE hModule = m_hCurrentInstanceHandle;

WCHAR rgchFullModulePath[MAX_PATH + 2];
rgchFullModulePath[_countof(rgchFullModulePath) - 1] = 0;
rgchFullModulePath[_countof(rgchFullModulePath) - 2] = 0;
DWORD dw = GetModuleFileNameW(hModule, rgchFullModulePath, _countof(rgchFullModulePath)-1);
if (dw == 0)
{
return;
}
if (rgchFullModulePath[_countof(rgchFullModulePath) - 2] != 0)
{
SetLastError(ERROR_BUFFER_OVERFLOW);
return;
}
//First try ID 2 and then ID 1 - this is to consider also a.dll.manifest file
//for dlls, which ID 2 ignores.
ACTCTXW actCtx;
actCtx.cbSize = sizeof(actCtx);
actCtx.dwFlags = ACTCTX_FLAG_RESOURCE_NAME_VALID | ACTCTX_FLAG_HMODULE_VALID;
actCtx.lpSource = rgchFullModulePath;
actCtx.lpResourceName = MAKEINTRESOURCEW(ISOLATIONAWARE_MANIFEST_RESOURCE_ID);
actCtx.hModule = hModule;
m_hActCtx = AfxCreateActCtxW(&actCtx);
if (m_hActCtx == INVALID_HANDLE_VALUE)
{
actCtx.lpResourceName = MAKEINTRESOURCEW(ISOLATIONAWARE_NOSTATICIMPORT_MANIFEST_RESOURCE_ID);
m_hActCtx = AfxCreateActCtxW(&actCtx);
}
if (m_hActCtx == INVALID_HANDLE_VALUE)
{
actCtx.lpResourceName = MAKEINTRESOURCEW(CREATEPROCESS_MANIFEST_RESOURCE_ID);
m_hActCtx = AfxCreateActCtxW(&actCtx);
}
if (m_hActCtx == INVALID_HANDLE_VALUE)
{
m_hActCtx = NULL;
}
}


Looks pretty straight forward. Find the filename of the current module. Next,
in order of ISOLATIONAWARE_MANIFEST_RESOURCE_ID,
ISOLATIONAWARE_NOSTATICIMPORT_MANIFEST_RESOURCE_ID, then
CREATEPROCESS_MANIFEST_RESOURCE_ID, try to find an embedded manifest and create
a manifest that get gets stored in the AfxState of this module.


Now that we have an ActCtx, when does it get activated?
Well, let look at an MFC USERDLL call:

STDMETHODIMP CVxMesh3D::LoadMesh(BSTR filename)
{
AFX_MANAGE_STATE(AfxGetStaticModuleState())

HRESULT hr=S_OK;

hr=GetMesh3DObject()->LoadMesh(filename);

return hr;
}


Looks like a standard COM wrapper call. There is an internal Cpp object accessed
by GetMesh3DObject(). An InterOp or a COM client calls in with a BSTR of the
mesh we want to load in the scenegraph. All the magic happens in the
AFX_MANAGE_STATE(AfxGetStaticModuleState()) call.


After a bit of digging, you'll find that that macro actually makes an instance
of the class "AFX_MAINTAIN_STATE2." During that object's construction the
following call is made:


AFX_MAINTAIN_STATE2::AFX_MAINTAIN_STATE2(AFX_MODULE_STATE* pNewState) throw()
{
#ifdef _AFXDLL
m_pThreadState = _afxThreadState.GetData();
ASSERT(m_pThreadState);
if(m_pThreadState)
{
m_pPrevModuleState = m_pThreadState->m_pModuleState;
m_pThreadState->m_pModuleState = pNewState;
}
else
{
// This is a very bad state; we have no good way to report the error at this moment
// since exceptions from here are not expected
m_pPrevModuleState=NULL;
m_pThreadState=NULL;
}
#endif

if (AfxGetAmbientActCtx() &&
pNewState->m_hActCtx != INVALID_HANDLE_VALUE)
{
m_bValidActCtxCookie = AfxActivateActCtx(pNewState->m_hActCtx, &m_ulActCtxCookie);
}
else
{
m_bValidActCtxCookie = FALSE;
}
}


Pretty straight forward. If there is a stored ActCtx, then it is activated.
ActCtx's must always be pushed and popped if the module is not using the
caller's ActCtx, so where does he get deactivated?


If we look in that class's destructor:

// AFX_MAINTAIN_STATE2 functions
_AFXWIN_INLINE AFX_MAINTAIN_STATE2::~AFX_MAINTAIN_STATE2()
{
#ifdef _AFXDLL
// Not a good place to report errors here, so just be safe
if(m_pThreadState)
{
m_pThreadState->m_pModuleState = m_pPrevModuleState;
}
#endif

if (m_bValidActCtxCookie)
{
BOOL bRet;
bRet = AfxDeactivateActCtx(0, m_ulActCtxCookie);
ASSERT(bRet == TRUE);
}
}


Again straight forward. If there is an activated ActCtx, turn if off (popping it
off the stack). There is a really elegant use of the language here. Placing the
activation and deactivation in the constructor and destructor makes it such
that we never have to worry about manually making the push and pop calls. Even
if there is an exception!

Say GetMesh3DObject() was NULL in the above sample. That would cause a throw.
Say my calling application is and old VC6 client. My module was compiled with
the VS2005 C runtime. If the ActCtx is not deactivated, when the exception is
caught by the caller, any runtime calls he makes will be routed to the 2005
runtime, instead of the msvcrt.dll version 6. That is an obvious recipe for a
crash. Since the deactivation code is in the destructor, it is guaranteed to be
called!

In my last post I mentioned not trusting the ide's use of mt.exe to embed your
module's manifest. The reason is that you really want to decide which of the
three enumeration values you want to place it at. Also, again, if you let it do
the embed it will not leave a finalized copy of the manifest on the file system.
We need that file for other steps of this the Reg Free process.


In my solution I turned off the Embed, and use this call in the post build
step: mt /nologo -manifest "$(TargetPath).manifest" -outputresource:"$(TargetPath)";2

This uses the command line manifest tool and embeds my manifest as
ISOLATIONAWARE_MANIFEST_RESOURCE_ID.

At this point, you really haven't made any changes that move us toward Reg Free
COM. We have the standard manifest that the linker produced for us, but all it
should contain is the references to the c runtime. We are still going to have
to make our other manifests. I'll go into these steps in my next entry.
Mathucub

Mathucub

0
 

What is a manifest?

There are a few types of manifest formats that I understand: Application,
Assembly, SxS Assembly, and Dependency.


Application manifests will usually appear as "myprogram.exe.manifest" or
embedded in the exe as an RT_MANIFEST (more on this later).


Assembly manifests are analogous to Application manifests but they
typically associated with dll's. SxS Assembly's are a special case of
this in that they go in the \windows\winsxs directory and have some special
properties.

Dependency manifests are small, usually single entry manifests, that allow
assemblies to advertise their existence. If you take an application (or
assembly) manifest, and merge in a bunch of dependency manifests: when the
application starts, it will look for those dependencies.


Visual Studio 2005 introduced manifest creation at build time. Microsoft moved
the c runtime to the \windows\winsxs directory so that it can be versioned. This
means that if your using Visual Studio 2005 or higher--your already interacting
with the ActCtx API and using manifests!


If you go to your project properties, and go down to the "Manifest Tool" drop
down and go to "Input and Output" you will see and option "Embed Manifest."

Here is where the problems start, and where we have to start going off road.
If the manifest tool (mt.exe) embeds your manifest, you won't have it to use
yourself. Also, where exactly is it embeding it?--does it make sense that you
just trust this tool? (No...)


Next entry I'll explain the options for embedding, what they mean, and what they
do.
Mathucub

Mathucub

0
 

Real world implementation of Reg Free Com (WinSxS)

.NET has taken over much of Windows development; however, for those of us that
have large applications in COM we can't simply jump ship. COM/C++ is still a
valid development platform. The major drawback of COM was actually once one of
its selling points: there can only be one component on a system with a
particular GUID.
That makes running multiple versions of the same component impossible. For
shops like ours; where our front end application is a wrapper around COM dll's,
this has become a serious problem. Its not an acceptable workflow to expect
clients to uninstall/reinstall different versions of the software to work on
different projects.
Windows XP SP2 introduced the solution to this with a technology called
fusion. It allows for COM components to be run "Reg Free," or not registered in
the windows registry. Most people have encountered an aspect of fusion in the
form of application manifest files used to program the "Activation Context API"
and to run with elevated privileges on Vista/7.
Making an end to end solution with this technology is non-trivial and
information about how to do each step of it is sparse and mostly not to the
point. The documentation is written from the viewpoint of already understanding
how the underlying technology works, not from the ground up.


Items to cover:
What is a manifest? - different formats, and how to generate them.
What is the Activation Context (ActCtx) API and why does he matter?
What are the options for running in Reg Free mode?
What is the WinSxS directory and how to I install to him?
Hows does the ActCtx interact with .NET InterOp?
Mathucub

Mathucub

2
Sign in to follow this  
Followers 0
  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

  I accept

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!