not sure how this code works, the foreach loop should throw an InvalidOperationException, as you modify the collection on the loop...
the problem i can see is that this will load the assembly in your appDomain, maybe you want to create another, so you can unload them after?
also, shouldn't it be a List ? it seens that you're trying to load multiple assemblies from one file...
maybe this method can help in a first attempt to removo possible problematic assemblies:
http://msdn.microsoft.com/en-us/library/system.reflection.assembly.getreferencedassemblies.aspx
using it, you can check if the assembly references your own, then you can check the version it references, if it's lower than the current one, you remove it. this way you'll not have an exception.
at this point i'm just throwing suggestions on the table, but maybe we can get something that works for you, my current understanding is that you're receiving an exception everytime you load an plugin that was compiled with an old version of your main project, that isn't compatible anymore...
this question on StackOverflow has an example use of the method i described:
http://stackoverflow.com/a/2135347
i really couldn't find a way to only skip the assembly if it isn't compatible, as in "if it uses an older version, but is still compatible, load it", all solutions i found are either based on version numbers or in type loader exceptions...