This is probably an excellent opportunity to profile rather than making a premature optimization. In general LINQ performs reasonably well if used correctly -- especially with the more recent versions of .NET -- but only you can do the testing to find out if it's presenting a bottleneck in your particular case. It certainly isn't something that's known to be such a problem in the majority of cases that you should simply avoid it outright, and it does allow very elegant solutions -- especially given you're only using small portions where it provides a real benefit to readability and isn't used extensively throughout the entire code-base.
I'd suggest you continue to use it for now and consider replacing it later on if you find it's a cause of performance problems.
Hope that's helpful!