I experience that on several different machines, with plugins, without plugins, with VB.net or c# solutions of many different sizes, closing the solution in VS 2005 generally takes significantly more time than actually building the solution.
This has always been the case for me since I started using Visual Studio 2005, so I have learned to live with it, but I am curious:
What on earth is visual studio doing when you have actually told it to shut down? Is it significant? Is it configurable, can you turn it off?