Developing within SharePoint 2010 - all the latest updates for it are installed (SP2 etc.)
Standard farm with 2 application servers, 2 front-end servers, Active Directory server and 2 SQL servers. All this stuff is hosted by Windows Azure Virtual machines, within Virtual network.
While performing simple
SPWebApplication.Lookup() noticed that it takes very-very long to complete - about 16 seconds. To compare - locally it takes about 1 second. And on another very similar farm, also hosted in Azure - about 2 seconds.
What attempts were made to fix performance degradation:
- Checked configs and network settings, pings etc. - looks 100% OK.
- Profiled with SQL Profiler - no bottlenecks found - there is no hard SQL for this request actually.
- Double-check that all the servers and DBs are upgraded and up to date.
- Kick off all the possible errors that were found in ULS and Windows logs - now it's clear there.
- Investigation of metrics with Metalogix Diagnostics manager - as result nothing critical was found. It only sometimes showed that processor queue length is big. But as I know, normal number for it is #of cores +1. So 4-5 in my case is fine. Also it's needed to note that from my perspective - for VM it's also normal to have such number.
- Wrote very simple console app that performs lookup of web app. Profiled with Ants profiler. Noticed that call tree differs from the result received locally. Maybe that's OK, cause locally I have standalone installation. The result at farm is not optimistic - several calls have a huge Hits count. Though, it's clear where the bottleneck in the call tree - all the ideas about the source have already finished. Profiling result as follows: http://1drv.ms/1kYT3rT
It would be great if you could advice.
Thanks in advance.