Date Published: March 2008.

Author: Claudio Vacalebre.

Soundtrack: For Test, Author: Claudio Vacalebre.

Today I'm touching a hot topic: scalability and performance testing with HMC 4.5.

Testing is not an easy task when touching "Solutions" as they are basically a set of client and server products that must work together, smoothly; testing a solution means several things; from Component testing where each integrated feature is tested individually to understand if the solution requirements and the associated functionalities can be validated, to Integration Testing where features are tested together to validate the End to End functions which represent hypothetical Customer scenarios; this approach worked very well since few years ago.

However things have changed and the testing approach has changed too. Services Testing has become more and more important as it represents the new SaaS (Software as a Service) world; Service Providers face on going set up (not anymore: when you have installed your application you have done) where they face constantly changing and upgrading issues while providing new services to their users.

Starting from HMC 4.0 the testing approach has been influenced by these new world and has also helped in identifying testing limitations in product testing (Exchange, Sharepoint, OCS, etc.) vs Solutions testing such as multi-tenant functionalities, Provisioning, Deployment process, Multi-tenant and multi-protocol scalability and performance, End to End scenarios testing.

As a result new testing methodologies include, among other, putting under stress a given site from three different internal and external locations (or type of interface), Provisioning Load (through the provisiong engine), the Managed User Load (namespaces) and the End User Load (products). Not an easy task!

You can easily identify four main variables that could affect the scalability testing: with regard to Exchange Server 2007 the total number of users, the total number of organizations, the number of Offline Address Books and the total number of SMTP domains are very important factors to consider.

Not to mention the need to perform multi-protocol testing (MAPI, POP3, OWA) which add complexity into the testing environment; the number of combinations could be thousands...

From a Provisioning Service perspective the capability of adding a certain number of organizations and users per minute is also extremely interesting; how many password resets and account detail updates per minute can be supported; how many SMPTP domain can be added or removed every minute.

I have just mentioned a couple of core HMC 4.5 services but I could continue with Sharepoint Services and Office Communication Server sizing parameters...

Depending on the combination of these variables you can deploy testing environments that accurately represents a given scenario.

HMC 4.5 release date is currently end of May 2008 and I do not have any negative evidence which should affect this deadline, everything seems going well and in the right direction; as soon as HMC 4.5 will be released the scalability testing will start. It will include multi-tenancy and multi-protocol testing for all the core components and services including Provisioning, Exchange, Sharepoint and OCS.

I'll try to keep you update on the results as soon as they will be available.

In the meantime you can download the HMC 4.0 Scalability Report!