A few weeks ago TeraTech's ColdFusion server tuning guru Ajay Sathuluri gave a webinar on How to Cure Slow and Crashing ColdFusion servers. He looked at how to diagnose problems and some common ways to heal a sick ColdFusion server. He also discussed what tools you can use to prevent problems from occurring.

If you missed the webinar you can get the notes from it by completing a short server survey here.

Ajay has been using ColdFusion for more than 10 years and has spoken on server tuning and load testing at CFUnited and MDCFUG.

We had overwhelming interesting this week's webinar on How to Cure a Slow or Crashing ColdFusion server with over 200 people registering. Many attendees asked if we could do future webinars in more detail on some of the following topics

• Staging environment best practices
• JVM settings, memory configuration
• Garbage collection
• Monitoring tools
• Load testing

we are planning on doing some of these in the next few months. If you have other topics you would like to see let me know.

Of the attendees on the webinar the majority were using CF 8 Enterprise, with CF9 coming in a close second, followed by CF 7 and earlier.

• §

• Are your ColdFusion applications running slow or even crashing the server?
• Are you concerned about what increasing load will do the the reliability of your application?
• Do you want to protect your organizations reputation for quality on the web?

Then join TeraTech's ColdFusion server tuning guru Ajay Sathuluri for this webinar on what to do when your server is slow or sick. We will look at how to diagnose problems and some common ways to heal a sick ColdFusion server. We will also discuss what tools you can use to prevent problems from occurring. 

Ajay has been using ColdFusion for more than 10 years and has spoken on server tuning and load testing at CFUnited and MDCFUG.

The webinar on "How to cure slow and crashing ColdFusion servers"  is on Thursday Dec 3, 2009 1:00 PM  EST. The webinar will cover fixing slow servers, performance bottlenecks location and diagnosis tips. It will be approximately 45 minutes including time for Q and A. The webinar is free. You can register at  https://www1.gotomeeting.com/register/750953064 Hope to see you there!

System Requirements
PC-based attendees
Required: Windows® 2000, XP Home, XP Pro, 2003 Server, Vista

Macintosh®-based attendees
Required: Mac OS® X 10.4 (Tiger®) or newer

My CFUnited presentation on "Dynamic Load Testing with Webserver Stress Tool" went well. Thanks to everyone for the good feedback. 

My presentation, source, and testing scripts will be posted on the CFUnited site soon after the conference.

Congrats to the 3 raffle winners: Steven Focht (Webserver Stress Tool 7), Abram Adams (WST), Bryan Modders (FusionReactor).

Paessler will be contacting the WST winners soon with the license info.

Thanks again to Paessler and Intergal.

CFUnited is next week!

I'll be presenting on Wednesday on "Dynamic Load Testing with Webserver Stress Tool". I'll be covering Basic, parameterized, and scripted load testing. It's a fairly inexpensive tool, but can do some powerful testing.

This week I also found out I'll be able to raffle 3 software licenses: 2 licenses for Paessler Webserver Stress Tool, and 1 for FusionReactor server monitor! ($1000 worth of software). Licenses sponsored by Paessler and Intergral, respectively.

Topics:

• Test Types
• setting up a set of URLs
• Multiple users
• URLs bound to a dataset (parameterize the tested URLs, merge with a dataset; one set for each user)
• Scripting, instead of list of URLs
• Simulate multiple simultaneous CF sessions
• How to simulate AJAX calls
• Test results and logs; where's the most useful information

My presentation on HA-Clustering at CFUnited actually consisted of two parts.  Firstly there was the PowerPoint bit which I uploaded to the CFUnited server and then there was the practical part which I would like to overview here for those who could not make the event.

I had intended to use two VMware copies of Windows 2003 Enterprise (32Bit) with Windows Network Load Balancing (NLB) running as the web server clustering mechanism,  ColdFusion Enterprise 8.01 and MySQL 5.0.  I attempted this on my Windows XP notebook with 3GB Ram and after several attempts I gave up because I just could not get enough responsiveness, I could never have gotten through the presentation effectively.  However, I do intend to attempt this again, in future.

Instead I reverted to running the tests on my notebook and the details are as follows:

• System - Dell Inspiron E1705 with 3GB Ram running Windows XP. 
• CF Version - ColdFusion Enterprise 8.01, installed in the multi instance manner with two instances clustered with RoundRobin algorithm.
• Servlet Container - JRun 4 Updater 6.
• JVM - Sun 1.6 (aka 6).
• Web Server Apache 2.0.59
• Test Application - cfwhisperer blog lab copy based on Mango blog.
• Load Test Tool - Paessler Web Stress Tool.
• Server Utility - SeeFusion version 4.0.7

I created a load test script by simply browsing the lab copy of the cfwhisperer blog in the URL recorder inside the Paessler Web Server Stress Tool and saving the URLs; this took me about 5 minutes total  and I had a very usable test script, that is one of the reasons I really like the Paessler tool.  I recorded 18 clicks in total as I have found around 20 to be optimal, for most test scenarios. As my presentation was only 60 minutes total I only showed one 10 minute test.   I kicked off the test which was running against the Apache web server which I connected to the two instance CF cluster with the wsconfig GUI utility.  So this was a 10 minute load test with 20 concurrent vUsers with 8 second think time between clicks.  As the test was running I first stopped one instance and we immediately observed the response times climb from an average of 300 milliseconds to a peak of over 20 seconds and then fall back to 300 milliseconds, this took around 30 seconds and although response times went up we still responded to requests.  I left this for a couple of minutes then restarted the stopped instance there was no similar slowdown as the instance came back, but we did see a slight rise to 600 milliseconds.  I then repeated the exercise by stopping the other instance and we observed similar results. 

My intention is to at least repeat this at cfdevcon in Brighton, England in September, 2008 and hopefully extend it a bit if I can.

This is the second in a series of blog posts leading up to my presentation on Clustering at CFUnited on June 20, 2008 at 2:00PM.  Clustering has been available to us in two apparently different forms.  Hardware Clustering and Software Clustering.  I say apparently different as some digging reveals interesting items, particularly as they relate to the current Software Clustering offered with ColdFusion – JRun.

Before looking more deeply into those issues, I wanted to lay out some basic differences between Hardware and Software Clustering.  In my view a Hardware Clustering system exists when there is a dedicated piece of hardware whose only job is to handle Clustering (Fail-Over and/or Load-Balancing).  One detail of note here, the Hardware Clustering device needs Clustering Software to operate. 

Software Clustering typically means that the Clustering Software is installed on an existing server, not a dedicated Hardware device.  One noted example of Software Clustering is Windows Network Load Balancing (aka NLB).  This is a bit of a misnomer, as this is a Clustering mechanism which includes Load-Balancing as well as Fail-Over.   Another example of Software clustering is the Clustering offered in ColdFusion-JRun which is actually based on J2EE standards.

If we dig a little deeper into the current J2EE clustering used in ColdFusion and JRun; since CF moved to Java with the MX version forward.   We see what I would class as a subset of full Software Clustering.  The reason I believe this is that J2EE clustering as applied the CF-JRun is a sub-set of Software Clustering is that it is purely peer-to-peer with no overall service watching all Cluster members.  Typically we would not only have multiple ColdFusion severs-instances but also multiple web servers or we would not have full redundancy.  Fail-Over over is not covered at the CF-JRun level if a Web server fails.  There has to be a higher level system to do this such as Windows NLB or a Hardware Clustering device.  As an historical point;  prior to MX, ColdFusion-JRun had a much more fully featured Software Clustering system called “ClusterCATS”, this did embrace to concept of a central monitoring-management service and Web server Fail-Over.  This was always tricky though if ColdFusion was deployed in the “distributed-mode” where the Web server and ColdFusion were on different physical devices. 

The last consideration in this piece, is when would we use Hardware or Software Clustering or both.  Typically Hardware Clustering scenarios are much more robust than Software Clustering, as that is their only job in life.  Having said that; Windows NLB claims support for up to 32 servers.  Going back to the Allaire-ColdFusion clustering, ClusterCATS, our testing showed that 8 servers was a good point to consider Hardware Clustering. 

Lastly, the overall term here is “Clustering” and within Clustering we have Fail-Over and Load-Balancing so the often used term “Load Balancing Device” should actually be “Clustering Device”.

I will be presenting at CFUnited on Friday June 20, 2008, on the subject of High Availability (HA)-Clustering for ColdFusion/JRun applications and I intend to make this as practical as possible. Having spent many years travelling the world helping to fix slow or unresponsive ColdFusion applications, I see HA as a natural progression to this and in fact Load-Balancing, which is a part of Clustering, has a direct impact on improving performance.

There is a point, often overlooked by even the manufacturers of clustering/load balancing equipment. Clustering is the overall term which, in my opinion, applies whenever two items or more appear as one, to the users. In our world, that typically means multiple web servers, with multiple application and database servers.

With this aspect of Clustering there are two services which are a part of the Clustering; Fail-Over and Load-Balancing. In my experience Fail-Over is always present, meaning if one member of the Cluster fails the remaining members ensure that continuity of service is maintained. This is a prime function of a Cluster.

Load-Balancing is the apportioning of load around members of the Cluster, typically an even distribution of the load is what is required. The most even distribution is via Round-Robin which means each single request moves around the Cluster members, like this (this example shows a 3 member Cluster):

• USER 1 > REQUEST1 > CLUSTERMEMBER1

• USER 2 > REQUEST1 > CLUSTERMEMBER1

• USER 1 > REQUEST2 > CLUSTERMEMBER2

• USER 2 > REQUEST2 > CLUSTERMEMBER2

• USER 1 > REQUEST3> CLUSTERMEMBER3

• USER 2 > REQUEST3> CLUSTERMEMBER3

• USER 1 > REQUEST4 > CLUSTERMEMBER1

• USER 2 > REQUEST4 > CLUSTERMEMBER1

This is the most evenly balanced Load Balancing algorithm and as I mentioned above is the Round-Robin algorithm. Problems can occur with that algorithm if there are user specific items in memory on one of the Cluster members, for instance in memory session state variables. If USER1 has logged in to CLUSTERMEMBER1 above and their details are in session variables on CLUSTERMEMBER1 when users next request takes them to CLUSTERMEMBER2 those in memory session state variables will not be there. My preference for the optimal Load-Balancing algorithm is Round-Robin with Sticky Sessions. In the case a user “sticks” to a Cluster member as follows: 

• USER 1 > REQUEST1 > CLUSTERMEMBER1

• USER 2 > REQUEST1 > CLUSTERMEMBER2

• USER 1 > REQUEST2 > CLUSTERMEMBER1

• USER 2 > REQUEST2 > CLUSTERMEMBER2

• USER 1 > REQUEST3> CLUSTERMEMBER1

• USER 2 > REQUEST3> CLUSTERMEMBER2

• USER 1 > REQUEST4 > CLUSTERMEMBER1

• USER 2 > REQUEST4 > CLUSTERMEMBER2

  This is not quite as evenly balanced as Round-Robin alone but unless there is a failure of one Cluster member the user will not lose their session state variables and the load balances across all Cluster members eventually.

This article serves as the first in a series of posts leading up to the CFUnited presentation and my next one will delve into differences between Hardware and Software Clustering.

We were recently working on a site that was growing rapiding in traffic. How should it be clustered and why? In putting together this recommendation we took into account the anticipated increase in traffic volumes and also are striving to avoid too much distribution of assets across a network.  By avoiding the use of Network Attached Storage (NAS) or Storage Area Network (SAN) we believe you can achieve maximum efficiency and maximum redundancy. 

Initially with two physical ColdFusion servers and two physical SQL Servers, a failure of one will not cause a total loss of service.  If we employ either a NAS or SAN device, that could be a single point of failure.  The only complication of having two Web-ColdFusion-File Servers is in making sure that both systems are fully replicated.  If this is seen as too considerable a job we could easily add a file-server connected via Gigabit Ethernet, however that once again would introduce a single point of failure should the file server fail. There is software available to automate the copying of code across a server farm (more on that in a later blog post)

One other major point, relating to clustering.  The software clustering in ColdFusion cannot handle fail-over from server to server.  So in the event that Server 1 of 2 in a cluster fails, all users on Server 1 would lose their sessions and work.  In addition, without a hardware clustering device,  there is no effective way to share incoming traffic between the two ColdFusion servers.  In order to deal with this we recommended a hardware clustering device which would handle all incoming traffic and who’s main job will be to send users to Server 2 should server 1 fail.

We are also taking account of another good feature in database design and use.  There are two distinct uses for databases:

OLTP : On Line-Transactional-Processing - This is what we would characterize as the ongoing day-to-day functional copy of the database. It is where data is added and updated but never overwritten or deleted.  This what most companies have and what they use for everything, including, typically heavy, reporting needs.

OLAP: On Line-Analytical-Processing - In this model data is stored in a format which enables the efficient creation of data mining/reports. OLAP design should accommodate reporting on very large record sets with little degradation in operational efficiency.

Our proposal involves two SQL Server databases which will operate in an Active-Passive method for OLTP work, the ongoing adding and updating of data.  So Database Server 1 will handle all the OLTP traffic unless it fails in which case all traffic will go to Database Server 2 which will have an up to date replicated copy.  Database Server 2 will function as a read-only reporting server in an OLAP way and in the case of a failure of Database Server 2 traffic can be sent to Database Server 1 for OLTP and OLAP duties until Database Server 2 recovers.

Using this method there is good redundancy-failure coverage and also some distribution of load for reporting needs.

I was talking with TeraTech’s server tuning consultant Mike Brunt about servers and SANs (Storage Area Networks). When should you use a SAN?

1/ As a SAN systems are typically simply a large number of hard-drives, optical drives etc in an enclosure, it is very easy to add capacity quickly as needed as opposed to adding a complete new server which takes time. So if you database is growing fast you might chose to put the data on a SAN.

2/ If there is a need to allow access to a large volumes of data from multiple systems.  A typical case here would be large media files such as photographs-images, movies, music, games etc.

It is generally not a good idea to put CF code on a SAN to share among a cluster of servers - the network traffic will be much slower than accessing the CF code on a local hard drive. Instead use automated tools to copy updated code between servers in the cluster. More on that later.