My topic for CFUNITED 2010 was selected! I'll be speaking about the new Excel generation features in CF9; much easier and more flexible than generating the XML or HTML used to be.

Thank you to everyone who voted for my topic.

See you all at Lansdowne.

• Are your ColdFusion applications running slow or even crashing the server?
• Are you concerned about what increasing load will do the the reliability of your application?
• Do you want to protect your organizations reputation for quality on the web?

Then join TeraTech's ColdFusion server tuning guru Ajay Sathuluri for this webinar on what to do when your server is slow or sick. We will look at how to diagnose problems and some common ways to heal a sick ColdFusion server. We will also discuss what tools you can use to prevent problems from occurring. 

Ajay has been using ColdFusion for more than 10 years and has spoken on server tuning and load testing at CFUnited and MDCFUG.

The webinar on "How to cure slow and crashing ColdFusion servers"  is on Thursday Dec 3, 2009 1:00 PM  EST. The webinar will cover fixing slow servers, performance bottlenecks location and diagnosis tips. It will be approximately 45 minutes including time for Q and A. The webinar is free. You can register at  https://www1.gotomeeting.com/register/750953064 Hope to see you there!

System Requirements
PC-based attendees
Required: Windows® 2000, XP Home, XP Pro, 2003 Server, Vista

Macintosh®-based attendees
Required: Mac OS® X 10.4 (Tiger®) or newer

My CFUnited presentation on "Dynamic Load Testing with Webserver Stress Tool" went well. Thanks to everyone for the good feedback. 

My presentation, source, and testing scripts will be posted on the CFUnited site soon after the conference.

Congrats to the 3 raffle winners: Steven Focht (Webserver Stress Tool 7), Abram Adams (WST), Bryan Modders (FusionReactor).

Paessler will be contacting the WST winners soon with the license info.

Thanks again to Paessler and Intergal.

CFUnited is next week!

I'll be presenting on Wednesday on "Dynamic Load Testing with Webserver Stress Tool". I'll be covering Basic, parameterized, and scripted load testing. It's a fairly inexpensive tool, but can do some powerful testing.

This week I also found out I'll be able to raffle 3 software licenses: 2 licenses for Paessler Webserver Stress Tool, and 1 for FusionReactor server monitor! ($1000 worth of software). Licenses sponsored by Paessler and Intergral, respectively.

Topics:

• Test Types
• setting up a set of URLs
• Multiple users
• URLs bound to a dataset (parameterize the tested URLs, merge with a dataset; one set for each user)
• Scripting, instead of list of URLs
• Simulate multiple simultaneous CF sessions
• How to simulate AJAX calls
• Test results and logs; where's the most useful information

Recently we worked on an application for a large association in DC that generates about 60% of the association's revenue. The application had grown in features and users over several years and the code was in need of tuning, with slow performance and frequent server crashes. It was running on CF 7 on Linux.

Improvements

We recently helped setup the development server with a 64-bit operating system, and the 64-bit version of ColdFusion 8.0.1. Additionally, we also did performance and memory tuning.

CF8 itself has been shown to have better object creation performance as well as general speed improvements across the board (see Adobe’s performance brief www.adobe.com/products/coldfusion/pdfs/cf8_performancebrief.pdf). We tuned JVM configuration parameters (memory and garbage collection), recommended and helped setup the updated Java runtime 1.6 updater 12, which fixes a major performance bug and allowed us to take full advantage of CF 8’s and Java 1.6’s performance and memory improvements. We even discovered CPU and memory issues that were fixed by applying a cumulative hotfix for ColdFusion.

The result of all this is much more manageable CPU and memory usage, as well as better initialization and search performance. Furthermore, many cases of CFC methods that were missing var scoping on local variables were fixed; this affecting stability and performance.

Maintainability

Here we added Application-level service objects to Coldspring, to make the code more maintainable. We also had Coldspring create more of the sub-objects, to make the object hierarchy more visible.

Generally, where these service objects called global objects directly (configuration or other service objects), changed to pass those in via the Coldspring configuration. Many places were able to reduce to passing in specific configuration properties (like a DSN), rather than passing in the whole object. This goes a long way towards reducing interdependency, which makes them better able to test.

We removed some reserved words in method names, so we could use automatic CFC documentation tools. We also had consistent use of init() methods; adding to objects that didn’t have them, and then using consistently when creating. This better enables use of Coldspring, and makes usage overall more readable.

We then made a switch to Fusebox 5.5, which allowed us to move customized code out of the core files, and into the index.cfm, which calls it; this included application-specific parameters retrieved from configuration or CGI variables. This will make it much simpler to swap out the framework for later versions.

Scalability

Finally, we developed and ran a profiling tool, which temporarily modified all of the components to save logging information on when objects are created and methods are called. We ran on some sample searches, logged the number of times objects are created and methods are called. We were then able to reduce use of some classes used in application.

There was one class that was created 160 times for each search result page that we were able to convert to a structure generated by a service object; this service object only needs to be created once per session, saving a whole lot of object creation.

SQL Injection can damage your website’s data and spread to other sites in your organization. This article explains how it works and how you can prevent it.

Exploited vulnerabilities:

A SQL Injection attack relies on the someone sending an HTTP request (web site visitor) being able to add SQL commands to a URL or form variable, and have it sent in a SQL query so that the SQL Server runs it as another SQL command. Any text passed along in URL or form variables can be modified by the user, or automated. URL/Form variables should only be treated as data, and never trusted.  (See http://www.forta.com/blog/index.cfm/2005/12/21/SQL-Injection-Attacks-Easy-To-Prevent-But-Apparently-Still-Ignored for a quick description). If text entered in the URL can get run on the database server as a SQL command, then a malicious user can pass any SQL commands that that SQL login has access to, including making malicious edits, reading from system tables for the database, bypassing site login code, or possibly (if the SQL login has permissions) creating or deleting tables. 

Over the past few months (since about May 2008) there’s been an automated SQL Injection attack running against first just ASP sites, then others, and now ColdFusion sites (searching for URLs containing “.cfm” files). It tries to pass a SQL script in a URL variable to each page, on the hope that at least some pages will pass it directly to the database in a way that the database server will execute them. (See http://www.coldfusionmuse.com/index.cfm/2008/7/18/Injection-Using-CAST-And-ASCII for details on how this SQL works.) 

The malicious SQL script used in this attack loops over system tables (sysobjects, syscolumns) to find character fields in all user tables, and append an HTML string to all character fields, assuming that at least some of those columns will be output directly on some other web page. This HTML string includes a JavaScript block, which will get the browser to download and run that JavaScript URL. (This JS URL will probably change quite often, but will be available enough to be downloaded.) Apparently, this script then turns the browser hitting the page displaying that data into a “bot” to hit other servers, and can do anything that Javascript in the browser can do, especially exploiting browser vulnerabilities.

It only requires successfully hitting one page on one site to get into that database, then the HTML would be displayed on whatever site uses those tables and columns.

The best way to prevent, is to make sure any data passed to SQL queries is only treated as data. All CF Queries should be parameterized (use CFQueryparam to pass the values in).

Action taken for containment:

The best permanent fix is for each and every query to be parameterized, so that no user-supplied text is passed to the database directly.   

There are some ways to block some of the SQL keywords in URL and form variables; this is not guaranteed to match, and may have some false positives, but will buy time to protect all queries. 

We set up some CF code on most public sites, set up in Application.cfm (runs at beginning of each request) to search for these SQL keywords in the URL and then abort the page.

Action taken for resolution and recovery:

The best permanent fix is for each and every query to be parameterized, so that no user-supplied text is passed to the database directly.   

In the meantime, block those SQL keywords either through CF code (we put on most public sites) or through rewrite rules. 

To repair the infected database, we are running a SQL script similar to the injected SQL to remove the appended HTML from those columns it was added to.

Further recommendations:

• Have daily backups of all databases.
• Regular backups of code.
• Do a security audit on your site (TeraTech can do this)
• Better CF regex filtering of url/form variables at application level
  • May block this set of attacks enough to buy time.
  • Not a replacement for parameterizing all queries.
• Blocking on webserver level, using rewrite rules. We haven’t quite found a set of rules that work reliably, but will keep looking.
  • May block this set of attacks enough to buy time.
  • Not a replacement for parameterizing all queries.
  • open-source Ionics ISAPI rewrite filter http://www.codeplex.com/IIRF/
    • Filter requires a copy of the DLL and INI file in separate folder for each site, then point each site to that DLL
  • Helicon ISAPI Rewrite (commercial): http://www.isapirewrite.com/
  • Backup the rule configuration files
• Optionally, look at an application firewall that can block certain types of requests. Rules are options. Not guaranteed to block everything, and can have false positives, but protects the whole server and individual sites, and is highly configurable. Again, not a replacement for parameterizing all queries.
  • Applicure dotDefender; $4000 for a server
    • http://www.applicure.com/
• Each and every query should pass variables to database using CFQueryParam. Use QueryParam scanner (http://qpscanner.riaforge.org/) on source folders to find queries that might be missing.
• Any code that uses URL or Form-supplied values to determine SQL columns or sort order, should check against a list of valid columns etc. Don’t use those values directly in the query without checking.
• If you have cached queries that get user-entered values, CF before 8.01 doesn’t allow caching queries that use cfqueryparam. Consider other options, like caching the query without that WHERE clause and then doing a query on that query.
• Validate form fields; check values of ID fields entered in URL before getting to query, then display nicer message to user and don’t need to send an error message.

Resources: 

CF-Talk discussions on the issue:

http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:57221

http://www.houseoffusion.com/groups/cf-talk/thread.cfm/threadid:57241

http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:310594

Good description on how this attack works:

http://www.coldfusionmuse.com/index.cfm/2008/7/18/Injection-Using-CAST-And-ASCII

on Ben Forta’s blog:

http://www.forta.com/blog/index.cfm/2005/12/21/SQL-Injection-Attacks-Easy-To-Prevent-But-Apparently-Still-Ignored

http://www.forta.com/blog/index.cfm/2008/7/22/For-Goodness-Sake-Use-CFQUERYPARAM-Already

http://www.forta.com/blog/index.cfm/2008/7/23/Hacker-Webzine-Recommends-Use-Of-CFQUERYPARAM

QueryParam scanner (search CF code for queries that are probably missing cfqueryparam):

http://qpscanner.riaforge.org/

HP Scrawlr: finds web pages vulnerable to SQL injection attacks on your own sites.

http://www.communities.hp.com/securitysoftware/blogs/spilabs/archive/2008/06/23/finding-sql-injection-with-scrawlr.aspx

My presentation on HA-Clustering at CFUnited actually consisted of two parts.  Firstly there was the PowerPoint bit which I uploaded to the CFUnited server and then there was the practical part which I would like to overview here for those who could not make the event.

I had intended to use two VMware copies of Windows 2003 Enterprise (32Bit) with Windows Network Load Balancing (NLB) running as the web server clustering mechanism,  ColdFusion Enterprise 8.01 and MySQL 5.0.  I attempted this on my Windows XP notebook with 3GB Ram and after several attempts I gave up because I just could not get enough responsiveness, I could never have gotten through the presentation effectively.  However, I do intend to attempt this again, in future.

Instead I reverted to running the tests on my notebook and the details are as follows:

• System - Dell Inspiron E1705 with 3GB Ram running Windows XP. 
• CF Version - ColdFusion Enterprise 8.01, installed in the multi instance manner with two instances clustered with RoundRobin algorithm.
• Servlet Container - JRun 4 Updater 6.
• JVM - Sun 1.6 (aka 6).
• Web Server Apache 2.0.59
• Test Application - cfwhisperer blog lab copy based on Mango blog.
• Load Test Tool - Paessler Web Stress Tool.
• Server Utility - SeeFusion version 4.0.7

I created a load test script by simply browsing the lab copy of the cfwhisperer blog in the URL recorder inside the Paessler Web Server Stress Tool and saving the URLs; this took me about 5 minutes total  and I had a very usable test script, that is one of the reasons I really like the Paessler tool.  I recorded 18 clicks in total as I have found around 20 to be optimal, for most test scenarios. As my presentation was only 60 minutes total I only showed one 10 minute test.   I kicked off the test which was running against the Apache web server which I connected to the two instance CF cluster with the wsconfig GUI utility.  So this was a 10 minute load test with 20 concurrent vUsers with 8 second think time between clicks.  As the test was running I first stopped one instance and we immediately observed the response times climb from an average of 300 milliseconds to a peak of over 20 seconds and then fall back to 300 milliseconds, this took around 30 seconds and although response times went up we still responded to requests.  I left this for a couple of minutes then restarted the stopped instance there was no similar slowdown as the instance came back, but we did see a slight rise to 600 milliseconds.  I then repeated the exercise by stopping the other instance and we observed similar results. 

My intention is to at least repeat this at cfdevcon in Brighton, England in September, 2008 and hopefully extend it a bit if I can.

This is the second in a series of blog posts leading up to my presentation on Clustering at CFUnited on June 20, 2008 at 2:00PM.  Clustering has been available to us in two apparently different forms.  Hardware Clustering and Software Clustering.  I say apparently different as some digging reveals interesting items, particularly as they relate to the current Software Clustering offered with ColdFusion – JRun.

Before looking more deeply into those issues, I wanted to lay out some basic differences between Hardware and Software Clustering.  In my view a Hardware Clustering system exists when there is a dedicated piece of hardware whose only job is to handle Clustering (Fail-Over and/or Load-Balancing).  One detail of note here, the Hardware Clustering device needs Clustering Software to operate. 

Software Clustering typically means that the Clustering Software is installed on an existing server, not a dedicated Hardware device.  One noted example of Software Clustering is Windows Network Load Balancing (aka NLB).  This is a bit of a misnomer, as this is a Clustering mechanism which includes Load-Balancing as well as Fail-Over.   Another example of Software clustering is the Clustering offered in ColdFusion-JRun which is actually based on J2EE standards.

If we dig a little deeper into the current J2EE clustering used in ColdFusion and JRun; since CF moved to Java with the MX version forward.   We see what I would class as a subset of full Software Clustering.  The reason I believe this is that J2EE clustering as applied the CF-JRun is a sub-set of Software Clustering is that it is purely peer-to-peer with no overall service watching all Cluster members.  Typically we would not only have multiple ColdFusion severs-instances but also multiple web servers or we would not have full redundancy.  Fail-Over over is not covered at the CF-JRun level if a Web server fails.  There has to be a higher level system to do this such as Windows NLB or a Hardware Clustering device.  As an historical point;  prior to MX, ColdFusion-JRun had a much more fully featured Software Clustering system called “ClusterCATS”, this did embrace to concept of a central monitoring-management service and Web server Fail-Over.  This was always tricky though if ColdFusion was deployed in the “distributed-mode” where the Web server and ColdFusion were on different physical devices. 

The last consideration in this piece, is when would we use Hardware or Software Clustering or both.  Typically Hardware Clustering scenarios are much more robust than Software Clustering, as that is their only job in life.  Having said that; Windows NLB claims support for up to 32 servers.  Going back to the Allaire-ColdFusion clustering, ClusterCATS, our testing showed that 8 servers was a good point to consider Hardware Clustering. 

Lastly, the overall term here is “Clustering” and within Clustering we have Fail-Over and Load-Balancing so the often used term “Load Balancing Device” should actually be “Clustering Device”.

I will be presenting at CFUnited on Friday June 20, 2008, on the subject of High Availability (HA)-Clustering for ColdFusion/JRun applications and I intend to make this as practical as possible. Having spent many years travelling the world helping to fix slow or unresponsive ColdFusion applications, I see HA as a natural progression to this and in fact Load-Balancing, which is a part of Clustering, has a direct impact on improving performance.

There is a point, often overlooked by even the manufacturers of clustering/load balancing equipment. Clustering is the overall term which, in my opinion, applies whenever two items or more appear as one, to the users. In our world, that typically means multiple web servers, with multiple application and database servers.

With this aspect of Clustering there are two services which are a part of the Clustering; Fail-Over and Load-Balancing. In my experience Fail-Over is always present, meaning if one member of the Cluster fails the remaining members ensure that continuity of service is maintained. This is a prime function of a Cluster.

Load-Balancing is the apportioning of load around members of the Cluster, typically an even distribution of the load is what is required. The most even distribution is via Round-Robin which means each single request moves around the Cluster members, like this (this example shows a 3 member Cluster):

• USER 1 > REQUEST1 > CLUSTERMEMBER1

• USER 2 > REQUEST1 > CLUSTERMEMBER1

• USER 1 > REQUEST2 > CLUSTERMEMBER2

• USER 2 > REQUEST2 > CLUSTERMEMBER2

• USER 1 > REQUEST3> CLUSTERMEMBER3

• USER 2 > REQUEST3> CLUSTERMEMBER3

• USER 1 > REQUEST4 > CLUSTERMEMBER1

• USER 2 > REQUEST4 > CLUSTERMEMBER1

This is the most evenly balanced Load Balancing algorithm and as I mentioned above is the Round-Robin algorithm. Problems can occur with that algorithm if there are user specific items in memory on one of the Cluster members, for instance in memory session state variables. If USER1 has logged in to CLUSTERMEMBER1 above and their details are in session variables on CLUSTERMEMBER1 when users next request takes them to CLUSTERMEMBER2 those in memory session state variables will not be there. My preference for the optimal Load-Balancing algorithm is Round-Robin with Sticky Sessions. In the case a user “sticks” to a Cluster member as follows: 

• USER 1 > REQUEST1 > CLUSTERMEMBER1

• USER 2 > REQUEST1 > CLUSTERMEMBER2

• USER 1 > REQUEST2 > CLUSTERMEMBER1

• USER 2 > REQUEST2 > CLUSTERMEMBER2

• USER 1 > REQUEST3> CLUSTERMEMBER1

• USER 2 > REQUEST3> CLUSTERMEMBER2

• USER 1 > REQUEST4 > CLUSTERMEMBER1

• USER 2 > REQUEST4 > CLUSTERMEMBER2

  This is not quite as evenly balanced as Round-Robin alone but unless there is a failure of one Cluster member the user will not lose their session state variables and the load balances across all Cluster members eventually.

This article serves as the first in a series of posts leading up to the CFUnited presentation and my next one will delve into differences between Hardware and Software Clustering.

I was talking with TeraTech’s server tuning consultant Mike Brunt about ColdFusion tuning and clustering which we did for a client to improve stability and performance.  I thought others might be interested in what we did with server metrics, threads, logs and garbage collection in the Java Virtual Machine (JVM). We edited two files, as follows... 

1/ {JRun4_root}servers\default\SERVER-INF\jrun.xml

We edited the following lines Lines 126 through 130, we uncommented this section (it is sometimes already uncommented) made sure it was set to true

<!-- This Service provides metrics information -->

<!-- ================================================================== -->

<service class="coldfusion.server.jrun4.metrics.MetricsServiceAdapter" name="MetricsService">

    <attribute name="bindToJNDI">true</attribute>

</service>

We then edited the section from line 148 through 151 as follows...

<!-- You may also need to uncomment MetricsService if you want metrics enabled -->

    <attribute name="metricsEnabled">true</attribute>

    <attribute name="metricsLogFrequency">60</attribute>

    <attribute name="metricsFormat">Web threads (busy/total): {busyTh}/{totalTh} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory}</attribute>

We changed <attribute name="metricsEnabled">true</attribute> to true which turns on Metrics Logging we made no more changes to this section but it is good to note what they do. <attribute name="metricsLogFrequency">60</attribute> sets the frequency of the metrics logging to 60 second intervals, that can be changed, in my experience, 60 seconds is typically a good time interval.

The line below that determines what is displayed; {busyTh}/{totalTh} will show all busy threads and total threads in use.  The busy threads value shows you how many threads are actually in use when this snapshot was taken and the total the actual busy threads along with threads in every other state.  There is a good write up of different thread states here - http://www.bpurcell.org/blog/index.cfm?mode=entry&entry=934.  The total thread number shows all threads in every state, as a very broad rule of thumb, the greater the difference between busy and total the more efficiently the application is running.

Sessions: {sessions} shows the number of Java-J2EE sessions running which is enabled by turning on "Use J2EE Session Variables" in the Memory Variables section of CF Admin.  This will show you how many J2EE sessions are running every 60 seconds.  This is an indication of activity and this number will rise and fall as user sessions begin and end.

Total Memory={totalMemory} Free={freeMemory} is fairly self explanatory and shows the memory state in 60 second intervals, ideally we do not want to see free memory drop too low, I would get uncomfortable if it drops below 75 MB or 10% of the total.  Also, we want to see any used memory being released as quickly as possible.

We next edited the way logs are generated and created a split out of the logs by adding {log.level} to line 155 

<attribute name="filename">{jrun.rootdir}/logs/{jrun.server.name}-{log.level}-event.log</attribute> 

This last change enables more targeted logs which will be written to {JRun4_root}\logs.

 I recommend making these changes on all Production CFMX 6.1 systems and leaving them on, the information produced can be invaluable for troubleshooting.

2/ {JRun4_root}\bin\jvm.config 

This files is the main configuration file for the JVM and is loaded at start time.  After observing application behavior metrics from the logging changes above we applied the following changes to jvm.config...

-Xms768m -Xmx768m - This sets the start and maximum memory allotted to the heap at 768MB.  I typically set this number as high as is needed by the metrics logging and verbose garbage collection information.  In practical terms, on 32 bit Windows, it cannot be set any higher than 1.4GB. 

-XX:PermSize=128m -XX:MaxPermSize=128m - Here we added a start size for the permanent generation which I where the classes for JRun-CF are stored.  We did this because we observed all of this memory space being fully consumed many times, in the verbose garbage collection log. 

-Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 - These arguments generate an explicit Full Garbage Collection (Full GC) at 10 minute intervals.  We applied these because we observed Full GC's taking place at one minute intervals and in some cases at 30 second intervals.  This is far too often as when there is a Full GC, everything else on the server stops.

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:teratechgc.log - This argument set turns on logging of all garbage collections in a verbose manner and this is where we got our information for the memory settings along with the metrics information.  We turned this logging off after making the changes and checking that all was OK.