Benchmarks revisited

After my last blog post regarding performance differences between various functions and between accessing them with cfscript vs. tag based calls, I got an email from reader Robert Froehling, who said the following:

Your article, Benchmarks – try VS isdefined() & script VS tag, got me thinking about how the various functions, that can be used to determine a variables existence or value, perform.  So I decided to do a little testing myself.  I setup the tests almost exactly the way you did, just using different functions.  I used Len(), IsDefined(), Compare(), and StructKeyExists().  Personally, I’ve been using Len() for a while now.  Obviously, it requires you to use <cfparam> for all of the potential variables.  But this doesn’t bother me much.  I ran each test 10 times to come up with an average.  It turned out that Len() and Compare() using <cfscript> are the leaders at an average of 60 milliseconds.  Attached are my results.

Just thought I’d share.

Fro

It appears that his tests too seem to add weight to cfscript being notably faster on the functions he tested.

With his permission to share, here are his results:

Len() IsDefined() Compare() StructKeyExists()
Block Script Block Script Block Script Block Script
1 547 156 469 140 344 140 265 187
2 172 250 172 219 125 219 344 235
3 125 31 172 79 125 32 203 110
4 125 16 172 78 125 31 203 109
5 125 16 171 94 141 31 203 109
6 125 31 171 79 125 31 203 110
7 141 31 172 79 110 31 203 109
8 125 31 172 78 109 16 203 109
9 125 15 156 78 109 31 204 109
10 109 31 172 79 109 31 203 94
Average 171.9 60.8 199.9 100.3 142.2 59.3 223.4 128.1

Benchmarks – try VS isdefined() & script VS tag

I was discussing with a friend yesterday about some code I had written where I was passing values from a form where the form elements may or may not exist depending on the customer’s configuration.  I took the approach of using <cftry> vs isdefined() like so…

<cfscript>

try  {
myObject.setProperty(“form.Property”);
}
catch(Any excpt)   {
// the form variable didn’t exist
}

</cfscript>

My reasoning was that a) I felt like doing something different, and b) I thought it might be faster overall not having to compare before it accessed it, given the fact that 90% of the time the value will exist.

So this morning I did some benchmarks with isdefined() vs cftry and found results that surprised me.  I looped though 100,000 records and took an average processing time.

_______________

First I tested a case where the value didn’t exist using the cfty method:

<cfset start = getTickCount() />
<cfloop from=1 to=100000 index=”i”>
<cftry>
<cfset new = old />
<cfcatch>
<!— old didn’t exist —>
</cfcatch>
</cftry>
</cfloop>
Done in <cfoutput>#getTickCount() – start# milliseconds.</cfoutput>

I found that this averaged out to about 2797ms on our modest development server.

I then pitted that against using isdefined() as so:

<cfset start = getTickCount() />
<cfloop from=1 to=100000 index=”i”>
<cfif isdefined(“old”)>
<cfset new = old />
</cfif>
</cfloop>
Done in <cfoutput>#getTickCount() – start# milliseconds.</cfoutput>

This was considerably faster averaging 2140ms.

_______________

Then I decided I would benchmark the two methods when the value did exist.    The first thing that surprised me is how much quicker it was for both methods!  I figured that in this case <cftry> might win since in my mind (which I now see was probably wrong), it wouldn’t have to do a test before it accessed it.  Here were my results:

First the cftry method:

<cfset start = getTickCount() />
<cfset old = 1 />
<cfloop from=1 to=100000 index=”i”>
<cfif isdefined(“old”)>
<cfset new = old />
</cfif>
</cfloop>
Done in <cfoutput>#getTickCount() – start# milliseconds.</cfoutput>

This one ran on an average of 130ms.

Then the isdefined() approach:

<cfset start = getTickCount() />
<cfset old = 1 />
<cfloop from=1 to=100000 index=”i”>
<cfif isdefined(“old”)>
<cfset new = old />
</cfif>
</cfloop>
Done in <cfoutput>#getTickCount() – start# milliseconds.</cfoutput>

Using the isdefined() approach netted very similar numbers averaging 125ms.

_______________

So then while I was in the testing mode I decided I would pit tag based code against script code.  Another shocker came out of this.  When the variable didn’t exist, it was slightly faster, but I when it did exist, it was about 10x faster!  Here is what I found:

Using the try approach when the variable didn’t exist:

<cfscript>
start = getTickCount();
for (i=1;i LTE 100000;i=i+1)    {
try    {
new = old;
}
catch(Any excpt)    {
// old didn’t exist
}
}
writeOutput(“Done in ” & getTickCount() – start & “ms.”);
</cfscript>

This method averaged out to 2575ms, a little more than 200ms quicker than tag based.

I then ran the script based isdefined() approach:

<cfscript>
start = getTickCount();
for (i=1;i LTE 100000;i=i+1)    {
if (isdefined(“old”))    {
new = old;
}
}
writeOutput(“Done in ” & getTickCount() – start & “ms.”);
</cfscript>

This method averaged out to 1844ms, a little more than 300ms quicker than its tag based version, but over 700ms faster than the script based try version.

_______________

Then I tried the script versions when the variable did exist, and wow!

<cfscript>
start = getTickCount();
old = 1;
for (i=1;i LTE 100000;i=i+1)    {
try    {
new = old;
}
catch(Any excpt)    {
// old didn’t exist
}
}
writeOutput(“Done in ” & getTickCount() – start & “ms.”);
</cfscript>

This method averaged out to 16ms!  This was about 8 times faster than using tag based code.

And using the isdefined() version was virtually the same:

<cfscript>
start = getTickCount();
old = 1;
for (i=1;i LTE 100000;i=i+1)    {
if (isdefined(“old”))    {
new = old;
}
}
writeOutput(“Done in ” & getTickCount() – start & “ms.”);
</cfscript>

Which averaged to 15ms, again *far* faster than its tag based counterpart.

_______________

So to sum up the tests I ran I found that isdefined() is indeed faster than just doing a try/catch, and (at least for the code I ran) cfscript was markedly faster than cf tags, the latter of which was a surprise to me since I felt that once it was compiled into byte code it would be identical.

Hope someone else finds this remotely interesting! :)

Strange issue with suspected latency… anyone???

I have several applications on a server that is exhibiting a long string of oddball behavior that is often unreproducible on other machines.   I have exhausted all that I know to search for on the current issue and thought I would post it here in case anyone happened to have some thoughts about it.

This server (Windows 2003/IIS/CFMX6.1) has been running somewhat normally for a year or so with the code that is on it.  Lately however, when many pages load, they behave as though they are missing some of the HTML or Javascript.  Javascript-dependent menus fail, or sometimes no text at all will be displayed on the screen.  If you view the source, all the code is there.  If you copy that source into a new template and run it, it works fine.  It is almost like the browser is just receiving instructions in the wrong order or something and doesn’t know what to do.  This problem is completely reproducible on the pages that it happens on, but only on that server.  This problem also only happens with Internet Explorer.

My thoughts are that it can’t have anything to do with ColdFusion.  I base this on the fact that the HTML is clearly being written.  It seems to me that it has to be something going on with IIS and how it is delivering the content to the browser.

If anyone has any experience with similar problems, I would love to hear some opinions.

EDIT:
After taking this server offline and bringing it back to the office from the colocation facility, we found that the problem lied with IIS.  We backed up the sites, removed IIS and reinstalled it and it magically worked again.   I will refrain from using the opportunity to express how this has never happened to me using Apache… oh ummm… well, I guess I just did.

How do visitors view your site?

Sometime back I came across this article on the viewing patterns of people visiting websites.  I think this is a very valuable piece of information and is worth considering when you are planning the development of your next site.

The article is very interesting, and they also offer suggestions for where you want to place your content to be seen by your visitors.

Go check it out!

Just in case you were all caught up… AFLAX!

This was sent to me by fellow DFWCFUGger Ryan Everhart.   As if my head was not already spinning from Flash remoting, Flex, AJAX…. introduce yourself to AFLAX!  I know nothing more about it than its definition:

“AFLAX is a method through which developers may use JavaScript and Flash together to create AJAX-type applications, but with a much richer set of vector drawing controls than are available in either Internet Explorer or FireFox. Developers using this library have access to the full range of Flash features, but without ever touching the Flash IDE.”

Sounds awesome, but where oh where can I find the time!

… is it just me or do I hear a duck quacking?

Maddening server problem… Resolved!!!

Starting about 4-5 days ago, one of our web servers began experiencing nightmarish performance issues. This was such a strange problem to deal with, and I wasn’t able to Google anyone having the same issue, so I thought that I better document this in hopes that it may serve someone in the future.

Our ColdFusion MX 6.1 server would run for a while, the suddenly requests would just stop returning results at all. Occasionally a user would finally get a response to the browser that was a ‘<’ sign, followed by varying strings of seemingly random characters. It would hang like this failing to answer any future requests until the services were restarted. This system has some obvious suspect trouble points. I will list a few:

  • About 120 remote (and I mean very remote) DBs, and 1 DB on the same subnet. Those remote databases are housed in various data centers and client sites.
  • Of the remote DBs, over half are using and ODBC Socket to Sybase (Yes, JConnect is a much better idea and is in the works).
  • Of the remote DBs, a handful of them are using VPN connections.
  • A proprietary framework has been used on a couple of the applications on the server that has a number of examples of questionable coding practice in it.
  • A new company was recently migrated to the proprietary framework application that has 40,000 users.
  • Client variables were being used and were being stored in the registry.

I viewed all of the above as places that could potentially have some type of effect on the problem we were experiencing.

Any guesses so far?

Well for starters we had Fusion Reactor on the system.  We had it set to notify us when a request took longer than 30 seconds, and kill that request gracefully.  There was no consistency whatsoever to the pages in the requests that it as notifying us on.  Even simple ‘welcome’ pages would be in that list.  This didn’t appear to be helping us on this problem so to isolate the issue I removed it from the server.

We finally got on the phone with Macromedia…errr.. Adobe and dealt with Ted Zimmerman (and later Swathi C.), who went out of his way to help us and even stayed well after he was supposed to have left for the day. One of the first things we did was to change the Client Variables setting in CF Admin so that ColdFusion stored client variables in a DB rather than the registry. Also since we have no real need for persisting client variables through multiple sessions, we set ColdFusion to store them for 1 day as opposed to the 90 day default. This didn’t seem to fix anything and if anything our problem just seemed worse and worse. Instead of the uptime for 10-20 minutes after restarting, if anything, it seemed to be bombing more and more rapidly byt he minute.

With Ted’s assistance, I learned a great new technique for viewing threads at a very low level and even being able to track process IDs over a period of time and seeing if they had hung at a particular memory address. We did this by running ColdFusion from the console and outputting to a text file. That is a very interesting and informative exercise!

We kept finding a process that was talking to a remote database that seemed to be just hanging indefinitely. I did some better trapping on that page by not only catching database exceptions and logging them, but by adding in a timeout to the query, which is something I have not typically done in my code. I found that by doing this I no longer saw that particular page in the metrics information as a trouble maker, but our greater problem still existed. Now, in our thread stack dumps, another page seemed to show up pretty frequently. I put the same traps around that process and still remained in the same boat. That said, even though we didn’t fix the problem with that, we did fix a couple of issues that were definitely in need of attention.

So, any guesses yet?
I need to preface this next section by stating that the way that this company is structured and due to the fact that the systems store extremely sensitive data related to both loan origination and loan servicing, I do not have any direct access to the production system under normal circumstances, and even in this circumstance, I have no access to the database server at all. (Ironic when you consider the power of code and how counterproductive this approach can be, but I will save that for another discussion).

That said, in the middle of all of this work, we received an email from our network center that the SQL Server had reported an error notification of less than 1% of memory available. I assumed that could have just been a weird spike that had happened, but nonetheless out tech group looked into it immediately. We found that our SQL Server was showing that it had 16MB of available physical memory. As if that wasn’t a big enough “Uh Oh!”, we were told that it only had 300MB of space available on the C drive which (against my wishes) is where the virtual memory resides.

Upon learning this, our network tech immediately robbed memory from an internal development server and drove down to our NOC to upgrade the memory. Immediately the system went back to normal and has been blazing ever since!

It was interesting thinking back to when we altered the client variables setting. The change didn’t seem obvious at the moment, but in hindsight, it definitely exasperated the problem, although it was clearly the right thing to do.

So, now… 18 hours after the memory upgrade, things are still rocking!

Model Glue…. for Java!

I picked this up off of the Model Glue list.  Chris Scott has made a Java implementaion of Model Glue.  One of his reasons was to perhaps make Java development more reachable for ColdFusion developers who have had trouble getting their heads around the environment in general.  His thinking is that if you have worked with an MVC framework before that ideally looking at a a familiar framework in Java might help the transition.

This means that Model Glue is now available for 3 languages! (there is a .NET port as well).  This is great validation for Joe Rinehart’s hard work on the framework.