Parsing CSV files with Grails

One of the arguments that I often make for my use of CFML is how you can do so much with so little code. Seemingly every time I attempt to do something that I haven’t done previously with Grails, I find that argument holds less water than I thought, as I can often do it even easier in Grails.

For a current project, we have an occasionally updated CSV document that contains codes related to the customer’s industry. Given that this file will be changing with additional codes being added, while this app is in early development, we decided that we would just keep it in our application config directory and ensure that any new codes are added during the application bootstrap routine.  Here is what I came up with:


// Insert new codes
def csv = new File("grails-app/conf/code.list.csv")
def code
csv.splitEachLine(',') { row ->
   code = Code.findByLabel(row[1]) ?: new Code(
      code: row[0],
      label: row[1]
   ).save(failOnError: true, flush: true)
}

Essentially, the above is saying:

  • Read the CSV file
  • Loop each line, where each line is referred to as “row” in the closure.
  • Search in the database for codes with the same label
  • If the code does not yet exist in the system, create a new instance of Code passing in property values from the row in the CSV file.
  • Save the new code to the database.

As you see, I have added line breaks for readability, but I was able to get the result I was looking for in THREE lines of code! I figured I would share in case anyone is looking for a similar solution.

JRun wsconfig error- Security alert: attempt to connect to JRun server from host

I was experimenting with the Railo 3.3 installer, which includes an IIS connector to Tomcat, which works really well.   Too well in fact!  When I ran it, it actually unmapped all my existing IIS ISAPI mappings to JRun and was sending all requests to Tomcat.

I decided the quickest fix to this would be to simply open up /JRun4/bin/wsconfig.exe and remap the sites that were no longer connected.  However, when I did this, I received the following error:

Could not connect to JRun/ColdFusion servers on host localhost.

Knowing perfectly well that I had an instance of JRun running, I went to the terminal to look at the standard out and saw this:

Security alert: attempt to connect to JRun server from a 10.252.11.207 host

In case that is too hard to read, it says: “Security alert: attempt to connect to JRun server from a 10.252.11.207 host”.   I suspect that because I am attached to a WIFI connection with an IP Address on 192.168.*, and then VPN’d into my company with a second address of 10.252.*, JRun assumes that the connection attempt is coming from outside the subnet.

I went digging through files in JRun4/lib and came across security.properties.  In this file, there is a default setting:

jrun.subnet.restriction=255.255.255.0
jrun.trusted.hosts=
jrun.subnet.restriction.ipv6=[ffff:ffff:ffff:ffff:0:0:0:0]

I altered that restriction  setting from “255.255.255.0″ to “*” like this:

jrun.subnet.restriction=*
jrun.trusted.hosts=
jrun.subnet.restriction.ipv6=[ffff:ffff:ffff:ffff:0:0:0:0]

Once I did this and restarted the server, I was able to use wsconfig without issue.  And my ACF sites are pointed to JRun, my Railo sites are pointed to Tomcat, and all is right in the world again.

NOTE: DO NOT DO THIS ON A PRODUCTION MACHINE!   If you do, I strongly recommend that it is a very temporary change.

CFML wishlist: All collections should extend Iterator

Have you ever really given a second thought to the fact that in ColdFusion/CFML you have to loop queries, arrays, and structures in completely different ways?   For example, in each of these things, we are essentially doing the same thing:

<!--- looping our query --->
<cfloop query="myQuery">
	<cfset doStuff() />
</cfloop>
<!--- looping our array --->
<cfloop array="#myArray#" index="i">
	<cfset doStuff() />
</cfloop>
<!--- or --->
<cfloop from="1" to="#ArrayLen(myArray)#" index="i">
	<cfset doStuff() />
</cfloop>

<!--- looping our structure --->
<cfloop collection="#myStruct#" item="i">
	<cfset doStuff() />
</cfloop>

In each of these cases, we are essentially doing the same thing, that being looping a collection that contains multiple items and acting on each iteration.  I have always liked the fact that the ColdFusion array can be converted to a Java iterator like this:

<cfset iterator = myArray.iterator() />
<cfloop condition="#iterator.hasNext()#">
	<cfset thisIteration = iterator.next() />
	<cfset doStuff() />
</cfloop>

However, given the fact that <cfloop array=”#myArray#” index=”i”> is already an abstraction, it doesn’t make sense to use this in most cases.   But wouldn’t it be cool if you could call myQuery.iterator() or myStruct.iterator() and have the same functionality?  Or even better, why not have those collections all extend an iterator class so that it could be simplified even futher with myQuery.hasNext() or myStruct.hasNext().

Keep in mind, this discussion is only coming from the perspective of the programming interface itself, and I am not going to get into the differences behind the scenes of how a query result is actually a set of arrays of columns, or how an array differs from a struct.  My point is simply that if we have these abstractions, it sure would be cool if they were consistent, but we were still able to call specific functions on them like ArrayFind() , StructDelete(), etc.  If we had this ability, our loops in CFSCRIPT would be a lot more consistent as well.

With that said…. I will leave you with my wishful implementation for writing loops in CFML:

<cfloop condition="#myQuery.hasNext()#">
	<cfset thisRow = myQuery.next() />
	<cfset doStuff() />
</cfloop>

<cfloop condition="#myArray.hasNext()#">
	<cfset thisItem = myArray.next() />
	<cfset doStuff() />
</cfloop>

<cfloop condition="#myStruct.hasNext()#">
	<cfset thisItem = myStruct.next() />
	<cfset doStuff() />
</cfloop>

Piecing together optional view elements at runtime with Mach-II

Often in web development, you run across a case where there is a display that contains optional view elements that are derived at runtime.  Perhaps there is a section of a form that is only available to residents of the US.  Maybe, only users with a certain level of group access have the ability to see a section of a page that can be partially viewed by other user types.  I am sure you can think of numerous cases that you have come across in your own work.   In a current project at our company, one of our developers was tasked with rewriting an old piece of legacy code in which logged in agents can select one or more of multiple reports to display.  The legacy code was a complete nightmare that would probably be worthy of an entire series of what not to do, but that is for another day!  To boil this piece down a bit, essentially the agent has a series of checkboxes of specific reports, and “to” and “from” date inputs to provide a date range.   Depending on what the agent selects, the submission page might show a single report, or a series of several reports in line.

One approach to this would be to have an event defined in which you compile each piece of data into some kind of data collection

if ( [Report1 was selected] )
     get data for Report1
if  ( [Report2 was selected] )
     get data for Report2
(... and so on...)

Then on the view, you could do something like:

if ([we have report data for Report1])
show Report1
if ([we have report data for Report2])
     show Report2
(... and so on ....)

Well, we could but it would be wrong!  Why?  For one thing, we now have conditional logic about each report built into multiple places in our application.  From a complexity and maintenance standpoint, you have just made it, at a minimum, twice as complex as it needs to be.  There is also a strong argument that could be made (and I would make it!) that your view shouldn’t be responsible for determining what it’s supposed to display.  It should simply display!

So what is another approach to this?  How could we employ MVC techniques without the individual components involved becoming intertwined, creating yet another administrative issue.  Here is the solution that I proposed to our developer:
First, let’s start with our form, which is remarkably simple:

<h3>Select the reports you would like to view</h3>
<cfoutput>
<form name="reportform" action="#buildUrl( "viewreports" )#" method="post">
<input type="checkbox" name="reportList" value="report1" /> Report 1<br />
<input type="checkbox" name="reportList" value="report2" /> Report 2<br />
<input type="checkbox" name="reportList" value="report3" /> Report 3<br />
<input type="checkbox" name="reportList" value="report4" /> Report 4<br />
<input type="checkbox" name="reportList" value="report5" /> Report 5<br />
<input type="submit" value="run reports">
</form>
</cfoutput>

As you can see, we are going to load up an event-arg on the submission named “reportList” that will be a comma separated list of reports that we will be displaying. For instance, if we make the selections you see below, on the viewreports event, event.getArg( “reportList” ) will be: report1,report3,report5

Report Selection form output

I decided that generally I wanted it to behave with a flow like this:

Reports flow diagram

It is a good goal in application development to move as much specific knowledge of the flow of the application outside of any component (view, service, or otherwise) that is not responsible for it to avoid coupling issues.  For instance, our report display page shouldn’t understand flow should it?   (hint: “no“)   Our service layer that is responsible for retrieving data shouldn’t should it? (you guessed it: “no”)

So where does that responsibility lie?  I place it squarely on the front controller framework at hand, namely Mach-II in this case.

If that is the approach we are going to follow, then how can we have freely operating pieces, and create a composite view without the individual pieces having any knowledge of each other, nor any knowledge of their role in the bigger picture?   We achieve this by creating small encapsulated pieces that are individually responsible for their limited role, and count on our framework to do the rest.

If you look at the flow diagram above, you will see that we start with a conditional statement on our submission event: “Has form output been generated?”

In our Mach-II configuration we can achieve this by doing the following:

<event-handler event="viewreports" access="public">
 <event-mapping event="noData" mapping="multi_report1" />
 <filter name="checkForReportData" />
 <view-page name="selectreports" contentArg="form" />
 <view-page name="reports" />
</event-handler>

Let’s talk about what those pieces are doing.  First, we are defining an event-mapping “noData“.  What this means is that anywhere further in this event, if someone announces “noData“, the event that we are really going to announce is “multi_report1“.   By doing this, we don’t bury specific knowledge into our component responsible for the announcing, but more on that in a moment.  Next, we are calling a filter named checkForReportData. Filters are Mach-II components that contain a single public method filterEvent() which returns a boolean value telling Mach-II whether or not it should continue further within this event.  In the code above, if the filter returns “false”, the <view-page/> nodes will not be processed.   So let’s take a look at the filterEvent() method.

<cffunction name="filterEvent" access="public" returntype="boolean" output="false" hint="I am invoked by the Mach II framework.">
     <cfargument name="event" type="MachII.framework.Event" required="true" hint="I am the current event object created by the Mach II framework." />
     <cfargument name="eventContext" type="MachII.framework.EventContext" required="true" hint="I am the current event context object created by the Mach II framework." />

     <cfset var result = event.isArgDefined( "reportOutput" ) />

     <cfif NOT result>
          <cfset announceEvent( "noData", event.getArgs() ) />
     </cfif>

     <cfreturn result />
</cffunction>

Very simply, we are saying: Is there an event-arg named reportOutput defined? If there is, we are returning true, telling the event to continue.  If not we are going to announce an event noData, and returning false.   By announcing a generic event named “noData”, and then defining what “noData” means in the XML config, we have just insulated this filter from change.  For instance, right now the <event-mapping/> says that this means that we should announce “multi_report1“.  If this ever changes to another report, then we only have to change the config.  Additionally, we might be able to repurpose this filter another way in the future and announce a completely different event by using a different event-mapping.

So in our example, we have no reportOutput on our first pass through this method, so we are being rerouted to the event “multi_report1“.  Here is what it looks like:

<event-handler event="multi_report1" access="private">
     <event-arg name="reportName" value="report1" />
     <event-mapping event="nextEvent" mapping="multi_report2" />
     <filter name="checkIncludeReport" />
     <notify listener="ReportListener" method="getData" resultArg="data" />
     <view-page name="reports.report1" contentArg="reportOutput" append="true" />
     <announce event="nextEvent" copyEventArgs="true" />
 </event-handler>

On the second line, all we are doing is defining an event-arg named “reportName” and assigning a value of “report1“.   We will be using this value in a moment.  Before we get to that, and now that you understand what event-mappings are doing, the third line should be clear.  We are just telling Mach-II “if someone or something announces nextEvent within the context of this event, announce multi_report2 instead“.  Again this allows our components to announce generic events which are explicitly defined in the config.   Next, we are calling a filter to see if report1 has been selected in the form by calling a filter named checkIncludeReport.   If the report was not selected in the form, we will kick out and announce nextEvent aka multi_report2.   However, if the report is included, we will continue down the line calling a method on our listener to retrieve data, and then using that data in a view named “reports.report1“.  We take that generated HTML and append it into an event-arg named “reportOutput“.   If you look at our code above, you will be reminded that this is the argument we were testing for in the checkForReportData filter.   Here is a look at our checkIncludedReport filter which makes the decision to include this report or not.

<cffunction name="filterEvent" access="public" returntype="boolean" output="false" hint="I am invoked by the Mach II framework.">
     <cfargument name="event" type="MachII.framework.Event" required="true" hint="I am the current event object created by the Mach II framework." />
     <cfargument name="eventContext" type="MachII.framework.EventContext" required="true" hint="I am the current event context object created by the Mach II framework." />    

     <cfset var result = ListFindNoCase( event.getArg( "reportList" ), event.getArg( "reportName" ) ) />

     <cfif NOT result>
          <cfset announceEvent( "nextEvent", event.getArgs() ) />
     </cfif>

     <cfreturn result />   
</cffunction>

All this filter is doing is checking in the event-arg reportList, which is a comma separated list of reports, and seeing if the value of event-arg reportName (which was defined on line 2 above) exists in the list.  Based on our example of selecting reports 1, 3, and 5, the plain English translation of this comparison is:  If the list “report1,report3,report5″ contains “report1″, return true, otherwise announce “nextEvent” and return false. As you surely know by now, in this case “nextEvent” translates to “multi_report2

Essentially we just repeat this exact pattern for the next 4 events, with a minor change in the last event:

<event-handler event="multi_report2" access="private">
    <event-arg name="reportName" value="report2" />
    <event-mapping event="nextEvent" mapping="multi_report3" />
    <filter name="checkIncludeReport" />
    <notify listener="ReportListener" method="getData" resultArg="data" />
    <view-page name="reports.report2" contentArg="reportOutput" append="true" />
    <announce event="nextEvent" copyEventArgs="true" />
</event-handler>

<event-handler event="multi_report3" access="private">
     <event-arg name="reportName" value="report3" />
     <event-mapping event="nextEvent" mapping="multi_report4" />
     <filter name="checkIncludeReport" />
     <notify listener="ReportListener" method="getData" resultArg="data" />
     <view-page name="reports.report3" contentArg="reportOutput" append="true" />
     <announce event="nextEvent" copyEventArgs="true" />
</event-handler>

<event-handler event="multi_report4" access="private">
     <event-arg name="reportName" value="report4" />
     <event-mapping event="nextEvent" mapping="multi_report5" />
     <filter name="checkIncludeReport" />
     <notify listener="ReportListener" method="getData" resultArg="data" />
     <view-page name="reports.report4" contentArg="reportOutput" append="true" />
     <announce event="nextEvent" copyEventArgs="true" />
</event-handler>

<event-handler event="multi_report5" access="private">
     <event-arg name="reportName" value="report5" />
     <event-mapping event="nextEvent" mapping="viewreports" />
     <filter name="checkIncludeReport" />
     <notify listener="ReportListener" method="getData" resultArg="data" />
     <view-page name="reports.report5" contentArg="reportOutput" append="true" />
     <announce event="nextEvent" copyEventArgs="true" />
</event-handler>

As I mentioned, there is a slight change in multi_report5 in that nextEvent is defined as “viewreports“.   By doing this, we have then ended our report generation and are redirecting the flow back to the initial event that kicked this process off.  Since we now have reportOutput data, we are directed to the page that ouputs it all.  Quite simply, our big, massive, magnificent multi-form display page looks like this:

these are the reports:

<cfoutput>#event.getArg( "reportOutput" )#</cfoutput>

There is no conditional nonsense, and the view simply outputs all of the generated output that was appended into the event-arg reportOutput.   Additionally, if you reflect on the things we have done, no where are we explicitly saying “if the user selected report1, do something“.  We have left it all fairly generic and hopefully have created some potentially reusable components.    For instance, let’s say that we now have a requirement for an event that only displays report2. No problem!  All we need to do is add an additional event like this:

<event-handler event="report2" access="public">
     <notify listener="ReportListener" method="getData" resultArg="data" />
     <view-page name="reports.report2" />
</event-handler>

Easy, huh!

Lastly,  I know that some of the more astute of you may have noticed a fatal flaw in the design above.  What happens when no reports are selected?   In the interest of keeping this example as stripped down as I could, I let that one go, but it is a very simple fix.  What would you do?  Where would you put it?   Feel free to post your fix in the comments, along with any other thoughts you have on this solution.

download fully-functional example files – NOTE: doesn’t include the Mach-II framework

Help the homeless. Scriptalizer.com is being evicted!

I am cross posting this for AaronScriptalizer is a GREAT project that needs a home.  If you have a server, VPS, etc running a CFML engine (Railo, OpenBlueDragon, or ColdFusion), and wouldn’t mind adding another site to your server, I am sure Aaron would be grateful.

For details contact Aaron in the comments of this post.

Railo, Structures, Keys and Case

After hundreds and hundreds of development hours, Aaron and I are on the verge of releasing the completely rewritten InstantSpot to beta this week.  I plan on a more detailed post of the technology later, but in a nutshell, we are releasing it as a Linux/Apache2.2.x/Tomcat/Railo2/Mach-II application.

Until today, we have found only one place that we have had to make a Railo workaround and that was using some pretty fringe functionality doing XMLRPC for desktop blogging.

Last night I ran into the second point where we have to make a workaround, although from an architecture standpoint, again it is pretty slight.  This time, it came about as I was changing our Mach-II config to use the ColdspringProperty rather than the ColdspringPlugin.  When I did, the application died a horrible death with errors stating that it couldn’t find our imported ColdSpring XML files that are defined in our main ColdSpring XML.

After tearing into it this bit, I found that the DefaultXMLBeanFactory.cfc calls a recursive method to build a structure of the main config XML, with additional keys for each imported XML file which are the full physical path to the file itself.   I could see that just prior to adding to the structure, the case was correct.  However, when I look at the key that was created, it was converted to lower case.

Later in the process as the Mach-II ColdspringProperty.cfc tries to loop through these keys, the config files cannot be found if there are any capital letters that exist in the real path.

I then tried modifying the ColdSpring code to use structure bracket notation for adding the key, rather than using StructInsert(), since this is one known way of preserving key case in ColdFusion.  However, I found that no matter what I did, the structure key that is created is all lowercase.

Stepping back a bit, I decided to do a simple test which confirmed this functionality:

code:

Here were the results:

Struct
thisonelast
number 3
thisonesecond
number 2
thisonefirst
number 1

As you can see, no matter which approach I took, the key is lowercase.

So, what does this mean for our release?  Not much… back to using the ColdspringPlugin.  However, it is something that I hope the fine developers of the Railo project keep in mind as a future fix.

Railo RC 2.0 is released and WOW!

The Railo team has announced the release of its 2.0 release candidate, which brings them up to CFMX7 compatibility regarding tags/functions, with a few inconsequential (to us!) exceptions.  Coincidentally the past few days, Aaron and I have been playing with setting up Apache webserver using the jk_mod connector to Tomcat.  Using this setup we have been swapping in all the different CFML processing engines and playing with them as we work on the next big release of InstantSpot.

This morning I took a sandbox app that I have set up, which uses a basic Mach-II 1.5 application with ColdSpring which we are using to wire together our concrete data model.  I have a sample event that trees through several levels of child relationships, and outputs/dumps information along the way.

In ColdFusion, this event has been taking about 650ms with debugging turned off.  I dropped that same code into my Railo 2.0 instance this morning and after the initial load of the application, that same event ran in 42ms!!!! Even when I turned debugging on it ran in under 100ms.  With that kind of performance gain, it would be silly not to consider using Railo in our future.

If you are interested, I am pasting the text of the Railo 2.0 announcement from Michael Streit that was sent to the Railo email list:

Railo RC 2.0.0.003 is avaiable for download at http://www.railo.ch/de/index.cfm?treeID=98
Fixed many bugs and also done some performance issues.


we have add some cf 8 features
- add support for ‘onMissingMethod’ to Components
- add support for attribute “statuscode” to tag location

and a particular support for the coldfusion.runtime.ServiceFactory (experimental) and other coldfusion.* specific classes (more to come)
- add Wrapper class coldfusion.runtime.JavaProxy
- add particular support for coldfusion.runtime.ServiceFactory -> coldfusion.server.DataSourceService (experimental)


tnx for all your inputs Michael



fixes for 2.0.0.002
- fixed Bug in FileHandling “BOM (Byte Order Mask) will not be removed in any case”
- change default date parsing from simple do advanced parser except in comparsion
- fixed bug FileResource “Opening Stream of a none existing Resource on win32″
- fixed bug in HTTP/FTP ResourcePath Util “cant handle path with no path”
- fixed bug in java reflector “cant cast content of one array to a other”
- fixed bug in caster “can not cast calendar to date object”
- fixed bug in tag directory “railo does not ignore invalid input for attribute sort”
- add support for “Lazy Serializer” to Struct and Array Object for method “toString()”
- change return value of method iterator from Array Object from a key Iteration to a value iteration (similar to java.util.List Object)
- add Wrapper class coldfusion.runtime.JavaProxy
- fixed bug in Application.cfc handling “only allow named argument with name ‘targetPage’”
- fixed bug in error template error.cfm “error template does not check if errorcode exists”
- add support for implizit casting from String to Resource
- add support for “cfdocument” scope to tag “cfdocument”
- fixed bug in tag http “can no read quoted charset definition from response”
- fixed bug in tag http “post with enctype ‘application/x-www-form-urlencoded’ (multipart=false) has a invalid ‘Content-Type’ Header Entry”
- fixed bug in tag http “query string part of value from attribute url should not be encodedgcal1″
- fixed bug in handling of the applicatin.cfc “cfc throws a exception when using tag cfabort”
- fixed “bug” in tag function “when abort execution of function body with tag abort, “output” attribute value “false” should be ignored”
- fixed bug in tag queryparam “sql date date does not accept null valus (empty string)”
- fixed bug in tag logout “tag does not remove session value”
- fixed bug in Application.cfc handling “onApplicationStart is called when the first time the application Scope is invoke by the code”
- fixed bug in Application.cfc handling “when onApplicationStart return false the execution of the page does not stop”
- fixed bug in tag procparam “attribute maxlength and scale is of invalid type (int instead of double)”
- fixed bug in tag lock “tag does not run without scope or name attribute”
- fixed bug in tag query “cant handle multiple resultsets, if first is a update”
- fixed bug in tag query “attribute result produce a NullPointerxception when query does not return a result”
- add support for conversion of array to “java.util.Vector” (CFMX Array Type)
- add support for conversion of struct to “java.util.Hashtable” (CFMX Struct Type)
- add possibility to change general read/write access to tag admin (Web Administrator)
- add support for node objects to Comparator
- fixed bug in tag loop-index “works with integer instead of double value”
- fixed bug in components “super components has no acccess on the members of its children”
- fixed bug in components “instaneof not work properly in the components constructor”
- fixed bug in tag select “attribute multiple has invalid tld definition (type boolean instead of string)”
- fixed bug in java reflection “selection of right constructor can fail when there are more than 1 constructor”
- fixed bug in tag query “QOQ loose typing in case of union selects”
- add support for attribute “statuscode” to tag location
- extend debugging-comment template
- extend debugging-neo template
- fixed bug in tag form “attribute method allows only lower case inputs”
- fixed bug in debugging template “missing cgi.context_path for in img source”
- add support for ‘onMissingMethod’ to Components
- fixed bug in function javacast “type ‘null’ is not supported”
- add particular support for coldfusion.runtime.ServiceFactory -> coldfusion.server.DataSourceService (experimental)


fixes 2.0.0.003
- fixed bug in tag file-upload “attribute ‘filefield’ does not accept form field value as input (fileField = “#form.download#”)”“,1] ); //–>
- fixed bug in Component Body “code in body can access callers local scope”
- fixed bug in web administrator “can not create archive when language is english”
- fixed bug in server administator “can not set init password”
- fixed bug in tag query “if username is a empty string, admin defaults username should be used”
- fixed bug in bytecode writer “return operation does not handle finally block the right way”
- fixed bug in Webservice Client “conflict in handling timezones of remote date objects”

Installing CFMX7 & Apache2.2.2 on Ubuntu 6.06

One week into being an Ubuntu convert and I still have ZERO complaints.  This distro of Linux is so comfortable to use that not once in the past week have I thought “Man, if I was just using Windows!”.   This weekend, I setup CFMX7 Developer edition with Apache 2.2.2 which just became officially supported by Adobe a couple of weeks ago.

There were a few tricks to setting up ColdFusion with Apache though so I thought I might write about it to save someone else a few minutes if they follow the same path.

***DISCLAIMER*** I am not a long-time Linux guy.  I do not claim to be doing things the “right” way.  I realize there are likely other more efficient ways to achieve the things I am doing, but I don’t know about them! :)   That said…

  • Installing Apache 2.2.2 – One of the painless things about setting up new software in Ubunutu (and other Debian distros) is the apt-get command.  For intance if you want to install Apache, just open a command propmet and type in:

    >$ sudo apt-get install apache2

    Then in a few minutes you have an instance of Apache running.   ***HOWEVER***….. this is not what you want to do in this case.  By doing apt-get, you (read “I”) do not have the ability to recompile it to suit your needs.  I found that when I tried to use the ColdFusion connector tool, it failed due to the installation type of Apache that I had.  So, to remove this I did:

    >$ sudo apt-get remove apache2

    I then removed the startup scripts from /etc/init.d.  Once this was complete found that there were a few prerequisites I needed as I went through the Apache compile/install.

    1. Make sure you have GCC installed, which was not installed on my system.  To do this run:
      >$ sudo apt-get install gcc
    2. You also need a C compiler installed.  I am not sure this is the most efficient method, but what worked quite simply for me was to install Build Essential like this:
      >$ sudo apt-get install build-essential
    3. I also found that I needed ZLib installed.  I pulled down the source from the ZLib project page and did the following:
      • extract tar to a directory and terminal into it.
      • run >$ sudo ./configure
      • run >$ sudo make
      • run >$ sudo make install
    4. Once these prerequisite steps are taken, you are ready to install Apache.  Do the following:
      • Download the UNIX source here
      • Extract the tar file into a working directory and terminal into it.
      • run >$ sudo ./configure –prefix=/usr/local/apache2 –enable-mods-shared=all
      • run >$ sudo make
      • run >$ sudo make install

      You can start Apache now by running >$ sudo /usr/local/apache2/bin/apachectl start and test your installation by pulling up http://localhost/.

  • Installing ColdFusion MX7 – Now that Apache is intalled we can install ColdFusion.  Pull down the latest ColdFusion binary from www.adobe.com.  There is nothing exceptionally tricky during the installation process except for a couple of key points.
    • During the pre-installation checklist you may get a warning regarding a missing C++ compatability pack.  This is used for C++ custom tags, and I believe for Verity as well.  I disregarded this warning and moved on.  I did choose to install “Search Services” when prompted however, and will address that issue in the near future with a Verity project that I have on the horizon.  When I do that I will come back and update this post.
    • Another point to note is that I chose the multi-server installation.  I am not sure how the server configuration might differ from these steps.
    • Lastly, when you get to the point of choosing which webserver you would like to use, choose the internal webserver that ships with ColdFusion.
  • Once the installation is complete and you start the ColdFusion server by running:
    >$ sudo /opt/jrun4/bin/jrun start cfusion
    … and then testing your installation by going to http://localhost:8300/CFIDE/administrator/
  • Now it is time to connect ColdFusion to Apache.  You need to have the updated wsconfig.jar that was released in May 2006, and can be downloaded here.  Make a backup of /opt/jrun4/lib/wsconfig.jar ($> sudo mv wsconfig.jar wsconfig.jar.bak) and replace it with the wsconfig.jar in that zip file.
  • Stop and start the jrun process so that it picks up the new wsconfig.jar.
  • Now open the connecter by running: >$ sudo /opt/jrun4/bin/wsconfig.  Once it opens make the following changes:
    1. For “Web Server” choose Apache
    2. For configuration directory, choose: /usr/local/apache2/conf
    3. Check the box for “Configure webserver for ColdFusion MX applications”
    4. Click OK and accept the prompt to restart the webserver.
  • Copy the CFIDE directory into your webroot so that you have access to the ColdFusion administrator.  Run the following:
    >$ sudo cp -R /opt/jrun4/servers/cfusion/cfusion-ear/cfusion-war/CFIDE/ /usr/local/apache2/htdocs/

At this point you should be able to log into the ColdFusion administrator by going to:

http://localhost/CFIDE/administrator/index.cfm.

Have fun!

Creating, Modifying, Deleting with Reactor

In my last entry, I gave a showed how easy Reactor makes accessing records in your database.  Even if that was all Reactor did I think it would be a great tool.  However, as one would expect, you can do much more.  In this entry, I will show you how to use Reactor to create new records, modify data in those records.

Let say we are again working with a UserAccount record, like our example from last time.  Our UserAccount object has the following properties: UserId PK, UserName, Password, Email, DateRegistered (and of course our table as these columns as well).

The first thing we need to do is create an instance of the Reactor Factory.  As I mentioned last time, this should probably be done in the application scope or something similar so you don’t have to continually reinitialize it.  Let’s go ahead and put that in our application scope:

<cfscript>
// create an instance of Reactor
application.Reactor = CreateObject(“Component”,        “reactor.reactorFactory”).init(expandPath(“/config/reactor.xml”));
</cfscript>

Once that is instantiated, anytime we need it we can copy it into the local scope of the template we are working on.   So, let’s create a new UserAccount object in our system and populate it with our new user ‘dshuck’.

<cfscript>
// create local instance of the Reactor Factor
Reactor = application.Reactor;

// create a new empty instance of our UserAccount object
UserAccount = Reactor.createRecord(“UserAccount”);

// now populate the UserAccount object properties
UserAccount.setUserName(“dshuck”);
UserAccount.setPassword(“secret”);
UserAccount.setEmail(“dshuck@gmail.com”);
UserAccount.setDateRegistered(createODBCdatetime(now()));
</cfscript>

So now we have a UserAccount object that whose properties hold the data that we added.  If you wanted to at this point you could output something such as the username by:

<cfoutput>#UserAccount.getUserName()#</cfoutput>

If you look in the database at this point you will notice something though.  There is no ‘dshuck’ user in the UserAccount table.   The reason is that the UserAccount is currently held in memory, but we need to actually persist it to the database.  To do so, we call the save() method in the UserAccount object like so:

<cfscript>
UserAccount.save();
</cfscript>

Now if you look in the table you will see our new record.  Pretty cool!

<infomercialAnnouncerVoice>
But wait… there’s more!
</infomercialAnnouncerVoice>

Often times when we do some sort of insert action like this, we need to do something with the primary key value that is created on insert.  Typically this is done with some sort of “SELECT max(UserId) …” query, or doing a “SELECT @@identity….” via a stored procedure.  Well as you can probably guess, Reactor makes this easy as well.  How do you do it?

You already did!

Your UserAccount object already holds the new UserId value, so if you wanted to access it, you could call it like this:

<cfoutput>#UserAccount.getUserId()#</cfoutput>

By this point you should be able to plainly see the ease and speed at which you can not only read, but create new records as well.

So say we are now on a new page and would like to change ‘dshuck’s password to something tricky like “password”.  First we would need to load the record.  Once it is loaded we would alter just the password property, then save the document.

Here it is in code, and let’s assume the UserId PK value of UserAccount is 3:

<cfscript>
// create local instance of the Reactor Factor
Reactor = application.Reactor;

// load the user ‘dshuck’ as an instance of the UserAccount class
UserAccount = Reactor.createRecord(“UserAccount”).load(UserId=3);

// change the password
UserAccount.setPassword(“password”);

// persist it to the database
UserAccount.save();
</cfscript>

So there you go.  Altering property state in 4 lines.  Of course, if you are familiar with OOP in general this isn’t exceptionally groundbreaking.  What is groundbreaking however is that still, we have not coded a single CFC.  That is just ridiculous!

Well, the time has come to say goodbye to our friend ‘dshuck’.  Let’s say, we are now on a new page and have been passed ‘dshuck’s UserId.  As I am sure you are guessing by now, we are not in for an enormous workload to make that happen.   So…  here we go:

<cfscript>
// create local instance of the Reactor Factor
Reactor = application.Reactor
Reactor.createGateway(“UserAccount”).delete(UserId=3);
</cfscript>

And with that we have now Created, Modified, and Delete with ease.  Reactor has many more features, including Data Gateways to access multiple records and return query record sets.  I am still just scratching the surface myself, but what I have seen so far has been very impressive.

I am sure this won’t be the last time you hear me talk about it!

Maddening server problem… Resolved!!!

Starting about 4-5 days ago, one of our web servers began experiencing nightmarish performance issues. This was such a strange problem to deal with, and I wasn’t able to Google anyone having the same issue, so I thought that I better document this in hopes that it may serve someone in the future.

Our ColdFusion MX 6.1 server would run for a while, the suddenly requests would just stop returning results at all. Occasionally a user would finally get a response to the browser that was a ‘<’ sign, followed by varying strings of seemingly random characters. It would hang like this failing to answer any future requests until the services were restarted. This system has some obvious suspect trouble points. I will list a few:

  • About 120 remote (and I mean very remote) DBs, and 1 DB on the same subnet. Those remote databases are housed in various data centers and client sites.
  • Of the remote DBs, over half are using and ODBC Socket to Sybase (Yes, JConnect is a much better idea and is in the works).
  • Of the remote DBs, a handful of them are using VPN connections.
  • A proprietary framework has been used on a couple of the applications on the server that has a number of examples of questionable coding practice in it.
  • A new company was recently migrated to the proprietary framework application that has 40,000 users.
  • Client variables were being used and were being stored in the registry.

I viewed all of the above as places that could potentially have some type of effect on the problem we were experiencing.

Any guesses so far?

Well for starters we had Fusion Reactor on the system.  We had it set to notify us when a request took longer than 30 seconds, and kill that request gracefully.  There was no consistency whatsoever to the pages in the requests that it as notifying us on.  Even simple ‘welcome’ pages would be in that list.  This didn’t appear to be helping us on this problem so to isolate the issue I removed it from the server.

We finally got on the phone with Macromedia…errr.. Adobe and dealt with Ted Zimmerman (and later Swathi C.), who went out of his way to help us and even stayed well after he was supposed to have left for the day. One of the first things we did was to change the Client Variables setting in CF Admin so that ColdFusion stored client variables in a DB rather than the registry. Also since we have no real need for persisting client variables through multiple sessions, we set ColdFusion to store them for 1 day as opposed to the 90 day default. This didn’t seem to fix anything and if anything our problem just seemed worse and worse. Instead of the uptime for 10-20 minutes after restarting, if anything, it seemed to be bombing more and more rapidly byt he minute.

With Ted’s assistance, I learned a great new technique for viewing threads at a very low level and even being able to track process IDs over a period of time and seeing if they had hung at a particular memory address. We did this by running ColdFusion from the console and outputting to a text file. That is a very interesting and informative exercise!

We kept finding a process that was talking to a remote database that seemed to be just hanging indefinitely. I did some better trapping on that page by not only catching database exceptions and logging them, but by adding in a timeout to the query, which is something I have not typically done in my code. I found that by doing this I no longer saw that particular page in the metrics information as a trouble maker, but our greater problem still existed. Now, in our thread stack dumps, another page seemed to show up pretty frequently. I put the same traps around that process and still remained in the same boat. That said, even though we didn’t fix the problem with that, we did fix a couple of issues that were definitely in need of attention.

So, any guesses yet?
I need to preface this next section by stating that the way that this company is structured and due to the fact that the systems store extremely sensitive data related to both loan origination and loan servicing, I do not have any direct access to the production system under normal circumstances, and even in this circumstance, I have no access to the database server at all. (Ironic when you consider the power of code and how counterproductive this approach can be, but I will save that for another discussion).

That said, in the middle of all of this work, we received an email from our network center that the SQL Server had reported an error notification of less than 1% of memory available. I assumed that could have just been a weird spike that had happened, but nonetheless out tech group looked into it immediately. We found that our SQL Server was showing that it had 16MB of available physical memory. As if that wasn’t a big enough “Uh Oh!”, we were told that it only had 300MB of space available on the C drive which (against my wishes) is where the virtual memory resides.

Upon learning this, our network tech immediately robbed memory from an internal development server and drove down to our NOC to upgrade the memory. Immediately the system went back to normal and has been blazing ever since!

It was interesting thinking back to when we altered the client variables setting. The change didn’t seem obvious at the moment, but in hindsight, it definitely exasperated the problem, although it was clearly the right thing to do.

So, now… 18 hours after the memory upgrade, things are still rocking!