Thursday, March 29, 2007

Template reports

Today I wrote a reporting tool for our Web 2.0 project. I wanted a simple solution that enabled the end customer to create his own reports based on the ones we creates during development.

To make the system flexible I used the Smarty template engine. Smarty works by compiling templates describing layout into PHP files that is when combined with data from an application into a presentation. The normal use of Smarty is to create HTML pages for viewing in a browser, but the code is well written and can be used in many more ways.

I also wanted the SQL queries to be configurable, to allow even more flexibility in the design of the reports. The solution was to use a feature in Smarty that is normally used to avoid hard coding in the templates, the config files. In my solution the report is a config file that specifies the content of the report. A simple report looks like this:



name = Users
desc = This report list all active users in the system.

[list]
Users = SELECT iuser.cn FROM iuser WHERE istatus = 'active'

[screen]
users = users_table.tpl

[print]
head = print_header.tpl
users = users_table.tpl
foot = print_footer.tpl

[excel]
users = users_table.tpl

The application can easily build a list of all the reports in the system by looking at all the files *.rep in the reports directory, using the glob PHP function. The reports supports to be rendered in three different ways, screen, print or excel.

To init the Smarty package, I use:

$smarty = new Smarty;
$smarty->compile_check = true;
$smarty->debugging = false;

$smarty->config_dir = "reports";
$smarty->template_dir = "reports";

$conf = new Config_File("reports");

And to get information about a specific report, I use the Smarty Config_File class to query the parameters from the .rep file:

$res["name"] = $conf->get($report_file,null,'name'); // Get report name
$res["desc"] = $conf->get($report_file,null,'desc'); // Get report description

To generate the report, the PHP code fetches and executes the different queries configured in the report. Two different kinds of queries are supported, queries returning lists and queries returning a single value.

$lists = $conf->get($report_file,'list');
foreach($lists as $name => $query) {

$res = iDbSelect($query);

$smarty->assign($name,$res);
}

$vars = $conf->get($report_file,'var');
foreach($vars as $name => $query) {

$res = iDbSelectOne($query);

$smarty->assign($name,$res);
}

After all the data has been read in from the database, Smarty is called to render the report. The $show variable holds the type of report to generate; this could be screen, print or excel. This makes in possible to have some parts of the report generated the same way independently if the report is intended for the printer, screen or for Excel.

$displays = $conf->get($report_file,$show);
foreach($displays as $display) {
$smarty->display($display);
}

By using this solution we can easily design reports made up from smaller parts that the customer later will be able to reuse in designing his own custom reports.

Sunday, March 25, 2007

History reloaded

My last post was about Really Simple History and how to get it to work in IE7. I forgot to mention the problems you easily get then using dhtmlHistory.js and IE if you serve your pages from a script language. This is because for dynamic history to work, IE must preserve the content of a form between page reloads. This can only happend if the web server returns a status code 304 (Not modified) on the page access.

In my case I used PHP as the server side script language to build my pages and had problems getting F5 to reload the page and stay on the current page. After a page reload, dhtmlHistory.js considered it a first load and the home page was shown again.

The solution I found after a while was to check if any of the pages had been modified and return the 304 status code from PHP. I also use an ETag header to make the checking for modifications simple.

The index.php file creates the HTML by listing all page_*.php files in a directory. For this I use the glob PHP function. After that I check for the file that was modified last, this is because this time will be used as modification date for the generated page. Then serving the generated page an ETag header will also be generated. This ETag will be constructed by using the md5 PHP function on the modification time.

On hitting F5 the web browser will send this ETag back in a header (if-none-match) to check for modified content. In PHP this header will be called $_SERVER["HTTP_IF_NONE_MATCH"] and the code checks if the current modification time is the same, if so a 304 status is returned.

Here is the code from index.php:


$pages = glob("page_*.php");

$mtime = filemtime("index.php");
foreach($pages as $page_file) {
$m = filemtime($page_file);
if ($m > $mtime) {
$mtime = $m;
}
}


$etag=(isset($_SERVER['HTTP_IF_NONE_MATCH']))? $_SERVER['HTTP_IF_NONE_MATCH']:"";

$etime = $mtime + 3600; // How long can a cache server save this content without asking again? (1 hour)

if ($etag == md5($mtime)) {
header('HTTP/1.0 304 Not Modified');
die;
}

header("Last-Modified: " . gmdate('D, d M Y H:i:s', $mtime) . ' GMT');
header("Expires: ". gmdate('D, d M Y H:i:s', $etime) . ' GMT');
header("ETag: " . md5($mtime));

Thursday, March 22, 2007

History restored

I currently work on a Web2.0 project for a customer where we use AJAX extensively. We download all of the HTML code and JavaScript from index.html and every part of the application after that are implemented using DHTML and AJAX requests to the server.

To give the user the standard Web experience we are using a JavaScript library called Really Simply History. By using this library we can control that will happen if the user presses the back button in the browser.

The library worked great until IE7 started to appear. Suddenly we got strange "white-outs" of the pages. Then clicking a button or a link all HTML of the page was removed and only two buttons of the UI was left.

After some digging using Google I found a post in Spanish describing that went wrong and a fix. The problem seems to be that dhtmlHistory.js breaks the DOM in some way. I implemented the fix suggested by jorgemaestre and it worked, the whiteouts was removed.

But instead I got other problems. Pressing F5 for a page reload did not work as expected, after hitting F5 the site reloaded but I was taken back to the home page. This means the history function did not work. I did some debugging and discovered that for dhtmlHistory.js to work, the form field values needed to be preserved. I also found another post on the Internet about IE not saving the values of a form if it's created after the page has loaded.

The solution is to let the dhtmlHistory.js create it's form during normal page loading but as the last part of the page, after all other content. To implement the fix:

1) Open the dhtmlHistory.js and remove the last rows saying:


/** Initialize all of our objects now. */
window.historyStorage.init();
window.dhtmlHistory.create();


2) To your main index.html or whatever that's using dhtmlHistory.js add as the last part before </body>


</script>
/** Initialize all of our objects now. */
window.historyStorage.init();
window.dhtmlHistory.create();
</script>

Monday, March 19, 2007

IE7 does not Excel

The statistical module on one of our web applications has a function to download the data directly into Microsoft Excel. This function has been working like a charm since the application was deployed last year, but now we got problem reports from IE7 users that were unable to use the function.

I did a quick test and found no problem, a not so uncommon situation for anyone involved in software development. I thought about reporting the usual "It works in development" back to the customer when I figured I should do another test using the full test environment. I now got the reported failure; IE7 fails to download the file and a popup is shown instead. Aha! Its HTTPS that's causing this.

After some digging around I found the answer in the header() documentation on php.net. To get file downloads to work for IE7 you must add two more headers to the response:


header('Pragma: private');
header('Cache-control: private, must-revalidate');

This will tell IE7 to allow the encrypted file to be saved locally, a requirement to be able to open the file using Excel. I have not found any pointers about why the functionality was changed between IE6 and IE7.

I hope this saves an hour for you!

Friday, March 16, 2007

Hard disk slowfox

I've had some problems for the last weeks with the performance of my laptop and was looking around to buy a new one. But since it's a lot of comparing to do, to decide witch one to buy I was not able to find the perfect one so I started to investigate that was causing this. I ignored the usual helpfull comments about Windows slowing down over time, and questions about when I last reinstalled Windows.

I did the standard scans using Ad-Aware and did not find anything alarming. I removed all unneeded startup items from the registry and the start-up folders and stopped all services I could be without, but the computer still felt like an old 386. I was near or at 100% CPU utilisation without getting much work done, something must be broken.

I downloaded a performance measurement tool called PerformanceTest 6.1 to try to get a measurement of how slow my laptop was, and indeed it was slow. I got 1.2MB/s transfer rate to my hard disk. That is not much. I tried the program on my other machine and got 40MB/s. Now I was convinced something was terrible broken.

After a couple of hours of scanning for rootkits and examining all class filters for the hard disk device stack my brother suggested that I checked the DMA settings on the IDE channel. I said sure, let's check, because DMA is always active and I did not suspect it could be disabled. I was running my hard disk on PIO-mode! How could that be?

After another hour of investigations I had found the Microsoft Knowledge Base article KB817472 that explains the problem. If the driver receives errors from the hard disk, it tries to lower the transfer speed over the IDE channel. The real problem starts then using a laptop and you only hibernate the system since you are likely to get some errors then starting up again. My machine had lowered the transfer mode to the lowers PIO-mode where the CPU is used to output a byte at the time to the hard disk, at high cost of CPU cycles and very slow throughput.

The "fix" to this problem was simple, just delete the Primary IDE channel in device manager and reboot, and the system will plug and play the device back again with restored performance. Running PerformanceTest again gave me 25MB/s to the disk.

Microsoft has also added and option to disable this "feature" in the driver, and I think you can guess my setting...

Thursday, March 15, 2007

LiveUpdate blues

Today i decided to do a backup of my laptop; it had been a while since the last time. So I enabled all the Symantec services and started Norton Ghost to get the backup running. I also decided to check for updates at the same time, just to be sure. I launched LiveUpdate from the Norton Ghost main menu and it checked for updates and stopped with a cryptic:

LU1848: Couldn't create callback object

OK, lets ask Google. I got a lot of hits. But the common recommendation seems to be to uninstall all Symantec software and reinstall it again. A lot of people also recommended to ignore the second step and buy something different.

I thought this was a good opportunity to test out our next product. Application Inspector. Application Inspector is an easy to use merge of regmon and filemon, built for non developers. It works by recording a lot of systems API calls the application does and checks for errors. I started the luall.exe using application inspector and after fixing a couple of bugs (it's still in development) I got a result. LiveUpdate was looking for a COM object with the CLSID of:

{DBBC1D05-9B24-42BE-9AB9-EDFEB039806A}

A quick search using Google this CLSID gave my the faulting Symantec product. It should be pointing toward a SymIDSLU.LUCallback object that was missing from my system. I had noticed before a reference to an IDS program in the list of application that LiveUpdate was trying to update. I think this must be a leftover from an old Norton Internet Security installation that Dell shipped with the computer.

OK, now how to fix this? I had some time, the computer was just doing backup anyway and it was almost to slow to do any real work. I started to poke around among the LiveUpdate files and looked at the configuration files for LiveUpdate. But the Product.Inventory.LiveUpdate file that keeps a list of installed applications that LiveUpdate is supposed to update is encrypted or something, it's not readable. I when found the ProductRegCom_2_6.DLL module and started to look at the TypeLibrary embedded in that file. The API was simple and I soon had a small test program working to dump all the information I needed, and a small fix again I had that faulty application removed from LiveUpdate repository. Nice!

Here is the code:


#include "stdafx.h"

#import "c:\program files\symantec\LiveUpdate\ProductRegCom_2_6.dll" named_guids

using namespace PRODUCTREGCOMLib;

_COM_SMARTPTR_TYPEDEF(IEnumString, __uuidof(IEnumString));

int main(int argc, char* argv[])
{
HRESULT hr;

argc--;
argv++;

hr = CoInitialize(0);
if (FAILED(hr)) {
printf("Failed to init COM\n");
exit(1);
}

IluProductRegPtr p;

hr = CoCreateInstance(CLSID_luProductReg,NULL,CLSCTX_INPROC_SERVER,IID_IluProductReg,(void**)&p);
if (FAILED(hr)) {
printf("Failed to create LiveUpdate Product Reg object\n");
exit(1);
}


if (argc) {
hr = p->DeleteProduct(*argv);
if (FAILED(hr)) {
printf("Failed to delete product\n");
exit(1);
}
return 0;
}

IEnumStringPtr t;

t = p->GetProductMonikerEnum();

LPOLESTR prod;
LPOLESTR prop;

while(t->Next(1,&prod,NULL) == S_OK) {
printf("%S\n",prod);

IEnumStringPtr x;

x = p->GetPropEnum(prod);

while(x->Next(1,&prop,NULL) == S_OK) {
printf(" %S\n",prop);

VARIANT v;

VariantInit(&v);

p->GetProperty(prod,prop,&v);

printf(" %S\n",v.bstrVal);
}
}

return 0;
}