Aug 06

Update: November 3, 2008: I gave up on trying to migrate my thermd deployment to the NAS server and instead assembled a new low power home server.

My goal was to move my home temperature reporting system from an older power hungry PC to my QNAP TS-109 Pro NAS server, a device which has an advertised power consumption of only 14W at full speed.  (Since I have an attached external drive for real-time mirroring, my consumption is more than 14W, but likely way below a full on PC).  I suspect I can save a fair amount of money through decreased power consumption and heat generation by decommissioning this extra PC.

My home monitoring system uses Dan Klein’s most excellent thermd utility to poll an HA7Net 1-wire host adapter which in turn talks to a bunch of 1-wire temperature and humidity sensors spread throughout the house.

To accomplish this goal, I had to:

  1. Get all of the dependencies required by thermd running on my NAS server
  2. Get the Perl-based thermd monitoring application running on my NAS server
  3. Set up scheduled publishing of home systems reports to our public web site

This article covers steps 1 and 2 of the project.   In a later post I’ll cover step 3 which involves setting up recurring jobs on the NAS server.

Although those are the top level tasks, there were a lot of details involved in this effort. I also took the opportunity to upgrade to the latest version of thermd. Newer versions have several features that I will likely use in the near future.

One of the purchasing requirements for my home NAS server was some degree of extensibility. I knew that I wanted to consolidate some long running applications such as the home systems monitoring application on the device. During my review of the QNAP NAS server, I discovered that there was the ability to obtain Perl and other popular open source software via a set of package repositories.  So I started with some optimism that turned out to be well founded.

Where to Start?

Although the QNAP is running a form of Linux and there are many methods of adding open source software to Linux systems, I wanted to take the easiest route possible to add the software required by the Perl-based thermd application.  One fly in the ointment of using pretty much any super low power consuming device is that commodity x86 processors are the exception.  ARM and other non-x86 processors are the norm for small, low power devices such as home NAS servers and wireless access points/routers.  Which means that all of those x86 binaries aren’t applicable to my circumstances. The good news is that there’s quite a cottage industry of people running custom Linux distributions and packaging systems on these devices.

Over the past year the QNAP forums had included instructions about how a software packaging system and environment could be installed on the QNAP. I had started following these instructions, but I eventually realized that a recent software update for the QNAP includes a supported version of the packaging system. With the v2.1.0 firmware update QNAP has introduced a meta software delivery and management system call QPKG. The packaging system of interest to me is one of the applications available via the QPKG feature.

1. Install Latest Firmware to Obtain QPKG Support

Upgrade your QNAP to the latest firmware.

2. Install Optware iPKG

After updating your QNAP to the latest frimware, you should see a “QPKG” entry at the bottom of the “System Tools” menu. Click on the “QPKG” menu item and then click on “Get QPKG”.

Now you’ll see a web page displayed with a list of QPKG applications. One of the applications is:

“Optware IPKG (Itsy Package Management System)”

Click on this entry and download the file associated with your QNAP server model.

Once you’ve unzipped the download, go to the “System Update” menu item and upload the .qpkg file to your server.

After installation has completed, click on the new QPKG entry for the Optware iPKG software. Click “Enable” to activate the application.

What’s this Optware Stuff?

The Optware package system is a set of software package repositories containing content packaged and installed using the iPKG technology. These package repositories were initially set up by a community of Linksys NSLU2 NAS server (aka “Slug”) users to enable them to easily obtain and install many open source software components on open Linux versions of their NAS servers. Not surprisingly, Linksys’ version of Linux hasn’t kept as up-to-date as the overall Linux community. This limitation couple with a lack of flexibility in the software content and extensibility of the pre-installed Linux-based NAS OS, compelled a community to form and deliver a range of newer, more flexible Linux distributions for this popular NAS device. The iPKG technology and the Optware package repositories provide a flexible package management facility for these replacement Linux distributions.

Given the similarity of the QNAP NAS servers to the NSLU2, QNAP users are able to benefit from the investments made by the NSLU2 community. Based on using the chroot technology of Linux and the example of the NSLU2 custom Linux distros, Andy was able to predefine what amounts to a lightweight Linux operating environment that is installed on top of the existing QNAP Linux OS. He has pre-configured that environment to act as an iPKG client to a specific iPKG package repository. The repository contains hundreds of useful packages. As a result of the installation instructions below, the following package repository is made readily available:

http://ipkg.nslu2-linux.org/feeds/optware/cs05q3armel/cross/stable/

Other repositories are available. The IpkgFind search facility enables users to easily find and obtain packages of interest from the repositories. Since package often contain binaries that are compatible with certain processor architectures, care must be exercised when attempting to pull packages from other repositories. For example, the QNAP TS-109 NAS server uses an ARM processor. Binaries for Intel-based processors will not work on this system.

The chroot Environment

The Optware iPKG installation results in “chroot” environment being installed in the user data portion of the internal drive:

/share/HDA_DATA/optware

A chroot environment is basically a semblance of an installation of the operating system. In this example, the QNAP’s underlying Linux kernel and other low level software is still used, but all applications installed via the iPKG system are installed in the chroot environment. On a small appliance such as this NAS server, it’s important to avoid overloading the real OS partition with applications that aren’t absolutely necessary for the operation of the NAS server. There’s only 500 MB or so allocated to house theQNAP’s underlying operating environment. That space would be eaten up fairly quickly as applications are added to it. Use of a choort environment housed in the user data portion of the internal drive opens up gigabytes of space for you to add applications.

A nice side effect of installing applications in a chroot environment that is housed in the user data portion of your drive is that, if you’re using the Q-RAID1 feature, the entire choot environment is automatically backed upto the external drive along with your other data.

Although ipkg and at least some of the other commands installed in the chroot environment may be executed either inside or outside of the chroot, you may find that some commands fail to run properly when executed outside of the chroot due to library incompatibilities between the native QNAP Linux and the libraries installed in the chroot environment. To be on the safe side, I execute chroot whenever I manage the content of the chroot environment and I schedule applications to run from the chroot area of the filesystem.

3. Fix Some Gaps in the chroot Environment

Although many of your root environment’s settings are carried over to the new chroot environment, I found a few settings that I had to manually synchronize.

Ensure chroot environment has same timezone setting as main root:

# cp  localtime  /share/HDA_DATA/optware/etc

Also, some of your network-based applications won’t work properly without a protocols file under /etc/protocols:

# cp /etc/protocols /share/HDA_DATA/optware/etc/protocols

4. Access the chroot Environment

After logging into the QNAP, execute the chroot command as follows:

# /usr/sbin/chroot /share/HDA_DATA/optware /opt/bin/bash

Now you’re in the chroot environment.  Note that your PATH include /opt/bin/ before some of the other areas.

As a result of the installation of the Optware iPKG environment, you’ll notice some garbage entries in the root of the chroot environment.

You can use Midnight Commander to remove these files:

# mc

Within Midnight Commander, browse to the root directory and select each of the garbage files. Then press the F8 key to delete each file.

5. Update the iPKG Database

The following command reloads the catalog of available packages:

# ipkg update

BTW, the remote package feeds or repositories are listed in this file:

/opt/etc/ipkg.conf

Let’s look at the list of packages that were installed as part of the set up of the iPKG installation:

# ipkg list_installed
bash – 3.2.33-1 – A bourne style shell
e2fslibs – 1.40.3-5 – Ext2 Filesystem Libraries
e2fsprogs – 1.40.3-5 – Ext2 Filesystem Utilities
glib – 2.12.13-1 – The GLib library of C routines.
mc – 4.6.1-3 – Midnight Commander File Manager
readline – 5.2-2 – The GNU Readline library provides a set of functions for use by applications that allow users to edit command lines as they are
slang – 2.1.3-1 – S-Lang is a multi-platform library designed to allow a developer to create robust multi-platform software.
Successfully terminated.

Running the following command will show you the packages that are available in the network package repository:

# ipkg list

Performing a grep on this output is an easy way to find packages of interest to you.

6. Install thermd and its Dependencies

Since thermd is a Perl application, the first package to install is perl.

Installing Perl Using ipkg

Let’s install Perl and the packages on which it depends:

# ipkg install perl
Installing perl (5.8.8-17) to root…
Downloading http://ipkg.nslu2-linux.org/feeds/optware/cs05q3armel/cross/stable/perl_5.8.8-17_arm.ipk
Installing libdb (4.2.52-3) to root…
Downloading http://ipkg.nslu2-linux.org/feeds/optware/cs05q3armel/cross/stable/libdb_4.2.52-3_arm.ipk
Installing gdbm (1.8.3-2) to root…
Downloading http://ipkg.nslu2-linux.org/feeds/optware/cs05q3armel/cross/stable/gdbm_1.8.3-2_arm.ipk
Configuring gdbm
Configuring libdb
Configuring perl
Successfully terminated.

Although this screen capture shows Perl 5.8.8-17 being installed, I had to install -18 in order for networking to work properly via Perl.  The iPKG repositories have the -18 version as of a week or so ago.  -17 will not work properly.

Install Perl Modules Required by thermd
The first Perl module dependency to resolve is HTTP::Date. Fortunately, the iPKG repository already has this package:

# ipkg install perl-libwww

Unfortunately, the other Perl modules required by thermd are not available in the iPKG repository. An alternative method of obtaining Perl modules is to use the CPAN facility.  CPAN is a Perl-oriented repository of source level modules.  Via the cpan command, you can “install” a wide variety of Perl modules.  Installation entails downloading, compilation and installation of the resulting binaries and scripts to the proper location.

However, in order to run cpan, you need to install some more packages from the iPKG repository:

# ipkg install tar

# ipkg install gcc

# ipkg install unzip

# ipkg install make

# ipkg install lynx

# ipkg install ncftp

# ipkg install gnupg

# ipkg install perl-libwww

# ipkg install perl-gd

# ipkg install imagemagick

# ipkg install ghostscript\

# ipkg install cairo

# ipkg install vim (built-in vi is flaky – perhaps due to my terminal settings)

Resulting Packages Installed

After installing the packages listed above and troubleshooting the use of Perl’s CPAN utility and trying out various thermd features, here’s the set of packages I currently have installed:

# ipkg list_installed
bash – 3.2.33-1 – A bourne style shell
binutils – 2.17-2 – The GNU assembler and linker and related tools
bzip2 – 1.0.5-1 – Very high-quality data compression program
c-ares – 1.5.2-1 -

cups – 1.3.8-1 – Common Unix Printing System
cyrus-sasl-libs – 2.1.22-2 – Provides client or server side authentication (see RFC 2222).
diffutils – 2.8.1-6 – contains gnu diff, cmp, sdiff and diff3 to display differences between and among text files
e2fslibs – 1.40.3-5 – Ext2 Filesystem Libraries
e2fsprogs – 1.40.3-5 – Ext2 Filesystem Utilities
expat – 2.0.1-1 – XML Parser library
fontconfig – 2.5.0-0 – Font configuration library
freetype – 2.3.6-1 – Free truetype library
gcc – 3.4.4-4 – The GNU Compiler Collection.
gconv-modules – 2.3.6-1 – Provides gconv modules missing from the firmware.  These are used by glibc iconv() implementation.
gdbm – 1.8.3-2 – GNU dbm is a set of database routines that use extensible hashing. It works similar to the standard UNIX dbm routines.
gdchart – 0.11.5dev-1 – Easy to use C API for create charts and graphs in PNG, GIF and WBMP format.
ghostscript – 8.62-1 – An interpreter for the PostScript (TM) languageglib – 2.12.13-1 – The GLib library of C routines.
gnupg – 1.4.9-1 -
gzip – 1.2.4a-3 – GNU Zip data compression program
imagemagick – 6.4.1.8-1 – A set of image processing utilities.
libc-dev – 2.3.6-3 – libc development files.
libcurl – 7.18.2-3 – Curl is a command line tool for transferring files with URL syntax, supporting FTP, FTPS, HTTP, HTTPS, GOPHER, TELNET, DICT, FI
libdb – 4.2.52-3 – Berkeley DB Libraries
libgd – 2.0.35-2 – An ANSI C library for the dynamic creation of images
libgpg-error – 1.4-1 – Error handling library for libgcrypt
libjpeg – 6b-2 – collection of jpeg tools
liblcms – 1.15-1 – A small-footprint, speed optimized color management engine.
libnsl – 2.3.6-4 – Network Services Library
libpng – 1.2.29-1 – Portable Network Graphics Libraries
libstdc++ – 6.0.3-6 – Standard C++ library, needed for dynamically linked C++ programs
libtiff – 3.8.2-1 – Tag Image File Format Libraries
libtool – 1.5.24-1 – Library tools.
libusb – 0.1.12-1 – Library for interfacing to the USB subsystem.
lynx – 2.8.6-1 – A text browser for the World Wide Web
make – 3.81-2 – examines files and runs commands necessary for compilation
mc – 4.6.1-3 – Midnight Commander File Manager
md5deep – 3.0-1 -
ncftp – 3.2.1-1 – Nice command line FTP client
ncurses – 5.6-3 – NCurses libraries
openldap-libs – 2.3.38-1 -
openssl – 0.9.7m-4 – Openssl provides the ssl implementation in libraries libcrypto and libssl, and is needed by many other applications and librari
perl – 5.8.8-18 – Practical Extraction and Report Language.
perl-compress-zlib – 1.42-2 – Compress-Zlib – Interface to zlib compression library.
perl-digest-perl-md5 – 1.8-1 – Digest-Perl-MD5 – Perl implementation of Ron Rivests MD5 Algorithm
perl-digest-sha – 5.47-1 – Digest-SHA – Perl extension for SHA-1/224/256/384/512.
perl-digest-sha1 – 2.11-3 – The module allows you to use the NIST SHA-1 message digest algorithm.
perl-gd – 2.35-2 – GD – Interface to Gd Graphics Library
perl-html-parser – 3.55-2 -
perl-html-tagset – 3.04-4 – This module contains data tables useful in dealing with HTML.
perl-libwww – 5.805-4 – libwww-perl – The World-Wide Web library for Perl
perl-uri – 1.35-3 – URI – <module_description>
procmail – 3.22-3 – Versatile email processor.
readline – 5.2-2 – The GNU Readline library provides a set of functions for use by applications that allow users to edit command lines as they are
slang – 2.1.3-1 – S-Lang is a multi-platform library designed to allow a developer to create robust multi-platform software.
tar – 1.20-1 – heavyweight version of the Tape ARchiver
unzip – 5.52-3 – A (de)compression library for the ZIP format
vim – 7.1-1 – Yet another version of the vi editor.
zlib – 1.2.3-3 – zlib is a library implementing the ‘deflate’ compression system.
Fix /opt/bin/ld Issue

Some of the Perl modules that contain native code require the “ld” command to produce binaries.  Based on the installation of the gcc package or binutils (I forget which one), the “arm-none-linux-gnueabi-ld” command gets installed, but there’s no symlink to a companion “ld” command.  Hence the makefiles executed by through the CPAN process can fail when the “ld” command is not found.

# cd /opt/bin
# ln -s ld arm-none-linux-gnueabi-ld

Install Rest of the Perl Modules

Here are the Perl modules I ended up installing via CPAN:
# cpan

The first time cpan is run, you will need to configure it.  Choose the manual option such that you can override the default install locations of the necessary tools.  Often, you will want to specify /opt/bin/… rather than /usr/bin/.  Since I could not find a general purpose ftp program, I did not specify one in the cpan configuration.

Here’s what my cpan configuration looks like:

cpan> o conf
CPAN::Config options from /opt/lib/perl5/5.8.8/CPAN/Config.pm:
commit             Commit changes to disk
defaults           Reload defaults from disk
init               Interactive setting of all options

build_cache        10
build_dir          /root/.cpan/build
cache_metadata     1
cpan_home          /root/.cpan
dontload_hash
ftp
ftp_proxy
getcwd             cwd
gpg                /opt/bin/gpg
gzip               /opt/bin/gzip
histfile           /root/.cpan/histfile
histsize           100
http_proxy
inactivity_timeout 0
index_expire       1
inhibit_startup_message 0
keep_source_where  /root/.cpan/sources
lynx               /opt/bin/lynx
make               /opt/bin/make
make_arg
make_install_arg
makepl_arg
ncftp
ncftpget           /opt/bin/ncftpget
no_proxy
pager              /bin/more
prerequisites_policy ask
scan_cache         atstart
shell              /bin/sh
tar                /opt/bin/tar
term_is_latin      1
unzip              /opt/bin/unzip
urllist
ftp://mirror.nyi.net/CPAN/
ftp://mirror.rit.edu/CPAN/
ftp://mirror.sit.wisc.edu/pub/CPAN/
wget               /bin/wget
Here are the modules I installed via CPAN:

cpan> install Config::General
cpan> install Digest::CRC
cpan> install LWP::UserAgent
cpan> install GD
cpan> install GD::Graph
cpan> install Image::Magick

Download thermd

# cd /opt/bin

# wget http://www.klein.com/thermd/thermd

Edit the thermd script to refer to the location of perl:

#!/opt/bin/perl

Make thermd Publishing Area Accessible to chroot

All of the data generated by the therm application will be written to one of my existing file shares on the NAS device. The advantage of doing so is that all of the data will automatically be backed up and readily accessible to other systems on our network.

This share needs to be accessible in the chroot environment.

In root environment, edit:

# vi /etc/mtab

Add the following line:
/share/HDA_DATA/documents /share/HDA_DATA/optware/mnt/ext/documents none rw,bind 0 0

And then mount the share:

# mount /share/HDA_DATA/optware/mnt/ext/documents

(In my next blog, I’ll show how to preserve this setting across reboots).

Modify thermd.conf

I modified my thermd.conf file to override some of the default filesystem locations for various dynamically generated file:

RSS feed location: <RSS /mnt/ext/documents/home-systems/www/thermd>

PIDFile: /opt/var/run/thermd.pid

LogWrite:    /mnt/ext/documents/home-systems/thermd

Modify thermd Program

Remove references to “nice” command: Even though the NAS server is running a form of Linux, not all of the commands available in a typical Linux installation are present.  For example, the thermd script refers to the “nice” command in several areas.  I had to remove references to this command because it is not present on my NAS server.  Nor is it available in the iPKG package repository.

Avoid Startup Issues: For now, since thermd would immediately fail upon startup, I am using the “-nofork” option when starting thermd:

# ./thermd -daemon -nofork -config /opt/etc/thermd.conf

When I get some time, I’ll troubleshoot that problem further.

Modify Timeout Values When Reading Sensor Values: Although I was able to get thermd to start up properly, the application would die after running for a few minutes or a few hours. After much debugging, I finally found a workaround that seems to address the problem:

The “my_select()” function includes two timeout values:

alarm(3)

and

unstick() if time – $start > 1;

I modified these to be:

alarm(40)

and

unstick() if time – $start > 30;

Basically, providing more time for the code draining the file descriptor containing sensor readings to wait on the process that is reading data from the HA7Net and writing sensor readings to the shared file descriptor. In my environment, it might take 20-30 seconds for the polling process to receive a response from the HA7Net device.

7. Run thermd!

# ./thermd -daemon -nofork -config /opt/etc/thermd.conf &

Next Steps: Scheduling Custom Jobs

My next project is to schedule thermd to start automatically when the NAS server starts and to schedule file transfer jobs to send graph image files up to our public web site.  The goods news is that the following site seems to provide some useful tooling to make this happen:

http://forum.qnap.com/viewtopic.php?f=94&t=1219

http://technologytreason.blogspot.com/2007/11/adding-cron-jobs-to-qnap-server.html

Leave a Reply

preload preload preload