A picture worth 1000 MB

So many things about this picture make me happy, thanks to http://spot.fedorapeople.org/steam/ for the repo to make steam client installation painless.

Monitoring apache-status on aegir servers with nagios / check_mk

Apache server-status can produce interesting performance information that can be useful for server and application tuning, getting access to this information and graphing it with nagios is not terribly hard but add in check_mk and the Aegir platform and things get a little bit more complicated.

In the following steps I will demonstrate how to install the check_mk agent, install the check_apachestatus_auto.pl script and dependencies, add the proper stanza for server-status to aegir and finally add the check into mrpe (check_mk replacement for nrpe).

Assumptions: RedHat/CentOS/Scientific Linux, 64-bit, EPEL, Aegir, root access.

# Install check_mk agent:
*First install the check_mk agent rpm:

yum install http://mathias-kettner.de/download/check_mk-agent-1.2.0p3-1.noarch.rpm --nogpg

*We don't want just anyone to poll the data from check_mk, so modify the /etc/xinetd.d/check_mk by adding the ip of your nagios server to the 'only_from' line:

only_from     = 127.0.0.1 nagios_server_ip

*The check_mk agent operates through xinetd on port 6556, verify that xinetd will start at boot, and make sure it is currently running:

chkconfig xinetd on ; service xinetd start

*Hopefully you are running a firewall, to poke a hole in an iptables based firewall you can add a rule similar to:

-A INPUT -s nagios_server_ip -p tcp -m tcp --dport 6556 -j ACCEPT 

to /etc/sysconfig/iptables, then restart iptables with the service command:

service iptables restart

# Install the check_apachestatus_auto.pl plugin
* Install the nagios-plugins-perl rpm from EPEL, this will provide the /usr/lib64/nagios/plugins/utils.pm file, as well as creating a directory structure:

yum install nagios-plugins-perl

* Download plugin from http://blog.spreendigital.de/nagios/?#check_apachestatus_auto to /usr/lib64/nagios/plugins, modify it to find utils.pm in /usr/lib64/nagios/plugins:

wget -O /tmp/check_apachestatus_auto.tgz http://blog.spreendigital.de/wp-content/uploads/2009/07/check_apachestatus_auto.tgz
tar zxvf /tmp/check_apachestatus_auto.tgz -C /usr/lib64/nagios/plugins/
sed -i 's/\/usr\/local\/nagios\/libexec/\/usr\/lib64\/nagios\/plugins/g' /usr/lib64/nagios/plugins/check_apachestatus_auto.pl

# Modify apache to display server-status
* Typically you could enable server-status by un-commenting the correct stanza in /etc/httpd/conf/httpd.conf, but with an aegir system any get request for http://localhost/server-status will be fulfilled by aegir. If you do some digging you will fine the /var/aegir/config/server_master/apache/pre.d directory which is included before any virtual hosts, this is where you need to put a config file for server-status.

cat << EOF >> /var/aegir/config/server_master/apache/pre.d/nagios.conf
<VirtualHost *:80>
ServerName localhost
<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
</Location>
</VirtualHost>
EOF

* Reload apache to read your new config file:

service httpd reload

* Verify it's working using curl, this should dump the raw html from the server-status page to your screen:

curl localhost/server-status

# Setup mrpe to execute the plugin
* This is one of the easier steps, simply create /etc/check_mk/mrpe.cfg, and add a line with the check alias and location:

mkdir /etc/check_mk
echo "Apache_Status /usr/lib64/nagios/plugins/check_apachestatus_auto.pl -H localhost" >> /etc/check_mk/mrpe.cfg

# Now that we have installed the check_mk agent, the check_apachestatus_auto.pl script and the mrpe.cfg file you can re-inventory the node from your check_mk server, note the mrpe line in the following output:

cpu.loads         1 new checks
cpu.threads       1 new checks
df                7 new checks
diskstat          1 new checks
kernel            3 new checks
kernel.util       1 new checks
lnx_if            1 new checks
mem.used          1 new checks
mounts            7 new checks
mrpe              1 new checks
ntp.time          1 new checks
postfix_mailq     1 new checks
tcp_conn_stats    1 new checks
uptime            1 new checks

# Once you have reloaded nagios, check_mk will watch the server-status page and produce nice graphs like this:

Installing Dwarf Fortress and Dwarf Therapist on 64-bit fedora 16/17

Because losing is fun.

Dwarf Fortress:
 sudo yum install -y SDL{,_image}.i686 gtk2.i686 mesa-libGLU.i686 SDL_ttf.i686
Grab the latest linux pack for Phoebus with DF pre-installed from http://dffd.wimbli.com/file.php?id=2944, untar and you should be good to go.


Dwarf Therapist:
sudo yum install -y mercurial qt-devel gcc-c++
Checkout a copy of the Dwarf Therapist code:
 cd ~/Downloads ; hg clone https://code.google.com/p/dwarftherapist/
Build with qmake/c++
 cd ~/Downloads/dwarftherapist/ ; qmake-qt4 && make && sudo make install
Therapist should now be in your path, and runnable from any directory with the 'dwarftherapist' command.
The only way I have been able to get this to run lately is from the directory it was built in, and not using the wrapper script (which is 'dwarftherapist' in all lower case):
 cd ~/Downloads/dwarftherapist ; DwarfTherapist

AIX BoF

Home Directories
I have not been able to have home directories automagically created when logging in with LDAP, I have added many of the user home dir's to the default AIX build but you may run into some that don't exist, you can easily make you home directory using sudo.
If you are adding LDAP to a AIX box with local accounts already established it is best practice to remove the account for LDAP users. Before you run this command you must MAKE SURE THAT SECLDAPCLNTD IS NOT RUNNING, if it is still running you will remove all of the admin accounts from LDAP!:
stop-secldapclntd && for i in ; do rmuser -p $i ; done

Here are two one liners to create all the current Unix, App, and DB admins home folders:



start-secldapclntd
for i in ; do
 mkdir /home/$i;
 chown -R $i. /home/$i;
done

for i in ; do
 mkdir /home/$i;
 chown -R $i.IRM /home/$i;
done



AIX 5.3 and 6.1

# Installing and configuring LDAP client service on AIX 5.3/6.1
Installation
All packages are available on the /media/software nfs share, instructions reference that location for installation.

# Install Java
If java is not currently installed you can install Java5 64 bit, this version is for AIX 5.3 AND 6.1:

 installp -acgXYd /media/software/AIX/java/ Java5_64.sdk
Once java is installed you may need to add it to your path, the following command will do this temporarily:

 export PATH=/usr/java5_64/bin:$PATH


# Install gskit
Source: https://www14.software.ibm.com/webapp/iwm/web/reg/pick.do?source=gskitupdt&S_PKG

 installp -acgXd /media/software/AIX/GSKIT/ gsksa.rte gskta.rte


# Install ITDS packages
Remove any previous version of ldap that are installed, if the following command returns with filesets they need to be removed through smit:

  lslpp -aL *ldap* # will list installed ldap packages


Source http://www-01.ibm.com/support/docview.wss?rs=767&context=SSPREK&uid=swg27009778#ver62
6.2 packages

 installp -acgXd /media/software/AIX/ITDS/6.2.0.2-TIV-ITDS-AIX-IF0002/images/ idsldap.clt32bit62 \
 idsldap.clt64bit62 idsldap.cltbase62 idsldap.msg62.en_US idsldap.clt_max_crypto32bit62 \
 idsldap.clt_max_crypto64bit62
Once those filesets are installed run 'lslpp -aL idsldap.*' to verify, output should be similar to this:

 lslpp -aL idsldap.*
  Fileset                      Level  State  Type  Description (Uninstaller)
  ----------------------------------------------------------------------------
  idsldap.clt32bit62.rte     6.2.0.n    C     F    Directory Server - 32 bit
                                                   Client
  idsldap.clt64bit62.rte     6.2.0.n    C     F    Directory Server - 64 bit
                                                   Client
  idsldap.clt_max_crypto32bit62.rte
                             6.2.0.n    C     F    Directory Server - 32 bit
                                                   Client (SSL)
  idsldap.clt_max_crypto64bit62.rte
                             6.2.0.n    C     F    Directory Server - 64 bit
                                                   Client (SSL)
  idsldap.cltbase62.adt      6.2.0.n    C     F    Directory Server - Base Client
  idsldap.cltbase62.rte      6.2.0.n    C     F    Directory Server - Base Client
  idsldap.msg62.en_US        6.2.0.n    C     F    Directory Server - Messages -
                                                   U.S. English (en)



# Configuration
methods.cfg
Paste the following into a root cli: (this assumes that you use TSM for backups and that the client is configured)

 if [ $(grep -c LDAP /usr/lib/security/methods.cfg) -eq "0" ];
  then
   echo "Backing up current methods.cfg, please wait"
   dsmc i /usr/lib/security/methods.cfg
   cat <>/usr/lib/security/methods.cfg

LDAP:
        program = /usr/lib/security/LDAP
        program_64 = /usr/lib/security/LDAP64
EOF
 else
   echo "Found LDAP stanza in /usr/lib/security/methods.cfg, assuming that it is correct."
 fi


CA Certs
Get certs from the repos

 cd /etc/security/ldap/ && wget .crt && wget .crt

Create and list key database for ssl, if you run into java errors check that your version of java is at least 1.4.2

 gsk7cmd -keydb -create -db /etc/security/ldap/key.kdb -pw -type cms
 gsk7cmd -cert -list CA -db /etc/security/ldap/key.kdb -pw

Add CA cert from to key database

gsk7cmd -cert -add -db /etc/security/ldap/key.kdb -file /etc/security/ldap/.crt -format ascii -label " CA cert" -pw -trust enable


# mksecldap command / ldap.cfg file
Run the mksecldap command to tie the system into LDAP, this may take a minute or more.
The mkseclsap command cannot be used to setup anonymous bind configurations, instead of using that command create the /etc/security/ldap/ldap.cfg using the following command:

cat </etc/security/ldap/ldap.cfg
ldapservers:,
authtype:ldap_auth
userattrmappath:/etc/security/ldap/2307user.map
groupattrmappath:/etc/security/ldap/2307group.map
userbasedn:
groupbasedn:
useSSL:yes
ldapsslkeyf:/etc/security/ldap/key.kdb
ldapsslkeypwd:
EOF


# IDS links
IDS needs certain libraries and binaries linked from /opt/IBM/ldap//... to /usr/lib, etc.
First step is to remove any current links

 /opt/IBM/ldap/V6.2/bin/idsrmlink -i -l 64 -s fullsrv
 /opt/IBM/ldap/V6.2/bin/idsrmlink -i -l 32 -s fullsrv

Second step is to create new links to the 32 bit binaries

 /opt/IBM/ldap/V6.2/bin/idslink -i -l 32 -s base


# Start secldap at reboot
Remove existing inittab entries

rmitab ldapclntd

Create new inittab entry

 mkitab 'ldapclntd:23456789:wait:/usr/sbin/start-secldapclntd  > /dev/console  > 2&>1


# /etc/security/user
Add ldap compatability to /etc/security/user, without this ldap login will not work

 chsec -f /etc/security/user -s default -a "SYSTEM=LDAP or compat"


# sudo
Add your sudo config, I haven't had success with the AIX Linux Toolbox sudo rpm, try the sudo-noldap package (also on AIX Linux Toolbox)

# Start secldap client
Run the following command to start the ldap client

start-secldapclntd








#
# Syslog
#

AIX pulls its logging configuration from /etc/syslog.conf, the standard AIX build as of May 2009 writes most messages to /var/adm/SYSLOG and does not include rotation. I like the logging to be more Linux-like so I append this to the end of the /etc/syslog.conf file, consider commenting out the existing logging if you don't want to write to /var/adm/SYSLOG:

 # Linux-ify the AIX logging setup and enable automagic rotation
 # Everything but mail and auth to messages
 *.info;mail.none;auth.none      /var/log/messages       rotate size 10m files 10 compress
 # Auth to secure
 auth.debug                      /var/log/secure         rotate size 10m files 10 compress
 # Mail to maillog
 mail.debug                      /var/log/maillog        rotate size 10m files 10 compress
 # Emergency messages to all users
 *.emerg         *ss
 *.info;mail.none                @

This will send:
All messages except mail and authentication to /var/log/messages.
All authentication messages to /var/log/secure.
All mail messages to /var/log/maillog.
All emergency messages to all users.
All messages except mail to

If you don't like seeing the "Message forwarded from hostname:" message all over splunk you need to run the following command line as root:
 chssys -s syslogd -a "-n"
All the files need to exist prior to syslog writing to them, it will not create them on its own:
 sudo touch /var/log/messages /var/log/secure /var/log/maillog
You must then restart syslog
 stopsrc -s syslogd && startsrc -s syslogd

Enabling console access on fully virt guests (virsh/virt-manager)

Using virsh or virt-manager you can gain access to the console by editing grub.conf, this allows you to watch the machine boot from a terminal.

Add the following to the kernel line in /etc/grub.conf and reboot:

console=tty0 console=ttyS0

Converting kvm guests from lvm to qcow2, base images and snapshots

lvm based kvm guests are fast but you lose some flexibility, playing with fedora/kvm on my laptop I prefer to use file based images. Converting from lvm images to qcow2 isn't hard but the documentation is sparse.

1. use qemu-img to convert from an lvm to qcow2 format:

qemu-img convert -O qcow2 /dev/vg_name/lv_name/ /var/lib/libvirt/images/image_name.qcow2

If you want the image compressed add '-c' right after the word convert.

2. edit the xml for the image

virsh edit image_name

modify the disk stanza, adding a type to the driver line; on the source line change 'dev' to 'file' and modify the path:

driver name='qemu' type='qcow2'
source file='/var/lib/libvirt/images/image_name.qcow2'

Creating images from with a base image allows quick rollouts of many boxes based on an single install - for example I have a 'golden image' of centos, I can stop that VM and create 2 servers using the original VM disk as a base file and writing changes to different files.

qemu-img create -b original_image.qcow2 -f qcow2 clone_image01.qcow2
qemu-img create -b original_image.qcow2 -f qcow2 clone_image02.qcow2

Taking this further I can then snapshot both images so once I start making changes, rolling back to a point in time prior to the changes is very easy:

qemu-img snapshot -c snapshot_name vm_image_name.qcow2

references:
http://www.linux-kvm.com/content/how-you-can-use-qemukvm-base-images-be-more-productive-part-1

DHCP hostnames

On my home LAN Windows and Ubuntu boxes push their hostnames into local DNS which makes them easily resolvable - my router lists the names associated with MACs in its web interface, this is one of those small things that is really convenient. For some reason my fedora boxes have not been doing this lately and I finally got annoyed enough to look it up, turns out the fix is very simple.

First off figure out which interface you are using, if you are using more then 1 then I will assume you know which one you want the hostname pushed from, easily enough you can run 'ipconfig' or 'ip addr show' and note the interface name. In most cases this will be eth0 for copper and something like wlan0 for a wireless connection. Once you have the interface name substitute it into the following command:

echo "DHCP_HOSTNAME=$HOSTNAME" >> /etc/sysconfig/network-scripts/ifcfg-

Then restart networking:

sudo service network restart

In the above example I am using '$HOSTNAME' which is the system hostname variable but you could set this to a static name if you desired.

Is Hyper Threading enabled?

Needed a quick way to check if Hyper Threading was enabled on some RHEL boxes, ended up writing a quick "script" that can be copied onto the command line.

I'll go through it line by line just for fun:

First we grab all lines matching "core id" from /proc/cpuinfo, sort them (in case the id's where not listed in numeric order), list the unique values and count them

cores=`grep "core id" /proc/cpuinfo|sort|uniq|wc -l`

Using grep I count the number of lines matching "processor" from /proc/cpuinfo

procs=`grep -c "processor" /proc/cpuinfo`

If we fine less cores the processors then Hyper Threading must be on

if [[ "$cores" -lt "$procs" ]]; then
echo -e "\n$HOSTNAME: cores=$cores, processors=$procs\n    HyperThreading: Enabled"

If we find the same number of processors and cores the Hyper Threading is off

elif [[ "$cores" -eq "$procs" ]]; then
echo -e "\n$HOSTNAME: cores=$cores, processors=$procs\n    HyperThreading: Disabled"

If neither case matches then we have run into a failure, or our math doesn't work on this particular box

else
echo "epic failure"
fi

And the whole thing...

cores=`grep "core id" /proc/cpuinfo|sort|uniq|wc -l`
procs=`grep -c "processor" /proc/cpuinfo`
if [[ "$cores" -lt "$procs" ]]; then
echo -e "\n$HOSTNAME: cores=$cores, processors=$procs\n    HyperThreading: Enabled"
elif [[ "$cores" -eq "$procs" ]]; then
echo -e "\n$HOSTNAME: cores=$cores, processors=$procs\n    HyperThreading: Disabled"
else
echo "epic failure"
fi

How I learned to Stop Worrying and Love the Bomb istat

I was recently tasked with organizing 137k+ small .jpg files into a folder structure based on year and quarter, why? users opening this directory with a ftp client complained that it took a "long time" to get a directory listing... apparently 15 - 20 minutes each time they opened the directory, honestly if a program didn't return anything in 15 minutes I would probably kill it and blame the server!

I really didn't think too much of the problem, in my head I though "i'll just use 'find' and 'stat'", which would have worked perfectly EXCEPT that I had to do this on a AIX 4.3 server and mounting the filesystem remotely was not an option.

A few problems with AIX 4.3 - no 'stat' command, in AIX 5.x you can install the coreutils rpm from the AIX toolbox to overcome this problem but you are up the creek without a paddle in 4.3! Also 'find' doesn't have all of the options you would usually have available on a newer version of linux - this was an issue in my case since I had to put the files into subdirectories (example: /basedirectory/2008/Q3) which meant that when searching for files to process in the basedirectory I did not want to descend into the yearly and quarterly subdirectories, easy with the -maxdepth option - which is not available in 4.3.

I ended up getting around the lack of -maxdepth in the find command by using the -prune option to remove subdirectories from processing, because the basedirectoy did not contain any directories except 200{8,9}/Q{1..4} this task was simplified even further by providing a common directory 'Q*'.

The lack of the 'stat' command had me banging my head against 'ls' for a day or so... The problem I have with 'ls' is in trying to get the year from 'ls -l', it works great for files older then 180 days but files less then 180 days are listed with the file modification timestamp in place of the year. I toyed with awk'ing the year/modifaction time column and checking if the value was an integer, which does work but you run into issues if your script is running within 180 days of the end of the year and examining files from the previous year, all the files will have timestamps which would cause you to examine the value of the current month vs. the month of the file being examined to determine the correct year - logic that I was uninterested in writing out.

Enter in my new most loved command in AIX: 'istat'
I was lucky enough to find a post mentioning 'istat' which "displays the i-node information for a particular file". 'istat' is simliar to the linux 'stat' command although it does not allow you modify the output using command line switches - nothing a little grep and awk won't fix! What 'istat' does do is handily format data about file creation, modification and access in an unambiguous matter - dates are always shown in the same format, unlike 'ls -l'. Without this tool I was writing a longer and longer script to deal with corner cases dealing with files modified 180 days ago and files modified around the last 3 months of the year - with 'istat' I was able to make my script much simpler and rely on the computer to hand me information in an consistent format.

I would be surprised if anyone has to solve this same problem but I will post the script anyways, as a warning this script is slow - 'istat' is not a tool for performance! Also working 'xargs' into the mix would make a more elegant solution in-place of 'find' and 'cat'.

In the following script I have disabled the actual move command - this will only print what would happen! uncomment the line beginning with 'mv' and it will move files.

#!/usr/bin/ksh
#
# organize files ending in $fileextension in $basedir
# by moving them into subdirectories $basedir/$year/$quarter
#


# backdate variable controls how many days old a file must be before
# it is considered for processing, 92 days is approx 3 months
# if you don't believe me ask google "3 months in days"
backdate=92

fileext=YOUR_FILE_EXTENTION
outfile=/tmp/jpg_organizer.out
basedir=YOUR_BASE_DIRECTORY

errors=0

# function to calulate which quarter a month lives in
calculate_quarter() {
case $month in
      Jan|Feb|Mar)
              quarter="Q1"
              ;;
      Apr|May|Jun)
              quarter="Q2"
              ;;
      Jul|Aug|Sep)
              quarter="Q3"
              ;;
      Oct|Nov|Dec)
              quarter="Q4"
              ;;
esac
}

# rudimentary error checking
error_check() {
let errors="$errors + $?"
if [[ $errors -gt 0 ]]; then
  echo "encountered an error, exiting"
  exit $?
fi
}

# find files older then $backdate and move them into $basedir/$year/$quarter directories
find $basedir -name Q\* -prune -o -name \*$fileext -mtime +$backdate -type f -print > $outfile
error_check
for i in `cat $outfile` ; do
filename=$i
fileattrib=`istat $i | grep "Last modified:"`
month=`echo $fileattrib | awk '{print $4}'`
year=`echo $fileattrib | awk '{print $7}'`
calculate_quarter
if [[ ! -d $basedir/$year/$quarter ]]; then
  mkdir -p $basedir/$year/$quarter
  error_check
fi
echo "moving:$filename to $basedir/$year/$quarter/"
#mv $filename $basedir/$year/$quarter/
error_check
done

rm $outfile

exit 0

AIX syslogd and splunk (and more)

AIX is what I would call a 'batteries not-included' OS; the vanilla DVD install leaves you with a functioning system that has telnet (with root access) enabled, no OpenSSL/OpenSSH, korn shell without autocomplete (must be enables 'set -o vi'), no logging, etc...
Since I work around a lot of RedHat boxes I tend to modify the AIX servers to have a simlar setup to RHEL, here are some of the steps I take:

Install the following rpm's from the aix toolbox:
bash (add /usr/bin/bash to /etc/security/login.cfg)
curl
coreutils
less
lsof
python
rsync
sudo
unzip
wget

Install OpenSSL and OpenSSH

Change root home directory to /root and change shell to bash:

mkdir /root && chuser home=/root shell=/usr/bin/bash root

Modify prompt for all users:

# Set bash prompt to be much more linux like
if [[ "$TERM" == "xterm" ]];then
if [[ "$SHELL" == "/usr/bin/bash" || "$SHELL" == "/bin/bash" ]];then
  if [[ "$UID" -eq 0 ]];then
    PS1="\[\033]0;\u@\h:\w\007\][\[\033[31;1m\]\u\[\033[0m\]@\h \W]# "
  else
    PS1="\[\033]0;\u@\h:\w\007\][\u@\h \W]\$ "
  fi
fi
fi

Change logging setup:

# Linux-ify the AIX logging setup and enable automagic rotation
# Everything but mail and auth to messages
*.info;mail.none;auth.none      /var/log/messages       rotate size 10m files 10 compress
# Auth to secure
auth.debug                      /var/log/secure         rotate  size 10m files 10 compress
# Mail to maillog
mail.debug                      /var/log/maillog        rotate size 10m files 10 compress
# Emergency messages to all users
*.emerg         *
*.info;mail.none                @NETWORK_LOG_SERVER

Remove "Message forwarded from hostname:" from remote logging output:

chssys -s syslogd -a "-n" ; stopsrc -s syslogd ; startsrc -s syslogd

Run aixpert to enable a much higher level of security:
aixpert -l high

Lotus Notes 8.5 on Fedora 10 x86_64

32-bit packages required for notes 8.5 to work on Fedora 10 x86_64. Notes will install fine without these but will not run.

libxkbfile-1.0.4-5.fc9.i386
libgnomecanvas-2.20.1.1-4.fc10.i386
libgnomeprint22-2.18.5-1.fc10.i386
libgnomeprintui22-2.18.3-1.fc10.i386
gnome-vfs2-2.24.0-3.fc10.i386
libgnome-2.24.1-9.fc10.i386
libgnomeui-2.24.0-2.fc10.i386
libXScrnSaver-1.1.3-1.fc10.i386
libcanberra-gtk2-0.10-3.fc10.i386
gtk-nodoka-engine-0.7.2-1.fc10.i386

Of course installing these also carries a lot of dependency baggage.

And a handy one liner:
sudo yum -y install libxkbfile-1.0.4-5.fc9.i386 libgnomecanvas-2.20.1.1-4.fc10.i386 libgnomeprint22-2.18.5-1.fc10.i386 libgnomeprintui22-2.18.3-1.fc10.i386 gnome-vfs2-2.24.0-3.fc10.i386 libgnome-2.24.1-9.fc10.i386 libgnomeui-2.24.0-2.fc10.i386 libXScrnSaver-1.1.3-1.fc10.i386 libcanberra-gtk2-0.10-3.fc10.i386

ardour vsti support

To enable vsti support in ardour you must compile it from source, this has to do with licensing of the steinberg vst sdk. Here are my basic instructions for doing this on Fedora 10, note that I have already installed the ccrma repository and kernel as well as many packages referenced in my earlier post.

cd ~/Download/ && wget http://releases.ardour.org/ardour-2.7.1.tar.bz2

bunzip2 ardour-2.7.1.tar.bz2 && tar xf ardour-2.7.1.tar && cd ardour-2.7.1

get vst2.3 zip from steinberg, put in ~/Download/ardour-2.7.1/libs/fst

yum install liblrdf-devel libgnomecanvas-devel aubio-devel fftw-devel libsxlt-devel gcc-c++ boost-devel

cd ~/Download/ardour-2.7.1/
scons VST=1
scons install

Installing IBM Tape System Reporter

On Sept. 8th I got an email from IBM notifying me that Tape System Reporter had been released, it is supposed to:

The IBM Tape System Reporter (TSR) application enables operators and administrators of the TS3500 Tape Library to monitor and report on storage devices in an enterprise environment

I have two TS3500 with 12 drives between them so this sounded pretty good, I thought I would install it and see what it can offer. I am a big fan of reporting since usually I can gain some ground with management to buy more stuff if I have pretty graphs in my hands!

Unfortunately I was in for a ride on this, I should have known when I read this:

It is not the intent of this documentation to explain how to download and use
Derby to establish a database that contains the authorizations for using the IBM
Tape System Reporter application.

Which means that I had to learn how to install a new application (apache Derby) with little help from IBM... I was able to accomplish this - though at the end I had not read the requirements well enough and found that I did NOT have ALMS licensed so even though I had the app installed correctly I couldn't get the data out of it... Either way here are the steps I went through to install this app.

You need Windows XP or 2000 for the install so I booted up a XP virtual machine using Virtual Box.

1. Download and install the latest version of java from http://www.java.com, derby is a java database...
2. Install Adobe Reader from http://www.adobe.com/products/acrobat/readstep2.html, the install docs are in PDF and copy/paste from my main OS to the virtual box doesnt always work...
3. Download the latest version of derby from http://db.apache.org/derby/derby_downloads.html
4. Extract derby, I chose to go with c:\derby_10\ like the doc shows, earlier I had tried putting it in c:\program files\derby\ but didn't have much luck - I started to wonder if the %PATH% variables where getting stuck on the spaces in the directory structure.
5. Time to set some variables, you can set these on the command line for one time use or set them in the global profile, I chose the later:
1. Right click on "My Computer", click on properties
2. select the 'Advanced' tab, click on Environment variables
   
3. on the lower half of the window that opens (System Variables) click 'New'
4. Variable name = DERBY_HOME, Variable value = C:\Derby_10 (or the directory you expanded the derby zip into). Click OK
5. Click on 'Path' in the System Variables section, choose 'Edit'
   
6. To the end of the Variable value add the following: ;%DERBY_HOME%\bin
6. Good time to validate those variables, open a command prompt (Start->Run->cmd) and type the following:
1. echo %DERBY_HOME%
   
• output should be the variable value you set in step 5, in my case c:\derby_10
2. ij
• this command is part of the derby package, output should look something like:
  

  version 10.4
  
  ij>

• if that is working simply type 'quit;' to exit the ij shell, if you dont see the ij prompt your system variables are not set correctly!
  
7. At this point the directions from IBM start to lose their usefulness, some of the files they mention don't exist, other required files are not mentioned... good thing they had that disclaimer at the beginning of the document!
8. Navigate to %DERBY_HOME\bin and copy the derby_common, startNetworkServer and stopNetworkServer scripts to the main derby folder (one folder down). The instructions mention a derby.properties file, i believe this would only exist if you had previously used derby so if it doesn't exist you can create it in the next step
   
9. create a new file and save it as derby.properties in %DERBY_HOME%, remember that if you create the file with notepad (as I did) that you must set the "Save as type:" to "All Files" or windows will magically append .txt to the filename.
10. Add the following to your %DERBY_HOME%/derby.properties file, in this example I am using tsruser as the username and tsrpass as the password - adjust accordingly.
    

    derby.connection.require Authentication=true
    
    derby.authentication.provider=BUILTIN
    
    derby.user.tsruser=tsrpass
    
    derby.databasedefaultConnectionMode=fullAccess

11. Now you need to edit the startNetworkServer script in %DERBY_HOME%, if you have your CLASSPATH setup for derby you can follow the IBM instructions, I did not so I had a much longer string to enter. Note that in my example I am setting the directory for the database to be created in as %DERBY_HOM%\tsrdb. Add the following to the end of the script (should be one long line):
    

    java -classpath %DERBY_HOME%\lib\derby.jar;%DERBY_HOME%\lib\derbynet.jar;%DERBY_HOME%\lib\derbyclient.jar;%DERBY_HOME%\lib\derbytools.jar;%DERBY_HOME%\lib\derbyrun.jar -Dderby.system.home=%DERBY_HOME%\tsrdb\ org.apache.derby.drda.NetworkServerControl start -h localhost -p 1527
    

12. Add a vary similar line to %DERBY_HOME%\stopNetworkServer:
    

    java -classpath %DERBY_HOME%\lib\derby.jar;%DERBY_HOME%\lib\derbynet.jar;%DERBY_HOME%\lib\derbyclient.jar;%DERBY_HOME%\lib\derbytools.jar;%DERBY_HOME%\lib\derbyrun.jar -Dderby.system.home=%DERBY_HOME%\tsrdb\ org.apache.derby.drda.NetworkServerControl shutdown -h localhost -p 1527
    

13. You can test the start and stop scripts at this point by double clicking on them, the start script should open a command window that accepts no input and the stop script should open a command window and then close both the start and stop windows. If everything is working correctly go to the next step, otherwise double check everything.
14. Time to create the database - start the derby server by double clicking on %DERBY_HOME%\startNetworkServer, open a command window and get an ij prompt by typing 'ij'. Enter the following text to create the database - I am using tsrdb as the database name, tsruser as the username and tsrpass as the password
    

    connect 'jdbc:derby://localhost:1527/tsrdb;create=true;user=tsruser;pass=tsrpass';
    

15. Check the %DERBY_HOME% directory, you should see a folder matching your database name (tsrdb in my examples). If you do then you should have the derby portion of the install complete!
16. Install the DB2 Run-Time Client Lite that is mentioned as a prereq in the docs, you can find it at http://www-01.ibm.com/support/docview.wss?uid=ssg1S4000680. I took the defaults and did a 'Typical' install which worked fine.
    
17. Download the TSR zip file, it can be found on the page mentioned in step 16. Extract it to a directory of your choosing, I like c:\Program Files\tsr
18. Now we can check on the database connectivity and create the table to store data in:
• Double click on the tsr executable, from the menu choose 'Database->Setup'.
  
• Enter your database name (tsrdb) the IP (localhost) and port (1527)
• Click Test, enter the username (tsruser) and password (tsrpass) and click OK
• The system will churn for a minute and should say 'Test Passed'.
• Select the 'Table' tab while still in the Setup window and choose a table name, according to IBM this table will be used for storing the library performance data, I chose 'tsrdata'.
• Click create, you will need to enter your username and password again (tsruser/tsrpass) and click OK
• Output should be 'Table created successfully', click OK and then click OK again.
19. Time to connect to the database and begin collecting data
• Click 'Database->Connect'
• enter username and password, click OK, should get 'Connection successful'
  
• Click 'File->Start Monitoring'
• enter the IP address or dns entry of the TS3500 you would like to monitor and click OK
• A window should open and have some output in it like 'Starting Monitor on Tape Library yourlibnamehere'
• cant say much more, without ALMS licensed this is as far as I got :( but hopefully it works!

Hopefully this helps someone, if I get an ALMS license soon I would like to post some screenshots of the app actually displaying useful data. Maybe someone can send me some!

Fedora Multimedia Workstation

My default Fedora 10 install packages plus ccrma repos and meta package/kernel for low latency audio workstation with windows vsti's. With this setup I can run an alesis trigger io and trigger samples in XLN Audio Addictive Drums without having to run windows, plus I get SUPER low latency with low cost sound cards (even onboard is below 3ms), currently running a turtle beach riviera on an old Pentium 4 and getting 1.6ms

# setup yum repos
sudo rpm -ivh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm \
http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm \
http://rpm.livna.org/livna-release-9.rpm \
http://linuxdownload.adobe.com/adobe-release/adobe-release-i386-1.0-1.noarch.rpm

# install apps
sudo yum -y install bash-completion nautilus-open-terminal gstreamer-plugins-bad gstreamer-plugins-ugly gstreamer-ffmpeg k3b-extras-freeworld lame easytag mplayer gnome-mplayer gecko-mediaplayer mencoder libdvdcss flash-plugin AdobeReader_enu clusterssh compat-libstdc++-33 gcc wine wine-devel grip unrar vnc

# 64-bit flash support
yum install flash-plugin nspluginwrapper.x86_64 nspluginwrapper.i386 alsa-plugins-pulseaudio.i386 libcurl.i386

# nvidia driver
sudo yum -y install akmod-nvidia

# ati driver
sudo yum -y install akmod-fglrx

# ccrma repos add extra audio workstation tools and includes the low latency kernel
sudo rpm -Uvh http://ccrma.stanford.edu/planetccrma/mirror/fedora/linux/planetccrma/10/i386/planetccrma-repo-1.1-2.fc10.ccrma.noarch.rpm

# meta package for all major ccrma apps, this is a LARGE compilation and will take a while to download on a slow connection! (in my case 259mb with dependencies)
sudo yum -y install planetccrma-apps

# low latency kernel, first step is to allow Fedora to keep more kernels which can be done by changing the 'installonly_limit=3' line to 'installonly_limit=0', then running the following command
sudo yum -y install planetccrma-core

# get rid of "Could not load Mozilla. HTML rendering will be disabled." when running wine
wine iexplore http://www.winehq.com

Rebinding TSM archives so they do not expire

I have some TSM archives on an AIX host with a short expiration period of 14 days that I needed to extend for an unknown amount of time, I thought this would be an easy task but it took me a minute to figure out exactly how to quickly and efficiently get this done. To stop expiration on an archive you have to use the 'set event type=hold' command in from 'dsmc', the documentation on the command is complete but sparse with few examples so I had play with it to understand it. The most important thing I learned was that you cannot use a '*' to specify all files in an archive having a specific description - but you can specify just the base directory (in TSM terms 'filespace_name' from the archives table) and append a '/' (example: '/directory/) and it will pick up all of the files in the archive under the base directory.

First build the file list, this can be done easily a sql select statement from within dsmadmc :

select distinct(filespace_name) from archives where node_name='node name' and description='archive description here' > outfile

Then I needed to add a '/' character to the end of each line from the command line using awk:

# cat outfile | awk '{ print $1"/"}' > filelist.out

Alternately you could have selected filespace_name and hl_name and used awk to print both columns without a space between, either way the results should be the same...

Now I had a file that looked similar to this:

/aaaa/
/bbbb/
/cccc/
/db/abcd/
/db/efgh/
/db/ijkl/
/db/mnop/
/db/qrst/
/db/uvwx/
/db/yz/
/eeee/
/ffff/

each entry is a seperate mount point for a filesystem which is why /db/ would not have worked correctly.

Load this file into dsmc using the set event command:

set event -type=hold -filelist=filelist.out -description="Some unique description here"

output like:
....
ANS1899I ***** Examined 35,000 files *****
ANS1899I ***** Examined 36,000 files *****
ANS1899I ***** Examined 37,000 files *****
ANS1899I ***** Examined 38,000 files *****
ANS1899I ***** Examined 39,000 files *****
....

Total number of objects archived: 0
Total number of objects failed: 0
Total number of objects rebound: 82781
Total number of bytes transferred: 0 B
Data transfer time: 0.00 sec
Network data transfer rate: 0.00 KB/sec
Aggregate data transfer rate: 0.00 KB/sec
Objects compressed by: 0%
Elapsed processing time: 00:04:26
tsm>

After the files were rebound I jumped onto dsmadmc to make sure I had gotten every file rebound, I did this by selecting the number of objects from the archive and comparing it to the number of objects rebound:

select count(*) as "# of archived objects" from archives where description='Some unique description here' and node_name='node_name'

# of archived objects
---------------------
82781

Looks good!

This was by far the fastest way to rebind all these objects. I also tried selecting each individual object from the tsm database and loading that as the filelist into dsmc (a file with 82781 lines), after about 10 hours of processing dsmc core dumped. My other thought was to use the filelist containing every object and running it in a for loop so each object was processed individually - this would have worked but the time for completion would have been much longer.

Installing projectM with amarok support on Fedora 8

I wanted projectM to work with amarok on my Fedora 8 box... I had done it before but didn't remember the exact steps...

Dependencies: projectM isn't in the standard fedora or livna repos so we have to compile, because compiling doesn't automagically install requisite software like yum does we need to have some packages pre-installed. This list is probably not complete but it's what I had to add (plus dependencies for these packages which are handled by yum):
pulseaudio-libs-devel amarok-visualisation, qt4-devel, cmake, ftgl-devel, glew-devel and subversion

# sudo yum -y install pulseaudio-libs-devel amarok-visualisation qt4-devel cmake ftgl-devel glew-devel subversion

Next we need to download projectM from sourceforge, I like to install the latest bleeding edge software since it generally causes me more pain so I chose the to go with the latest branch from subversion:

# mkdir ~/Download/projectM && cd ~/Download/projectM
# svn co https://projectm.svn.sf.net/svnroot/projectm/trunk projectM-Trunk
# cd projectM-Trunk/src
# ccmake .

cmake loads. press 'c' to configure: change CMAKE_BUILD_TYPE to 'Release' and CMAKE_INSTALL_PREFIX to /usr/
press 'c' again to configure and g to generate makefile

Time to compile:

# make && sudo make install

Now 'projectM-pulseaudio' should be in your Applications ->Video menu (if your using gnome) and it will bump along to any sounds that are playing through pulse. If you have amarok open just restart it and you can launch it from the visualizations menu!

I can't get it to play nice with compiz on an ati card running dual monitors, but with compiz turned off it's fine.

I ripped off a bunch of this from the ubuntu forum (thanks!):
http://ubuntuforums.org/showthread.php?t=749793
as well as the projectM page itself:
http://projectm.wiki.sourceforge.net/Installation+Instructions

Tivoli Storage Manager Admin Center on rhel5

If you install the ISC and Admin Center for Tivoli Storage Manager on a RHEL5 box and you get the "portlet unavailable" error try installing libXp
$ yum install libXp
then stopping and starting the ISC again (by default $isc_home is /opt/IBM/ISC601/):
cd $isc_home/PortalServer/bin
./stopISC.sh ISC_Portal iscadmin iscpassword

check for runaway processes
$ ps -ef | grep -i java

Then startup the the ISC again...
./startISC.sh ISC_Portal

Fedora 8 and Lotus Notes 8 - Java issues!

I finally got around to installing Fedora 8 on my laptop after spending all last week in Dallas at LISA '07 and I am still ironing out some issues (compiz i'm looking in your direction). One thing I have solved is the Lotus 8 installer failing! Essentially the problem boils down to incompatibilities between Sun's Java and F8 - I noticed on the mjmwired page that to use sun java a patch needed to be applied to libmawt.so, and I knew that the lotus installer was using a packaged version of java that was extraced at install time so....

$ cd /home/nate/Desktop/lotus\ 8/
$ sudo ./setup.sh
When the installer runs you will see

Initializing Wizard........
Extracting Bundled JRE.

then

Initializing Wizard........
Installing Bundled JRE.

then

Initializing Wizard........
Verifying JVM.

at this stage (or at Extracing Installation Archive.) press ctrl-z to pause and background the process
$ cd /tmp/istemp{randomnumbershere}/_bundledJRE_/jre
$ for i in `find ./ -name libmawt.so` ; do echo "Found $i, applying patch..." ; \
sudo sed -i 's/XINERAMA/FAKEEXTN/g' $i ; done
$ fg

From this point the installer *should* work correctly and Lotus 8 will install! of course your mileage may vary... let me know if this helps you!

Noticed another possible problem with Notes 8 on Fedora - Incorrect permissions on ~/lotus directory causing lotus to fail to start after a successful installation. To fix that simply remove the folder and start notes again.

Powered by ScribeFire.

Self signed ssl certificates with oracle oc4j and ssl wallets

After beating my head against the wall for about 2 hours I got this to work - the documentation from oracle did not contain one important little gotcha which would have saved me at least an hour! Essentially what I am doing here is creating a self-signed certificate for developers who are making SOAP or access calls to seperate secure web servers AND the secure web server is using Oracle Apache/Webcache and SSL Wallets, since the application has no way of asking if the (self-signed) certificate should be accepted the app will fail with something like this:

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Ok, onto the work:

• server1 = OC4J containers - application deployed here, making https calls to server2
  
• server2 = Oracle Apache/WebCache server, app calls to https://server2/ causing errors
  

1. Become your own Certificate Authority
• Easy enough, follow these instructions: http://www.onlamp.com/pub/a/onlamp/2003/02/06/linuxhacks.html
• In Fedora the package that provides CA.pl (yum whatprovides CA.pl) is openssl-perl from the core repos.
2. OWM (Oracle Wallet Manager) on server2
   
• Start a vnc session (Im sure you could also use X forwarding, but I'm lazy) to server2
• open a terminal window
• $ cd $ORACLE_HOME/webcache/wallets/
• cp -pR defaults backup
  
• $ cd $ORACLE_HOME/bin
• $ ./owm
• Wallet -&gt; New
• "Your default wallet directory does not exist. Do you want to create it? NO
• Create a password for you wallet, click YES
• Do you want to create a certificate request at this time? YES
• Enter the information for your cert, remember that the "Common Name" must match the url that is serving https requests, if it does not you will receive certificate errors - example.com and www.example.com are different in this case! Click OK when you have entered all the info and click OK again on the "certificate request has been created" dialog box
• Under you wallet you should now see an entry like "Certificate:[Requested]", click on this and copy the Certificate request (including the BEGIN NEW CERTIFICATE REQUEST line)
• open a terminal on the box you created your Certificate Authority on and paste the Certificate Signing Request (csr) into a text file, in my case /home/nate/CA/certs/example.com.csr
  
• $ CA.pl -sign after renaming your .csr to newreq.pem
  
• enter your CA password
• Sign the certificate? [y/n]: Y
• commit? [y/n] Y
• copy the newly created certificate from the BEGIN CERTIFICATE line to the END CERTIFICATE line
• Back in the vnc session in the owm application
  
• Operations -&gt; Import User Certificate -&gt; Paste the certificate, click OK
  
• Paste the the newly created cert in the dialog box and click OK
• OH NO, an error... we can fix this "User certificate import has failed because the CA certificate does not exist. Do you want to import CA certificate now? YES" -&gt; Paste the certificate, click OK
  
• on your CA box look in the CA dirctory for a file called cacert.pem
• $ cat /home/nate/CA/cacert.pem # copy from BEGIN CERTIFICATE to END CERTIFICATE
• Back in the vnc session in the owm application
• paste the CA certificate into the dialog box and click OK
• At this point your Wallet should say "Certificate:[Ready]" and your CA cert should be listed under the Trusted Certificates
• Since I am using webcache on the apache box I am going to save this in the webcache directory but if I was using apache then it would be the apache directory and I would also possibly need to modify the SSLWallet directive in the ssl.conf file, if you are using apache without webcache checkout the references to the oracle docs at the end of this document since they cover this part well... what they don't cover - and what I had issues with is using the ssl wallet with webcache so....
• one of the first steps created a backup of $ORACLE_HOME/webcache/wallets/default so we can overwrite those files without worry
• Wallet -&gt; Auto Login # this allows apache (and I assume webcache) to startup without asking for the wallet password... very important
• Wallet -&gt; Save As -&gt; $ORACLE_HOME/webcache/wallets/default
• Do you want to overwrite it? YES (you made a backup right?)
• Now in $ORACLE_HOME/webcache/wallets/default on server2 you should have two files: cwallet.sso and ewallet.p12 that contain your certificate, if you want to verify try the orapki command
• $ORACLE_HOME/bin/orapki wallet display -wallet $ORACLE_HOME/webcache/wallets/default/ewallet.p12
• enter your wallet password and BAM there is the stuff...
• You can now bounce WebCace ($ORACLE_HOME/bin/opmnctl stopproc ias-component=WebCache ; $ORACLE_HOME/bin/opmnctl startproc ias-component=WebCache) and navigate to https://server2/ in a web browser and YOUR cert should pop up as untrusted - this is ok, as long as it is your cert and not the default Oracle cert!
3. Importing your certificate into the cacerts file on server1
• On your CA server copy the self-signed certificate that was created in the previous steps
• Logon to server2 and navigate to the OC4J container jdk/jre/bin directory
• $ cd $ORACLE_HOME/soa1/jdk/jre/bin
• paste the cert into a text file
  
• $ ./keytool -v -import -alias example.com -file ./example.com.crt -keystore ../lib/security/cacerts
• default password on oracle keystores is: changeit
• In my case the OC4J containers are clustered so I needed to duplicate this
• $ cd $ORACLE_HOME/soa2/jdk/jre/bin
• $ ./keytool -v import -alias example.com -file ../../../../soa1/jdk/jre/bin/example.com.crt -keystore ../lib/security/cacerts
• Bounce the OC4J containers
  
4. Done!

References:

• http://oraclelon1.oracle.com/docs/cd/B14099_15/core.1012/b13995/wallets.htm#i1010609
• http://oraclelon1.oracle.com/docs/cd/B14099_15/web.1012/b14007/ssl.htm#CHDFCADD
• http://www.onlamp.com/pub/a/onlamp/2003/02/06/linuxhacks.html
• http://www.lifeaftercoffee.com/2006/03/29/secure-certificate-management-in-oracle-application-server/
• http://www.openssl.org/docs/apps/CA.pl.html