Archive for June, 2009

Using JSVC with Tomcat on a 64-bits machine with a 32-bits JVM.

The Problem

After a good amount of research and issues that  looked like memory leaks I’m moving my jvms to 32-bits JVMs for the servers that do not need a heap larger than 2GB. This was ok and a fast switch of the /usr/local/java symlink, BUT it was not as easy with JSVC that I use to run Tomcat under the user tomcat. The problem is that when I compile jsvc on a 64-bits machine it will auto-magically find the box as x86_64 and then expect that your jvm is also 64 bits.

The Fix

What I found that needs to be done is to force the compiler to use 32bit flags, so first lets run the configure string

[root ~]# cd /usr/local/tomcat/bin && tar -zxf jsvc.tar.gz
[root ~]# cd jsvc-src && chmod +x configure
[root ~]# CFLAGS=-m32 CPPFLAGS=-m32 CCASFLAGS=-m32 LDFLAGS="-L/usr/lib -L/lib" ./configure --build=i686-pc-linux-gnu

now the next step is to run make

[root ~]# make

What you will now see is that the last step of the make will fail horribly with the following error:

gcc -L/usr/lib -L/lib -ldl -lpthread jsvc-unix.o libservice.a -o ../jsvc
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libdl.a when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.a when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
/usr/bin/ld: skipping incompatible /usr/lib/libc.a when searching for -lc
/usr/bin/ld: warning: i386 architecture of input file `jsvc-unix.o' is incompatible with i386:x86-64 output
/usr/bin/ld: warning: i386 architecture of input file `libservice.a(arguments.o)' is incompatible with i386:x86-64 output
/usr/bin/ld: warning: i386 architecture of input file `libservice.a(debug.o)' is incompatible with i386:x86-64 output
/usr/bin/ld: warning: i386 architecture of input file `libservice.a(help.o)' is incompatible with i386:x86-64 output
/usr/bin/ld: warning: i386 architecture of input file `libservice.a(home.o)' is incompatible with i386:x86-64 output
/usr/bin/ld: warning: i386 architecture of input file `libservice.a(java.o)' is incompatible with i386:x86-64 output
/usr/bin/ld: warning: i386 architecture of input file `libservice.a(location.o)' is incompatible with i386:x86-64 output
/usr/bin/ld: warning: i386 architecture of input file `libservice.a(replace.o)' is incompatible with i386:x86-64 output
/usr/bin/ld: warning: i386 architecture of input file `libservice.a(dso-dlfcn.o)' is incompatible with i386:x86-64 output
make[1]: Leaving directory `/usr/local/tomcat/bin/jsvc-src/native'

Looks like the last gcc left out the -m32 and therefor fails.. Lets do the last task manually

[root ~]# cd native && gcc -m32 -L/usr/local/java/lib -L/usr/lib -L/lib -ldl -lpthread jsvc-unix.o libservice.a -o ../jsvc && cd ..

now if you do the following you should have a correctly compiled 32-bits jsvc executable that points to your 32 bits jvm

[root ~]# file jsvc
jsvc: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped

Nice!

Resources

http://www.mail-archive.com/users@tomcat.apache.org/msg36176.html

Installing Apache HTTP Server with a quick-start config

The Problem

How to install Apache HTTP Server with a good baseline configuration file starter.

The Fix

When I install a Apache HTTP Server for a client or myself, small or large I follow a “standard” configuration setup which is very easy to build on later. For the most part I use CentOS or Redhat Enterprise edition servers but these steps should work on any Unix system. This might not be true for AIX which requires a little more hand-holding to make sure the compiler is installed correctly.

Content of this article

  • Download the source code from the Apache project website. ( Currently 2.2.11 )
  • Execute the configure, make and make install installation steps. ( With a few custom switches )
  • Setup the httpd.conf and associated files
  • Start your newly built Apache server
  • Done!

Download Apache

First step is to download the Apache source code and not binaries or rpms. I believe that using the source code will give the best performing most flexible installation of Apache. If you follow a few steps the actual “installation” procedure is not difficult and it will give you a good foundation to add/remove features later.

Go to this url http://httpd.apache.org/download.cgi and get the latest tgz or bz2 file, currently 2.2.11.

Compiling/installing the source for apache

I like to keep my source download in ~/Software/ so that I can go back and just re-compile and re-install the binaries if I need to add a module or two.

[root@coco ~]# cd ~/Software/httpd-2.2.11
[root@coco httpd-2.2.11]# “./configure” \
“–enable-ssl” \
“–enable-proxy” \
“–enable-proxy-balancer” \
“–enable-rewrite” \
“–enable-headers” \
“–enable-deflate” \
“–enable-cache” \
“–enable-expires” \
“–enable-mem-cache” \
“–enable-disk-cache” \
“–enable-file-cache” \
“–with-mpm=worker” \
“–disable-cgi –disable-asis” \
“–disable-autoindex” \
“–disable-userdir”

Let me explain

enable-ssl This is so that you can enable a secure port later.

enable-proxy/enable-proxy-balancer This is here to setup a connection to a backend server like Tomcat/Thin/Mongrel

enable-rewrite We are always going to need rewrite rules in the config file.

enable-headers Need this to enable monitoring of the server, and for mod_proxy we need to manipulate the header.

enable-deflate The old gzip module which will allow us to setup some content to be compressed with gzip

enable-cache/expires/mem-cache/disk-cache/file-cache Are all there so that we can enable the expires module.

with-mpm=worker I’m choosing to use the worker MPM as my default since most server I work with do have more than 1 CPU. If you are working on servers that only has one cpu use the prefork MPM.

[root@coco httpd-2.2.11]# make && make install

You will now have the apache server installed in /usr/local/apache2, this is the default install directory and if you would want to change this you need to add the –prefix=/my/directory/apache2 switch to the configure string and make && make install to install apache to the correct directory.

Apache startup script

The easiest and fastest way is to copy /usr/local/apache2/bin/apachectl to /etc/init.d/apache this will allow you todo /etc/init.d/apache start|stop|restart.

If you want a script that has more feedback you can use the following script:

#!/bin/bash
# httpd        Startup script for the Apache HTTP Server
# chkconfig: 2345 85 15
# description: Apache is a World Wide Web server.  It is used to serve \
#              HTML files and CGI.
# processname: httpd
# config: /usr/local/apache2/conf/httpd.conf
# pidfile: /var/run/apache2.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Start httpd in the C locale by default.
HTTPD_LANG=${HTTPD_LANG-"C"}

# This will prevent initlog from swallowing up a pass-phrase prompt if
# mod_ssl needs a pass-phrase from the user.
INITLOG_ARGS=""

# Path to the apachectl script, server binary, and short-form for messages.
apachectl=/usr/local/apache2/bin/apachectl
httpd=${HTTPD-/usr/local/apache2/bin/httpd}
prog=httpd
pidfile=${PIDFILE-/var/run/apache2.pid}
lockfile=${LOCKFILE-/var/lock/subsys/apache2}
RETVAL=0

start() {
 echo -n $"Starting $prog: "
 LANG=$HTTPD_LANG daemon $httpd $OPTIONS
 RETVAL=$?
 echo
 [ $RETVAL = 0 ] && touch ${lockfile}
 return $RETVAL
}
stop() {
 echo -n $"Stopping $prog: "
 killproc $httpd
 RETVAL=$?
 echo
 [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}
reload() {
 echo -n $"Reloading $prog: "
 if ! LANG=$HTTPD_LANG $httpd $OPTIONS -t >&/dev/null; then
 RETVAL=$?
 echo $"not reloading due to configuration syntax error"
 failure $"not reloading $httpd due to configuration syntax error"
 else
 killproc $httpd -HUP
 RETVAL=$?
 fi
 echo
}

# See how we were called.
case "$1" in
 start)
 start
 ;;
 stop)
 stop
 ;;
 status)
 status $httpd
 RETVAL=$?
 ;;
 restart)
 stop
 start
 ;;
 condrestart)
 if [ -f ${pidfile} ] ; then
 stop
 start
 fi
 ;;
 reload)
 reload
 ;;
 graceful|help|configtest|fullstatus)
 $apachectl $@
 RETVAL=$?
 ;;
 *)
 echo $"Usage: $prog {start|stop|restart|condrestart|reload|status|fullstatus|graceful|help|configtest}"
 exit 1
esac

exit $RETVAL

Apache configuration file

I will step through the different sections of the main configuration file that I use as the template for my Apache servers.

# =================================================
# Basic settings
# =================================================
ServerName %{SERVER_NAME}
ServerRoot "/usr/local/apache2"
PidFile "/var/run/apache2.pid"
# =================================================
# Performance settings
# =================================================
Timeout 30
KeepAlive On
MaxKeepAliveRequests 500
KeepAliveTimeout 2
<IfModule mpm_prefork_module>
 StartServers            1
 MinSpareServers         1
 MaxSpareServers         10
 MaxClients              25
 MaxRequestsPerChild     1000
</IfModule>
<IfModule mpm_worker_module>
 ServerLimit             16
 StartServers             2
 MaxClients              40
 MinSpareThreads          5
 MaxSpareThreads         20
 ThreadsPerChild         20
 MaxRequestsPerChild   5000
</IfModule>

The Basic section is just there for Apaches Root directory. Next we have a Timeout of 30 seconds which is enough for most setups the default of 300 is way too long. We enable keepalive BUT the keepalive timeout is only 2 seconds which allows each user to get their own connection but the connection will close as soon as they download the page they requested ( You can play with this timeout and have it set somewhere in the 1-5 sec range ). Then I setup perfork and worker depending on the number of cpus that are installed on the Apache server.

# =================================================
# General settings
# =================================================
Listen 80
# Listen 443
User www
Group www
ServerAdmin webmaster@openlogic.com
UseCanonicalName Off
ServerTokens Prod
ServerSignature Off
HostnameLookups Off
ExtendedStatus On
# =================================================
# Modules
# =================================================
#LoadModule dummy_module /usr/lib/apache2/modules/mod_dummy.so

Next sections we listen to port 80 but also have 443 if we want.( I will show you later how to setup a https/SSL/Secure virtualhost ) User and Group is the www user which is a system user. ( On Redhat you create a system user with the -r switch. adduser -r www ). We dont want the server to lookup hostname or show a signature to our users. The ExtendedStatus is on for monitoring reasons.The dummy module is there if we want to install php later down the road.

# =================================================
# Access control
# =================================================
<Directory />
 Options FollowSymLinks
 AllowOverride None
 Order deny,allow
 Deny from all
</Directory>
<DirectoryMatch "^/.*/\.svn/">
 ErrorDocument 403 /404.html
 Order allow,deny
 Deny from all
 Satisfy All
</DirectoryMatch>
<FilesMatch "^\.ht">
 Order allow,deny
 Deny from all
 Satisfy All
</FilesMatch>
# =================================================
# MIME encoding
# =================================================
DefaultType text/plain
TypesConfig conf/mime.types
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl

First things first, Deny from all. This makes sure we have to allow access to any directory that is used in the Apache configuration. Then we make sure that users don’t have access to .svn directories or .ht files. We also have a minimal mimetype setup. ( for the deflate and ssl modules. )

# =================================================
# Logs
# =================================================
LogLevel warn
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
ErrorLog /usr/local/apache2/logs/error_log
# Mark requests for the robots.txt file
SetEnvIf Request_URI "^/robots\.txt$" dontlog
SetEnvIf Request_URI "^\/monit\/token$" dontlog
# =================================================
# SSL Configuration
# =================================================
SSLPassPhraseDialog  builtin
SSLSessionCache        shmcb:/usr/local/apache2/logs/ssl_scache(512000)
SSLSessionCacheTimeout  300
SSLMutex  file:/usr/local/apache2/logs/ssl_mutex
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin

Now we setup the logformat for use in our virtualhosts and the server error log file. We also mark two dontlog Env which removes the robot.txt and monit/token hits from the log. When we created the virtualhost I will show how this is used. We also setup a default ssl configuration for the server.

# =================================================
# Mod status for monitoring
# =================================================
<VirtualHost 127.0.0.1:80>
 <Location /server-status>
 SetHandler server-status
 Order Deny,Allow
 Deny from all
 Allow from localhost
 Allow from 127.0.0.1
 </Location>
</VirtualHost>
# =================================================
# Include extra configs
# =================================================
Include conf/extra/httpd-myblog.com.conf

Now the server monitoring setup. We allow only access from localhost and it will only listen to the 127.0.0.1 ip. This is a good setup for tools like Groundworks and Hyperic. The last line includes a virtualhost configuration file. Now lets have a look at the virtualhost

Virtualhosts using a namebased setup

I like to have my httpd.conf with server-wide settings and free of actual content hosting elements or mod_proxy/mod_jk configurations. In this example I have a blog that’s running on a Ruby on Rails backend with three thin ( Thin is one RoR application server ) servers listening to ports 8000- 8002.

# --------------------------------------------------------
# Always keep the host header
# --------------------------------------------------------
ProxyPreserveHost On
# --------------------------------------------------------
# Rails Cluster
# --------------------------------------------------------
<Proxy balancer://rails-cluster>
  BalancerMember http://127.0.0.1:8000
  BalancerMember http://127.0.0.1:8001
  BalancerMember http://127.0.0.1:8002
</Proxy>

This setup has three servers in a proxy_balancer cluster that you can access using balancer://rails-cluster/ just as it was one server.

# --------------------------------------------------------
# name-based virtual hosting.
# --------------------------------------------------------
NameVirtualHost *:80

<VirtualHost *:80>
 DocumentRoot "/var/www/myblog.com/current/public"
 ServerName www.myblog.com
 ServerAlias myblog.com

 # -------------------------------------------------
 # Rewrite rules
 # -------------------------------------------------
 RewriteEngine on

 # Force www.myblog.com and make sure we use a 301 HTTP code for the redirect. This is a SEO must.
 RewriteCond %{HTTP_HOST}   !^www\.myblog\.com [NC]
 RewriteCond %{HTTP_HOST}   !^$
 RewriteRule ^/(.*)         http://www.myblog.com/$1 [L,R=301]

 # --------------------------------------------------------
 # List of urls not to proxy
 # --------------------------------------------------------
 ProxyPass /system !
 ProxyPass /images !
 ProxyPass /stylesheets !
 ProxyPass /javascripts !
 ProxyPass /monit/token !
 # Send everything else to the proxy_balancer cluster of rails servers
 ProxyPass / balancer://rails-cluster/
 ProxyPassReverse / balancer://rails-cluster/

 <Directory "/var/www/myblog.com/current/public">
  Options FollowSymLinks
  AllowOverride None
  Order allow,deny
  Allow from all
 </Directory>
 # Before you restart the server you need to create the logs/myblog.com directory.
 # We are also adding the dontlog environment variable here to stop logging the set entries. (This is configured in your httpd.conf)
 ErrorLog  "logs/myblog.com/error_log"
 CustomLog "logs/myblog.com/access_log" combined env=!dontlog

 # --------------------------------------------------------
 # Deflate Module Configuration
 # --------------------------------------------------------
 <IfModule deflate_module>
  AddOutputFilterByType DEFLATE text/plain
  AddOutputFilterByType DEFLATE text/xml
  AddOutputFilterByType DEFLATE application/xhtml+xml
  AddOutputFilterByType DEFLATE text/css
  AddOutputFilterByType DEFLATE application/xml
  AddOutputFilterByType DEFLATE image/svg+xml
  AddOutputFilterByType DEFLATE application/rss+xml
  AddOutputFilterByType DEFLATE application/atom_xml
  AddOutputFilterByType DEFLATE application/javascript
  AddOutputFilterByType DEFLATE application/x-javascript
  AddOutputFilterByType DEFLATE application/x-httpd-php
  AddOutputFilterByType DEFLATE application/x-httpd-fastphp
  AddOutputFilterByType DEFLATE application/x-httpd-eruby
  AddOutputFilterByType DEFLATE text/html
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  BrowserMatch ^Mozilla/4.0[678] no-gzip
 </IfModule>
 # =============================================
 # Configure expires Module
 # =============================================
 <IfModule mod_expires.c>
  ExpiresActive On
  ExpiresDefault "access plus 1 seconds"
  ExpiresByType text/html "access plus 1 seconds"
  ExpiresByType image/gif "access plus 1 week"
  ExpiresByType image/jpeg "access plus 1 week"
  ExpiresByType image/png "access plus 1 week"
  ExpiresByType text/css "access plus 1 week"
  ExpiresByType text/javascript "access plus 1 week"
  ExpiresByType application/x-javascript "access plus 1 week"
  ExpiresByType text/xml "access plus 1 seconds"
 </IfModule>
</VirtualHost>

Here there is alot of information lets try to take it step by step. First we setup a server with the name http://www.myblog.com that also listens to myblog.com but by using mod_rewrite we force everyone to http://www.myblog.com with a 301 redirect. Then we setup all of the static content that we want apache to server from the local filesystem using ProxyPass with a ! to say “do not proxypass” these directories and then we send everything else to the balancer cluster. We setup the access rights to the static directory where our content is stored ( images, javascript, uploaded files and other things like css. ) Now we setup the virtualhosts log file in its own directory inside the logs directory. The mod_deflate and mod_expires configurations work for most setups BUT this piece needs to be monitored and tuned to your setup. I have seen the mod_expires setup give me problems using rails and authentication.

Now of to a secure.myblog.com virtualhost

<VirtualHost _default_:443>
 DocumentRoot "/var/www/myblog.com/current/public"
 ServerName secure.myblog.com
 ServerAlias www.myblog.com myblog.com
 RewriteCond %{HTTP_HOST}   !^secure\.myblog\.com [NC]
 RewriteCond %{HTTP_HOST}   !^$
 RewriteRule ^/(.*)         https://secure.myblog.com/$1 [L,R=301]
 # --------------------------------------------------------
 # List of urls not to proxy
 # --------------------------------------------------------
 ProxyPass /system !
 ProxyPass /images !
 ProxyPass /stylesheets !
 ProxyPass /javascripts !
 ProxyPass / balancer://rails-cluster/
 ProxyPassReverse / balancer://rails-cluster/

 ErrorLog  "logs/myblog.com/error_log"
 CustomLog "logs/myblog.com/access_log" combined env=!donlog

 # --------------------------------------------------------
 # SSL Certificates
 # --------------------------------------------------------
 SSLEngine on
 SSLCertificateFile    /usr/local/apache2/ssl/secure.myblog.com.crt
 SSLCertificateKeyFile /usr/local/apache2/ssl/secure.myblog.com.key
 # --------------------------------------------------------
 # Deflate Module Configuration
 # --------------------------------------------------------
 <IfModule deflate_module>
  AddOutputFilterByType DEFLATE text/plain
  AddOutputFilterByType DEFLATE text/xml
  AddOutputFilterByType DEFLATE application/xhtml+xml
  AddOutputFilterByType DEFLATE text/css
  AddOutputFilterByType DEFLATE application/xml
  AddOutputFilterByType DEFLATE image/svg+xml
  AddOutputFilterByType DEFLATE application/rss+xml
  AddOutputFilterByType DEFLATE application/atom_xml
  AddOutputFilterByType DEFLATE application/javascript
  AddOutputFilterByType DEFLATE application/x-javascript
  AddOutputFilterByType DEFLATE application/x-httpd-php
  AddOutputFilterByType DEFLATE application/x-httpd-fastphp
  AddOutputFilterByType DEFLATE application/x-httpd-eruby
  AddOutputFilterByType DEFLATE text/html
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  BrowserMatch ^Mozilla/4.0[678] no-gzip
 </IfModule>
 # =============================================
 # Configure expires Module
 # =============================================
 <IfModule mod_expires.c>
  ExpiresActive On
  ExpiresDefault "access plus 1 seconds"
  ExpiresByType text/html "access plus 1 seconds"
  ExpiresByType image/gif "access plus 1 week"
  ExpiresByType image/jpeg "access plus 1 week"
  ExpiresByType image/png "access plus 1 week"
  ExpiresByType text/css "access plus 1 week"
  ExpiresByType text/javascript "access plus 1 week"
  ExpiresByType application/x-javascript "access plus 1 week"
  ExpiresByType text/xml "access plus 1 seconds"
 </IfModule>
 # --------------------------------------------------------
 # Document root /
 # --------------------------------------------------------
 <Directory "/var/www/myblog.com/current/public">
  Options FollowSymLinks
  AllowOverride None
  Order allow,deny
  Allow from all
 </Directory>
 # -------------------------------------------------
 # Fixing yet abother IE 6 bug
 # -------------------------------------------------
 BrowserMatch ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
 # -------------------------------------------------
 # Add this to the request header so that
 # rails puts the correct redirect in place
 # -------------------------------------------------
 RequestHeader set X_FORWARDED_PROTO 'https'
</VirtualHost>

This is very similar to the port 80 virtualhost of the same name. the biggest difference is the ssl certificates and then the bottom ssl/https settings to fix issues with Mongrel/rails and ie6. You can’t configure mod_expires and mod_deflate in the main configuration file and have the virtuallhost inheret the configuration so the best solution to be dry would be to put these settings in their own mod_deflate.conf and mod_expires.conf and then include the named configuration files in each virtualhosts configuration file like so:

Include conf/mod_deflate.conf
Include conf/mod_expires.conf

Conclusion

Apache with mod_proxy rocks 🙂 What we have here is Apache that is ready to be expanded to a high performance webserver or proxy server, or both. I like to start with this setup and then build from here but if you had to get more concurrent clients and throughput through your Apache server I would look at your available memory and cpu cycles and maybe do something like so:

<IfModule mpm_worker_module>
  ThreadLimit 100
  StartServers 5
  MaxClients 1000
  MinSpareThreads 100
  MaxSpareThreads 1000
  ThreadsPerChild 100
  MaxRequestsPerChild 0
</IfModule>

This is a high threads and low processes setup and to get the number of processes that Apache will use simply divide MaxClients by ThreadPerChild. So this gives us 10 processes each with a maximum of 100 threads, with a maximum of 1000 clients total. Depending on the server and type of content that you are serving you can load test and increase these settings if you need more than 1000 concurrent users.

I’m a very big fan of mod_proxy and I use mod_proxy_ajp inplace of mod_jk every chance I get. ( I also talk my customers into using mod_proxy over mod_jk if they are using Apache 2.2.x )

Using this setup to include the virtualhosts that the server runs I feel it is easy to add/remove new websites to this setup and it also gives me a good overview of what is running on the server.

iPhone OS 3.0 and none functional calendar integration

Upgraded to iPhone OS 3.0 today, nice!

The changes are minor but I like the new stuff…

There is ONE thing however that I do NOT like or understand. When I get an invite from someone using ical ( Apples calendering tool. ) in a .ics file (or any other calendar tool… ) to my iPhone I can’t look/respond/reject the meeting/event. My phone only shows an ics icon and can’t open the file?!?!?

I also have an exchange server at work that I sync the iPhone too and now that works like it should! I get a invite or changes to an event and I can modify/add/reject the event .

Why can’t Apple get this to work!!!

Am I asking too much from the iPhone? I have had this function on the Blackberry for years… (Been a iPhone user for 2 years, Blackberry for 5)

Search! great

cut’n past great!

Add an invite to your calendar not working !>!>! Unless you use a Microsoft product?

If you are like me and feel this is ?  Then go here and post a feature request… http://www.apple.com/feedback/iphone.html

mail hangs on leopard

The Problem

Started my mail client today and all I got was the menu bar with a nice spinning color wheel 😦

Not sure if this started right after the 10.5.7 upgrade or if there was something else…

The Fix

After running a few different clenup and fix-it tools (and reboots) I almost gave up on my mail but had to try one more last thing. I Deleted the mail folder in my home/Library folder. Then force killed mail and voila! mail booted.

Looking in my trash where the 6GB mail folder was located I found my signatures and all I need todo to get them was quit mail copy the signature folder over the NEW signature folder and start mail again…

I’m glad all my email accounts support IMAP.

Fancy Mac (OS-x) Prompt

The Problem

Boring promt

The Fix

– Open a Terminal window.

– Edit your .profile or create a file like so vi ~/.profile

Add the following to the file:

alias vi=’vim’
3
4 alias ol=’ssh root@192.168.10.206′
5 #alias cade=’ssh 74.86.232.5′
6 alias cade=’ssh 74.63.12.158′
7 alias oce=’ssh root@74.208.45.99′
8
9 # colors
10 export CLICOLOR=1
11 export TERM=xterm-color
12 export LSCOLORS=gxgxcxdxbxegedabagacad  # cyan directories
13 export PS1=’\[33[01;32m\]\u@\h\[33[00m\]:\[33[01;36m\]\w\[33[00m\]\$ ‘

# Starts the extended vi when using vi
alias vi=’vim’
# Nice listing using ll
alias ll=’ls -Alhp’

export CLICOLOR=1
export TERM=xterm-color
# Information about the colors at the bottom of the article
export LSCOLORS=gxgxcxdxbxegedabagacad  # cyan directories
export PS1=’\[33[01;32m\]\u@\h\[33[00m\]:\[33[01;36m\]\w\[33[00m\]\$ ‘

This will make sure that you prompt will have ansi colors and look something like this:
freddy@svn:~/Documents$ ll drwxr-xr-x  17 freddy  staff   578B Feb 10  2008 cadechristian.com_project/ -rw-r--r--@  1 freddy  staff   112K Mar  1  2008 server.docx drwxrwxrwx   4 freddy  staff   136B Jan 28  2008 server_backup/ -rw-r--r--@  1 freddy  staff   559K Mar  3  2008 sunJVM-on-intel-multicoreservers.pdf -rw-r--r--@  1 freddy  staff   556K Mar  1  2008 tomcat_performance_tuning_20071015.ppt freddy@svn:~/Documents$ 

LSCOLOR information

These are the available ANSI colors:

a     black
b     red
c     green
d     brown
e     blue
f     magenta
g     cyan
h     light grey
A     bold black, usually shows up as dark grey
B     bold red
C     bold green
D     bold brown, usually shows up as yellow
E     bold blue
F     bold magenta
G     bold cyan
H     bold light grey; looks like bright white
x     default foreground or background 

Note that the above are standard ANSI colors. The actual display may differ depending on the color capabilities of the terminal in use. The order of the attributes in the LSCOLORS variable is as follows:

  1. directory
  2. symbolic link
  3. socket
  4. pipe
  5. executable
  6. block special
  7. character special
  8. executable with setuid bit set
  9. executable with setgid bit set
  10. directory writable to others, with sticky bit
  11. directory writable to others, without sticky bit

They are set in pairs, foreground (f) then background (b), i.e. fbfbfbfbfbfbfbfbfbfbfb for all 11 settings. The default is exfxcxdxbxegedabagacad, i.e. blue foreground and default background for regular directories, black foreground and red background for setuid executables, etc.

Graphing Mysql Performance in Groundworks

The Problem

Just installed GroundWorks Community Edition 5.3 and this is a great product but when comparing the built-in graphing capabilities to Hyperics community editon it falls short. Adding monitoring to MySQL, Apache or Tomcat does not give you graphs out-of-the-box but this CAN and will be fixed…

FYI I’m assuming that groundworks is installed and working, and also that you have the ability to monitor cpu, memory and more using ssh to your mysql server. If you don’t have this setup then some steps in this article will NOT work.

The Fix

Quick rundown of fix

  • Create a service for Nagios under the configuration screen.
  • Add the new service to your MySQL server
  • Go to performance and setup a “Graph” template for the MySQL service
  • Wait
  • Watch graphs

Ok that was the quick run-down and that was what I just could NOT get to work the first-time around. I’ll go though how you can add multiple graph points and also how to debug the issue if like happened to me NOTHING is graphed or even that the graph is not created.

Setup check_mysql

First we need a user on the MySQL server that has STATUS only access

mysql> grant usage on *.* to ‘nagios’@’groundworks-host’ identified by ‘password’;

Lets create a service and a command that uses the check_mysql script that is supplied with Groundworks.

Command definition: $USER1$/check_mysql -H $HOSTADDRESS$ -u “$ARG1$” -p “$ARG2$” -P “$ARG3$”
Usage: check_mysql_status!ARG1!ARG2!ARG3
Command line: ?

The add your mysql server in the test box and click “test”

Test: Host:
/usr/local/groundwork/nagios/libexec/check_mysql -H 192.168.2.2 -u "nagios" -p "password" -P "3306"

Uptime: 1862427  Threads: 7  Questions: 195920  Slow queries: 2  Opens: 166  Flush tables: 1  Open tables: 159  Queries per second avg: 0.105

Command returned exit status 0

Nice that works! but that is the easy part. Now we need to look at the “performance” area and actually create the graph setup.

Creating the graph

Go to Performance – configure. You will see a long list of the default graphs and now we need to add our own.

The top one should be Current Load click on the copy button and create a new performance graph that will look like this:

Graph Label: MySQL Stats
Service: mysql_status-port-330[1-9]
Use Service as a Regular Expression ON
Host: *
Status Text Parsing Regular Expression: [\w\d\s:]+Threads: (\d+)[\w\d\s:]+avg: ([\d\.]+)
Use Status Text Parsing instead of Performance Data ON
RRD Name /usr/local/groundwork/rrd/$HOST$_$SERVICE$.rrd
RRD Create Command $RRDTOOL$ create $RRDNAME$ –step 60 –start n-1yr DS:Threads:GAUGE:1800:U:U DS:QPS:GAUGE:1800:U:U RRA:AVERAGE:0.5:1:8640 RRA:AVERAGE:0.5:12:9480 2>&1
RRD Update Command $RRDTOOL$ update $RRDNAME$ $LASTCHECK$:$VALUE1$:$VALUE2$ 2>&1
Custom RRDtool Graph Command
Enable ON

I have MySQL servers listening to multiple ports so when I created the graph I made the service name a regular expression.

mysql_status-port-330[1-9]

This matches the three service names that I have created

mysql_status-port-3306
mysql_status-port-3307
mysql_status-port-3308

The name of the service IS VERY IMPORTANT when you create the graph. When the performance data is imported in the performance graph Groundworks does a search through the database to find the service data using the service name. Since I setup the service with multiple ports for the same mysql_status service it adds the port-3306… to each of the services and that is the name stored in the database. Here is how I setup the multiple ports

In the service check of the hosts mysql_status I added three instances at the bottom of that screen.

Instance Name Suffix Status Arguments
Active
Active
Active

Ok back to the graph setup…

After you have saved the new performance graph there are two directories that you need to worry about.

/usr/local/groundworks/rrd
and
/usr/local/groundworks/nagios/eventhandler

rrd

In the rrd directory you will get a file that is named something like mysqlservername.cadechristian.com_mysql_status-port-3306.rrd, this file is created using the RRD Create Command from the performance template. That will will also be updated using the RRD Update Command.

Lets talk about the RRD Create command

$RRDTOOL$ create $RRDNAME$ –step 60 –start n-1yr DS:Threads:GAUGE:1800:U:U DS:QPS:GAUGE:1800:U:U RRA:AVERAGE:0.5:1:8640 RRA:AVERAGE:0.5:12:9480 2>&1

The important part is

–step (how often to plot on the graph) 300 is a good production number but for testing I like 60

DS:Threads:GAUGE:1800:U:U – First datasource
DS:QPS:GAUGE:1800:U:U – Second Datasource

2>&1 put errors to standard out log.

The datasources are found through the regex that I setup that parses the output from the mysql command ( Each datasource is one regex group ):

[\w\d\s:]+Threads: (\d+)[\w\d\s:]+avg: ([\d\.]+)

The (\d+) is how you create a group with the () around what you are matching. So in this regex we get the first group with the number of threads and the second group with the Queries per second.

Now it is important that we also have 2 values in the RRD update command.

$RRDTOOL$ update $RRDNAME$ $LASTCHECK$:$VALUE1$:$VALUE2$ 2>&1

If you need to change the RRD create command you need to MANUALLY DELETE the RRD file in the /usr/local/groundworks/rrd directory!!!

The rrd file is created in a default setup after 1-2 min and the performance data will be parsed and visible in the graph maybe after 5min.

Debugging

The “event” scripts are located in /usr/local/groundworks/nagios/eventhandlers and inside of the scripts there is a variable called $debug that you should set to 3 ( default = 0 ) if you want to get a more descriptive output in the logs.

The  logfile /usr/local/groundworks/nagios/eventhandlers/service_perfdata.log has very light debug information but it shows when the service is pulled for a name host

1244226293 den2mysql002.openlogic.com mysql_status-port-3306 Uptime: 1864121  Threads: 7  Questions: 196033  Slow queries: 2  Opens: 166  Flush tables: 1  Open tables: 159  Queries per second avg: 0.105
1244226296 den2mysql002.openlogic.com tcp_ssh SSH OK – OpenSSH_4.3 (protocol 2.0)
1244226297 olex-perfdb icmp_ping PING OK – Packet loss = 0%, RTA = 0.13 ms
1244226293 mysqlserver mysql_status-port-3306 Uptime: 1864121  Threads: 7  Questions: 196033  Slow queries: 2  Opens: 166  Flush tables: 1  Open tables: 159  Queries per second avg: 0.105 1244226296 mysqlserver tcp_ssh SSH OK - OpenSSH_4.3 (protocol 2.0) 1244226297 mysqlserver icmp_ping PING OK - Packet loss = 0%, RTA = 0.13 ms

This is ok information BUT it will not tell us if something did NOT work. So lets add $debug = 3 to the process_service_perf(_db AND _db_file).pl file and wait for the process_service_perf.log file to be created in the eventhandlers directory

And here is a good entry from the debug log:

Fri Jun  5 14:43:14 MDT 2009
Host: mysqlserver
Svcdesc: mysql_status-port-3306
Lastcheck: 1244234573
Statustext: Uptime: 1872401  Threads: 7  Questions: 196614  Slow queries: 2  Opens: 166  Flush tables: 1  Open tables: 159  Queries per second avg: 0.105

Perfdata:
No exact service name mysql_status-port-3306. Query database for service pattern matches.mysql_status-port-3306 matches database service pattern mysql_status-port-330[1-9]. Using this entry.

Match in status text for regular expression (?-xism:[\w\d\s:]+Threads: (\d+)[\w\d\s:]+avg: ([\d\.]+))

SQL = SELECT hs.host_service_id FROM host_service as hs, datatype as dt WHERE (hs.host=’mysqlserver’ AND hs.service=’mysql_status-port-3306′ AND dt.type=’RRD’ AND dt.location=’/usr/local/groundwork/rrd/mysqlserver_mysql_status-port-3306.rrd’ AND hs.datatype_id=dt.datatype_id)

Table host_service, host=mysqlserver, service=mysql_status-port-3306 already has an exisitng entry for location /usr/local/groundwork/rrd/mysqlserver_mysql_status-port-3306.rrd. New entry not added.
/usr/local/groundwork/common/bin/rrdtool update /usr/local/groundwork/rrd/mysqlserver_mysql_status-port-3306.rrd 1244234573:7:0.105 2>&1

Return:
Label Threads found in rrd create string: $RRDTOOL$ create $RRDNAME$ –step 60 –start n-1yr:GAUGE:1800:U:U DS:QPS:GAUGE:1800:U:U RRA:AVERAGE:0.5:1:8640 RRA:AVERAGE:0.5:12:9480 2>&1
Label QPS found in rrd create string: $RRDTOOL$ create $RRDNAME$ –step 60 –start n-1yr:GAUGE:1800:U:U:GAUGE:1800:U:U RRA:AVERAGE:0.5:1:8640 RRA:AVERAGE:0.5:12:9480 2>&1

Posting data to Foundation
performancedatalabel=Threads
performancevalue=7
performancedatalabel=QPS
performancevalue=0.105
Elapsed Execution time = 4.50323104858398 seconds

You see that it finds the service name using the regex for the service, then we find two values that are stuffed into the two labels that we have (Threads, QPS)

One of the biggest issues I had was with the collected performance data. After I found that I could use regex everything started working much better.

Conclusion

Using Groundworks to monitor and get graphs is NOW easy but it was a pain to get to this point. I turning point was finding the eventhandler debug log. I hope this will be helpful to someone out there…

check_apache.pl and perl module issue

The problem

[nagios@den2lweb002 libexec]$ ./check_apache.pl
Can’t locate LWP/UserAgent.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at ./check_apache.pl line 32.
BEGIN failed–compilation aborted at ./check_apache.pl line 32.

[nagios@myserver libexec]$ ./check_apache.pl -H localhost

Can’t locate LWP/UserAgent.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at ./check_apache.pl line 32.

BEGIN failed–compilation aborted at ./check_apache.pl line 32.

The fix

Lets find the missing LWP/UserAgent.pm module. After searching around on Google I could not find that module alone but I found the Bundle::LWP, so I started this command after a quick su to root…

[root@myserver libexec]# perl -MCPAN -e ‘install Bundle::LWP’

And after about 2 minutes the bundle was installed

Then I dropped back to the nagios user and tested the check_apache.pl command again…

[nagios@myserver libexec]$ ./check_apache.pl -H localhost

HTTPD ok: average response time 184 milliseconds

We are back in business!

Versions used

  • Linux Red Hat Enterprise Linux Server release 5.3
  • Perl v5.8.8 built for x86_64-linux-thread-multi
  • Ground Works 5.3 community edition