joining a critsit in a large bank in Athens – WebSphere Portal 5.1, WCM, TAM ,TDS, TDI, ESB, db2, etc. Massive performance issues caused by inadequately tuning the environment, lack of system resources – basically returned to default values following tuning guides. Primary intention was to stabilize environment, as well as providing mid- to long term recommendations about how to manage the infrastructure and systems / applications.

Athens is a beast with traffic collapsing – would go mad living here. Akropolis area (Plaka) very touristic (what did I expect), but nice to switch off from a busy day (apart from all the little gangsters trying to get hold of your money).

edit /etc/apache2/mod_log_config.conf and add a new logformat (plots) – (“mod_log_config.conf” is referenced by “httpd.conf”)

LogFormat “%h %l %u %t \”%r\” %>s %b \ \”%{Referer}i\” \”%{User-Agent}i\”" combined
LogFormat “%h %{%d.%m.%Y:%H:%M:%S}t %D %U” plots

The characteristics of the “%” directives are:

%…h Remote host
%…{format}t time & date
%…D time taken to serve the request, in microseconds.
%…U URL path requested

Add another “customlog” directive to your “virtualhost” section in “httpd.conf”:

DocumentRoot /home/h/hensler.net/public_html/bernhard/
ServerName bernhard.hensler.net
IndexOptions
DirectoryIndex index.htm index.html index.shtml start.htm start.html start.shtm index.php

CustomLog “/usr/local/visas/logfiles/hensler.net/%Y/%m/%d/access_log” vhost_combined
CustomLog “/usr/local/visas/logfiles/hensler.net/bernhard.access_log” plots

Concatenate logs from all virtual hosts e.g.: cat hensler.access_log niko.access_log bernhard.access_log max.access_log > plot_log (sample line: 66.249.111.111 30.08.2009:14:15:17 4372853 /blog/) and start gnuplot from the command line:

$ gnuplot

reset
set terminal png small color
set output “2dplot.png”
set title “average response time”

set style data points
set pointsize 1
set grid

set xlabel “time”
set timefmt “%d.%m.%Y:%H:%M:%S”
set format x “%H:%M\n%d/%b”
set xdata time
set xrange [ "30.08.2009:00:00" : "30.08.2009:23:59" ]

set ylabel “response time”
set yrange [ 0 : 10000 ]

plot “/usr/local/visas/logfiles/hensler.net/plot_log” using 2:3 title “2d”

gnuplot 3d

Then read this excellent article about “A New Visualization for Web Server Logs” and create a perl script:

#
# prepare-for-gnuplot.pl: convert access log files to gnuplot input
# Raju Varghese. 2007-02-03

use strict;

my $tempFilename = “./tmp/temp.dat”;
my $ipListFilename = “./tmp/iplist.dat”;
my $urlListFilename = “./tmp/urllist.dat”;

my (%ipList, %urlList);

sub ip2int {
my ($ip) = @_;
my @ipOctet = split (/\./, $ip);
my $n = 0;
foreach (@ipOctet) {
$n = $n*256 + $_;
}
return $n;
}

# prepare temp file to store log lines temporarily
open (TEMP, “>$tempFilename”);

# reads log lines from stdin or files specified on command line

while (<>) {
chomp;
my ($ip, $time, $D, $url, $sc) = split;
$time =~ s/\[//;
next if ($url =~ /(gif|jpg|png|js|css)$/);
print TEMP "$ip $time $D $url $sc\n";
$ipList{$ip}++;
$urlList{$url}++;
}

# process IP addresses

my @sortedIpList = sort {ip2int($a) <=> ip2int($b)} keys %ipList;
my $n = 0;
open (IPLIST, ">$ipListFilename");
foreach (@sortedIpList) {
++$n;
print IPLIST "$n $ipList{$_} $_\n";
$ipList{$_} = $n;
}
close (IPLIST);

# process URLs

my @sortedUrlList = sort {$urlList {$b} <=> $urlList {$a}} keys %urlList;
$n = 0;
open (URLLIST, ">$urlListFilename");
foreach (@sortedUrlList) {
++$n;
print URLLIST "$n $urlList{$_} $_\n";
$urlList{$_} = $n;
}
close (URLLIST);

close (TEMP); open (TEMP, $tempFilename);
while () {
chomp;
my ($ip, $time, $D, $url, $sc) = split;
print "$time $ipList{$ip} $urlList{$url} $sc\n";
}
close (TEMP);

Run this perl script and redirect output to a file from the command line:

$ perl gnuplot.pl "/usr/local/visas/logfiles/hensler.net/bernhard.access_log" > gnuplot.input

The fields in gnuplot.input, the output file of the Perl script, are date/time, ip rank, url rank.

Run gnuplot from the command line: $ gnuplot and the following commands:

reset
set terminal png small color
set output "3dplot.png"
set style data dots
set xdata time
set timefmt "%d.%m.%Y:%H:%M:%S"
set zlabel "Content"
set ylabel "IP address"
splot "gnuplot.input" using 1:2:3 title "3d"

Image taken from oreillynet, my website is not producing sufficient data …

• X, the time axis–a full day from midnight to midnight of November 16.
• Y, the requester’s IP address, with the conventional dotted decimal format sorted and given an ordinal number between 1 and 120,000, representing the number of clients that accessed the web server.
• Z, the URL (or content) sorted by popularity. Of the approximately 60,000 distinct pages on the site, the most popular URLs are near the zero point of the Z-axis and the least popular ones at the top.

http://www.ibm.com/developerworks/linux/library/lgnuplot

http://www.oreillynet.com/pub/a/sysadmin/2007/02/02/3d-logfile-visualization.html?page=1

http://phasorburn.com/index.php/archive/excel-0-gnuplot-1

A final step will cover loadrunner tools like openSTA and jmeter.

See also Part I of this tutorial.

<VirtualHost *:443>
DocumentRoot “var/local/wwwroot/bernhard/”
ServerName bernhard.hensler.net
ErrorLog /var/log/apache2/error_log
TransferLog /var/log/apache2/access_log
SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+
LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile /etc/apache2/ssl.crt/server.crt
SSLCertificateKeyFile /etc/apache2/ssl.key/server.key
</VirtualHost>

Fortunately there was no need to go through the painful process of creating my own CA etc., but instead I was able to use the “built-in” certifcate of my provider. To read a SSL certificate: openssl x509 -noout -text -in certificate.crt

Signature Algorithm: md5WithRSAEncryption
Issuer: C=DE, O=hensler.net, OU=IT, CN=bernhard.hensler.net/emailAddress=bhensler at gmail dot com
Validity
Not Before: Aug 24 17:31:07 2009 GMT
Not After : Aug 22 17:31:07 2019 GMT
Subject: C=DE, O=hensler.net, CN=bernhard.hensler.net/emailAddress=bhensler at gmail dot com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (4096 bit)
Modulus (4096 bit):

check Apache server version by creating a simple e.g. serverinfo.php file with following statement: <?php phpinfo(); ?>

and open in your browser: http://your host/serverinfo.php. Find the version info in the Apache Version section:

Apache Version – Apache/2.0.53 (Linux/SUSE)

or run from the command line:

/usr/sbin # apache2ctl -v

Server version: Apache/2.0.53

Server built: Aug 30 2006 13:14:23

h969344:/usr/sbin #

Next add mod_status to your APACHE_MODULES by editing /etc/sysconfig/apache2 configuration – the Status module allows a server administrator to find out how well a server is performing. A HTML page is presented that gives the current server statistics in an easily readable form:

APACHE_MODULES=

“access actions alias auth auth_dbm autoindex cgi

dir env expires headers include log_config mime

mod_status negotiation setenvif

ssl suexec userdir php4 rewrite”

In the same file and if extended logging is needed, set APACHE_EXTENDED_STATUS=”on” – be careful here, this will degrade performance.

Edit httpd.conf and add a Location section to your virtualhost section (remove Include /etc/apache2/mod_status.conf):

<VirtualHost *:80>

….

<Location /server-status>

SetHandler server-status

# Order deny,allow

# Deny from all

Allow from .yourdomain

</Location>

….

</VirtualHost>

restart the HTTP server: /usr/sbin/rcapache2 restart

You should see your Apache server status now by issuing this URL: http://yourHost/server-status providing the following information:

• The number of worker serving requests
• The number of idle worker
• The status of each worker, the number of requests that worker has performed and the total number of bytes served by the worker (*)
• A total number of accesses and byte count served (*)
• The time the server was started/restarted and the time it has been running for
• Averages giving the number of requests per second, the number of bytes served per second and the average number of bytes per request (*)
• The current percentage CPU used by each worker and in total by Apache (*)
• The current hosts and requests being processed (*)

The lines marked “(*)” are only available if ExtendedStatus is On

awstats

To display HTTP server log information in a graphical form, install awstats:

AWStats is a free powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically.

Copy awstats to a directory of your choice, installation defaults to /usr/local/awstats. Then run the awstats configuration process: perl awstats_configure.pl, which will add directives to your httpd.conf and create a configuration file (default location: /etc/awstats)

<Directory “/usr/local/awstats/wwwroot”>

Options all

AllowOverride None

Order allow,deny

Allow from all

</Directory>

Alias /awstatsclasses “/usr/local/awstats/wwwroot/classes/”

Alias /awstatscss “/usr/local/awstats/wwwroot/css/”

Alias /awstatsicons “/usr/local/awstats/wwwroot/icon/”

ScriptAlias /awstats/ “/usr/local/awstats/wwwroot/cgi-bin/”

Once done run the data import process from the cgi-bin folder of your awstats installation (you can automate this in the config file): perl awstats.pl config=yourdomain. See results of your site: http://yourdomain/awstats/awstats.pl or at this demo site

This link to part II of the tutorial.

WASX7209I: Connected to process “WebSphere_Portal” on node wcl using SOAP connector; the type of process is: unManagedProcess; WASX7029I: For help, enter: “$Help help”

wsadmin> set jvm [$AdminControl completeObjectName type=JVM,process=WebSphere_Portal,*]

WebSphere:name=JVM,process=WebSphere_Portal,
platform=dynamicproxy,node=wcl,j2eeType=JVM,
J2EEServer=WebSphere_Portal,version=6.0.2.17,
type=JVM,mbeanIdentifier=JVM,cell=wcl

wsadmin>$AdminControl invoke $jvm generateHeapDump
wsadmin>$AdminControl invoke $jvm dumpThreads

IBM’s support assistant (ISA) is an Eclipse based tool with quite a number of plugins to analyze heapdumps – a good developerworks article can be found here (part I) and here (part II).

To analyze WebSphere Portal performance issues, although this document is applicable to isolate performance problems around Java based application servers in general , see here.

see also here

Auf Anfrage bei click&buy heisst es lediglich, dass ich wohl mit meinen Passwörtern nicht sorgsam genug umgegangen bin – stellt sich die Frage nach dem Huhn und dem Ei; ich meine, dass  eine Internet Bank so sicher sein muss, dass keinerlei Daten dechiffrier- oder hackbar sind – zumal meine mailverbindungen aussschliesslich über SSL laufen. Wie auch immer, der Kunde ist der Depp – bleibt nur der Rat: Finger weg von click&buy !

Ich glaub’, ich frage mal in 3 Monaten wieder nach … ob es einen neuen Account auf meinen Namen bzw. mail Adresse gibt.

• ssh login to your CS407
• ipkg -force-depends install bind (provided you have configured optware for your CS407) — link –
• I pretty much followed the instructions for setting up and configuring a primary DNS server, so nothing to add from my side
• reverse DNS lookup of any IP address in the local zone doesn’t work so far (any hint appreciated)

Alternatively a DNSMasq package is available …

• ssh login to your CS407
• ipkg -force-depends install openldap (provided you have configured optware for your CS407) — link –
• cd /opt/etc/openldap/ and edit slapd.conf and add the following lines (take a look at the README to define which schema files are appropriate for your environment) – below listed schema files are made available through the installation process:

include         /opt/etc/openldap/schema/core.schema
include         /opt/etc/openldap/schema/cosine.schema
include         /opt/etc/openldap/schema/inetorgperson.schema
include         /opt/etc/openldap/schema/rfc2307bis.schema
include         /opt/etc/openldap/schema/ppolicy.schema

• then start the ldap daemon with the following command: /opt/libexec/slapd
• I use “Apache Directory Studio” (an Eclipse based LDAP Browser and Directory client) to manage and administer the openldap (you should find the credentials in slapd.conf).
• create your base DN (e.g.: dc=private,dc=lan)
• once a base DN has been created you should find the following lines in slapd.conf:

##################
# BDB database definitions
##################

database        bdb
suffix          “dc=private,dc=lan”
rootdn          “cn=administrator,dc=private,dc=lan”
rootpw          ***********
directory       /opt/var/openldap-data
checkpoint 1024 5
cachesize 10000
# Indices to maintain
checkpoint 1024 5
cachesize 10000
index objectClass,uidNumber,gidNumber eq
index member,mail eq,pres
index cn,displayname,uid,sn,givenname sub,eq,pres

• once your base DN definitions are OK, you can continue creating / importing your user / groups definitions (e.g. by creating ldif files)
• if you need to add your own objectclass or attribute definitions, take a look at a schema extension file I created to provide basic Lotus Notes/Domino LDAP attributes (attibute definitions must be defined first):

objectidentifier DominoOC 2.16.840.1.113678.2.2.2.1.1
objectidentifier DominoAT 2.16.840.1.113678.2.2.2.2.1

##
## Attribute Section
##

attributetype ( DominoAT:1 NAME ( ‘MailServer’ )
EQUALITY caseIgnoreMatch
SUBSTR caseIgnoreSubstringsMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )

attributetype ( DominoAT:2 NAME ( ‘MailFile’ )
EQUALITY caseIgnoreMatch
SUBSTR caseIgnoreSubstringsMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )

attributetype ( DominoAT:3 NAME ( ‘HTTP-HostName’ )
EQUALITY caseIgnoreMatch
SUBSTR caseIgnoreSubstringsMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )

attributetype ( DominoAT:4 NAME ( ‘HTTP-Port’ )
EQUALITY caseIgnoreMatch
SUBSTR caseIgnoreSubstringsMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )

attributetype ( DominoAT:5 NAME ( ‘notesDN’ )
DESC ‘attribute to uniquely identify a domino user’
EQUALITY caseIgnoreMatch
SUBSTR caseIgnoreSubstringsMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )

##
## Objectclass Section
##

objectclass ( DominoOC:1 NAME ( ‘dominoPerson’ ) SUP top AUXILIARY
DESC ‘represents the dominoPerson object class’
MAY ( notesDN $ MailServer $ MailFile ) )

objectclass ( DominoOC:2 NAME ( ‘dominoServer’ ) SUP top STRUCTURAL
DESC ‘represents the dominoServer object class’
MAY  ( cn $ displayName $ description $ HTTP-HostName $ HTTP-Port))

• add the schema extensions using the include command to your slapd.conf
• the installation also adds a script to automatically start the ldap daemon when rebooting your cs407 (/opt/etc/init.d/S58slapd)
• if you need to stop the ldap process: killall slapd

1. download and extract openvpn-2.x.zip
2. copy easy-rsa folder to /opt/etc/openvpn/easy-rsa
3. edit vars in folder /opt/etc/openvpn/easy-rsa
   export KEY_CONFIG=/opt/etc/openvpn/easy-rsa/openssl.cnf
   export KEY_DIR=/opt/etc/openvpn/private.lan/keys
   export KEY_COUNTRY=YOURCOUNTRY
   export KEY_PROVINCE=YOURPROVINCE
   export KEY_CITY=YOURCITY
   export KEY_ORG=”YOURCOMPANY”
   export KEY_EMAIL=”YOUREMAILADDRESS”
4. switch to bash shell: bash-3.2#
5. . vars
6. ./clean-all

As you create certificates, keys, and certificate signing requests, understand that only .key files should be kept confidential.  .crt and .csr files can be sent over insecure channels such as plaintext email.

• Start with building your own Certificate Authority
• ./build-ca

Generating a 1024 bit RSA private key, writing new private key to ‘ca.key’
You are about to be asked to enter information that will be incorporated into your certificate request. This information is retrieved from your vars file
For the Common Name use  e.g. a combination of your server name and MAC address: CS4071101AF2018964

• Create your server key files
• ./build-key-server server

Generating a 1024 bit RSA private key, writing new private key to ’server.key’
Common Name (eg, your name or your server’s hostname) :cs407
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y

• Then create your client key files
• ./build-key thinkpad

Generating a 1024 bit RSA private key, writing new private key to ‘thinkpad.key’
Common Name (eg, your name or your server’s hostname) []:thinkpad
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y

• Generate Diffie Hellman parameters
• ./build-dh

Using tls-auth requires that you generate a shared-secret key that is used in addition to the standard RSA certificate/key:

• openvpn –genkey –secret ta.key

In the server configuration file (server.ovpn), add:

tls-auth ta.key 0
max-clients 5 (or any number)

Copy the following files to the CS407 folder /opt/etc/openvpn/config:

1. ca.crt
2. dh1024.pem
3. server.crt
4. server.key
5. server.ovpn
6. ta.key

Copy the following files to your client folder C:\Program Files\OpenVPN\config

1. ca.crt
2. thinkpad.crt
3. thinkpad.key
4. client.ovpn
5. ta.key

In the client configuration file (client.ovpn), add:

tls-auth ta.key 1

Restart openvpn on your CS407 and connect your openvpn client:

cd /opt/etc/init.d
sh S24openvpn

In case you need to revoke access for specific users:

. vars
./revoke-full “clientname”