Archive for June, 2010

Optimize and Tweak High-Traffic Servers

June 30th, 2010

Summary

If you are reaching the limits of your server running Apache serving a lot of dynamic content, you can either spend thousands on new equipment or reduce bloat to increase your server capacity by anywhere from 2 to 10 times. This article concentrates on important and poorly-documented ways of increasing capacity without additional hardware.

Problems
There are a few common things that can cause server load problems, and a thousand uncommon. Let’s focus on the common:

1. Drive Swapping – Too many processes (or runaway processes) using too much RAM
2. CPU – poorly optimized DB queries, poorly optimized code, runaway processes
3. Network – Hardware limits, moron attacks

Solutions:
Briefly, and for completeness, here are the most obvious solutions:
1. Use “TOP” and “PS axu” to check for processes that are using too much CPU or RAM.
2. Use “netstat -anp | sort -u” to check for network problems.

Solutions: Apache’s RAM Usage
First and most obvious, Apache processes use a ton a RAM. This minor issue becomes a major issue when you realize that after each process has done its job, the bloated process sits and spoon-feed data to the client, instead of moving on to bigger and better things. This is further compounded by a bit of essential info that should really be more common knowledge:

If you serve 100% static files with Apache, each httpd process will use around 2-3 megs of RAM.
If you serve 99% static files & 1% dynamic files with Apache, each httpd process will use from 3-20 megs of RAM (depending on your MOST complex dynamic page).

This occurs because a process grows to accommodate whatever it is serving, and NEVER decreases again unless that process happens to die. Quickly, unless you have very few dynamic pages and major traffic fluctuation, most of your httpd processes will take up an amount of RAM equal to the largest dynamic script on your system. A smart web server would deal with this automatically. As it is, you have a few options to manually improve RAM usage.

Reduce wasted processes by tweaking KeepAlive
This is a tradeoff. KeepAliveTimeout is the amount of time a process sits around doing nothing but taking up space. Those seconds add up in a HUGE way. But using KeepAlive can increase speed for both you and the client – disable KeepAlive and the serving of static files like images can be a lot slower. I think it’s best to have KeepAlive on, and KeepAliveTimeout very low (like 1-2 seconds).

Limit total processes with MaxClients
If you use Apache to serve dynamic content, your simultaneous connections are severely limited. Exceed a certain number, and your system begins cannibalistic swapping, getting slower and slower until it dies. A web server should automatically take steps to prevent this, but instead they seem to assume you have unlimited resources. Use trial & error to figure out how many Apache processes your server can handle, and set this value in MaxClients. Note: the Apache docs on this are misleading – if this limit is reached, clients are not “locked out”, they are simply queued, and their access slows. Based on the value of MaxClients, you can estimate the values you need for StartServers, MinSpareServers, & MaxSpareServers.

Force processes to reset with MaxRequestsPerChild
Forcing your processes to die after a while makes them start over with low RAM usage, and this can reduce total memory usage in many situations. The less dynamic content you have, the more useful this will be. This is a game of catch-up, with your dynamic files constantly increasing total RAM usage, and restarting processes constantly reducing it. Experiment with MaxRequestsPerChild – even values as low as 20 may work well. But don’t set it too low, because creating new processes does have overhead. You can figure out the best settings under load by examining “ps axu –sort:rss”. A word of warning, using this is a bit like using heroin. The results can be impressive, but are NOT consistent – if the only way you can keep your server running is by tweaking this, you will eventually run into trouble. That being said, by tweaking MaxRequestsPerChild you may be able to increase MaxClients as much as 50%.

Apache Further Tweaking
For mixed purpose sites (say image galleries, download sites, etc.), you can often improve performance by running two different apache daemons on the same server. For example, we recently compiled apache to just serve up images (gifs,jpegs,png etc). This way for a site that has thousands of stock photos. We put both the main apache and the image apache on the same server and noticed a drop in load and ram usage. Consider a page had about 20-50 image calls — the were all off-loaded to the stripped down apache, which could run 3x more servers with the same ram usage than the regular apache on the server.

Finally, think outside the box: replace or supplement Apache

Use a 2nd server
You can use a tiny, lightning fast server to handle static documents & images, and pass any more complicated requests on to Apache on the same machine. This way Apache won’t tie up its multi-megabyte processes serving simple streams of bytes. You can have Apache only get used, for example, when a php script needs to be executed.

Try Turck MMCache
Turck MMCache is a free open source PHP accelerator, optimizer, encoder and dynamic content cache for PHP. It increases performance of PHP scripts by caching them in compiled state, so that the overhead of compiling is almost completely eliminated. Also it uses some optimizations to speed up execution of PHP scripts. Turck MMCache typically reduces server load and increases the speed of your PHP code by 1-10 times. More details please see this article http://onaxer.com/blog/?p=530

Try HAproxy
You can use some open source or hardware loadblancer to devide the load among multiple servers. For open source Haproxy is best options (http://haproxy.1wt.eu/) as I am using Haproxy past couple of yesrs and i have very -2 good experience. There so many types of hardware Loadblancer like Cisco Local Director etc..

Try Varnish
We cab also use caching server like Varnish. Varnish is an HTTP accelerator designed for content-heavy dynamic web sites. In contrast to other HTTP accelerators, many of which began life as client-side proxies or origin servers, Varnish was designed from the ground up as an HTTP accelerator.
http://en.wikipedia.org/wiki/Varnish_%28software%29. Using Varnish we can handle thousands of users without any problems, most of company using caching technologies to enhance their performance.

Try lingerd
Lingerd takes over the job of feeding bytes to the client after Apache has fetched the document, but requires kernel modification. Sounds pretty good, haven’t tried it. lingerd – http://www.iagora.com/about/software/lingerd/

Use a proxy cache
A proxy cache can keep a duplicate copy of everything it gets from Apache, and serve the copy instead of bothering Apache with it. This has the benefit of also being able to cache dynamically generated pages, but it does add a bit of bloat.

Solutions: PHP’s CPU & RAM Usage
Compiling PHP scripts is usually more expensive than running them. So why not use a simple tool that keeps them precompiled? I highly recommend Turck MMCache. Alternatives include PHP Accelerator, APC, & Zend Accelerator. You will see a speed increase of 2x-10x, simple as that. I have no stats on the RAM improvement at this time.

Solutions: Optimize Database Queries
This is covered in detail everywhere, so just keep in mind a few important notes: One bad query statement running often can bring your site to its knees. Two or three bad query statements don’t perform much different than one. In other words, if you optimize one query you may not see any server-wide speed improvement. If you find & optimize ALL your bad queries you may suddenly see a 5x server speed improvement. The log-slow-queries feature of MySQL can be very helpful.

How to log slow queries:
# vi /etc/rc.d/init.d/mysqld

Find this line:
SAFE_MYSQLD_OPTIONS=”–defaults-file=/etc/my.cnf”

change it to:
SAFE_MYSQLD_OPTIONS=”–defaults-file=/etc/my.cnf –log-slow-queries=/var/log/slow-queries.log”

As you can see, we added the option of logging all slow queries to /var/log/slow-queries.log
Close and save mysqld. Shift + Z + Z

touch /var/log/slow-queries.log
chmod 644 /var/log/slow-queries.log

restart mysql
service myslqd restart
mysqld will log all slow queries to this file.

Cheers!!
Manoj Chauhan

View Comments »

Posted in Apache, CentOS, Cluster, Load balancing, HTTP accelerators, Haproxy, Linux, Memcache, PHP

Tags:

Yum installation on RHEL4

June 29th, 2010

Yum prerequistes are:

python-elementtree is needed by yum-2.4.2-0.4.el4.rf.noarch
python-sqlite is needed by yum-2.4.2-0.4.el4.rf.noarch
urlgrabber is needed by yum-2.4.2-0.4.el4.rf.noarch
and the prereques for Uvh python-sqlite is:
libsqlite-2.8.15-1.i386.rpm

These rpm are to be installed in the following order

# rpm -Uvh libsqlite-2.8.15-1.i386.rpm
# rpm -Uvh python-elementtree-1.2.6-7.el4.rf.i386.rpm

#rpm -Uvh python-sqlite-0.5.0-1.2.el4.rf.i386.rpm

# rpm -Uvh python-urlgrabber-2.9.7-1.2.el4.rf.noarch.rpm

# rpm -Uvh yum-2.4.2-0.4.el4.rf.noarch.rpm

we can also skip any dependency using –nodeps

To confirm Yum is installed and functional:
#yum -h

View Comments »

Posted in Linux

Tags:

Http monitoring using httptop

June 29th, 2010

One of the tools most Linux/Unix admins are used to is called “top”. “top” by itself is a very powerful tool.

Installation
Perl modules in cpan

install Term::ReadKey
install File::Tail
install Time::HiRes

Create a executable script
#touch /usr/bin/httptop
#chmod a+x /usr/bin/httptop
Copy & paste the following contents in the script

#!/usr/bin/perl -w
use Time::HiRes qw( time );
use File::Tail ( );
use Term::ReadKey;
use Getopt::Std;
use strict;
### Defaults you might be interested in adjusting.
my $Update = 2; # update every n secs
my $Backtrack = 250; # backtrack n lines on startup
my @Paths = qw(
%
/title/%/logs/access_log
/var/log/httpd/%/access_log
/usr/local/apache/logs/%/access_log
);
my $Log_Format = “combined”;
my %Log_Fields = (
combined => [qw/ Host x x Time URI Response x Referer Client /],
vhost => [qw/ VHost Host x x Time URI Response x Referer Client /]
);
### Constants & other thingies. Nothing to see here. Move along.
my $Version = “0.4.1″;
sub by_hits_per ( ) { $b->{Rate} $a->{Rate} }
sub by_total ( ) { $b->{Total} $a->{Total} }
sub by_age ( ) { $a->{Last} $b->{Last} }
my $last_field = “Client”;
my $index = “Host”;
my $show_help = 0;
my $order = \&by_hits_per;
my $Help = “htlwufd?q”;
my %Keys = (
h => [ "Order by hits/second" => sub { $order = \&by_hits_per } ],
t => [ "Order by total recorded hits" => sub { $order = \&by_total } ],
l => [ "Order by most recent hits" => sub { $order = \&by_age } ],
w => [ "Show remote host" => sub { $index = "Host" } ],
u => [ "Show requested URI" => sub { $index = "URI" } ],
f => [ "Show referring URL" => sub { $index = "Referer" } ],
d => [ "Show referring domain" => sub { $index = "Domain" } ],
‘?’ => [ "Help (this thing here)" => sub { $show_help++ } ],
q => [ "Quit" => sub { exit } ]
);
my @Display_Fields = qw/ Host Date URI Response Client Referer Domain /;
my @Record_Fields = qw/ Host URI Referer Domain /;
my $Max_Index_Width = 50;
my $Initial_TTL = 50;
my @Months = qw/ Jan Feb Mar Apr May Jun Jul Aug Sep Nov Dec /;
my %Term = (
HOME => “33[H",
CLS => "33[2J",
START_TITLE => "33]0;”, # for xterms etc.
END_TITLE => “07″,
START_RV => “33[7m",
END_RV => "33[m"
);
my ( %hist, %opt, $spec );
$SIG{INT} = sub { exit };
END { ReadMode 0 };
### Subs.
sub refresh_output
{
my ( $cols, $rows ) = GetTerminalSize;
my $show = $rows - 3;
my $count = $show;
my $now = (shift || time);
for my $type ( values %hist ) {
for my $peer ( values %$type ) {
# if ( --$peer->{_Ttl} > 0 ) {
my $delta = $now - $peer->{Start};
if ( $delta >= 1 ) {
$peer->{ Rate } = $peer->{ Total } / $delta;
} else {
$peer->{ Rate } = 0
}
$peer->{ Last } = int( $now - $peer->{ Date } );
# } else {
# delete $type->{$peer}
# }
}
}
$count = scalar( values %{$hist{$index}} ) - 1 if $show >= scalar values %{$hist{$index}};
my @list = ( sort $order values %{$hist{$index}} )[ 0 .. $count ];
my $first = 0;
$first = ( $first {$index}) : 0 } @list;
$first = $Max_Index_Width if $Max_Index_Width {_Ttl}++;
my $line = sprintf( “%-${first}s %6.3f %4d %3d %s”,
substr( $_->{$index}, 0, $Max_Index_Width ), @$_{(qw{ Rate Total Last }, $last_field)} );
if ( length($line) > $cols ) {
substr( $line, $cols – 1 ) = “”;
} else {
$line .= ” ” x ($cols – length($line));
}
print $line, “\n”;
}
print ” ” x $cols, “\n” while $count++ < $show;
}
sub process_line
{
my $line = shift;
my $now = ( shift || time );
my %hit;
chomp $line;
@hit{@{$Log_Fields{$Log_Format}}} = grep( $_, split( /”([^"]+)”|\[([^]]+)\]|\s/o, $line ) );
$hit{ URI } =~ s/HTTP\/1\S+//gos;
$hit{ Referer } = “” if not $hit{Referer} or $hit{Referer} eq “-”;
( $hit{Domain} = $hit{Referer} ) =~ s#^\w+://([^/]+).*$#$1#os;
$hit{ Client } ||= “”;
$hit{ Client } =~ s/Mozilla\/[\w.]+ \(compatible; /(/gos;
$hit{ Client } =~ s/[^\x20-\x7f]//gos;
# if $now is negative, try to guess how old the hit is based on the time stamp.
if ( $now $now, _Ttl => $Initial_TTL } );
@$peer{ @Display_Fields } = @hit{ @Display_Fields };
$peer->{ Total }++;
}
}
sub display_help {
my $msg = “httptop v.$Version”;
print @Term{qw/ HOME CLS START_RV /}, $msg, $Term{END_RV}, “\n\n”;
print ” ” x 4, $_, ” ” x 8, $Keys{$_}[0], “\n” for ( split “”, $Help );
print “\nPress any key to continue.\n”;
}
### Init.
getopt( ‘frb’ => \%opt );
$Backtrack = $opt{b} if $opt{b};
$Update = $opt{r} if $opt{r};
$Log_Format = $opt{f} if $opt{f};
$spec = $ARGV[0];
die <<End unless $spec and $Log_Fields{$Log_Format};
Usage: $0 [-f ] [-r ] [-b ]
Valid formats are: @{[ join ", ", keys %Log_Fields ]}.
End
for ( @Paths ) {
last if -r $spec;
( $spec = $_ ) =~ s/%/$ARGV[0]/gos;
}
die “No access_log $ARGV[0] found.\n” unless -r $spec;
my $file = File::Tail->new(
name => $spec,
interval => $Update / 2,
maxinterval => $Update,
tail => $Backtrack,
nowait => 1
) or die “$spec: $!”;
my $last_update = time;
my ( $line, $now );
# Backtracking.
while ( $Backtrack– > 0 ) {
last unless $line = $file->read;
process_line( $line, -1 );
}
$file->nowait( 0 );
ReadMode 4; # Echo off.
print @Term{“HOME”, “CLS”}; # Home & clear.
refresh_output;
### Main loop.
while (defined( $line = $file->read )) {
$now = time;
process_line( $line, $now );
while ( $line = lc ReadKey(-1) ) {
$show_help = 0 if $show_help;
$Keys{$line}[1]->( ) if $Keys{$line};
}
if ( $show_help == 1 ) {
display_help;
$show_help++; # Don’t display help again.
} elsif ( $now – $last_update > $Update and not $show_help ) {
$last_update = $now;
refresh_output( $now );
}
}
————————————————–

Now we can run httptop
#httptop -f combined -r 1 /usr/local/apache2/logs/access_log

Note: You can get help when u r in ,using “?”

View Comments »

Posted in Apache, CentOS, Linux, Monitoring

Tags:

Great Strategies for Using Memcached and MySQL Better Together

June 28th, 2010

The primero recommendation for speeding up a website is almost always to add cache and more cache. And after that add a little more cache just in case. Memcached is almost always given as the recommended cache to use.

What is Memcached?

Memcached is a general-purpose distributed memory caching system that was originally developed by Danga Interactive for LiveJournal, but is now used by many other sites. It is often used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source (such as a database or API) must be read. Memcached runs on Unix, Windows and MacOS and is distributed under a permissive free software license.

Memcached’s APIs provide a giant hash table distributed across multiple machines. When the table is full, subsequent inserts cause older data to be purged in least recently used (LRU) order. Applications using Memcached typically layer requests and additions into core before falling back on a slower backing store, such as a database.

Architecture

The system uses a client–server architecture. The servers maintain a key–value associative array; the clients populate this array and query it. Keys are up to 250 bytes long and values can be at most 1 megabyte large.

Clients use client side libraries to contact the servers which, by default, expose their service at port 11211. Each client knows all servers; the servers do not communicate with each other. If a client wishes to set or read the value corresponding to a certain key, the client’s library first computes a hash of the key to determine the server that will be used. Then it contacts that server. The server will compute a second hash of the key to determine where to store or read the corresponding value.

The servers keep the values in RAM; if a server runs out of RAM, it discards the oldest values. Therefore, clients must treat Memcached as a transitory cache; they cannot assume that data stored in Memcached is still there when they need it. A Memcached-protocol compatible product known as MemcacheDB provides persistent storage. There is also a solution called Membase from NorthScale that provides persistence, replication and clustering.

If all client libraries use the same hashing algorithm to determine servers, then clients can read each other’s cached data; this is obviously desirable.

A typical deployment will have several servers and many clients. However, it is possible to use Memcached on a single computer, acting simultaneously as client and server.

Security

Most deployments of Memcached exist within trusted networks where clients may freely connect to any server. There are cases, however, where Memcached is deployed in untrusted networks or where administrators would like to exercise control over the clients that are connecting. For this purpose Memcached can be compiled with optional SASL authentication support. The SASL support requires the binary protocol.

Example code

Note that all functions described on this page are pseudocode only. Memcached calls and programming languages may vary based on the API used.

Converting a database or object creation queries to use Memcached is simple. Typically, when using straight database queries, example code would be as follows:

function get_foo(int userid) {
result = db_select(“SELECT * FROM users WHERE userid = ?”, userid);
return result;
}

After conversion to Memcached, the same call might look like the following

function get_foo(int userid) {
/* first try the cache */
data = memcached_fetch(“userrow:” + userid);
if (!data) {
/* not found : request database */
data = db_select(“SELECT * FROM users WHERE userid = ?”, userid);
/* then store in cache until next get */
memcached_add(“userrow:” + userid,  data);
}
return data;
}

The server would first check whether a Memcached value with the unique key “userrow:userid” exists, where userid is some number. If the result does not exist, it would select from the database as usual, and set the unique key using the Memcached API add function call.

However, if only this API call were modified, the server would end up fetching incorrect data following any database update actions: the Memcached “view” of the data would become out of date. Therefore, in addition to creating an “add” call, an update call would be also needed, using the Memcached set function.

function update_foo(int userid, string dbUpdateString) {
/* first update database */
result = db_execute(dbUpdateString);
if (result) {
/* database update successful : fetch data to be stored in cache */
data = db_select(“SELECT * FROM users WHERE userid = ?”, userid);
/* last line could also look like   data = createDataFromDBString(dbUpdateString);   */
/* then store in cache until next get */
memcached_set(“userrow:” + userid, data);
}
}

This call would update the currently cached data to match the new data in the database, assuming the database query succeeds. An alternative approach would be to invalidate the cache with the Memcached delete function, so that subsequent fetches result in a cache miss. Similar action would need to be taken when database records were deleted, to maintain either a correct or incomplete cache.

Memcached and MySQL Go Better Together

There’s a little embrace and extend in the webinar as MySQL cluster is presented several times as doing much the same job as memcached, but more reliably. However, the recommended approach for using memcached and MySQL is:

1. Write scale the database by sharding. Partition data across multiple servers so more data can be written in parallel. This avoids a single server becoming the bottleneck.

2. Front MySQL with a memcached farm to scale reads. Applications access memcached first for data and if the data is not in memcached then the application tries the database. This removes a great deal of the load on a database so it can continue to perform it’s transactional duties for writes. In this architecture the database is still the system of record for the true value of data.

3. Use MySQL replication for reliability and read query scaling. There’s an effective limit to the number of slaves that can be supported so just adding slaves won’t work as scaling strategy for larger sites.

Using this approach you get scalable reads and writes along with high availability.

Given that MySQL has a cache, why is memcached needed at all?

1. The MySQL cache is associated with just one instance. This limits the cache to the maximum address of one server. If your system is larger than the memory for one server then using the MySQL cache won’t work. And if the same object is read from another instance its not cached.
2. The query cache invalidates on writes. You build up all that cache and it goes away when someone writes to it. Your cache may not be much of a cache at all depending on usage patterns.
3. The query cache is row based. Memcached can cache any type of data you want and it isn’t limited to caching database rows. Memcached can cache complex complex objects that are directly usable without a join.

Thanks
Manoj Chauhan

View Comments »

Posted in Apache, CentOS, HTTP accelerators, Linux, Memcache

Tags:

Tool for automatically creating the basic framework for a PHP module

June 25th, 2010

WHAT IT IS

It’s a tool for automatically creating the basic framework for a PHP module and writing C code handling arguments passed to your functions from a simple configuration file. See an example at the end of this file.

HOW TO USE IT

Very simple. First, change to the ext/ directory of the PHP 4/5 sources. If you just need the basic framework and will be writing all the code in your  functions yourself, you can now do

./ext_skel –extname=module_name

and everything you need is placed in directory module_name.

[ Note that GNU awk is likely required for this script to work.  Debian systems seem to default to using mawk, so you may need to change the #! line in skeleton/create_stubs and the cat $proto | awk line in ext_skel to use gawk explicitly. ]

We don’t need to find any php packages, we can create any package using above guidelines.

Thanks
Manoj Chauhan

View Comments »

Posted in Apache, CentOS, Linux, PHP

Tags:

How to do Large partions >2 TB, Gparted

June 20th, 2010

To Create a Partition of more than 2 TB, Gparted is used.

You must include GPT support in kernel in order to use GPT. If you don’t include GPT support in Linux kernelt, after rebooting the server, the file system will no longer be mountable or the GPT table will get corrupted. By default Redhat Enterprise Linux / CentOS comes with GPT kernel support. However, if you are using Debian or Ubuntu Linux, you need to recompile the kernel. Set CONFIG_EFI_PARTITION to y to compile this feature.

Creating Partion of 4 TB

# parted /dev/sdx
Here x can be b,c,d..
I take eg . with b
Creates a new GPT disklabel i.e. partition table
#mklabel gpt

Creating 4 TB partition
#mkpart primary 0 4001G
Quit and Save the changes
#quit

Formatting the partition with ext4 file system
#mke2fs -t ext4 /dev/sdb1
Also update fstab entry for boot time mounting.

If we have External storage disk (NAS,SAN,USB)
U must update File system checking ,in my case file system will be updated after 800 mounts

#tune2fs -c 800 /dev/sdb1

More Tricks, We can also use

# mkpart primary 0 -0

This will tell parted to fill up the entire drive with one volume.

EXT3 on most systems the max block size is 4096 bytes, which will still limit you to ~8TB for your filesystem.

View Comments »

Posted in CentOS, Linux

Tags:

How to use socat with haproxy

June 18th, 2010

Introduction:-

All you know about the haproxy, that its the one of the good opensource load balancing software and to check the fun stats of haproxy here we using ‘socat’ – Multipurpose relay (SOcket CAT)

about
—–

socat is a relay for bidirectional data transfer between two independent data channels. Each of these data channels may be a file, pipe, device (serial line etc. or a pseudo terminal), a socket (UNIX, IP4, IP6 – raw, UDP, TCP), an SSL socket, proxy CONNECT connection, a file descriptor (stdin etc.), the GNU line editor (readline), a program, or a combination of two of these. These modes include generation of “listening” sockets, named pipes, and pseudo
terminals.

socat can be used, e.g., as TCP port forwarder (one-shot or daemon), as an external socksifier, for attacking weak firewalls, as a shell interface to UNIX sockets, IP6 relay, for redirecting TCP oriented programs to a serial line, to logically connect serial lines on different computers, or to establish a relatively secure environment (su and  chroot) for running client or server shell scripts with network connections.

Many options are available to refine socats behaviour:
terminal parameters, open() options, file permissions, file and process owners, basic socket options like bind address, advanced socket options like IP source routing, linger, TTL, TOS (type of service), or TCP performance tuning.

More capabilities, like daemon mode with forking, client address check, “tail -f” mode, some stream data processing (line terminator conversion),
choosing sockets, pipes, or ptys for interprocess communication, debug and trace options, logging to syslog, stderr or file, and last but not least
precise error messages make it a versatile tool for many different purposes.

In fact, many of these features already exist in specialized tools; but until now, there does not seem to exists another tool that provides such a generic, flexible, simple and almost comprehensive (UNIX) byte stream connector.

install
——-

Get the tarball and extract it:
gtar xzf socat.tar.gz
cd socat-1.7.1.2
./configure
make
su
make install    # installs socat, filan, and procan in /usr/local/bin

For compiling socat, gcc (or egc) is recommended.
If gcc is not available, the configure script will fail to determine some features; then you’d better begin with one of the Makefiles and config.h’s
from the Config directory.

If you have problems with the OpenSSL library, you can apply the option
“–disable-openssl” to configure.

If you have problems with the readline library or (n)curses, you can apply the
option “–disable-readline” to configure.

If you have problems with the tcp wrappers library, you can apply the option
“–disable-libwrap” to configure.

If you still get errors or a tremendous amount of warnings you can exclude
the features for system call tracing and file descriptor analyzing by
applying the options “–disable-sycls –disable-filan” to configure.

You still need the functions vsnprintf and snprintf that are in the GNU libc,
but might not be available with some proprietary libc’s.

The configure script looks for headers and libraries of openssl, readline, and
tcp wrappers in the OS’es standard places and in the subdirectories include/
and lib/ of the following places:
/sw/
/usr/local/
/opt/freeware/
/usr/sfw/
and for openssl also in:
/usr/local/ssl/
In case of unexpected behaviour it is important to understand that configure
first searches for the appropriate include file and then expects to find the
library in the associated lib directory. That means, when e.g. a OpenSSL
installation resides under /usr/local and there is a symbolic link from
/usr/include/ssl/ssl.h to /usr/local/ssl/include/ssl/ssl.h, configure will find
the /usr/include/… header and will therefore expect libssl in /usr/lib
instead of /usr/local/…

If configure does not find a header file or library but you know where it is,
you can specify additional search locations, e.g.:
export LIBS=”-L$HOME/lib”
export CPPFLAGS=”-I$HOME/include”
before running configure and make.

For other operating systems, if socat does not compile without errors, refer to
the file PORTING.

platform specifics – redhat
—————————

On RedHat Linux 9.0, including openssl/ssl.h might fail due to problems with
the krb5-devel package. configure reacts with disabling openssl integration.
To solve this issue, help cpp to find the krb5.h include file:
CPPFLAGS=”-I/usr/kerberos/include” ./configure

EXAMPLES

http://www.dest-unreach.org/socat/doc/socat.html#EXAMPLES

How can we use |socat?

Step 1) Download ’socat’ from http://www.dest-unreach.org/socat/download/  latest version ~ “socat-2.0.0-b3.tar.gz”

manoj@server:~$ wget http://www.dest-unreach.org/socat/download/socat-1.7.1.2.tar.gz

manoj@server:~$ tar xvzf socat-1.7.1.2.tar.gz

manoj@server:~$ cd socat-1.7.1.2

NOTE ~ No need to install the ‘fipsld’ package if you got the below msg after running the ‘make’ just following steps for

compiling socat….

FIPSLD_CC=gcc fipsld -O -D_GNU_SOURCE -Wall -Wno-parentheses  -DHAVE_CONFIG_H -I.  -I.   -c -o socat.o socat.c
/bin/sh: fipsld: command not found
make: *** [socat.o] Error 127

manoj@server:~$ ./configure –disable-fips
manoj@server:~$ make

To install it login as root
manoj@server:~$ su -

manoj@server:~# make install

Step 2) Now you need to add stats socket PATH in Haproxy configuration and restart haproxy as per shown in following example,

where I have added it under in ‘global’ setting -

manoj@server:~# more /etc/haproxy/myhaproxy.cfg

#———–Start of haproxy Config file————–
global
log 127.0.0.1   local0
log 127.0.0.1   local1 notice
#log loghost    local0 info
maxconn 25000
#debug
#quiet
user manoj
group manoj
stats socket    /home/user/socket/haproxy.sock
defaults
option          contstats
timeout         connect 5s
timeout         client 25s
timeout         server 25s
maxconn         100

listen manojtestbed      0.0.0.0:80 ##manoj.com IP
mode            tcp
balance         roundrobin
server          web1 192.168.1.17
server          web2 192.168.1.12

listen stats
bind            0.0.0.0:8081
mode            http
#stats          uri /stat  #Comment this if you need to specify diff stat path for viewing stat page
stats enable
stats auth admin:admin ##Auth user pass

Step 3) Used /home/user/socket/haproxy.sock, Now you can send the commands to get stats from HAProxy -

Now time to use socat

echo “show info” |socat unix-connect:/home/user/socket/haproxy.sock stdio
This will give you information about the running HAProxy process such as pid, uptime and etc.

[manoj@server ~]# echo “show info” |socat unix-connect:/home/user/socket/haproxy.sock stdio
Name: HAProxy
Version: 1.3.23
Release_date: 2010/01/28
Nbproc: 1
Process_num: 1
Pid: 29958
Uptime: 0d 4h26m49s
Uptime_sec: 16009
Memmax_MB: 0
Ulimit-n: 50026
Maxsock: 50026
Maxconn: 25000
Maxpipes: 0
CurrConns: 66
PipesUsed: 0
PipesFree: 0
Tasks: 75
Run_queue: 1
node: appha1
description:

socat command options:-

Unknown command. Please enter one of the following commands only :
show info   : report information about the running process
show stat   : report counters for each proxy and server
show errors : report last request and response errors for each proxy
show sess   : report the list of current sessions

View Comments »

Posted in CentOS, Haproxy, Linux

Tags:

How to Install Munin on CentOS

June 18th, 2010

Introduction:-

Munin is a highly flexible and powerfull solution used to create graphs of virtually everything imaginable throughout your network, while still maintining a rattling easy of installation and configuration. Munin is a networked resource monitoring tool that can help analyze resource trends and “what just happened to kill our performance?” problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.

About Munin

Munin the monitoring tool surveys all your computers and remembers what it saw. It presents all the information in graphs through a web interface. Its emphasis is on plug and play capabilities. After completing a installation a high number of monitoring plugins will be playing with no more effort.

Using Munin you can easily monitor the performance of your computers, networks, SANs, applications, weather measurements and whatever comes to mind. It makes it easy to determine “what’s different today” when a performance problem crops up. It makes it easy to see how you’re doing capacity-wise on any resources.

Munin uses the excellent RRDTool (written by Tobi Oetiker) and the framework is written in Perl, while plugins may be written in any language. Munin has a master/node architecture in which the master connects to all the nodes at regular intervals and asks them for data. It then stores the data in RRD files, and (if needed) updates the graphs. One of the main goals has been ease of creating new plugins (graphs).

This site is a wiki as well as a project management tool. We appreciate any contributions to the documentation. While this is the homepage of the Munin project, we will still make all releases through Sourceforge.

I will use the following scenario:

munin-server.example – 192.168.1.254        # Munin Server
station1.example.com – 192.168.1.1            # Munin client
station2.example.com – 192.168.1.2            # Munin client

You can add many clients as much you want. Here i added two clients. Here on more thing to notice, using hostname is not necessary. We are not required any DNS resolution, it is just for clarity to recognize your client easily.

Server Side
Let start with Installation and configuration:

wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.i386.rpm
Enable The RPMforge Repository
yum install
yum install perl-Net-SNMP perl-Net-Server perl-HTML-Template perl-Log-Log4perl-RRDs perl-RRD-Simple

Install And Configure munin
yum install munin munin-node
chkconfig –levels 235 munin-node on
/etc/init.d/munin-node start

By default it will create the following directories,files.

Main Server main Configuration file:
/etc/munin/munin.conf

Munin cron file:
/usr/bin/munin-cron

Munin libraries:
/var/lib/munin

Munin logs:
/var/log/munin

Munin Document Root
/var/www/munin

Step 2 : Changing default Munin Document Root Path:
mv /var/www/munin /var/www/html/munin

Step 3 : Adding clients/nodes to monitor them in /etc/munin/munin.conf:

vi /etc/munin/munin.conf

[station1.example.com]
address 192.168.1.1
use_node_name yes
[station2.example.com]
address 192.168.1.2
use_node_name yes

Step 4 : Setting up permissions:

groupadd munhttp
usermod -G munhttp munin
usermod -G munhttp apache
chown -v apache.munhttp /var/www/html/munin

Step 4 : Setting up cron to schedule the munin to update every 5 minutes:
crontab -e
*/5 * * * * /usr/bin/munin-cron –force-root

Step 5 : Restart Apache:
service httpd restart

Step 6: Password protection for munin
AuthType Basic
AuthName “Members Only”
AuthUserFile /var/www/munin/.htpasswd
<limit GET PUT POST>
require valid-user
</limit>

htpasswd -c /var/www/munin/.htpasswd admin

Client Side
Here we go with installing the clients/nodes.

Step 1 : Installing the required packages :
yum install perl-Net-Server perl-Net-SNMP perl-Digest-HMAC perl-Digest-SHA1 munin-node

Step 2 : Editing /etc/munin/munin-node.conf for server ip address, you have to add it bottom of the file, below the line ” allow ^127.0.0.1$”
vi /etc/munin/munin-node.conf
allow ^192.168.1.254$

Step 3 : The following command will scan the system for available services and will auto configure it. e.g Sendmail, mysql.

munin-node-configure

Step 4 : Start Munin-node client to available for the server.
munin-node

Note : If you modify the configure file “/etc/munin/munin-node.conf” you will need to run the following command to re-read the configure file.
service munin-node restart

You can follow the “Client Side” configuration for the rest of clients/nodes.

Common Issues:

1. No values/stats are displayed or blank.
Please verify that munin-node service is running at machine. Verify it by telnet from machine itself:
# telnet localhost 4949
You should see munin prompt here, write fetch cpu and press ENTER. You should see various values related to your cpu. If you are not able to telnet to port 4949 (munin port) then possibly munin-node is not running or your iptable (firewall) or SELinux policy is preventing its usage.

After checking from localhost, go to Server machine and try to telnet from there:
# telnet 192.168.0.5 4949
You should get munin prompt and see values after issuing fetch cpu command there. If not, then firewall/selinux is preventing access this port or you didnt allowed this server from your munin-node.conf to get updates.

2. Why not compile from source?
Thats the best idea but when I tried the same, I got lots of perl modules dependencies/compile errors. Though, I managed to get its working but for a quick setup, yum is your friend.

3. I do not need all these values coming in graphs.
You can remove any plugin for which you do not want graph. Plugins are stored in /etc/munin/plugins directory and configured in plugins.conf. Configure these as per your requirements.

Thanks
Manoj

View Comments »

Posted in Apache, CentOS, Linux, Monitoring, Mysql, Network

Tags:

OpenEmm — Mass Mailing ,Compain server

June 17th, 2010

yum install mysql-server sendmail-cf MySQL-python libxml2

Download Jdk latest version from: wget http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/jdk-6u20-linux-

i586.bin?BundledLineItemUUID=Er9IBe.oOf0AAAEoay9bNk7a&OrderID=gddIBe.oe.0AAAEoXi9bNk7a&ProductID=guBIBe.oc_wAAAEnaDJHqPYe&FileName=/jdk-6u20-linux-i586.bin

mv jdk-6u20-linux-i586.bin?AuthParam=1272633583_0b435f520fad540d314411f50a185d7a&TicketId=B%2Fw4khmGSVNITB1FPFVTkwPl&GroupName=CDS&FilePath=%2FESD6%2FJSCDL%2Fjdk%2F6u20-b02%2Fjdk-6u20-linux-i586.bin&File=jdk-6u20-linux-i586.bin jdk-6u18-linux-i586.bin

Download openemm source file from : http://sourceforge.net/projects/openemm/files/OpenEMM%20software/OpenEMM%206.0.1/OpenEMM-6.0.1-bin.tar.gz/download

Create the ‘openemm’ User

Create a special group and user for OpenEMM:

groupadd openemm

useradd -m -g openemm -d /home/openemm -c “OpenEMM 6.x.y” openemm

Installation: Sun Java JDK

Copy the file to your /tmp directory: cp jdk-6u18-linux-i586.bin /tmp

- Change to the /tmp directory: cd /tmp

- Grant the file execution permission: chmod u+x jdk-6u18-linux-i586.bin

- Execute the file:
./jdk-6u18-linux-i586.bin

- Follow the onscreen instructions and confirm the license agreement

- Create a directory:
mkdir -p /opt/openemm.org/software

- Move the JDK-directory in there: mv jdk1.6.0_18 /opt/openemm.org/software

- Change to that directory:
cd /opt/openemm.org/software

- Create a symbolic link for the JDK:
ln -s jdk1.6.0_18 java

- Test the JDK: /opt/openemm.org/software/java/bin/java -version

You should get an output like this:

java version “1.6.0_18″

Java(TM) SE Runtime Environment (build 1.6.0_18-b01)

Java HotSpot(TM) Client VM (build 14.2-b01, mixed mode, sharing)

Installation of OpenEMM:

cd /home/openemm

tar xzvpf /tmp/OpenEMM-6.0.1.bin.tar.gz

mkdir -p /usr/share/doc/OpenEMM-6.0.1

mv USR_SHARE/* /usr/share/doc/OpenEMM-6.0.1

Read Access to Maillog

chmod 604 /var/log/maillog

chkconfig mysqld on

service mysqld start

give the root password for mysql: .

mysqladmin -u root password root123

Create data base for openemm:

mysqladmin -u root -p create openemm

mysql -u root -p openemm_cms < openemm_cms.sql

Replace the generic string “http://localhost:8080″ with the domain name of

your server (like “http://www.domain.com:8080″) in these files:

- /usr/share/doc/OpenEMM-6.0.1/openemm-6.0.1.sql (once)

- /home/openemm/webapps/core/WEB-INF/classes/emm.properties (twice)

o /home/openemm/webapps/core/WEB-INF/classes/cms.properties (once)

Add the data from sql file:

mysql -u root -p openemm_cms < /usr/share/doc/OpenEMM-6.0.1/openemm_cms.sql

Give full permission to user for acces the data bases:/

mysql -u root -p
GRANT DELETE, INSERT, UPDATE, LOCK TABLES, SELECT, ALTER, INDEX, CREATE TEMPORARY TABLES, DROP, CREATE ON openemm.* TO ‘agnitas’@'localhost’ IDENTIFIED BY ‘openemm’;
FLUSH PRIVILEGES;
quit

Change the line on sendmail:.

Open file /etc/mail/sendmail.mc and change the line

DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA’)dnl

to

dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA’)dnl

This will enable Sendmail to listen on all available network interfaces. By default Sendmail is listing only on the local interface lo0 for connections.

Add the following line at the end of the file:

INPUT_MAIL_FILTER(`bav’, `S=unix:/home/openemm/var/run/bav.sock, F=T’)dnl

This will enable the dynamic mail loop required by the bounce management to process delayed bounces.

If file /etc/mail/relay-domains does not exist, create the file – for example by

touch relay-domains

and add a line at the end of the file which specifies your DNS entry for the sender hostname (FQDN). In our example it is simply:

newsletter.pawan.in

open file /etc/mail/mailertable and add a line at the end which activates the bounce management for that FQDN:

news.openemm.org procmail:/home/openemm/conf/bav/bav.rc

to activate all Sendmail changes, run the following commands:

cd /etc/mail

make

and restart the Sendmail service by

/etc/init.d/sendmail restart

Launch OpenEMM

su – openemm

cd bin

sh OpenEMM.sh start

exit

View Comments »

Posted in Linux, Mail Servers, Security, Sendmail

Tags:

WARNING: You will need re2c 0.9.11 or later if you want to regenerate PHP parsers.

June 15th, 2010

I am getting this error “configure: WARNING: You will need re2c 0.9.11 or later if you want to regenerate PHP parsers.” while i tried to install json support with PHP 5.1.6

What is re2c?
re2c is a tool for writing very fast and very flexible scanners. Unlike any other such tool, re2c focuses on generating high efficient code for regular expression matching. As a result this allows a much broader range of use than any traditional lexer offers. And Last but not least re2c generates warning free code that is equal to hand-written code in terms of size, speed and quality.

I download the wget ftp://195.220.108.108/linux/dag/redhat/el4/en/x86_64/dag/RPMS/re2c-0.12.3-1.el4.rf.x86_64.rpm and installed on the server

#rpm -ivh re2c-0.12.3-1.el4.rf.x86_64.rpm
cd /tmp/json-1.2.1
./configure –with-php-config=/home/user/dev/php/bin/php-config> –with-json
make && make install
ll modules/json.so

OR

cp modules/json.so  /home/user/dev/php/lib/php/extensions/
/home/user/dev/php/bin/php -m
Edit
vim /home/user/dev/php/lib/php.ini and add extention extension=json.so

Restart apache
/home/user/dev/apache/bin/httpd -k  stop/restart/start

Thanks
Manoj

View Comments »

Posted in Apache, CentOS, Linux, PHP

Tags:

SMTP Redirection using Xinetd

June 2nd, 2010

1. Service sendmail stop
2.Create New File in /etc/xinetd.d/smtp
Insert the following contents

service smtp
{
disable = no
socket_type = stream
protocol = tcp
user = nobody
wait = no
server = /bin/nc
server_args = -w 2 xx.xx.xx.xx 25
}

Save and Close , also make sure to replace xxx with the SMTP server ip address.
/etc/xinetd.d/smtp file shows
how I forwarded incoming SMTP connections to the new host:

This causes all incoming connections on port 25 to be seamlessly redirected to the new host .

This Trick is used where , SMTP host is hidden from Users and
It removes worry about the delay in DNS propogation.

Thanks
Pawan Kumar

View Comments »

Posted in CentOS, Mail Servers, Sendmail

Tags:

InnoDB: Unable to lock ./ibdata1, error: 37

June 1st, 2010

InnoDB: Unable to lock ./ibdata1, error: 37

I recently faced the “error  InnoDB: Unable to lock ./ibdata1, error: 37 “ while moving my data directory to NFS from local disk , when I restarted the mysql , the restart failed and I got the error in the error log , all permissions were set properly so I was not sure whats going wrong , I was sure that there is some problem with the OS not able to lock the file and it was related to NFS so I

1) /etc/init.d/portmap restart
2) /etc/init.d/nfslock restart

Restarted the mysql and it worked .

View Comments »

Posted in CentOS, Linux, Mysql, NFS

Tags: