Archive for March, 2010

tnsping: error while loading shared libraries: libclntsh.so.10.1

March 31st, 2010

I am getting following error

user@servername$ /usr/local/nagios/libexec/check_oracle –tns servername
tnsping: error while loading shared libraries: libclntsh.so.10.1: cannot open shared object file: No such file or directory
No TNS Listener on servername

edit /home/user/.bashrc and following

export ORACLE_BASE=/mnt/home/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORACLE_SID=GLAMEURO
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib

Also edit vi /etc/group and add user in oinstall group

oinstall:x:505:user,nagios

Thanks
Manoj Chauhan

Puppet: Configuration,Management Tool, installation on RHEL 5

March 19th, 2010

Introduction:-

Prerequisite : Ruby and all ruby Library

Rpm Repository for Puppet

http://people.redhat.com/dlutter/yum/rhel5/

rpm -Uvh facter-1.3.7-1.el5.noarch.rpm puppet-0.22.4-1.el5.noarch.rpm
puppet-server-0.22.4-1.el5.noarch.rpm
Server Preparation

The server (puppetmasterd) requires a manifest to be in place before it’s able to run. Lets write a manifest that tells puppet to create a file “/tmp/testfile” on the client.

puppet:# vim /etc/puppet/manifests/site.pp

# Create “/tmp/testfile” if it doesn’t exist.

class test_class {

file { “/tmp/testfile”:

ensure => present,

mode   => 644,

owner  => root,

group  => root

}

}

# tell puppet on which client to run the class

node pclient {

include test_class

}

Managing nodes in Puppet

Once the puppet master is installed, we need to do some basic configuration and then start the daemon. Puppet operates not by executing scripts but by applying what are called manifests against target hosts. A manifest is a Puppet configuration document that describes the target configuration and the steps required to achieve it. Your manifest files also contain the definition of each of the hosts to be managed, which Puppet calls ‘nodes’.

Each puppet master server requires a master manifest, called a site manifest. The site manifest is stored in the file /etc/puppet/manifests/site.pp. You’ll need to create this directory like so:

# mkdir /etc/puppet/manifests
You can see an example of a site.pp file below.

class base_etc {
      file { “/etc/passwd”: owner => root, group => root, mode => 644}
      file { “/etc/sudoers”: owner => root, group => root, mode => 440}
}

node default {
        include base_etc
}
The site manifest file can contain all the instructions you want to use on your managed hosts or can import the contents of other files to enable you to better structure your configuration instructions. Take a look at this example:

import “nodes.pp”
import “classes/*”
The above configuration would import the file nodes.pp and all the files in the classes directory. All directories are assumed to be under the /etc/puppet/manifests directory.

In our example, site.pp file, we’ve specified a class called base_etc. Classes are containers for sets of instructions. In this case, the container holds two instructions, both using the file resource that sets the ownership and permission of the /etc/passwd and /etc/sudoers files. The file resource allows you to create files and directories or specify their permissions and ownership.

The file resource is one of a number of resource types, including types that can manage cron entries, host entries, manage packages, start and stop services or manage users. Each of these resource types can have multiple implementations, called providers, where each provider resource is designed for a different system. For instance, there are multiple file resource providers to cater for file operations on Linux, Unix, Windows and other platforms. This is one of the great strengths of Puppet; a single configuration language that can be interpreted appropriately on multiple target hosts running on different platforms.

We’ve also defined a node called default. The default node is a special definition that allows you to define sets of instructions that are applied to all nodes except those explicitly defined in your configuration. This allows you to specify a baseline configuration to all nodes without having to specifically define all nodes. We can also define other nodes in here (or in another file and import them).

A typical node definition looks like:
node goldfish {
       include apache
}
node ‘salmon.testing.com’ inherits default {
        include mysql}
Nodes are defined either by their simple name or a fully qualified domain name. If you use the fully-qualified domain name, then it must be encapsulated in single quotation marks. In our example, both the goldfish and salmon.testing.com nodes have been defined. Each node uses the include statement to specify particular classes that apply to them, which in this case are the apache and mysql classes, respectively. We’ve also specified that the salmon host also inherits the configuration of the default node.

Starting and configuring Puppet

Once you’ve created your site.pp file you can start the puppermasterd daemon. The Puppet RPMs come with an init script for Puppet, but we’re going to start it manually on the command line:
# /usr/bin/puppetmasterd –verbose
The –verbose option tells the puppetmasterd not to daemonize and to output all logging messages on the command line. If you omit this option, Puppet will daemonize by default. The Puppet master listens on port 8140 (you’ll need to ensure your clients have network connectivity and access through any firewalls to this port on your Puppet server). You can also change a variety of configuration options for your Puppet master daemon by editing the /etc/puppet/puppetmasterd.conf file.

Now you need to install Puppet on one of your client nodes. We also need to install Ruby and the Ruby libraries on the client node and then we need to download and install the Puppet and Facter RPMs.

# wget http://people.redhat.com/dlutter/yum/rhel5/puppet-0.22.4-1.el5.noarch.rpm
# wget http://people.redhat.com/dlutter/yum/rhel5/facter-1.3.7-1.el5.noarch.rpm
# rpm -Uvh facter-1.3.7-1.el5.noarch.rpm puppet-0.22.4-1.el5.noarch.rpm
Next, we need to configure the puppet.conf configuration file to tell it about our Puppet master server. The puppet.conf file is installed into the /etc/puppet directory. You can see an example of this file below.
[puppet]
    vardir = /var/lib/puppet
    logdir = /var/log/puppet
    rundir = /var/run/puppet
    ssldir = $vardir/ssl

[puppetd]
    classfile = $vardir/classes.txt
    localconfig = $vardir/localconfig
    server = puppetmaster.testing.com
Client Preparation
Clients by default will connect to a server on your network with a hostname of “puppet.” If your server’s hostname isn’t “puppet” a directive needs to be inserted into the puppetd configuration file “puppetd.conf.” Even though we don’t need to in this case, we’ll do so for demonstration purposes.

Open “/etc/puppet/puppetd.conf” with your favorite text editor and add “server = puppet.example.com” to the existing file as the example below indicates.

pclient:# vim /etc/puppet/puppetd.conf

[puppetd]
server = puppet.example.com

# Make sure all log messages are sent to the right directory
# This directory must be writable by the puppet user
logdir=/var/log/puppet
vardir=/var/lib/puppet
rundir=/var/run
Sign Keys
In order for the two systems to communicate securely we need to create signed SSL certificates. You should be logged into both the server and client machines for this next step.

On the client side run.

pclient:# puppetd –server puppet.example.com –waitforcert 60 –test

You should see the following message.

err: No certificate; running with reduced functionality.
info: Creating a new certificate request for pclient.example.con
info: Requesting certificate
warning: peer certificate won’t be verified in this SSL session
notice: Did not receive certificate

Next, on the server side, run the following command to verify the client is waiting for the cert to be signed.

puppet:# puppetca –list

pclient.example.con

Then sign the certificate.

puppet:# puppetca –sign pclient.example.com

Signed pclient.example.com

If everything went OK you should see this message on pclient.

info: Requesting certificate
warning: peer certificate won’t be verified in this SSL session
notice: Ignoring –listen on onetime run
info: Caching configuration at /etc/puppet/localconfig.yaml
notice: Starting configuration run
notice: //pclient/test_class/File[/tmp/testfile]/ensure: created
info: Creating state file /var/lib/puppet/state/state.yaml
notice: Finished configuration run in 0.11 seconds

Test
Check and make sure the file was created.

pclient:# ls -l /tmp/testfile

-rw-r–r– 1 root root 0 2007-02-18 18:28 /tmp/testfile

For a test lets edit the manifest and direct Puppet to modify the file mode. Change line, “mode => 644,” to “mode => 600,”

puppet:# vim /etc/puppet/manifests/site.pp

# Create “/tmp/testfile” if it doesn’t exist.
class test_class {
    file { “/tmp/testfile”:
       ensure => present,
       mode   => 600,
       owner  => root,
       group  => root
    }
}

# tell puppet on which client to run the class
node pclient {
    include test_class
}
On the client run puppetd in verbose mode (-v) and only once (-o).

pclient:# puppetd -v -o

You should see the following message, which states that /tmp/testfile changed from mode 644 to 600.

notice: Ignoring –listen on onetime run
info: Config is up to date
notice: Starting configuration run
notice: //pclient/test_class/File[/tmp/testfile]/mode: mode changed ’644′ to ’600′
notice: Finished configuration run in 0.26 seconds

To verify the work was completed properly.

pclient:# ls -l /tmp/testfile

-rw——- 1 root root 0 2007-02-18 18:28 /tmp/testfile

Conclusion
Congratulations, testing is complete and you have a working Puppet setup. Your next step is to create a functional
manifest, test some more, and then fire up the puppetd daemon on the client side. Puppetd by default will automatically poll the server every 30 minutes.

pclient:# /etc/init.d/puppet start

Thanks
Pawan Kumar

How imports historical Nagios log files into the mysql database

March 5th, 2010

NDO Utilities:-

The NDO utilities add-on, written by Nagios developer Ethan Galstad, is designed to output events and data from Nagios to standard files or to a Unix socket. It also comes with a module called NDO2DB that allows Nagios data to be written to a MySQL or PostgreSQL database The add-on is made up of the NDOMOD Event Broker module, which is loaded by Nagios at runtime. It dumps all events and data from Nagios to a regular file or a Unix domain socket. It also contains the ndo2db daemon, which reads data that has been sent for the NDOMOD module to a Unix domain socket and dumps it into a MySQL or PostgreSQL database. You can dump into multiple databases and have multiple instances of the NDOMOD module writing to the same domain socket. There is also a utility called FILE2SOCK, which reads data from a standard file and dumps it into a Unix domain socket. Suggested uses are to dump data from
NDOMOD that has been stored in a standard file into a Unix domain socket. Or if your Nagios server is remote from your database server, you can dump data into a standard file from NDOMOD, send the file via SSH or SFTP to the database server, and then dump the data into a Unix domain socket and from there into a database. Finally, there is the LOG2NDO utility, which imports historical Nagios log files into the ndo2db daemon and sends them to a Unix domain socket or to standard output.

To install the NDO add-on, you first need to download and unpack the module from the Nagios Sourceforge site at http://sourceforge.net/project/showfiles.php?group_id=26589&package_id=173832, as you can see here:
puppy# wget http://optusnet.dl.sourceforge.net/sourceforge/nagios/ndoutils-12272005.tar.gz

manoj# tar -zxf ndoutils-1.4b9.tar.gz
manoj# cd ndoutils-1.4b9

By default NDO utilities use Nagios user, but my requirements was diffrent, i want to compile NDO utilities with manoj user. So i have used the following options during the compile time
manoj#./configure –with-ndo2db-user=manoj –with-ndo2db-group=manoj –enable-mysql –disable-pgsql –with-mysql-lib=/usr/lib/mysql
manoj# make

You must grant the user you create, in this case nagios, the SELECT, INSERT, UPDATE, and DELETE privileges to the nagios database. Replace ‘password’ with an appropriate password for the database.

manoj#create user ‘nagios@%’ identified by ‘password’;
OR
manoj#create user nagios@’localhost’ identified by ‘password’;
manoj#grant insert,delete,select,update on *.* to nagios@’%’ ;
OR
manoj#grant all on *.*  to nagios@’%’ ;
manoj#grant all on *.* to nagios@’localhost’ ;
manoj#create database nagios_db
Change Mysql Root password
manoj#update user set password=PASSWORD(“new password”) where User=’root’;
manoj#flush privileges;

The NDO add-on contains a script to populate this newly created database with the required tables. For MySQL, it is called ndo-mysql.sql, and it is located in the db directory in the root of the package:
manoj# cd ndoutils-1.4b9
manoj(ndoutils-1.4b9/db)#./installdb  -u user -p password -h hostname -d database

To install the NDO module itself, install the compiled ndomod.omodule file located in the src directory. I recommend copying it into the Nagios bin directory, usually /usr/local/nagios/bin:

manoj# cp src/ndomod-3x.o /usr/local/nagios/bin/ndomod.o

You also need to copy the sample configuration file for the module, ndomod.cfg. It is located in the config directory in the NDO utilities package. I recommend installing it to the Nagios etc directory, usually /usr/local/nagios/etc:

manoj# cp config/ndomod.cfg-sample /usr/local/nagios/etc/ndomod.cfg

You also need to install the ndo2db daemon and its configuration file. They are also located in the src and config directories, respectively, and I suggest you copy them to the same locations in your Nagios installation:

manoj# cp src/ndo2db-3x /usr/local/nagios/bin/ndo2db
manoj# cp config/ndo2db.cfg-sample /usr/local/nagios/etc/ndo2db.cfg

Next, you need to modify your Nagios configuration file, nagios.cfg, to load the NDO module when Nagios starts. Add the following line to your nagios.cfg configuration file, usually located in /usr/local/nagios/etc:
broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg
(It should be on single line entry)

This configuration directive will load the ndomod.o NEB module when Nagios is started. You will need to restart Nagios to make the module active. The config_file part of the directive must be modified to specify the location of the module configuration file. You should ensure that the ownership and permissions of all these files is appropriate. They should generally all be owned by the user and group used by the Nagios server process and the configuration files only readable by that user:

manoj# chown prod:prod /usr/local/nagios/bin/ndo2db /usr/local/nagios/bin/ndomod.o
/usr/local/nagios/etc/ndo2db.cfg /usr/local/nagios/etc/ndomod.cfg
manoj# chmod 0600 /usr/local/nagios/etc/ndo2db.cfg /usr/local/nagios/etc/ndo2db.cfg

You may also want to modify the two configuration files, ndo2db.cfg and ndomod.cfg. By default, the ndomod.o NEB module outputs data to a Unix domain socket, /usr/local/nagios/var/ndo.sock, which is created by the ndo2db daemon when it is started. You will also need to modify the ndo2db.cfg configuration file to update it with the correct database name, username, and password to allow the ndo2db daemon to write to the database.

Now you can start ndo2db by using the following command
manoj# /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg

The daemon is launched with one command-line option, the location of the ndo2db daemon’s configuration file ndo2db.cfg. The daemon will create the Unix domain socket, /usr/local/nagios/var/ndo.sock. As you can see, I used the su command to change to the user nagios before launching. You should run the ndo2db daemon as the nagios user to allow the Unix domain socket to be created with the correct ownership. This will allow the ndomod.o module, which is run with the ownership and permissions of the Nagios server process, to write to that domain socket.

The module logs events and errors in the default Nagios log file, usually /usr/local/nagios/var/nagios.log OR /usr/local/nagios/var/ndo2db.debug. Check this file for errors and messages.

Thanks
Manoj Chauhan

Mysql slave giving duplicate entry error

March 4th, 2010

We can remove the duplicate entry error on slave by running below command on slave. If we will not removed the Duplicate errors from the slave, replication will be stopped.

stop slave;
SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1;
start slave;
show slave status\G;

Thanks
Manoj Chauhan

Nagios: ndomod: Still unable to connect to data sink

March 4th, 2010

I have Nagios 3.x running perfactly, I want to load all alerts data to Mysql for future reference so we can create reports  accordingly.

I have installed NDOUtils but after instalation I am getting the following error in the /var/log/messages
nagios: ndomod: Still unable to connect to data sink.  14613 items lost, 5000 queued items to flush.   

I have fixed the issue by changing socket_type=tcp to socket_type=unix  in /usr/local/nagios/etc/ndo2db.cfg

And restart Nagios service and start ndo2db  by using the following command /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg

After doing above change, now if you get following logs in /var/log/medssages which means everything alright

Mar  4 04:45:44 server1 nagios: ndomod: Successfully connected to data sink.  18134 items lost, 5000 queued items to flush.
Mar  4 04:45:47 server1 nagios: ndomod: Successfully flushed 5000 queued items to data sink.

Thanks
Manoj Chauhan