Linux Admin: March 2013

Tuesday, 19 March 2013

Tomcat - server.xml

Testing Tool:

curl, wget, telnet

Containers

Tomcat refers to Engine, Host, Context, and Cluster, as container. The highest-level is Engine; while the lowest-level is Context. Certain components, such as Realm and Valve, can be placed in a container.

Engine

A Engine is the highest-level of a container. It can contains one or more Hosts. You could configure a Tomcat server to run on several hostnames, known as virtual host.

<Engine name="Catalina" defaultHost="localhost">

The Catalina Engine receives HTTP requests from the HTTP connector, and direct them to the correct host based on the hostname/IP address in the request header.

Realm

A Realm is a database of user, password, and role for authentication (i.e., access control). You can define Realm for any container, such as Engine, Host, and Context, and Cluster.

<Realm className="org.apache.catalina.realm.LockOutRealm">
  <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>
</Realm>

The default configuration defines a Realm (UserDatabaseRealm) for the Catalina Engine, to perform user authentication for accessing this engine. It uses the JNDI name UserDatabase defined in the GlobalNamingResources.
Besides the UserDatabaseRealm, there are: JDBCRealm (for authenticating users to connect to a relational database via the JDBC driver); DataSourceRealm (to connect to a DataSource via JNDI; JNDIRealm (to connect to an LDAP directory); and MemoryRealm (to load an XML file in memory).

Hosts

A Host defines a virtual host under the Engine, which can in turn support many Contexts (webapps).

<Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">

The default configuration define one host called localhost. The appBase attribute defines the base directory of all the webapps, in this case, <CATALINA_HOME>\webapps. By default, each webapp's URL is the same as its directory name. For example, the default Tomcat installation provides four webapps: docs, examples, host-manager and manager under the webapps directory. The only exception is ROOT, which is identified by an empty string. That is, its URL is http://localhost:8080/.
The unpackWARs specifies whether WAR-file dropped into the webapps directory shall be unzipped. For unpackWARs="false", Tomcat will run the application from the WAR-file directly, without unpacking, which could mean slower execution.
The autoDeploy attribute specifies whether to deploy application dropped into the webapps directory automatically.

Cluster

Tomcat supports server clustering. It can replicate sessions and context attributes across the clustered server. It can also deploy a WAR-file on all the cluster.

Valve

A Valve can intercept HTTP requests before forwarding them to the applications, for pre-processing the requests. A Valve can be defined for any container, such as Engine, Host, and Context, and Cluster.
In the default configuration, the AccessLogValve intercepts an HTTP request and creates a log entry in the log file, as follows:

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
       prefix="localhost_access_log." suffix=".txt"
       pattern="%h %l %u %t &quot;%r&quot; %s %b" />

Other valves include:

RemoteAddrValve: which blocks requests from certain IP addresses,
RemoteHostValve: which blocks request based on hostnames,
RequestDumperValve: which logs details of the requests,
SingleSignOn Valve: when placed under a <host>, allows single sign-on to access all the webapp under the host.

Sample of server.xml

<Server port="8006" shutdown="SHUTDOWN" debug="0">
<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" debug="0"/>
The GlobalResourcesLifecycleListener enables the global resources, and makes possible the use of JNDI for accessing resources such as databases.
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" debug="0"/>

<GlobalNamingResources>
    
    <Environment name="simpleValue" type="java.lang.Integer" value="30"/>

The <GlobalNamingResources> element defines the JNDI (Java Naming and Directory Interface) resources, that allows Java software clients to discover and look up data and objects via a name.
The default configuration defines a JNDI name called UserDatabase via the <Resource> element, which is a memory-based database for user authentication loaded from "conf/tomcat-users.xml".

    
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml"
       description="User database that can be updated and saved">
    </Resource>
</GlobalNamingResources>

<Service name="portal">
    
    <Connector port="80"
               maxThreads="150"
               minSpareThreads="25"
               maxSpareThreads="75"
               enableLookups="false"
               acceptCount="100"
               debug="0"
               connectionTimeout="2000"
               disableUploadTimeout="true"
               compression="on"
               address="172.16.95.150"/>
     
     <Connector port="443"
                keystoreFile="/root/certificates/fams2012.p12"
                keystorePass="Fortinet0511#"
                keystoreType="PKCS12"
                SSLEnabled="true"
                maxThreads="150"
                minSpareThreads="25"
                enableLookups="false"
                disableUploadTimeout="true"
                scheme="https"
                secure="true"
                connectionTimeout="2000"
                clientAuth="false"
                maxKeepAliveRequests="20"
                sslProtocol="TLS"
                address="172.16.95.150"
                restrictedUserAgents="^.*MS Web Services Client Protocol.*$"/>

     
     <Connector port="443"
                keystoreFile="/root/certificates/fmz1.p12"
                keystorePass="fortinet123"
                keystoreType="PKCS12"
                SSLEnabled="true"
                maxThreads="150"
                minSpareThreads="25"
                enableLookups="false"
                disableUploadTimeout="true"
                scheme="https"
                secure="true"
                connectionTimeout="2000"
                clientAuth="false"
                maxKeepAliveRequests="20"
                sslProtocol="TLS"
                address="172.16.95.155"
                restrictedUserAgents="^.*MS Web Services Client Protocol.*$"/>
    
    <Connector port="8009"
               enableLookups="false"
               redirectPort="443"
               debug="0"
               protocol="AJP/1.3" />
    <Engine name="portal" defaultHost="portal" debug="0">

      
      <Logger className="org.apache.catalina.logger.FileLogger" prefix="catalina_log." suffix=".txt" timestamp="true"/>
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm" debug="0" resourceName="UserDatabase"/>
      <Host name="portal"
            debug="0"
            appBase="webapps"
            unpackWARs="true"
            autoDeploy="true"
            xmlValidation="false"
            xmlNamespaceAware="false">
        <Logger className="org.apache.catalina.logger.FileLogger" directory="logs" prefix="manager_log." suffix=".txt" timestamp="true"/>
        <Logger className="org.apache.catalina.logger.SystemOutLogger" timestamp="true"/>
        <Context path=""
                 docBase="/usr/local/portal/fams"
                 debug="0"
                 reloadable="true"
                 crossContext="true"
                 allowLinking="true"
                 privileged="true"/>
      </Host>
    </Engine>
</Service>
</Server>

Monday, 18 March 2013

Tomcat Architecture

Source: http://www.ntu.edu.sg/home/ehchua/programming/howto/Tomcat_More.html

Tomcat Architecture

Tomcat is an HTTP server. Tomcat is also a servlet container that can execute Java Servlet, and converting JavaServer Pages (JSP) and JavaServerFaces (JSF) to Java Servlet. Tomcat employs a hierarchical and modular architecture as illustrated:

Tomcat's Installed Directory Structure

Tomcat installation provides these directories:

bin: for Tomcat's binaries and startup scripts.
conf: global configuration applicable to all the webapps. The default installation provides:
- One Policy File: catalina.policy for specifying security policy.
- Two Properties Files: catalina.properties and logging.properties,
- Four Configuration XML Files: server.xml (Tomcat main configuration file), web.xml (global web application deployment descriptors), context.xml (global Tomcat-specific configuration options) and tomcat-users.xml (a database of user, password and role for authentication and access control).
The conf also contain a sub-directory for each engine, e.g., Catalina, which in turn contains a sub-sub-directory for each of its hosts, e.g., localhost. You can place the host-specific context information (similar to context.xml, but named as webapp.xml for each webapp under the host).
lib: Keeps the JAR-file that are available to all webapps. The default installation include servlet-api.jar (Servlet), jasper.jar (JSP) and jasper-el.jar (EL). You may also keep the JAR files of external package here, such as MySQL JDBC driver (mysql-connector-java-5.1.{xx}-bin.jar) and JSTL (jstl.jar and standard.jar).
logs: contains the engine logfile Catalina.{yyyy-mm-dd}.log, host logfile localhost.{yyyy-mm-dd}.log, and other application logfiles such as manger and host-manager. The access log (created by the AccessLogValve) is also kept here.
webapps: the default appBase - web applications base directory of the host localhost.
work: contains the translated servlet source files and classes of JSP/JSF. Organized in hierarchy of engine name (Catalina), host name (localhost), webapp name, followed by the Java classes package structure.
temp: temporary files.

Monday, 11 March 2013

Awk Introduction Tutorial – 7 Awk Print Examples

Source: http://www.thegeekstuff.com/2010/01/awk-introduction-tutorial-7-awk-print-examples/

This is the first article on the new awk tutorial series. We’ll be posting several articles on awk in the upcoming weeks that will cover all features of awk with practical examples.
In this article, let us review the fundamental awk working methodology along with 7 practical awk print examples.

Note: Make sure you review our earlier Sed Tutorial Series.

Awk Introduction and Printing Operations

Awk is a programming language which allows easy manipulation of structured data and the generation of formatted reports. Awk stands for the names of its authors “Aho, Weinberger, and Kernighan”
The Awk is mostly used for pattern scanning and processing. It searches one or more files to see if they contain lines that matches with the specified patterns and then perform associated actions.
Some of the key features of Awk are:

Awk views a text file as records and fields.
Like common programming language, Awk has variables, conditionals and loops
Awk has arithmetic and string operators.
Awk can generate formatted reports

Awk reads from a file or from its standard input, and outputs to its standard output. Awk does not get along with non-text files.

Syntax:

awk '/search pattern1/ {Actions}
     /search pattern2/ {Actions}' file

In the above awk syntax:

search pattern is a regular expression.
Actions – statement(s) to be performed.
several patterns and actions are possible in Awk.
file – Input file.
Single quotes around program is to avoid shell not to interpret any of its special characters.

Awk Working Methodology

Awk reads the input files one line at a time.
For each line, it matches with given pattern in the given order, if matches performs the corresponding action.
If no pattern matches, no action will be performed.
In the above syntax, either search pattern or action are optional, But not both.
If the search pattern is not given, then Awk performs the given actions for each line of the input.
If the action is not given, print all that lines that matches with the given patterns which is the default action.
Empty braces with out any action does nothing. It wont perform default printing operation.
Each statement in Actions should be delimited by semicolon.

Let us create employee.txt file which has the following content, which will be used in the
examples mentioned below.

$cat employee.txt
100  Thomas  Manager    Sales       $5,000
200  Jason   Developer  Technology  $5,500
300  Sanjay  Sysadmin   Technology  $7,000
400  Nisha   Manager    Marketing   $9,500
500  Randy   DBA        Technology  $6,000

Awk Example 1. Default behavior of Awk

By default Awk prints every line from the file.

$ awk '{print;}' employee.txt
100  Thomas  Manager    Sales       $5,000
200  Jason   Developer  Technology  $5,500
300  Sanjay  Sysadmin   Technology  $7,000
400  Nisha   Manager    Marketing   $9,500
500  Randy   DBA        Technology  $6,000

In the above example pattern is not given. So the actions are applicable to all the lines.
Action print with out any argument prints the whole line by default. So it prints all the
lines of the file with out fail. Actions has to be enclosed with in the braces.

Awk Example 2. Print the lines which matches with the pattern.

$ awk '/Thomas/
> /Nisha/' employee.txt
100  Thomas  Manager    Sales       $5,000
400  Nisha   Manager    Marketing   $9,500

In the above example it prints all the line which matches with the ‘Thomas’ or ‘Nisha’. It has two patterns. Awk accepts any number of patterns, but each set (patterns and its corresponding actions) has to be separated by newline.

Awk Example 3. Print only specific field.

Awk has number of built in variables. For each record i.e line, it splits the record delimited by whitespace character by default and stores it in the $n variables. If the line has 4 words, it will be stored in $1, $2, $3 and $4. $0 represents whole line. NF is a built in variable which represents total number of fields in a record.

$ awk '{print $2,$5;}' employee.txt
Thomas $5,000
Jason $5,500
Sanjay $7,000
Nisha $9,500
Randy $6,000

$ awk '{print $2,$NF;}' employee.txt
Thomas $5,000
Jason $5,500
Sanjay $7,000
Nisha $9,500
Randy $6,000

In the above example $2 and $5 represents Name and Salary respectively. We can get the Salary using $NF also, where $NF represents last field. In the print statement ‘,’ is a concatenator.

Awk Example 4. Initialization and Final Action

Awk has two important patterns which are specified by the keyword called BEGIN and END.

Syntax: 

BEGIN { Actions}
{ACTION} # Action for everyline in a file
END { Actions }

# is for comments in Awk

Actions specified in the BEGIN section will be executed before starts reading the lines from the input.
END actions will be performed after completing the reading and processing the lines from the input.

$ awk 'BEGIN {print "Name\tDesignation\tDepartment\tSalary";}
> {print $2,"\t",$3,"\t",$4,"\t",$NF;}
> END{print "Report Generated\n--------------";
> }' employee.txt
Name Designation Department Salary
Thomas   Manager   Sales           $5,000
Jason   Developer   Technology   $5,500
Sanjay   Sysadmin   Technology   $7,000
Nisha   Manager   Marketing   $9,500
Randy   DBA     Technology   $6,000
Report Generated
--------------

In the above example, it prints headline and last file for the reports.

Awk Example 5. Find the employees who has employee id greater than 200

$ awk '$1 >200' employee.txt
300  Sanjay  Sysadmin   Technology  $7,000
400  Nisha   Manager    Marketing   $9,500
500  Randy   DBA        Technology  $6,000

In the above example, first field ($1) is employee id. So if $1 is greater than 200, then just do the default print action to print the whole line.

Awk Example 6. Print the list of employees in Technology department

Now department name is available as a fourth field, so need to check if $4 matches with the string “Technology”, if yes print the line.

$ awk '$4 ~/Technology/' employee.txt
200  Jason   Developer  Technology  $5,500
300  Sanjay  Sysadmin   Technology  $7,000
500  Randy   DBA        Technology  $6,000

Operator ~ is for comparing with the regular expressions. If it matches the default action i.e print whole line will be performed.

Awk Example 7. Print number of employees in Technology department

The below example, checks if the department is Technology, if it is yes, in the Action, just increment the count variable, which was initialized with zero in the BEGIN section.

$ awk 'BEGIN { count=0;}
$4 ~ /Technology/ { count++; }
END { print "Number of employees in Technology Dept =",count;}' employee.txt
Number of employees in Tehcnology Dept = 3

Then at the end of the process, just print the value of count which gives you the number of employees in Technology department.

Tuesday, 5 March 2013

How to setup a slave DNS Nameserver with Bind

Source: http://www.stardothosting.com/blog/2009/05/how-to-setup-a-slave-dns-nameserver-with-bind/

When implementing redundancy as far as DNS is concerned, automated is always better. In a hosting environment, new zone files are constantly being created.
This need for a DNS master/slave implementation where new zone files are transferred between the master nameserver and the slave became apparent as operations grew and geographic DNS redundancy became apparent.
Obviously some commercial dns products provide this type of functionality out-of-the-box, but I will show you how to do this with a simple Bind DNS distribution.
I wrote this tutorial to help you, hopefully, to create an automated DNS slave / zone file transfer environment. Obviously you can create as many slave servers as you feel necessary.
MASTER Server
1. Edit /etc/named.conf and add the following to the options section where xx.xx.xx.xx is the ip of your slave server.:

allow-transfer { xx.xx.xx.xx; };

2. Create a script with the following, where somedirectory is the directory on your SLAVE server to store the slave zones and where yy.yy.yy.yy is your MASTER server ip and somewwwdir is a directory browsable via http and finally someslavefile.conf is the output file to write you slave config:

#!/bin/sh
#
for domain in `/bin/grep ^zone /etc/named.conf |/bin/grep "type master" |/bin/awk '{print $2}' |/bin/awk -F" '{print $2}'`

do

/usr/bin/printf "zone "${domain}" { type slave; file "/var/named/slaves/somedirectory/${domain}.db"; masters { yy.yy.yy.yy; }; };n"

done > /var/www/html/somewwwdir/someslavefile.conf

3. Test the script to ensure it is writing out the appropriate format.
4. Run the script as any user with permission to write to an http visible directory via cron.

0 4 * * * /path/to/script > /dev/null 2>&1

SLAVE SERVER
1. Transfer the rndc.key file from your master server to the slave :

scp MASTERSERVER:/etc/rndc.key /etc/ns1rndc.key

2. Edit ns1rndc.key and change the name of the key definition.
3. Edit named.conf and add the following to the options section:

allow-transfer { zz.zz.zz.zz; };

4. Append the following to the named.conf file:

include "/etc/ns1rndc.key";
include "/path/to/someslavefile.conf";

5. Run the following commands

touch /path/to/someslavefile.conf
mkdir /var/named/slaves/somedirectory/
chown -R named:named /var/named/slaves/somedirectory/
/etc/init.d/named restart

6. Create a script:

#!/bin/sh
/usr/bin/wget http://yy.yy.yy.yy/somewwwdir/someslavefile.conf -O /var/named/slaves/someslavefile.conf
/etc/init.d/named restart

7. Add to root’s crontab

0 4 * * * /path/to/script

In the second slave script, you see that the transfer is done via wget. This can be replaced by many other more secure methods. If ssh based key authentication is employed, a simple scp or even rsync can be utilized to accomplish the actual zone transfer.