User Migration (Redhat, CentOS, Fedora)

Following files/dirs are required for traditional Linux user management:

* /etc/passwd - contains various pieces of information for each user account

* /etc/shadow - contains the encrypted password information for user's accounts and optional the password aging information.

* /etc/group - defines the groups to which users belong

* /etc/gshadow - group shadow file (contains the encrypted password for group)

* /var/spool/mail - Generally user emails are stored here.

* /home - All Users data is stored here.

You need to backup all of the above files and directories from old server to new Linux server.
Commands to type on old Linux system

First create a tar ball of old uses (old Linux system). Create a directory:
# mkdir /root/move/
Setup UID filter limit:
# export UGIDLIMIT=500
Now copy /etc/passwd accounts to /root/move/passwd.mig using awk to filter out system account (i.e. only copy user accounts)
# awk -v LIMIT=$UGIDLIMIT -F: '($3>=LIMIT) && ($3!=65534)' /etc/passwd > /root/move/passwd.mig
Copy /etc/group file:
# awk -v LIMIT=$UGIDLIMIT -F: '($3>=LIMIT) && ($3!=65534)' /etc/group > /root/move/group.mig
Copy /etc/shadow file:
# awk -v LIMIT=$UGIDLIMIT -F: '($3>=LIMIT) && ($3!=65534) {print $1}' /etc/passwd | tee - |egrep -f - /etc/shadow > /root/move/shadow.mig
Copy /etc/gshadow (rarely used):
# cp /etc/gshadow /root/move/gshadow.mig
Make a backup of /home and /var/spool/mail dirs:
# tar -zcvpf /root/move/home.tar.gz /home
# tar -zcvpf /root/move/mail.tar.gz /var/spool/mail

Where,

* Users that are added to the Linux system always start with UID and GID values of as specified by Linux distribution or set by admin. Limits according to different Linux distro:
o RHEL/CentOS/Fedora Core : Default is 500 and upper limit is 65534 (/etc/libuser.conf).
o Debian and Ubuntu Linux : Default is 1000 and upper limit is 29999 (/etc/adduser.conf).
* You should never ever create any new system user accounts on the newly installed Cent OS Linux. So above awk command filter out UID according to Linux distro.
* export UGIDLIMIT=500 - setup UID start limit for normal user account. Set this value as per your Linux distro.
* awk -v LIMIT=$UGIDLIMIT -F: '($3>=LIMIT) && ($3!=65534)' /etc/passwd > /root/move/passwd.mig - You need to pass UGIDLIMIT variable to awk using -v option (it assigns value of shell variable UGIDLIMIT to awk program variable LIMIT). Option -F: sets the field separator to : . Finally awk read each line from /etc/passwd, filter out system accounts and generates new file /root/move/passwd.mig. Same logic is applies to rest of awk command.
* tar -zcvpf /root/move/home.tar.gz /home - Make a backup of users /home dir
* tar -zcvpf /root/move/mail.tar.gz /var/spool/mail - Make a backup of users mail dir

Use scp or usb pen or tape to copy /root/move to a new Linux system.
# scp -r /root/move/* user@new.linuxserver.com:/path/to/location
Commands to type on new Linux system

First, make a backup of current users and passwords:
# mkdir /root/newsusers.bak
# cp /etc/passwd /etc/shadow /etc/group /etc/gshadow /root/newsusers.bak

Now restore passwd and other files in /etc/
# cd /path/to/location
# cat passwd.mig >> /etc/passwd
# cat group.mig >> /etc/group
# cat shadow.mig >> /etc/shadow
# /bin/cp gshadow.mig /etc/gshadow

Please note that you must use >> (append) and not > (create) shell redirection.

Now copy and extract home.tar.gz to new server /home
# cd /
# tar -zxvf /path/to/location/home.tar.gz

Now copy and extract mail.tar.gz (Mails) to new server /var/spool/mail
# cd /
# tar -zxvf /path/to/location/mail.tar.gz

Now reboot system; when the Linux comes back, your user accounts will work as they did before on old system:
# reboot

Please note that if you are new to Linux perform above commands in a sandbox environment. Above technique can be used to UNIX to UNIX OR UNIX to Linux account migration. You need to make couple of changes but overall the concept remains the same.

How to Create a swap file in Linux partition

Linux divides its physical RAM (random access memory) into chucks of memory called pages. Swapping is the process whereby a page of memory is copied to the preconfigured space on the hard disk, called swap space, to free up that page of memory. The combined sizes of the physical memory and the swap space is the amount of virtual memory available.

Swapping is necessary for two important reasons. First, when the system requires more memory than is physically available, the kernel swaps out less used pages and gives memory to the current application (process) that needs the memory immediately. Second, a significant number of the pages used by an application during its startup phase may only be used for initialization and then never used again. The system can swap out those pages and free the memory for other applications or even for the disk cache.

Steps to create swap file

1. Use the "dd" command to create a file.

dd if=/dev/zero of=/aSwapFile bs=1024 count=65536

where dd: - is used to copy a specified number of bytes from an Input file (if) to an Output file (of).
Here the dd command copy null characters from the special file "/dev/zero" and copy it to the output file "aSwapFile" in the "/" directory. The "bs" specifies that the characters are read as BYTES. The "count" specifies the size of the bytes block that is to be created in the output file.

2. Use the "mkswap" command to set up the Linux swap area on the file created.

mkswap /aSwapFile

3. Use the "swapon" command to activate the swap file created.

swapon /aSwapFile

4. Edit the fstab "/etc/fstab" to enable the swap after a reboot.

vi /etc/fstab
(Add the following line to the fstab file.)
/aSwapFile swap swap defaults 0 0

Batch file renaming

Batch file renaming can be pretty handy as it saves loads of time when you have to rename thousands of files. There are many simple ways to do it though.

If you have files with same ending pattern it can be done as:

e.g. if you want all .htm to be renamed to .html

rename .htm .html *.htm

and On other Unixes, we'd have to do something like this to batch rename files:

for i in *.html
do
j=`echo $i | sed 's/.htm$/.html/'`
# or, in this simple case even just: j=$"i"l
mv $i $j
done

Though if we had bash or ksh, we could make that a little less cumbersome:

for i in *.htm
do
j=${i%.htm}.html
# or: j=${i}l
mv $i $j
done

The Linux "rename" isn't going to handle to more complex cases though. For example, I had to transfer mail files from one system to another recently. On the old system, each message would be named something like "1124993500.7359464636.e-smith". On the new system, they'd be "00000001.eml", with hexadecimal numbering going on up, so you'd get "00000009.eml" and the next message would be "0000000a.eml", and so on. There's no way for "rename" to do that, but a loop can:

for i in *
do
j=`printf "%0.9x.eml\n" $x`
mv $i $j
x=$((x+1))
done

And if you dont want to follow any pattern and simply add new extension to all the files in a directory:

for i in $(ls -lArt /path/to/directory |awk '{print $9}');do mv $i $i.txt; done

adding to it if files are placed recursively in directories:

for i in $(find /path/to/root -type f -print);do mv $i $i.txt; done

Analyze apache log with awk

Lets say we want to find the amount of times a specific ip address has hit your webserver,
on this example we are assuming your apache access_log is located in /usr/local/apache/logs

The full command would be:

awk '{print $1}' /usr/local/apache/logs/access.log | sort | uniq -c | sort -fr

and the output would be like this:

155 90.193.xx.xx
154 86.143.xx.xx
109 82.17.xx.xx
85 90.213.xx.xx
74 193.28.xx.xx

Merging 2 files using awk

When the contents of two files are required to be merged without repetition and neatly sorted AWK can help us do it on the fly.
The comparison of two files is done using the first column as parameter. e.g. there are two files as below with required ouput:

file1

827 111111111888811117122111
828 111189292929222222222222
830 112203030371818181811111
832 120811117777777777777777
833 110000000000000000811111

file2

829 561111111888811117122111
830 112203030371818181811111
831 199991772929222222222222
833 110000000000000000811111
834 166666666666666666777777

Merged_file

827 111111111888811117122111
828 111189292929222222222222
829 561111111888811117122111
830 112203030371818181811111
831 199991772929222222222222
832 120811117777777777777777
833 110000000000000000811111
834 166666666666666666777777

Here's the code:

# awk '{a[$1]=$0}END{for(i in a)print a[i]}' file1 file2 | sort > Merged_file

Find and calculate total size of files older than a month

For some applications it is very common to create temporary files or we can say cache files periodically, which if not removed regularly can consume a lot of disk space unnecessarily. Here is a tip which will save you from wondering how to find and get rid of these.

1. Here's a simple script to find and calculate total size of the files found if you just want to know if these exist under your application or not:

#!/bin/bash

export CACHE_DIR=/path/to/Cache/dir
export file_list=/var/log/cachecron/CacheCron.log.`date +"%Y%m%d_%H%M"` ;
export cache_report=/var/log/cachecron/CacheReport.log.`date +"%Y%m%d_%H%M"` ;
export size=0

#### List cache files #########

find $CACHE_DIR -type f -mtime +30 -exec ls -lAh {} \;| awk '{print $9}' > $file_list ;

##### Calculating the size of cache ##########

echo "CACHE Watch for SEVERNAME `date +"%d-%m-%Y"`" > $cache_report;
echo " " >> $cache_report;
echo "List of files is kept at $file_list" >> $cache_report;
echo "Total no. of files found `cat $file_list|wc -l`" >> $cache_report;
for file in `cat ${file_list}`; do
if [ -f "${file}" ]; then
file_size=`ls -l ${file} | awk '{print $5}'`
size=`expr ${size} + ${file_size}`
fi
done
echo " " >> $cache_report;

### Display Consolidated size in MB###########

echo "Overall Cache: $((${size} / 1048576)) MB" >> $cache_report;

#####Overnight mailer for daily log watch#####

mailx -s "CACHE Watch for SERVERNAME `date +"%d-%m-%Y"`" nixtrap@blogspot.com < $cache_report; #####################################

2. That was just to keep a check on log files and track on daily addition of cache files with respect to nuber and size. If this all seems crap to you and you just want to get rid of these without any intimation here's a one liner for you:

$ find /Cache/dir/path -type f -mtime +30 -exec rm -f {} \;

poof !! there goes the crap ... but before running that just make sure what you are going to erase. This one liner will only list the files found:

$ find /Cache/dir/path -type f -mtime +30 -exec ls -la {} \;

Rebooting the Magic Way

If you have ever had a hard drive fail on a remote server you may remember the feeling you had after trying to issue the following commands:

   
 # reboot
 bash: /sbin/reboot: Input/output error
 # shutdown -r now
 bash: /sbin/shutdown: Input/output error

Obviously, there is a problem with your drive. These commands are failing because the kernel is unable to load the /sbin/reboot and /sbin/shutdown binaries from the disk so that it can execute them.

A fsck on the next boot might be able to correct whatever is wrong with the disk, but first you need to get the system to reboot. If your machine is located at a managed hosting provider then you could submit a reboot ticket, but you'll have to wait for someone to take responsibility.

Wouldn't it be nice if there was a way to ask the kernel to reboot without needing to access the failing drive? Well, there is a way, and it is remarkably simple.

The "magic SysRq key" provides a way to send commands directly to the kernel through the /proc filesystem. It is enabled via a kernel compile time option, CONFIG_MAGIC_SYSRQ, which seems to be standard on most distributions. First you must activate the magic SysRq option:

   
 echo 1 > /proc/sys/kernel/sysrq

When you are ready to reboot the machine simply run the following:

    
echo b > /proc/sysrq-trigger

This does not attempt to unmount or sync filesystems, so it should only be used when absolutely necessary, but if your drive is already failing then that may not be a concern.

In addition to rebooting the system the sysrq trick can be used to dump memory information to the console, sync all filesystems, remount all filesystems in read-only mode, send SIGTERM or SIGKILL to all processes except init, or power off the machine entirely, among other things.

Also, instead of echoing into /proc/sys/kernel/sysrq each time you can activate the magic SysRq key at system boot time using sysctl, where supported:

    
echo "kernel.sysrq = 1" >> /etc/sysctl.conf

If you would like to learn more about magic SysRq you can read the sysrq.txt file in the kernel documentation.

Downloading an Entire Web Site with wget

If you ever need to download an entire Web site, perhaps for off-line viewing, wget can do the
job—for example:

$ wget \
  --recursive \
  --no-clobber \
  --page-requisites \
  --html-extension \
  --convert-links \
  --restrict-file-names=windows \
  --domains website.org \
  --no-parent \
     http://centos.yubis.org/5.3/  

This command downloads the Web site http://centos.yubis.org/5.3/

The options are:

• --recursive: download the entire Web site.

• --domains website.org: don't follow links outside website.org.

• --no-parent: don't follow links outside the directory tutorials/html/.

• --page-requisites: get all the elements that compose the page (images, CSS and so on).

• --html-extension: save files with the .html extension.

• --convert-links: convert links so that they work locally, off-line.

• --restrict-file-names=windows: modify filenames so that they will work in Windows as well.

• --no-clobber: don't overwrite any existing files (used in case the download is interrupted and
  resumed).

Copying a Filesystem between Computers

If you need to transfer an entire filesystem from one machine to another, for example, when you get a new computer, do the following steps.

1) Boot both PCs with any Linux live CD (for example, Knoppix), and make sure they can access each other via the network.

2) On the source machine, mount the partition containing the filesystem to be copied, and start the transfer using netcat and tar:

cd /mnt/sda1
tar -czpsf - . | pv -b | nc -l 3333

3) On the destination machine, mount the partition to receive the filesystem, and start the process:

cd /mnt/sda1
nc 192.168.10.101 3333 | pv -b | tar -xzpsf -

The nc (netcat) command is used for any kind of TCP connections between two hosts. The pv (progress viewer) command is used to display the progress of the transfer. tar is used to archive the files on the source machine and un-archive them on the destination.

Netstat for finding Out Applications Running on NIX box

Netstat is a handy tool which can also help out in finding Applications running on the box. Using netstat we can list down current listening sockets which infers to the applications running on the box.

$ netstat -alptn

-a, --all
-l, --listening
Show only listening sockets. (These are omitted by default.)
-p, --program
Show the PID and name of the program to which each socket belongs.
-n, --numeric
Show numerical addresses instead of trying to determine symbolic host, port or user names.
-t, --tcp
Limits the output to show only TCP connections

It can also be done using "nmap hostname/IP" but it is restricted sometimes due to security reasons.

AWK me !!!

AWK is really great Language if you want to play on with text or we can say with variables in a script. Let's just start with a few tips on text manipulation :

Print column1, column4 and column9 of a data file or output of any columns list

$awk ‘{print $1, $4, $9}’ /path/to/file
OR
$cat filename |awk ‘{print $1 $4 $9}’
OR
$ls –al |awk ‘{print $1, $4, $9}’ -- Prints file_permissions,group and file_name

Syntax of running an awk program

Awk ‘program’ input file(s)

List all files names whose file size greater than zero.

$ls –al |awk ‘$5 > 0 {print $9}’

List all files whose file size equal to 512bytes.

$ls –al |awk ‘$5 == 0 {print $9}’

"AWK is a language for processing files of text. A file is treated as a sequence of records, and by default each line is a record. Each line is broken up into a sequence of fields, so we can think of the first word in a line as the first field, the second word as the second field, and so on. An AWK program is of a sequence of pattern-action statements. AWK reads the input a line at a time. A line is scanned for each pattern in the program, and for each pattern that matches, the associated action is executed." - Alfred V. Aho

For more info on awk please visit awk.info -- the awk community portal