Remove blank/empty lines with awk

 How to remove empty or blank lines with awk.

Example 1:

command | awk '/^s*$/'

Example 2:

awk '/^s*$/' filename

Python substitute for netcat

When working across multiple platforms and flavours of IX, a system administrator  might not always have tools like telnet or netcat available to test network connections. One tool that is on all flavours of IX and is universal is python.

A quick python script for testing network connection.

I used this python script to send a message via UDP to a logstash server.

import socket
message = 'Flah blah bleh'
byteToSend = str.encode(message)
serverAddressPort = ('IP Address', port)
bufferSize = 1024
soc = socket.socket(family=socket.AF_INET, type=socket.SOCK_DGRAM)
soc.sendto(byteToSend, serverAddressPort)

Password validator

 Validating password with python.

Regex

import re

passRX = re.compile(r'[A-Za-z0-9@#$%^&+=]{8,}')

pp = passRX.search('dy_5mupV')

pp == None

True

pp = passRX.search('DEVGukvk')

pp == None

False

Reference:

https://stackoverflow.com/questions/2990654/how-to-test-a-regex-password-in-python

awk multiple field separators.

Using multiple field separators with awk and perl to convert "," to a newline.

Example:

echo "100(txt),528(smbw),529(smbt),530(smbn),10115(smbs)"|\
 perl -pe 's/,/\n/g'
100(txt)
528(smbw)
529(smbt)
530(smbn)
10115(smbs)

Using awk to just print the text between ().

echo "100(txt),528(smbw),529(smbt),530(smbn),10115(smbs)"|\
 perl -pe 's/,/\n/g'|\
 awk -F"[()]" '/smb/{print $2}'
smbw
smbt
smbn
smbs

Using awk to just print the text between [].

awk -F'[][]' and awk -F'[[]]' will not work.

Update remote /etc/hosts file using sudo

Sudo example

printf 'IP\tHost\n' | ssh -q host "sudo su - root -c 'cat - >> /etc/hosts'"

Debug bash scripts

Place the below lines near the top of the script.
Call debuggerIt in the script from a point where debugging should take place or at the start of the script execution.


OUTBASE="/tmp/${USER}/${0##*/}.log"

#-------------------------------------------
debuggerIt()
{
        test -d ${OUTBASE} || mkdir -p ${OUTBASE}
        exec 5> ${OUTBASE}/${0##*/}.debug
        BASH_XTRACEFD="5"
        PS4='$LINENO: '
        set -x
}
#-------------------------------------------

.

more functions

.

.

start of execution.

debuggerIt

more commands

Remote tar

Copy a files from remote location using a file list.

ssh host "cat fileList | xargs tar cf -" | tar xvf -

Using regex in vim

Search and replace using quantifiers.

Example:
This will erase all checkID's with single and/or double digits.

:%s/"checkID":[0-9]\{1,2\} //g

Before

Running command.

Split the $PATH variable into seperate lines.

To split the $PATH variable into separate lines can be useful, specially if you have a very long $PATH. Also good for showing double entries.




echo $PATH | awk '{split($0,a,":")}; END { for (p in a) printf "%s\n",a[p];}'

Man in the middle remote copy files.

To copy files between 2 remote servers using man in the middle server using tar.


ssh server1 "cd /data;tar cf - ." | ssh server2 "cd /data;tar xvf -"


 

Using commandline options in shell scripts

Sample code using getopt

while getopts :f:cuh OP
do
            case ${OP} in
                           f)         SERVERS=${OPTARG}
                                       ;;
                           c)        if [ "${SERVERS}" != "false" ]
                                      then
                                                check
                                       else
                                               usage
                                      fi
                                      ;;
                           u)       if [ "${SERVERS}" != "false" ]
                                     then
                                                cd ${BASE}
                                      fi
                                     ;;
                        *|h)      usage
                                     ;;
                     esac
done

Replace spaces with commas using sed

sed 's/ \{1,\}/,/g'

Use variable in search pattern in awk

Passing a variable to awk and using it in the search pattern.



awk -v string="$STRING" -f  script.awk  [filename]

Script extract:

$0 ~ string {
      
       print $0;
}

Note:

        On Solaris use nawk.

Match IP numbers with regex.

Print any IP address talking on port 21 in coming or out going.

Change the port number to any port that needs to be matched.

netstat -na | perl -ne '/((\d{1,3}\.){4}21)/ &&  print ;'

 

Variable assigments

Validating positional parameters.

The below example will print a usage message if there is no 2nd position paramaeter.

PACKAGE=${2:?$(usage ${SERVERLIST} [Package name:Version:Package file[:Dependent service]])}

awk and ifconfig

Print interface name and network config oneliner.

#ifconfig -a | awk '/^[a-z]/{intf=sprintf("%-10s", $1);getline; if($1=="inet") printf "%-6s %s\n", intf, $0;}'
          inet addr:172.19.2.26  Bcast:172.19.2.255  Mask:255.255.255.0
          inet addr:127.0.0.1  Mask:255.0.0.0

Colourise  your shell scripts

#!/usr/bin/ksh
#----------------------------------------------------------------------------------------------
#                       Define Colours for scripts.
#
#       Written : Stan Lovisa
#       Date    : 26-10-2012
#       Mod     :
#
# Black         0;30     Dark Gray      1;30
# Blue          0;34     Light Blue     1;34
# Green         0;32     Light Green    1;32
# Cyan          0;36     Light Cyan     1;36
# Red           0;31     Light Red      1;31
# Purple        0;35     Light Purple   1;35
# Brown         0;33     Yellow         1;33
# Light Gray    0;37     White          1;37
# 0 normal, 1 bold, 4 underline, 7 reverse video
#----------------------------------------------------------------------------------------------
#
#       Notes. How to use.
#
#       printf "${TXTGREEN}"    To switch on colour.
#       printf "${TXTNC}"       To switch off colour.
#
#----------------------------------------------------------------------------------------------
TXTRED="\033[1;31m"
TXTREDRV="\033[7;31m"
TXTLBLUE="\033[1;34m"
TXTBLUERV="\033[7;36m"
TXTGREEN="\033[0;32m"
TXTGREENRV="\033[7;32m"
TXTYELLOW="\033[1;33m"
TXTYELLOWRV="\033[7;33m"
TXTNC="\033[1;0m"

SMTP: Using smtp commands in a shell script when sendmail isn't setup.

LOG="/tmp/${0##*/}.log"

SUBJECT="${0##*/}"

MAILSERVER=mailhost

PORT="25"

MAILFROM="user1@domain user2@domain"

MAILLIST="email@somedomain"

#------------------------------------------------------------------------

mailto()

{

exec 3<>/dev/tcp/$MAILSERVER/$PORT

if [ $? -ne 0 ]

then

echo

echo "ERROR: Cannot connect to the Mail Server"

echo "Please check the servername and/or the port number"

exit

fi

echo -en "HELO ${MAILSERVER}\r\n" >&3

echo -en "MAIL FROM:$MAILFROM\r\n" >&3

for MAILTO in ${MAILLIST}

do

echo -en "RCPT TO:$MAILTO\r\n" >&3

done

echo -en "DATA\r\n" >&3

echo -en "Subject: $SUBJECT\r\n\r\n" >&3

echo -en "$(cat ${LOG})\r\n" >&3

echo -en ".\r\n" >&3

echo -en "QUIT\r\n" >&3

cat <&3

}

#------------------------------------------------------------------------

Some commands

.

.

mailto

AD group member listing.

List member of a AD group from XP

net group G_UNIX_SCHED /domain

or list user attributes.

net user <user id> /domain

Awk: Using getline to read a file in BEGIN

Example reading in the password file into an array for later use.

BEGIN {

        FS=":";

        while ((getline  < "/etc/passwd")  >;0 )

        {

                Alias[$1]=$0;

        }

}

Shell Scripting: basename variable

"${0##*/}" is the equivalent of "basename $0".
I.e. From /usr/bin/sh we get just sh.


Also try this to execute commands $(shell command)

Perl script options

Using getopt in perl scripts.
Sample code

#!/usr/bin/perl
#-------------------------------------------------------------------------
#       Using getop
#-------------------------------------------------------------------------
use Getopt::Std;
getopt('cw');
my @op = `./x.sh -c $opt_c -w $opt_w`;
foreach (@op)
{
        #printf "-c %s\t-w %s\n", $opt_c,$opt_w;
        print $_;
}

Output

./x.pl -c 3 -w 55

in ./x.sh.sh : -c 3 -w 55

-c 3

-w 55

or

my %options=();

#Pre set default values 

my $critv=95;
my $warnv=85;
getopts("c:w:", \%options);
$critv = $options{c} if defined $options{c};
$warnv = $options{w} if defined $options{w};

This method won't report an error if an option is emitted.

awk convert time string to time format

This will take a time string HHMMSS (121525) and format it as 12:15:25

A sample line from a.txt:

01201 095440 sd10 3393.2 1.1 13572.8 4.4 0.0 1.0 0.3 2 91

awk '{h=substr($2,1,2);\
      m=substr($2,3,2);\
      s=substr($2,5,2);\
      $2=sprintf("%s:%s:%s",h,m,s);\
      print $0;}' a.txt | head 

Will produce the following out put:

101201 09:54:40 sd10 3393.2 1.1 13572.8 4.4 0.0 1.0 0.3 2 91

Midnight commander

Startup Midnight commander with each panel in a pre-detirmined location.

mc /tmp /#ftp:guest@remote-host.com/pub/dir

Using awk to remove path and print file date

$ls -lat /current/*xml | \
         nawk '{printf "%s ", $8; system("basename " $9)}'

20:24 file1.xml

21:44 file2.xml

22:34 file3.xml
$

Translate upper case to lower case with vim

Translate upper case to lower case with vim.

:%s/[A-Z]/\L&/g

or

:%s/.*/\L&/

vi: Appending text to the end of each line

Appending text to the end of each line.

:%s/$/TextYouWantToAppend/

Add users from copied passwd file

for U in $(awk -F":" '{print $1}' passwd)
do
      echo "________________ $U __________________"
      UDIR=$(awk -F":" "/$U/"'{print $6}' passwd)
      mkdir $UDIR
      chown $U $UDIR
done

Simple compiling with gcc

Having trouble with a simple compile:

gcc -o data data.c
/tmp/ccCAfDR5.o: In function `main':
data.c:(.text+0x4c): undefined reference to `exp'
collect2: ld returned 1 exit status

Then:

gcc -lm -o data data.c

And it works, because math.h is just a header file and you need to link with the math's library.

rsync with ssh between servers

Rsync quicky with ssh between servers.

rsync -av -n source dir/ server:/target dir -n for testing. nothing happens.

rsync -av source dir/ server:/target dir -n removed. Do it for real.

Also note:
Source dir has a "/" after it to copy its contents and not the directory itself.

Rsync installed in a different location on the remote host:

rsync --rsync-path=/opt/csw/bin/rsync -av -n source dir/ server:/target

Perl. Replace text in file.

Substitute text in a file in-situ

perl -i -pe 's/1234/4321/g' filename

-i edits file in-situ.
-p loops through each filename and prints out each line.

Convert Upper case to lower case

perl -i -pe 'tr/A-Z/a-z/' x

Shell scripting: case statement & numbers

Selecting a numeric range with the case statement.

case ${NUMBER} in
        [0-9]|[0-6][0-9]|7[0-5])
                                echo "${NUMBER} <= 75"
       ;;
       [7-8][0-9]|9[0-5])
                               echo "75 < ${NUMBER}<=95"
       ;;
       [9][5-9]|100)
                                echo "95< ${NUMBER} <=100"
       ;;
esac

Shell scripting: Testing empty variables

$V="xNotEmpty"

if [ x$V = "xNotEmpty" ]
then
       echo "\$V not empty"
else
       echo"\$V is empty"
fi

Or:

if [ "$V" = "" ]
then
      echo "\$V is empty
fi

awk

Using shell variables with awk.

1. awk "/$func/"'{print $0}' filename

2. awk -v pat=$func '/pat/{print $0}' filename

Using fields in search pattern

For a similar match:

1. awk '$1~/pattern/{print $0}' filename

For a precise match:

2.  awk -F":" -v user=$U '$1 == user {print $0}' filename

Perl command line.

Using multiple perl commands on the command line.

perl -n -e 'perl cmd;'\
-e 'perlcmd'

Invoke the split command

perl -n -a -e 'cmd'

-n Places a loop around the command line.

My .vimrc file

syntax on
set viminfo^=h
set ruler
set nohls
set incsearch
set showmatch
set ts=4 sw=4
syntax enable
set hlsearch
"Use F4 to switch high lighting on/off
:noremap <F4> :set hlsearch! hlsearch<CR>

Shell Scripting: Special Variables

$?        Exit status or return code of last command.
$#        Number of arguments.
$@        Argument 1 thru n with Input Field Separator.
$*        "$1" $2" ... $n.
$!        Process id of last background process.
$$        Process id of shell running this script.
$-        The current shell flags.

Searching for SUID & SGID files.

find / \( -user root -perm -004000 -o -perm -002000 \) -type f -print