Search This Blog

Wednesday, July 12, 2023

Remove blank/empty lines with awk

 How to remove empty or blank lines with awk.

Example 1:

command | awk '/^s*$/'


Example 2:

awk '/^s*$/' filename


Monday, November 02, 2020

Python substitute for netcat

When working across multiple platforms and flavours of IX, a system administrator  might not always have tools like telnet or netcat available to test network connections. One tool that is on all flavours of IX and is universal is python.


A quick python script for testing network connection.

I used this python script to send a message via UDP to a logstash server.


import socket
message = 'Flah blah bleh'
byteToSend = str.encode(message)
serverAddressPort = ('IP Address', port)
bufferSize = 1024
soc = socket.socket(family=socket.AF_INET, type=socket.SOCK_DGRAM)
soc.sendto(byteToSend, serverAddressPort)



Tuesday, August 11, 2020

Password validator

 Validating password with python.


Regex

import re

passRX = re.compile(r'[A-Za-z0-9@#$%^&+=]{8,}')

pp = passRX.search('dy_5mupV')

pp == None

True


pp = passRX.search('DEVGukvk')

pp == None

False


Reference:

Thursday, June 18, 2020

awk multiple field separators.

Using multiple field separators with awk and perl to convert "," to a newline.

Example:

echo "100(txt),528(smbw),529(smbt),530(smbn),10115(smbs)"|\
 perl -pe 's/,/\n/g'
100(txt)
528(smbw)
529(smbt)
530(smbn)
10115(smbs)

Using awk to just print the text between ().
echo "100(txt),528(smbw),529(smbt),530(smbn),10115(smbs)"|\
 perl -pe 's/,/\n/g'|\
 awk -F"[()]" '/smb/{print $2}'
smbw
smbt
smbn
smbs

Using awk to just print the text between [].

awk -F'[][]' and awk -F'[[]]' will not work.

Wednesday, October 23, 2019

Update remote /etc/hosts file using sudo

Sudo example

printf 'IP\tHost\n' | ssh -q host "sudo su - root -c 'cat - >> /etc/hosts'"

Wednesday, August 28, 2019

Debug bash scripts


Place the below lines near the top of the script.
Call debuggerIt in the script from a point where debugging should take place or at the start of the script execution.


OUTBASE="/tmp/${USER}/${0##*/}.log"

#-------------------------------------------
debuggerIt()
{
        test -d ${OUTBASE} || mkdir -p ${OUTBASE}
        exec 5> ${OUTBASE}/${0##*/}.debug
        BASH_XTRACEFD="5"
        PS4='$LINENO: '
        set -x
}
#-------------------------------------------
.
more functions
.
.
start of execution.
debuggerIt
more commands

Thursday, April 25, 2019

Remote tar

Copy a files from remote location using a file list.


ssh host "cat fileList | xargs tar cf -" | tar xvf -

Wednesday, May 30, 2018

Using regex in vim

Search and replace using quantifiers.

Example:
This will erase all checkID's with single and/or double digits.

:%s/"checkID":[0-9]\{1,2\} //g

Before






Running command.



Wednesday, August 17, 2016

Split the $PATH variable into seperate lines.

To split the $PATH variable into separate lines can be useful, specially if you have a very long $PATH. Also good for showing double entries.




echo $PATH | awk '{split($0,a,":")}; END { for (p in a) printf "%s\n",a[p];}'



Monday, February 08, 2016

Man in the middle remote copy files.

To copy files between 2 remote servers using man in the middle server using tar.


ssh server1 "cd /data;tar cf - ." | ssh server2 "cd /data;tar xvf -"


 

Friday, September 18, 2015

Using commandline options in shell scripts

Sample code using getopt

while getopts :f:cuh OP
do
            case ${OP} in
                           f)         SERVERS=${OPTARG}
                                       ;;
                           c)        if [ "${SERVERS}" != "false" ]
                                      then
                                                check
                                       else
                                               usage
                                      fi
                                      ;;
                           u)       if [ "${SERVERS}" != "false" ]
                                     then
                                                cd ${BASE}
                                      fi
                                     ;;
                        *|h)      usage
                                     ;;
                     esac
done

Friday, May 08, 2015

Use variable in search pattern in awk

Passing a variable to awk and using it in the search pattern.



awk -v string="$STRING" -f  script.awk  [filename]

Script extract:

$0 ~ string {
      
       print $0;
}

Note:

        On Solaris use nawk.

Tuesday, April 14, 2015

Match IP numbers with regex.


Print any IP address talking on port 21 in coming or out going.
Change the port number to any port that needs to be matched.

netstat -na | perl -ne '/((\d{1,3}\.){4}21)/ &&  print ;'
 

Monday, January 26, 2015

Variable assigments

Validating positional parameters.

The below example will print a usage message if there is no 2nd position paramaeter.

PACKAGE=${2:?$(usage ${SERVERLIST} [Package name:Version:Package file[:Dependent service]])}

Wednesday, August 06, 2014

awk and ifconfig

Print interface name and network config oneliner.


#ifconfig -a | awk '/^[a-z]/{intf=sprintf("%-10s", $1);getline; if($1=="inet") printf "%-6s %s\n", intf, $0;}'
          inet addr:172.19.2.26  Bcast:172.19.2.255  Mask:255.255.255.0
          inet addr:127.0.0.1  Mask:255.0.0.0

Wednesday, October 16, 2013

Colourise  your shell scripts
#!/usr/bin/ksh
#----------------------------------------------------------------------------------------------
#                       Define Colours for scripts.
#
#       Written : Stan Lovisa
#       Date    : 26-10-2012
#       Mod     :
#
# Black         0;30     Dark Gray      1;30
# Blue          0;34     Light Blue     1;34
# Green         0;32     Light Green    1;32
# Cyan          0;36     Light Cyan     1;36
# Red           0;31     Light Red      1;31
# Purple        0;35     Light Purple   1;35
# Brown         0;33     Yellow         1;33
# Light Gray    0;37     White          1;37
# 0 normal, 1 bold, 4 underline, 7 reverse video
#----------------------------------------------------------------------------------------------
#
#       Notes. How to use.
#
#       printf "${TXTGREEN}"    To switch on colour.
#       printf "${TXTNC}"       To switch off colour.
#
#----------------------------------------------------------------------------------------------
TXTRED="\033[1;31m"
TXTREDRV="\033[7;31m"
TXTLBLUE="\033[1;34m"
TXTBLUERV="\033[7;36m"
TXTGREEN="\033[0;32m"
TXTGREENRV="\033[7;32m"
TXTYELLOW="\033[1;33m"
TXTYELLOWRV="\033[7;33m"
TXTNC="\033[1;0m"




Thursday, July 11, 2013

Scripting SMTP
SMTP: Using smtp commands in a shell script when sendmail isn't setup.




LOG="/tmp/${0##*/}.log"
SUBJECT="${0##*/}"
MAILSERVER=mailhost
PORT="25"
MAILFROM="user1@domain user2@domain"
MAILLIST="email@somedomain"
#------------------------------------------------------------------------
mailto()
{
exec 3<>/dev/tcp/$MAILSERVER/$PORT
if [ $? -ne 0 ]
then
echo
echo "ERROR: Cannot connect to the Mail Server"
echo "Please check the servername and/or the port number"
exit
fi
echo -en "HELO ${MAILSERVER}\r\n" >&3
echo -en "MAIL FROM:$MAILFROM\r\n" >&3
for MAILTO in ${MAILLIST}
do
echo -en "RCPT TO:$MAILTO\r\n" >&3
done
echo -en "DATA\r\n" >&3
echo -en "Subject: $SUBJECT\r\n\r\n" >&3
echo -en "$(cat ${LOG})\r\n" >&3
echo -en ".\r\n" >&3
echo -en "QUIT\r\n" >&3
cat <&3
}
#------------------------------------------------------------------------
Some commands
.
.
mailto





Tuesday, September 20, 2011

AD group member listing.

List member of a AD group from XP

net group G_UNIX_SCHED /domain

or list user attributes.


net user <user id> /domain

Tuesday, August 30, 2011

Awk: Using getline to read a file in BEGIN

Example reading in the password file into an array for later use.
BEGIN {
        FS=":";
        while ((getline  < "/etc/passwd")  >;0 )
        {
                Alias[$1]=$0;
        }
}



Tuesday, June 28, 2011

Shell Scripting: basename variable

"${0##*/}" is the equivalent of "basename $0".
I.e. From /usr/bin/sh we get just sh.


Also try this to execute commands $(shell command)

Thursday, December 23, 2010

Perl script options

Using getopt in perl scripts.
Sample code
#!/usr/bin/perl
#-------------------------------------------------------------------------
#       Using getop
#-------------------------------------------------------------------------
use Getopt::Std;
getopt('cw');
my @op = `./x.sh -c $opt_c -w $opt_w`;
foreach (@op)
{
        #printf "-c %s\t-w %s\n", $opt_c,$opt_w;
        print $_;
}

Output
./x.pl -c 3 -w 55
in ./x.sh.sh : -c 3 -w 55
-c 3
-w 55

or

my %options=();
#Pre set default values 
my $critv=95;
my $warnv=85;
getopts("c:w:", \%options);
$critv = $options{c} if defined $options{c};
$warnv = $options{w} if defined $options{w};

This method won't report an error if an option is emitted.

Wednesday, December 01, 2010

awk convert time string to time format

This will take a time string HHMMSS (121525) and format it as 12:15:25

A sample line from a.txt:

01201 095440 sd10 3393.2 1.1 13572.8 4.4 0.0 1.0 0.3 2 91

awk '{h=substr($2,1,2);\
      m=substr($2,3,2);\
      s=substr($2,5,2);\
      $2=sprintf("%s:%s:%s",h,m,s);\
      print $0;}' a.txt | head 
 

Will produce the following out put:

101201 09:54:40 sd10 3393.2 1.1 13572.8 4.4 0.0 1.0 0.3 2 91

Thursday, September 02, 2010

Midnight commander

Startup Midnight commander with each panel in a pre-detirmined location.

mc /tmp /#ftp:guest@remote-host.com/pub/dir

Thursday, April 08, 2010

Using awk to remove path and print file date

$ls -lat /current/*xml | \
         nawk '{printf "%s ", $8; system("basename " $9)}'
20:24 file1.xml
21:44 file2.xml
22:34 file3.xml
$

Sunday, March 21, 2010

Thursday, February 18, 2010

Tuesday, August 18, 2009

Add users from copied passwd file


for U in $(awk -F":" '{print $1}' passwd)
do
echo "________________ $U __________________"
UDIR=$(awk -F":" "/$U/"'{print $6}' passwd)
mkdir $UDIR
chown $U $UDIR
done

Friday, October 10, 2008

Simple compiling with gcc

Having trouble with a simple compile:

gcc -o data data.c
/tmp/ccCAfDR5.o: In function `main':
data.c:(.text+0x4c): undefined reference to `exp'
collect2: ld returned 1 exit status

Then:

gcc -lm -o data data.c

And it works, because math.h is just a header file and you need to link with the math's library.

Tuesday, August 12, 2008

rsync with ssh between servers

Rsync quicky with ssh between servers.

rsync -av -n source dir/ server:/target dir -n for testing. nothing happens.

rsync -av source dir/ server:/target dir -n removed. Do it for real.

Also note:
Source dir has a "/" after it to copy its contents and not the directory itself.

Rsync installed in a different location on the remote host:

rsync --rsync-path=/opt/csw/bin/rsync -av -n source dir/ server:/target

Tuesday, June 10, 2008

Perl. Replace text in file.

Substitute text in a file in-situ

perl -i -pe 's/1234/4321/g' filename

-i edits file in-situ.
-p loops through each filename and prints out each line.

Convert Upper case to lower case

perl -i -pe 'tr/A-Z/a-z/' x

Monday, September 17, 2007

Shell scripting: case statement & numbers

Selecting a numeric range with the case statement.
case ${NUMBER} in
[0-9]|[0-6][0-9]|7[0-5])
echo "${NUMBER} <= 75"
;;
[7-8][0-9]|9[0-5])
echo "75 < ${NUMBER}<=95"
;;
[9][5-9]|100)
echo "95< ${NUMBER} <=100"
;;
esac

Thursday, May 17, 2007

Shell scripting: Testing empty variables

$V="xNotEmpty"

if [ x$V = "xNotEmpty" ]
then
       echo "\$V not empty"
else
       echo"\$V is empty"
fi

Or:

if [ "$V" = "" ]
then
      echo "\$V is empty
fi

Wednesday, May 16, 2007

awk

Using shell variables with awk.

1. awk "/$func/"'{print $0}' filename

2. awk -v pat=$func '/pat/{print $0}' filename

Using fields in search pattern

For a similar match:

1. awk '$1~/pattern/{print $0}' filename

For a precise match:

2.  awk -F":" -v user=$U '$1 == user {print $0}' filename

Tuesday, March 20, 2007

Perl command line.

Using multiple perl commands on the command line.

perl -n -e 'perl cmd;'\
-e 'perlcmd'

Invoke the split command

perl -n -a -e 'cmd'

-n
Places a loop around the command line.

Tuesday, September 12, 2006

My .vimrc file

syntax on
set viminfo^=h
set ruler
set nohls
set incsearch
set showmatch
set ts=4 sw=4
syntax enable
set hlsearch
"Use F4 to switch high lighting on/off
:noremap <F4> :set hlsearch! hlsearch<CR>

Friday, September 01, 2006

Shell Scripting: Special Variables

  • $? Exit status or return code of last command.
  • $# Number of arguments.
  • $@ Argument 1 thru n with Input Field Separator.
  • $* "$1" $2" ... $n.
  • $! Process id of last background process.
  • $$ Process id of shell running this script.
  • $- The current shell flags.

Thursday, August 31, 2006