Eikonal Blog

2012.06.25

More unix tools

Filed under: scripting, unix — Tags: , , , , , , — sandokan65 @ 13:32
  • Joye’s “moreutils” collection – http://joeyh.name/code/moreutils/. Contains following:

    • chronic: runs a command quietly unless it fails
    • combine: combine the lines in two files using boolean operations
    • ifdata: get network interface info without parsing ifconfig output
    • ifne: run a program if the standard input is not empty
    • isutf8: check if a file or standard input is utf-8
    • lckdo: execute a program with a lock held
    • mispipe: pipe two commands, returning the exit status of the first
    • parallel: run multiple jobs at once
    • pee: tee standard input to pipes
    • sponge: soak up standard input and write to a file
    • ts: timestamp standard input
    • vidir: edit a directory in your text editor
    • vipe: insert a text editor into a pipe
    • zrun: automatically uncompress arguments to command
  • num-utils – http://suso.suso.org/programs/num-utils/. Contains:

    • average: A program for calculating the average of numbers.
    • bound: Finds the boundary numbers (min and max) of input.
    • interval: Shows the numeric intervals between each number in a sequence.
    • normalize: Normalizes a set of numbers between 0 and 1 by default.
    • numgrep: Like normal grep, but for sets of numbers.
    • numprocess: Do mathmatical operations on numbers.
    • numsum: Add up all the numbers.
    • random: Generate a random number from a given expression.
    • range: Generate a set of numbers in a range expression.
    • round: Round each number according to it’s value.
  • Scylla and Charybdis, Tools – http://www.scylla-charybdis.com/tool.php. Contains:

    • checkrun: Program watchdog to terminate a program with starving output)
    • cmpfast: Fast compare two files binary)
    • count: Copy lines, shows progress)
    • dbm: A little tool to access gdbm files from shell.)
    • dirlist: Primitive directory lister, quicker than ls, find and echo *)
    • diskus: Disk geometry checking and repair tool)
    • getrealpath: Print realpath to stdout)
    • histogram: Count bytes in file)
    • kdmktone: Make the console beep)
    • keypressed: Nonblocking, nondestructible test for waiting data on TTYs, sockets and probably pipes)
    • killmem: Protect some memory against paging until you need free memory)
    • lockdir: Create a directory for locking purpose)
    • lockrun: Exclusively run something by placing a file lock)
    • md5chk: Create md5sums for easy shell usage)
    • minicron: This is a program which starts other programs after some time)
    • mvatom: Move files by atomic rename instead of copy.)
    • printansi: Like /bin/echo but ANSI-escapes the output)
    • printargs: Like “hello world” but dumps the argc array)
    • ptybuffer: daemonize interactive tty line driven programs with output history)
    • runningfor: Return true until the given time periode is reached.)
    • slowdown: Slowdown processes or pipes.)
    • socklinger: Execute quick hack shell scripts connected to a socket.)
    • sq: SQLITE3 query tool for shell usage)
    • timeout: Execute a command or pipe only for a given duration.)
    • timestart: Start a program N-M times in parallel by running it it each A-B seconds)
    • tinohtmlparse: Simple HTML parser to extract information from HTML files by shell)
    • tinoseq: An integer seq implementation)
    • udevraw: Dump udev events for bash usage)
    • unbuffered: Copy stdin to stdout and stderr, unbuffered)
    • watcher: A Python2.6 script to watch files, pipes or Unix domain sockets)

2011.02.15

Unix tricks

Filed under: unix — Tags: , , , , , , , , — sandokan65 @ 15:38

Sources:

System information

Check cpu info:

    cat /proc/cpuinfo
    

Usernames

To get the list of usernames with its user ID in formatted way:

     awk -F":" '{ print "username: " $1 "\t\tuid:" $3 }' /etc/passwd
    

Filenames

Find the particular string from the list of files in current directory:

    cd /etc
    for i in $(find -type f); do grep -iH nfsnobody $i; done
    

Or

    grep -iH nfsnobody *
    

Counting words

Get the no of occurrences of particular word in file:

    awk '/ServerName/ {i=i+1} END {print i}' /etc/httpd/conf/httpd.conf
    grep ServerName /etc/httpd/conf/httpd.conf
    

Semaphores

To delete resources of semaphore arrays from memory:

    ipcs -s | 
    grep apache | 
    perl -e 'while () { @a=split(/\s+/); print`ipcrm sem $a[1]`}'
    

Convering Unix timestamp to human readable format

All one needs here is neatly summarized by Anton Olson in his blog posting “BASH: Convert Unix Timestamp to a Date” – http://www.antonolsen.com/2006/04/06/bash-convert-unix-timestamp-to-a-date/:

  • 1) perl -e “require ‘ctime.pl’; print &ctime($EPOCH);” – where ctime is a Perl module (available where?)
  • 2) perl -e “print scalar(localtime($EPOCH))”
  • 3) echo $EPOCH|awk ‘{print strftime(“%c”,$1)}’
  • 4) # date -d @1000000042 on Linux (and Cygwin)
  • 5) $ date -d ’1970-01-01 sec’. It is important to use the GMT tag:
    • #date -d ’1970-01-01 1000000000 sec GMT’ with output: Sat Sep 8 20:46:40 CDT 2001
    • #date -d ’1970-01-01 1000000000 sec’ gives output: Sun Sep 9 02:46:40 CDT 2001
  • 6) $ date -r 1229519950 – on FreeBSD

The third method works fine if you have available only shell (e.g. working in Cygwin) and does not require Perl interpreter. For example:

    In: echo 101000070 | awk '{print strftime("%c",$1)}'
    Out: Wed Mar 14 19:34:30 1973
    

Perl

To check whether perl module is installed correctly or not: if all is correct then output of this command nothing

    perl -e 'require Mail::SPF::Query'
    

To install CPAN module:

    cpan
    cpan> install Mail::SPF::Query
    CPAN: Storable loaded ok
    Going to read /root/.cpan/Metadata
    Database was generated on Thu, 24 Nov 2005 14:54:20 GMT
    Mail::SPF::Query is up to date.
    

IP Addresses

To get the list of IP addresses in the server:

    ifconfig | 
    grep -vw inet6 | 
    grep -w inet | 
    cut -d : -f 2 | 
    cut -d \ -f 1
    

Find list of IP address along with eth device and network mask:

    ifconfig | 
    cut -d " " -f1,12,16 | 
    grep -A 1 eth | 
    tr -d - | 
    tr -s "\n" |sed -e :a -e N -e 's/\n/ /'
    

hard disk

Know the performance of your HardDisk: change the device address as per your servers configuration

    hdparm -Tt /dev/sda
    

Logging

Get the customized output of raw accesslog of httpd: Navigate the folder where your http access log reside, then execute following:

    tail -f access_log | 
    awk '{if ($11 ~"\"-\"") print $1, $7, $12; else print $1, $10, $11, $12}'
    

Checking SSH failure log (in GNU/Linux Debian)

    /bin/cat /var/log/auth.log  |
    grep sshd:auth |
    grep failure |
    awk '{print $1"-" $2 "-" $3 "-->" $12 "->" $14 "->" $15}'
    

Open connections

The details of the present http connections can be found by using:

    netstat -plan | 
    grep ":80 " | 
    awk {'print $5'} |
    awk -F: {'print $1'}|
    sort
    
    cat /proc/net/ip_conntrack | 
    grep "port=80" | 
    wc -l
    

Number of connection from the particular IP addfess:

    netstat -ntu | awk '{print $5}'| cut -d: -f1 | sort | uniq -c | sort -nr | more
    

No of conections:

    netstat -alntp
    /sbin/ldconfig /usr/local/lib - Update the system linker cache
    

Real Time Network Activity Examples:

    root# watch -d "netstat -nalp |grep -v DGRAM |grep -v STREAM |grep -v LISTEN"
    root# watch "netstat -nalp"|grep ":TCP PORT Number"
    root# watch "netstat -nalp"|grep ":22"
    

Port scanning

Port scanning using nmap:
You can customized it to get more informative output

    nmap -sS localhost -
    

instead host localhost, it could be IP address of another server which is in question

Bash loops

You can execute bash command a certain number of times by using something similar to the following:

    n=0;while test -$n -gt -10; do echo n=$n; n=$[$n+1]; done
    

that code will print “n=0”, “n=1”, and so on 10 times.

Directory content

Only get the listing of directories:

    ls -F $1 | 
    grep \/ | 
    sed -e 's/\/$/4/g'
    

Killing processes

Kill program one time click base keyword:

    for a in $( ps aux | 
    grep  "some-thing-text" | 
    awk '{ print $2 }'); do kill $a; done;
    

Kill crontab processes:

    for a in $( ps aux | grep  "/USR/SBIN/CRON" | awk '{ print $2 }'); do kill $a; done;
    

Environment variables

CDPATH env variable, if it is set for instance

    [bash ~]$ export CDPATH=/usr/local/apache
    

when you enter from any directory you’re in

    [bash /var/log]$ cd htdocs
    

it will take you to /usr/local/apache/htdocs.

Bash completion, command repetition and command history

  • !! at the command shell executes the last executed command
  • !m – executes the last command in history that starts with m for instance mail
  • !m:p – will print what was the last command that starts with m

vi

  • hitting twice z while holding a shift key in vi in command mode saves the edited file and quits vi

File dates

Updating atime on file foo (e.g. to 21:00 of Jan 1 1970):

    touch -t 197001012100 foo

help, man, apropos

Redirect a rather big man page to a text file in easy readable format:

    man manpage | col -b > manpage.txt

Conversion DOS to UNIX and vice versa

Getting rid off annoying ^M (CTRL-M) in DOS-like files:

    tr -d "15"  /unixfile

In vi editor use search-and-replace command:

    :%s/^M//g

To get ^M (in vi)you press CTRL+V+M


Related here: Unix system administration – https://eikonal.wordpress.com/2011/03/02/unix-system-administration/ | Cygwin stuff – https://eikonal.wordpress.com/2010/07/12/cygwin-stuff | MS Windows Registry transversal by Cygwin – https://eikonal.wordpress.com/2011/03/01/registry-transversal-by-cygwin

Related here: Scripting languages – https://eikonal.wordpress.com/2010/06/15/awk-sed/ | Unix tricks – https://eikonal.wordpress.com/2011/02/15/unix-tricks/ | SED tricks – https://eikonal.wordpress.com/2010/10/05/sed-tricks/ | Memory of things disappearing > nmap stuff > getports.awk – https://eikonal.wordpress.com/2010/06/23/memory-of-things-disappearing-nmap-stuff-getports-awk/ | AWK – https://eikonal.wordpress.com/2011/09/30/awk/

2010.07.13

Command line based text replace

sed

  • sed 's/Mark Monre/Marc Monroe/' 1.txt > 2.txt
  • find ./* -type f -exec sed -i 's///g' {} \;

The “replace” command

  • Syntax:
    replace OLD-STRING NEW-STRING OUTPUT-FILE
  • Example:
    $ replace UNIX Linux  newfile
  • Example:
    $ cat /etc/passwd | replace : '|'
  • Partial support for regular expressions: \^ – matches start of line, and $ matches end of line.
  • Example: replace all IP address 192.168.1.2 start of line:
    $ replace \^192.168.1.2 192.168.5.10  newfile
  • a bash script, ‘fixer.sh’
    #!/bin/bash
    replace CHANGEFROM CHANGETO $1.tmp
    rm $1
    mv $1.tmp $1
    

    now run this command line:

    $ grep CHANGEFROM |cut -d':' -f1 |xargs -n 1 fixer.sh

    the results is that all files in the directory (or whatever you grep for) will be changed automagically.
    just make sure the grep doesn’t include the fixer script itself, or it will die half-way through changing when execute permissions are reset!


Perl


Sources:


Related: Regular expressions – https://eikonal.wordpress.com/2010/04/02/regular-expressions/ | Perl online – https://eikonal.wordpress.com/2010/02/15/perl-online/

2010.04.02

Regular expressions

Sites

Tools

Standalone tools:

Online testers:

Books

Tidbits

Sources: The above links.

  • [abc] – A single character: a, b or c
  • [^abc] – Any single character but a, b, or c
  • [a-z] – Any single character in the range a-z
  • [a-zA-Z] – Any single character in the range a-z or A-Z
  • ^ – Start of line
  • $ – End of line
  • \A – Start of string
  • \z – End of string
  • . – Any single character
  • \s – Any whitespace character
  • \S – Any non-whitespace character
  • \d – Any digit
  • \D – Any non-digit
  • \w – Any word character (letter, number, underscore)
  • \W – Any non-word character
  • \b – Any word boundary character
  • (…) – Capture everything enclosed
  • (a|b) – a or b
  • a? – Zero or one of a
  • a* – Zero or more of a
  • a+ – One or more of a
  • a{3} – Exactly 3 of a
  • a{3,} – 3 or more of a
  • a{3,6} – Between 3 and 6 of a
  • ^\s[ \t]*$ – Match a blank line
  • \d{2}-\d{5} – Validate an ID number consisting of 2 digits, a hyphen, and another 5 digits

Special common strings:

  • Personal Name: ^[\w\.\’]{2,}([\s][\w\.\’]{2,})+$
  • Username: ^[\w\d\_\.]{4,}$
  • Password at least 6 symbols: ^.{6,}$
  • Password or empty input: ^.{6,}$|^$
  • email: ^[\_]*([a-z0-9]+(\.|\_*)?)+@([a-z][a-z0-9\-]+(\.|\-*\.))+[a-z]{2,6}$
  • Email address: \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b[A-z0-9_.%+-]+@[A-z0-9_.%+-]+\.[A-z]{2,4}
  • US phone: \W?\d{3}\W?\d{3}\W?\d{4}
  • US Phone number: ^\+?[\d\s]{3,}$
  • US Phone with code: ^\+?[\d\s]+\(?[\d\s]{10,}$
  • URL: \W?\d{3}\W?\d{3}\W?\d{4}\b\w+://(\w|-|\.|/)+(/|\b)
  • US Social Security Number (SSN): \d{3}-\d{2}-\d{4}
  • US ZIP: \d{5}(-\d{4})?
  • IP (v4) address: \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b
  • IP (v4) address: \b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b
  • IP (v4) address: ^(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5]){3}$
  • IP (v4) address: \b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b
  • IP (v4) address: \b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b
  • IP (v6) address:
  • MAC address: ^([0-9a-fA-F][0-9a-fA-F]:){5}([0-9a-fA-F][0-9a-fA-F])$
  • Positive Integers: ^\d+$
  • Negative Integers: ^-\d+$
  • Integer: ^-{0,1}\d+$
  • Positive Number: ^\d*\.{0,1}\d+$
  • Negative Number: ^-\d*\.{0,1}\d+$
  • Positive Number or Negative Number: ^-{0,1}\d*\.{0,1}\d+$
  • Floating point number: [-+]?([0-9]*\.[0-9]+|[0-9]+)
  • Floating point number: [-+]?(?:\b[0-9]+(?:\.[0-9]*)?|\.[0-9]+\b)(?:[eE][-+]?[0-9]+\b)?
  • Roman number: ^(?i:(?=[MDCLXVI])((M{0,3})((C[DM])|(D?C{0,3}))?((X[LC])|(L?XX{0,2})|L)?((I[VX])|(V?(II{0,2}))|V)?))$
  • Domain Name: ^([a-zA-Z0-9]([a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,6}$
  • Domain Name: ^([a-z][a-z0-9\-]+(\.|\-*\.))+[a-z]{2,6}$
  • Windows File Name: (?i)^(?!^(PRN|AUX|CLOCK\$|NUL|CON|COM\d|LPT\d|\..*)(\..+)?$)[^\\\./:\*\?\”\|][^\\/:\*\?\”\|]{0,254}$
  • Date in format yyyy-MM-dd: (19|20)\d\d([- /.])(0[1-9]|1[012])\2(0[1-9]|[12][0-9]|3[01])
  • Date (dd mm yyyy, d/m/yyyy, etc.): ^([1-9]|0[1-9]|[12][0-9]|3[01])\D([1-9]|0[1-9]|1[012])\D(19[0-9][0-9]|20[0-9][0-9])$
  • Year 1900-2099: ^(19|20)[\d]{2,2}$

Related (here at this blog):
Command line based text replace – https://eikonal.wordpress.com/2010/07/13/command-line-based-text-replace/ |
Perl online – https://eikonal.wordpress.com/2010/02/15/perl-online/

Blog at WordPress.com.