Eikonal Blog

2012.06.25

More unix tools

Filed under: scripting, unix — Tags: , , , , , , — sandokan65 @ 13:32
  • Joye’s “moreutils” collection – http://joeyh.name/code/moreutils/. Contains following:

    • chronic: runs a command quietly unless it fails
    • combine: combine the lines in two files using boolean operations
    • ifdata: get network interface info without parsing ifconfig output
    • ifne: run a program if the standard input is not empty
    • isutf8: check if a file or standard input is utf-8
    • lckdo: execute a program with a lock held
    • mispipe: pipe two commands, returning the exit status of the first
    • parallel: run multiple jobs at once
    • pee: tee standard input to pipes
    • sponge: soak up standard input and write to a file
    • ts: timestamp standard input
    • vidir: edit a directory in your text editor
    • vipe: insert a text editor into a pipe
    • zrun: automatically uncompress arguments to command

  • num-utils – http://suso.suso.org/programs/num-utils/. Contains:

    • average: A program for calculating the average of numbers.
    • bound: Finds the boundary numbers (min and max) of input.
    • interval: Shows the numeric intervals between each number in a sequence.
    • normalize: Normalizes a set of numbers between 0 and 1 by default.
    • numgrep: Like normal grep, but for sets of numbers.
    • numprocess: Do mathmatical operations on numbers.
    • numsum: Add up all the numbers.
    • random: Generate a random number from a given expression.
    • range: Generate a set of numbers in a range expression.
    • round: Round each number according to it’s value.

  • Scylla and Charybdis, Tools – http://www.scylla-charybdis.com/tool.php. Contains:

    • checkrun: Program watchdog to terminate a program with starving output)
    • cmpfast: Fast compare two files binary)
    • count: Copy lines, shows progress)
    • dbm: A little tool to access gdbm files from shell.)
    • dirlist: Primitive directory lister, quicker than ls, find and echo *)
    • diskus: Disk geometry checking and repair tool)
    • getrealpath: Print realpath to stdout)
    • histogram: Count bytes in file)
    • kdmktone: Make the console beep)
    • keypressed: Nonblocking, nondestructible test for waiting data on TTYs, sockets and probably pipes)
    • killmem: Protect some memory against paging until you need free memory)
    • lockdir: Create a directory for locking purpose)
    • lockrun: Exclusively run something by placing a file lock)
    • md5chk: Create md5sums for easy shell usage)
    • minicron: This is a program which starts other programs after some time)
    • mvatom: Move files by atomic rename instead of copy.)
    • printansi: Like /bin/echo but ANSI-escapes the output)
    • printargs: Like “hello world” but dumps the argc array)
    • ptybuffer: daemonize interactive tty line driven programs with output history)
    • runningfor: Return true until the given time periode is reached.)
    • slowdown: Slowdown processes or pipes.)
    • socklinger: Execute quick hack shell scripts connected to a socket.)
    • sq: SQLITE3 query tool for shell usage)
    • timeout: Execute a command or pipe only for a given duration.)
    • timestart: Start a program N-M times in parallel by running it it each A-B seconds)
    • tinohtmlparse: Simple HTML parser to extract information from HTML files by shell)
    • tinoseq: An integer seq implementation)
    • udevraw: Dump udev events for bash usage)
    • unbuffered: Copy stdin to stdout and stderr, unbuffered)
    • watcher: A Python2.6 script to watch files, pipes or Unix domain sockets)

2011.02.15

Unix tricks

Filed under: unix — Tags: , , , , , , , , — sandokan65 @ 15:38

Sources:

System information

Check cpu info:

    cat /proc/cpuinfo

Usernames

To get the list of usernames with its user ID in formatted way:

     awk -F":" '{ print "username: " $1 "\t\tuid:" $3 }' /etc/passwd

Filenames

Find the particular string from the list of files in current directory:

    cd /etc
for i in $(find -type f); do grep -iH nfsnobody $i; done

Or

    grep -iH nfsnobody *

Counting words

Get the no of occurrences of particular word in file:

    awk '/ServerName/ {i=i+1} END {print i}' /etc/httpd/conf/httpd.conf
grep ServerName /etc/httpd/conf/httpd.conf

Semaphores

To delete resources of semaphore arrays from memory:

    ipcs -s | 
grep apache | 
perl -e 'while () { @a=split(/\s+/); print`ipcrm sem $a[1]`}'

Convering Unix timestamp to human readable format

All one needs here is neatly summarized by Anton Olson in his blog posting “BASH: Convert Unix Timestamp to a Date” – http://www.antonolsen.com/2006/04/06/bash-convert-unix-timestamp-to-a-date/:

  • 1) perl -e “require ‘ctime.pl’; print &ctime($EPOCH);” – where ctime is a Perl module (available where?)
  • 2) perl -e “print scalar(localtime($EPOCH))”
  • 3) echo $EPOCH|awk ‘{print strftime(“%c”,$1)}’
  • 4) # date -d @1000000042 on Linux (and Cygwin)
  • 5) $ date -d ’1970-01-01 sec’. It is important to use the GMT tag:
    • #date -d ’1970-01-01 1000000000 sec GMT’ with output: Sat Sep 8 20:46:40 CDT 2001
    • #date -d ’1970-01-01 1000000000 sec’ gives output: Sun Sep 9 02:46:40 CDT 2001
  • 6) $ date -r 1229519950 – on FreeBSD

The third method works fine if you have available only shell (e.g. working in Cygwin) and does not require Perl interpreter. For example:

    In: echo 101000070 | awk '{print strftime("%c",$1)}'
Out: Wed Mar 14 19:34:30 1973

Perl

To check whether perl module is installed correctly or not: if all is correct then output of this command nothing

    perl -e 'require Mail::SPF::Query'

To install CPAN module:

    cpan
cpan> install Mail::SPF::Query
CPAN: Storable loaded ok
Going to read /root/.cpan/Metadata
Database was generated on Thu, 24 Nov 2005 14:54:20 GMT
Mail::SPF::Query is up to date.

IP Addresses

To get the list of IP addresses in the server:

    ifconfig | 
grep -vw inet6 | 
grep -w inet | 
cut -d : -f 2 | 
cut -d \ -f 1

Find list of IP address along with eth device and network mask:

    ifconfig | 
cut -d " " -f1,12,16 | 
grep -A 1 eth | 
tr -d - | 
tr -s "\n" |sed -e :a -e N -e 's/\n/ /'

hard disk

Know the performance of your HardDisk: change the device address as per your servers configuration

    hdparm -Tt /dev/sda

Logging

Get the customized output of raw accesslog of httpd: Navigate the folder where your http access log reside, then execute following:

    tail -f access_log | 
awk '{if ($11 ~"\"-\"") print $1, $7, $12; else print $1, $10, $11, $12}'

Checking SSH failure log (in GNU/Linux Debian)

    /bin/cat /var/log/auth.log  |
grep sshd:auth |
grep failure |
awk '{print $1"-" $2 "-" $3 "-->" $12 "->" $14 "->" $15}'

Open connections

The details of the present http connections can be found by using:

    netstat -plan | 
grep ":80 " | 
awk {'print $5'} |
awk -F: {'print $1'}|
sort

cat /proc/net/ip_conntrack | 
grep "port=80" | 
wc -l

Number of connection from the particular IP addfess:

    netstat -ntu | awk '{print $5}'| cut -d: -f1 | sort | uniq -c | sort -nr | more

No of conections:

    netstat -alntp
/sbin/ldconfig /usr/local/lib - Update the system linker cache

Real Time Network Activity Examples:

    root# watch -d "netstat -nalp |grep -v DGRAM |grep -v STREAM |grep -v LISTEN"
root# watch "netstat -nalp"|grep ":TCP PORT Number"
root# watch "netstat -nalp"|grep ":22"

Port scanning

Port scanning using nmap:
You can customized it to get more informative output

    nmap -sS localhost -

instead host localhost, it could be IP address of another server which is in question

Bash loops

You can execute bash command a certain number of times by using something similar to the following:

    n=0;while test -$n -gt -10; do echo n=$n; n=$[$n+1]; done

that code will print “n=0”, “n=1”, and so on 10 times.

Directory content

Only get the listing of directories:

    ls -F $1 | 
grep \/ | 
sed -e 's/\/$/4/g'

Killing processes

Kill program one time click base keyword:

    for a in $( ps aux | 
grep  "some-thing-text" | 
awk '{ print $2 }'); do kill $a; done;

Kill crontab processes:

    for a in $( ps aux | grep  "/USR/SBIN/CRON" | awk '{ print $2 }'); do kill $a; done;

Environment variables

CDPATH env variable, if it is set for instance

    [bash ~]$ export CDPATH=/usr/local/apache

when you enter from any directory you’re in

    [bash /var/log]$ cd htdocs

it will take you to /usr/local/apache/htdocs.

Bash completion, command repetition and command history

  • !! at the command shell executes the last executed command
  • !m – executes the last command in history that starts with m for instance mail
  • !m:p – will print what was the last command that starts with m

vi

  • hitting twice z while holding a shift key in vi in command mode saves the edited file and quits vi

File dates

Updating atime on file foo (e.g. to 21:00 of Jan 1 1970): 

    touch -t 197001012100 foo

help, man, apropos

Redirect a rather big man page to a text file in easy readable format: 

    man manpage | col -b > manpage.txt

Conversion DOS to UNIX and vice versa

Getting rid off annoying ^M (CTRL-M) in DOS-like files: 

    tr -d "15"  /unixfile

In vi editor use search-and-replace command: 

    :%s/^M//g

To get ^M (in vi)you press CTRL+V+M

Related here: Unix system administration – https://eikonal.wordpress.com/2011/03/02/unix-system-administration/ | Cygwin stuff – https://eikonal.wordpress.com/2010/07/12/cygwin-stuff | MS Windows Registry transversal by Cygwin – https://eikonal.wordpress.com/2011/03/01/registry-transversal-by-cygwin

Related here: Scripting languages – https://eikonal.wordpress.com/2010/06/15/awk-sed/ | Unix tricks – https://eikonal.wordpress.com/2011/02/15/unix-tricks/ | SED tricks – https://eikonal.wordpress.com/2010/10/05/sed-tricks/ | Memory of things disappearing > nmap stuff > getports.awk – https://eikonal.wordpress.com/2010/06/23/memory-of-things-disappearing-nmap-stuff-getports-awk/ | AWK – https://eikonal.wordpress.com/2011/09/30/awk/

2010.07.12

Cygwin stuff

Filed under: cygwin, unix, windows — Tags: , , , , , — sandokan65 @ 15:38

Various Cygwin links

Portable Cygwin

Tricks

Setting effective HOME to your Windows “My Documents” folder.

  1. Start a Cygwin shell
  2. You should be located in your default directory which inside Cygwin looks like /home/thisuser, but from the Windows standpoint could be something like c:\cygwin\home\thisuser.
  3. Type ln -s /cygdrive/c/Documents and Settings/thisuser/My Documents/ ./docs
  4. Edit .profile and .bashrc to put at their ends following lines: 
      export HOME=./docs
cd $HOME
  5. that is it. Restart Cygwin. You should now be in the /home/thisuser/docs/ which is the same place as Windows’ c:\Documents and Settings\thisuser\My Documents/

Related: MS Windows Registry transversal by Cygwin – https://eikonal.wordpress.com/2011/03/01/registry-transversal-by-cygwin | Unix tricks – https://eikonal.wordpress.com/2011/02/15/unix-tricks/

Blog at WordPress.com.