Eikonal Blog

2012.06.25

More unix tools

Filed under: scripting, unix — Tags: , , , , , , — sandokan65 @ 13:32
  • Joye’s “moreutils” collection – http://joeyh.name/code/moreutils/. Contains following:

    • chronic: runs a command quietly unless it fails
    • combine: combine the lines in two files using boolean operations
    • ifdata: get network interface info without parsing ifconfig output
    • ifne: run a program if the standard input is not empty
    • isutf8: check if a file or standard input is utf-8
    • lckdo: execute a program with a lock held
    • mispipe: pipe two commands, returning the exit status of the first
    • parallel: run multiple jobs at once
    • pee: tee standard input to pipes
    • sponge: soak up standard input and write to a file
    • ts: timestamp standard input
    • vidir: edit a directory in your text editor
    • vipe: insert a text editor into a pipe
    • zrun: automatically uncompress arguments to command

  • num-utils – http://suso.suso.org/programs/num-utils/. Contains:

    • average: A program for calculating the average of numbers.
    • bound: Finds the boundary numbers (min and max) of input.
    • interval: Shows the numeric intervals between each number in a sequence.
    • normalize: Normalizes a set of numbers between 0 and 1 by default.
    • numgrep: Like normal grep, but for sets of numbers.
    • numprocess: Do mathmatical operations on numbers.
    • numsum: Add up all the numbers.
    • random: Generate a random number from a given expression.
    • range: Generate a set of numbers in a range expression.
    • round: Round each number according to it’s value.

  • Scylla and Charybdis, Tools – http://www.scylla-charybdis.com/tool.php. Contains:

    • checkrun: Program watchdog to terminate a program with starving output)
    • cmpfast: Fast compare two files binary)
    • count: Copy lines, shows progress)
    • dbm: A little tool to access gdbm files from shell.)
    • dirlist: Primitive directory lister, quicker than ls, find and echo *)
    • diskus: Disk geometry checking and repair tool)
    • getrealpath: Print realpath to stdout)
    • histogram: Count bytes in file)
    • kdmktone: Make the console beep)
    • keypressed: Nonblocking, nondestructible test for waiting data on TTYs, sockets and probably pipes)
    • killmem: Protect some memory against paging until you need free memory)
    • lockdir: Create a directory for locking purpose)
    • lockrun: Exclusively run something by placing a file lock)
    • md5chk: Create md5sums for easy shell usage)
    • minicron: This is a program which starts other programs after some time)
    • mvatom: Move files by atomic rename instead of copy.)
    • printansi: Like /bin/echo but ANSI-escapes the output)
    • printargs: Like “hello world” but dumps the argc array)
    • ptybuffer: daemonize interactive tty line driven programs with output history)
    • runningfor: Return true until the given time periode is reached.)
    • slowdown: Slowdown processes or pipes.)
    • socklinger: Execute quick hack shell scripts connected to a socket.)
    • sq: SQLITE3 query tool for shell usage)
    • timeout: Execute a command or pipe only for a given duration.)
    • timestart: Start a program N-M times in parallel by running it it each A-B seconds)
    • tinohtmlparse: Simple HTML parser to extract information from HTML files by shell)
    • tinoseq: An integer seq implementation)
    • udevraw: Dump udev events for bash usage)
    • unbuffered: Copy stdin to stdout and stderr, unbuffered)
    • watcher: A Python2.6 script to watch files, pipes or Unix domain sockets)

2011.09.30

awk

Filed under: scripting, unix — Tags: , — sandokan65 @ 11:03

Passing shell variables to AWK

Thing that works well for me: 

    awk '{print "'"$VARIABLE"'"}' 1 > 2

Related here: Scripting languages – https://eikonal.wordpress.com/2010/06/15/awk-sed/ | Unix tricks – https://eikonal.wordpress.com/2011/02/15/unix-tricks/ | SED tricks – https://eikonal.wordpress.com/2010/10/05/sed-tricks/ | Memory of things disappearing > nmap stuff > getports.awk – https://eikonal.wordpress.com/2010/06/23/memory-of-things-disappearing-nmap-stuff-getports-awk/

2011.07.08

Auditing Unix Security

Filed under: infosec, passwords, security hardening, security testing, unix, VA (Vulnerability Assessment) — Tags: , , , — sandokan65 @ 12:50

Misc

2011.05.03

2011.05.02

RPM (Red Hat Package Manager)

Filed under: unix — Tags: , , — sandokan65 @ 15:38

Misc

  • Interogating packages:
    • rpm -qa = lists installed packages
    • rpm -redhatrequires cron
    • rpm -redhatprovides cron
    • rpm -qpR cron *
  • Installing packages: -i = –install
    • rpm -i package1 package2 … = rpm –install package1 package2 …
    • rpm -i -nodeps package1 package2 … = install packages without their dependencies
    • rpm -U package1 package2 … = installs the latest versions of packages = updates the packages
  • Remove packages: -e
    • rpm -e package1 package2 …
    • rpm -e –force package1 … = forces uninstallation without regards for remaining dependency constraints
    • rpm -e –nodeps package1 …

Sites

Antimalware for Unix

Filed under: antimalware, antispyware, antivirus, infosec, unix — sandokan65 @ 14:33

2011.04.04

lastlog

Filed under: compliance, infosec, logging, Penetration Testing, security hardening, security testing, unix, VA (Vulnerability Assessment) — Tags: , — sandokan65 @ 15:48

Unix pages (at this blog)

Filed under: unix — Tags: — sandokan65 @ 15:45

2011.03.11

PAM (Pluggable Authentication Modules)

Filed under: infosec, unix — Tags: , , — sandokan65 @ 16:14

Articles

Roles of PAM files

  • /etc/pam.conf – all-in-one configuration file for early versions of PAM. It may still be used in some modern versions.
  • /etc/pam.d/ – directory containing configurations files for each of separately configured program
  • /etc/pam.d/other – the default config file regulating all files that do not have their own separate PAM config file
  • /etc/pam.d/login
  • /etc/pam.d/system-auth
  • /etc/pam.d/sshd
  • /etc/pam.d/su
  • /etc/pam.d/gdm – the GNOME Display Manager PAM file.
    • Example (from http://ubuntuforums.org/showthread.php?t=1506759):
      #%PAM-1.0
auth    requisite       pam_nologin.so
auth    required        pam_env.so readenv=1
auth    required        pam_env.so readenv=1 envfile=/etc/default/locale
auth    sufficient      pam_succeed_if.so user ingroup nopasswdlogin
@include common-auth
auth    optional        pam_gnome_keyring.so
@include common-account
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close
session required        pam_limits.so
@include common-session
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so open
session optional        pam_gnome_keyring.so auto_start
@include common-password

Syntax of config files

Each line has format: 

    module-type   control-flag   module-path   arguments

PAM modules

  • pam_deny.so module –
  • pam_permit.so module –
  • pam_warn.so module – used to interface to syslog

2011.03.02

Unix system administration

Filed under: unix — Tags: , , — sandokan65 @ 14:20

AIX

Specific tasks:

Related here: Unix tricks – https://eikonal.wordpress.com/2011/02/15/unix-tricks/

2011.03.01

MS Windows Registry transversal by Cygwin

Filed under: cygwin, unix, windows — Tags: , , , — sandokan65 @ 12:08

This may be an old thing, but it is new to me. Today I accidentally discovered that from inside Cygwin (i.e.e the Bash shell) one can transverse the Windows Registry. For example, one can list the content of HKEY_CLASSES_ROOT hive by simply typing:

    ls -la /proc/registry/HKEY_CLASSES_ROOT/

with partial output:

    total 0
dr-xr-x---     5 Administrators SYSTEM      0 Apr 26  2010 *
dr-xr-x---  7342 Administrators SYSTEM      0 Feb 18 10:32 .
dr-xr-xr-x     8 thisuser       mkgroup-l-d 0 Mar  1 12:02 ..
dr-xr-x---     3 Administrators SYSTEM      0 Feb 16  2010 .123
dr-xr-x---     2 Administrators SYSTEM      0 Feb  3  2009 .323
dr-xr-x---     3 Administrators SYSTEM      0 Feb  3  2009 .386
dr-xr-x---     2 Administrators SYSTEM      0 Nov 17  2009 .3g2
dr-xr-x---     2 Administrators SYSTEM      0 Nov 17  2009 .3gp
dr-x------     2 thisuser       ????????    0 Jun  7  2010 .5vw
dr-x------     4 thisuser       ????????    0 Nov 17  2009 .A1wish
dr-x------     2 ????????       None        0 Aug  5  2009 .ARC
dr-x------     2 ????????       None        0 Aug  5  2009 .ARJ
dr-xr-x---     3 Administrators SYSTEM      0 Feb  4  2009 .AddIn
dr-x------     3 ????????       ????????    0 Aug  7  2009 .AudioCD
dr-x------     2 ????????       None        0 Aug  5  2009 .B64
dr-x------     2 ????????       None        0 Aug  5  2009 .BHX
dr-x------     2 ????????       ????????    0 Aug  7  2009 .CTT
dr-xr-x---     3 Administrators SYSTEM      0 Feb  3  2009 .DeskLink
dr-x------     3 ????????       ????????    0 Aug  7  2009 .Folder
dr-xr-x---     2 Administrators SYSTEM      0 Feb  3  2009 .JSE
dr-x------     2 ????????       None        0 Aug  5  2009 .LZH
dr-xr-x---     3 Administrators SYSTEM      0 Feb  3  2009 .MAPIMail
dr-x------     2 ????????       None        0 Aug  5  2009 .MIM
...

Such feature is available in Microsoft’s PowerShell for several years now, but I did not know that this exists in the Cygwin’s Bash. When was this feature incorporated in the Cygwin? Was it inspired by PowerShell?

Also, the same feature works for /proc/registry32/ and /proc/registry64/

Related: Cygwin stuff – https://eikonal.wordpress.com/2010/07/12/cygwin-stuff//a> | MS Windows Registry transversal by Cygwin – https://eikonal.wordpress.com/2011/03/01/registry-transversal-by-cygwin | Unix tricks – https://eikonal.wordpress.com/2011/02/15/unix-tricks/

2011.02.15

Unix tricks

Filed under: unix — Tags: , , , , , , , , — sandokan65 @ 15:38

Sources:

System information

Check cpu info:

    cat /proc/cpuinfo

Usernames

To get the list of usernames with its user ID in formatted way:

     awk -F":" '{ print "username: " $1 "\t\tuid:" $3 }' /etc/passwd

Filenames

Find the particular string from the list of files in current directory:

    cd /etc
for i in $(find -type f); do grep -iH nfsnobody $i; done

Or

    grep -iH nfsnobody *

Counting words

Get the no of occurrences of particular word in file:

    awk '/ServerName/ {i=i+1} END {print i}' /etc/httpd/conf/httpd.conf
grep ServerName /etc/httpd/conf/httpd.conf

Semaphores

To delete resources of semaphore arrays from memory:

    ipcs -s | 
grep apache | 
perl -e 'while () { @a=split(/\s+/); print`ipcrm sem $a[1]`}'

Convering Unix timestamp to human readable format

All one needs here is neatly summarized by Anton Olson in his blog posting “BASH: Convert Unix Timestamp to a Date” – http://www.antonolsen.com/2006/04/06/bash-convert-unix-timestamp-to-a-date/:

  • 1) perl -e “require ‘ctime.pl’; print &ctime($EPOCH);” – where ctime is a Perl module (available where?)
  • 2) perl -e “print scalar(localtime($EPOCH))”
  • 3) echo $EPOCH|awk ‘{print strftime(“%c”,$1)}’
  • 4) # date -d @1000000042 on Linux (and Cygwin)
  • 5) $ date -d ’1970-01-01 sec’. It is important to use the GMT tag:
    • #date -d ’1970-01-01 1000000000 sec GMT’ with output: Sat Sep 8 20:46:40 CDT 2001
    • #date -d ’1970-01-01 1000000000 sec’ gives output: Sun Sep 9 02:46:40 CDT 2001
  • 6) $ date -r 1229519950 – on FreeBSD

The third method works fine if you have available only shell (e.g. working in Cygwin) and does not require Perl interpreter. For example:

    In: echo 101000070 | awk '{print strftime("%c",$1)}'
Out: Wed Mar 14 19:34:30 1973

Perl

To check whether perl module is installed correctly or not: if all is correct then output of this command nothing

    perl -e 'require Mail::SPF::Query'

To install CPAN module:

    cpan
cpan> install Mail::SPF::Query
CPAN: Storable loaded ok
Going to read /root/.cpan/Metadata
Database was generated on Thu, 24 Nov 2005 14:54:20 GMT
Mail::SPF::Query is up to date.

IP Addresses

To get the list of IP addresses in the server:

    ifconfig | 
grep -vw inet6 | 
grep -w inet | 
cut -d : -f 2 | 
cut -d \ -f 1

Find list of IP address along with eth device and network mask:

    ifconfig | 
cut -d " " -f1,12,16 | 
grep -A 1 eth | 
tr -d - | 
tr -s "\n" |sed -e :a -e N -e 's/\n/ /'

hard disk

Know the performance of your HardDisk: change the device address as per your servers configuration

    hdparm -Tt /dev/sda

Logging

Get the customized output of raw accesslog of httpd: Navigate the folder where your http access log reside, then execute following:

    tail -f access_log | 
awk '{if ($11 ~"\"-\"") print $1, $7, $12; else print $1, $10, $11, $12}'

Checking SSH failure log (in GNU/Linux Debian)

    /bin/cat /var/log/auth.log  |
grep sshd:auth |
grep failure |
awk '{print $1"-" $2 "-" $3 "-->" $12 "->" $14 "->" $15}'

Open connections

The details of the present http connections can be found by using:

    netstat -plan | 
grep ":80 " | 
awk {'print $5'} |
awk -F: {'print $1'}|
sort

cat /proc/net/ip_conntrack | 
grep "port=80" | 
wc -l

Number of connection from the particular IP addfess:

    netstat -ntu | awk '{print $5}'| cut -d: -f1 | sort | uniq -c | sort -nr | more

No of conections:

    netstat -alntp
/sbin/ldconfig /usr/local/lib - Update the system linker cache

Real Time Network Activity Examples:

    root# watch -d "netstat -nalp |grep -v DGRAM |grep -v STREAM |grep -v LISTEN"
root# watch "netstat -nalp"|grep ":TCP PORT Number"
root# watch "netstat -nalp"|grep ":22"

Port scanning

Port scanning using nmap:
You can customized it to get more informative output

    nmap -sS localhost -

instead host localhost, it could be IP address of another server which is in question

Bash loops

You can execute bash command a certain number of times by using something similar to the following:

    n=0;while test -$n -gt -10; do echo n=$n; n=$[$n+1]; done

that code will print “n=0”, “n=1”, and so on 10 times.

Directory content

Only get the listing of directories:

    ls -F $1 | 
grep \/ | 
sed -e 's/\/$/4/g'

Killing processes

Kill program one time click base keyword:

    for a in $( ps aux | 
grep  "some-thing-text" | 
awk '{ print $2 }'); do kill $a; done;

Kill crontab processes:

    for a in $( ps aux | grep  "/USR/SBIN/CRON" | awk '{ print $2 }'); do kill $a; done;

Environment variables

CDPATH env variable, if it is set for instance

    [bash ~]$ export CDPATH=/usr/local/apache

when you enter from any directory you’re in

    [bash /var/log]$ cd htdocs

it will take you to /usr/local/apache/htdocs.

Bash completion, command repetition and command history

  • !! at the command shell executes the last executed command
  • !m – executes the last command in history that starts with m for instance mail
  • !m:p – will print what was the last command that starts with m

vi

  • hitting twice z while holding a shift key in vi in command mode saves the edited file and quits vi

File dates

Updating atime on file foo (e.g. to 21:00 of Jan 1 1970): 

    touch -t 197001012100 foo

help, man, apropos

Redirect a rather big man page to a text file in easy readable format: 

    man manpage | col -b > manpage.txt

Conversion DOS to UNIX and vice versa

Getting rid off annoying ^M (CTRL-M) in DOS-like files: 

    tr -d "15"  /unixfile

In vi editor use search-and-replace command: 

    :%s/^M//g

To get ^M (in vi)you press CTRL+V+M

Related here: Unix system administration – https://eikonal.wordpress.com/2011/03/02/unix-system-administration/ | Cygwin stuff – https://eikonal.wordpress.com/2010/07/12/cygwin-stuff | MS Windows Registry transversal by Cygwin – https://eikonal.wordpress.com/2011/03/01/registry-transversal-by-cygwin

Related here: Scripting languages – https://eikonal.wordpress.com/2010/06/15/awk-sed/ | Unix tricks – https://eikonal.wordpress.com/2011/02/15/unix-tricks/ | SED tricks – https://eikonal.wordpress.com/2010/10/05/sed-tricks/ | Memory of things disappearing > nmap stuff > getports.awk – https://eikonal.wordpress.com/2010/06/23/memory-of-things-disappearing-nmap-stuff-getports-awk/ | AWK – https://eikonal.wordpress.com/2011/09/30/awk/

2011.02.10

SUDO

Filed under: infosec, security hardening, unix — Tags: , — sandokan65 @ 11:38

2011.01.13

Jaibreaking

Filed under: infosec, unix — Tags: , — sandokan65 @ 12:07

News:

2010.12.16

SSH, OpenSSH

Filed under: infosec, security hardening, unix — Tags: , , , , , , , , , , , , , , , , , , , , — sandokan65 @ 15:57

SSHFS (SSH FileSystem)

Related:

Authentication via public keys

SFTP

FTPS vs SFTP

Using SCP

  • Example syntax for Secure Copy (scp) – http://www.hypexr.org/linux_scp_help.php
    • Copy the file “foobar.txt” from a remote host to the local host: $ scp your_username@remotehost.edu:foobar.txt /some/local/directory
    • Copy the file “foobar.txt” from the local host to a remote host: $ scp foobar.txt your_username@remotehost.edu:/some/remote/directory
    • Copy the directory “foo” from the local host to a remote host’s directory “bar”: $ scp -r foo your_username@remotehost.edu:/some/remote/directory/bar
    • Copy the file “foobar.txt” from remote host “rh1.edu” to remote host “rh2.edu”: $ scp your_username@rh1.edu:/some/remote/directory/foobar.txt \
      your_username@rh2.edu:/some/remote/directory/
    • Copying the files “foo.txt” and “bar.txt” from the local host to your home directory on the remote host: $ scp foo.txt bar.txt your_username@remotehost.edu:~
    • Copy multiple files from the remote host to your current directory on the local host: $ scp your_username@remotehost.edu:/some/remote/directory/\{a,b,c\}. Also:: $ scp your_username@remotehost.edu:~/\{foo.txt,bar.txt\} .

Use of Expect with SSH suite applications

Password-less SFTP

Establish the SFTP connection to the system AAAA where the user account BBBB has password CCCC, and go to the directory DDDD, all without being prompted to enter the password:

    sftpToAAAA.expect 
    #!/bin/expect
# sftpToAAAA.expect

spawn sftp BBBB@AAAA
expect "password" {
   sleep 1
   send "CCCC\n"
}
send "cd DDDD\n"
interact

All normal warning on the danger of hard-wiring the password into scripts are in place here.

Password-less SCP

Use the SCP to upload connect system AAAA with user account BBBB (that has password CCCC), and upload the file EEEE to the directory DDDD, all without being prompted to enter the password:

    UploadEEEEtoAAAA.expect 
    #!/bin/expect
spawn scp EEEE BBBB@AAAA:DDDD/EEEE
expect "password" {
  send "CCCC\n"
}

More

2010.12.02

Unix “script” command

Filed under: unix — Tags: , , — sandokan65 @ 12:16

Implementation of the same functionality in Expect

Cygwin does not come with the “script” commands, so following implementation in Expect can do the same thing:

    script.expect 
    #!/bin/expect

spawn /bin/bash

if {[lindex $argv 0]==""} {
        log_file SESSION.txt
   } else {
        log_file [lindex $argv 0]
   }

interact

2010.11.15

vi editor

Filed under: unix — Tags: — sandokan65 @ 14:45

2010.08.07

Samba/SMB/CIFS security links

Filed under: infosec, unix, windows — Tags: , , — sandokan65 @ 13:04

2010.07.13

Command line based text replace

Filed under: perl, transformers, unix — Tags: , , , , , , , , , , — sandokan65 @ 13:36

sed

  • sed 's/Mark Monre/Marc Monroe/' 1.txt > 2.txt
  • find ./* -type f -exec sed -i 's///g' {} \;

The “replace” command

  • Syntax: 
    replace OLD-STRING NEW-STRING OUTPUT-FILE
  • Example: 
    $ replace UNIX Linux  newfile
  • Example: 
    $ cat /etc/passwd | replace : '|'
  • Partial support for regular expressions: \^ – matches start of line, and $ matches end of line.
  • Example: replace all IP address 192.168.1.2 start of line: 
    $ replace \^192.168.1.2 192.168.5.10  newfile
  • a bash script, ‘fixer.sh’
    #!/bin/bash
replace CHANGEFROM CHANGETO $1.tmp
rm $1
mv $1.tmp $1

    now run this command line: 

    $ grep CHANGEFROM |cut -d':' -f1 |xargs -n 1 fixer.sh

    the results is that all files in the directory (or whatever you grep for) will be changed automagically.
    just make sure the grep doesn’t include the fixer script itself, or it will die half-way through changing when execute permissions are reset!

Perl

Sources:

Related: Regular expressions – https://eikonal.wordpress.com/2010/04/02/regular-expressions/ | Perl online – https://eikonal.wordpress.com/2010/02/15/perl-online/

2010.07.12

Cygwin stuff

Filed under: cygwin, unix, windows — Tags: , , , , , — sandokan65 @ 15:38

Various Cygwin links

Portable Cygwin

Tricks

Setting effective HOME to your Windows “My Documents” folder.

  1. Start a Cygwin shell
  2. You should be located in your default directory which inside Cygwin looks like /home/thisuser, but from the Windows standpoint could be something like c:\cygwin\home\thisuser.
  3. Type ln -s /cygdrive/c/Documents and Settings/thisuser/My Documents/ ./docs
  4. Edit .profile and .bashrc to put at their ends following lines: 
      export HOME=./docs
cd $HOME
  5. that is it. Restart Cygwin. You should now be in the /home/thisuser/docs/ which is the same place as Windows’ c:\Documents and Settings\thisuser\My Documents/

Related: MS Windows Registry transversal by Cygwin – https://eikonal.wordpress.com/2011/03/01/registry-transversal-by-cygwin | Unix tricks – https://eikonal.wordpress.com/2011/02/15/unix-tricks/

Older Posts »

Blog at WordPress.com.