- Joye’s “moreutils” collection – http://joeyh.name/code/moreutils/. Contains following:
- chronic: runs a command quietly unless it fails
- combine: combine the lines in two files using boolean operations
- ifdata: get network interface info without parsing ifconfig output
- ifne: run a program if the standard input is not empty
- isutf8: check if a file or standard input is utf-8
- lckdo: execute a program with a lock held
- mispipe: pipe two commands, returning the exit status of the first
- parallel: run multiple jobs at once
- pee: tee standard input to pipes
- sponge: soak up standard input and write to a file
- ts: timestamp standard input
- vidir: edit a directory in your text editor
- vipe: insert a text editor into a pipe
- zrun: automatically uncompress arguments to command
- num-utils – http://suso.suso.org/programs/num-utils/. Contains:
- average: A program for calculating the average of numbers.
- bound: Finds the boundary numbers (min and max) of input.
- interval: Shows the numeric intervals between each number in a sequence.
- normalize: Normalizes a set of numbers between 0 and 1 by default.
- numgrep: Like normal grep, but for sets of numbers.
- numprocess: Do mathmatical operations on numbers.
- numsum: Add up all the numbers.
- random: Generate a random number from a given expression.
- range: Generate a set of numbers in a range expression.
- round: Round each number according to it’s value.
- Scylla and Charybdis, Tools – http://www.scylla-charybdis.com/tool.php. Contains:
- checkrun: Program watchdog to terminate a program with starving output)
- cmpfast: Fast compare two files binary)
- count: Copy lines, shows progress)
- dbm: A little tool to access gdbm files from shell.)
- dirlist: Primitive directory lister, quicker than ls, find and echo *)
- diskus: Disk geometry checking and repair tool)
- getrealpath: Print realpath to stdout)
- histogram: Count bytes in file)
- kdmktone: Make the console beep)
- keypressed: Nonblocking, nondestructible test for waiting data on TTYs, sockets and probably pipes)
- killmem: Protect some memory against paging until you need free memory)
- lockdir: Create a directory for locking purpose)
- lockrun: Exclusively run something by placing a file lock)
- md5chk: Create md5sums for easy shell usage)
- minicron: This is a program which starts other programs after some time)
- mvatom: Move files by atomic rename instead of copy.)
- printansi: Like /bin/echo but ANSI-escapes the output)
- printargs: Like “hello world” but dumps the argc array)
- ptybuffer: daemonize interactive tty line driven programs with output history)
- runningfor: Return true until the given time periode is reached.)
- slowdown: Slowdown processes or pipes.)
- socklinger: Execute quick hack shell scripts connected to a socket.)
- sq: SQLITE3 query tool for shell usage)
- timeout: Execute a command or pipe only for a given duration.)
- timestart: Start a program N-M times in parallel by running it it each A-B seconds)
- tinohtmlparse: Simple HTML parser to extract information from HTML files by shell)
- tinoseq: An integer seq implementation)
- udevraw: Dump udev events for bash usage)
- unbuffered: Copy stdin to stdout and stderr, unbuffered)
- watcher: A Python2.6 script to watch files, pipes or Unix domain sockets)
2012.06.25
More unix tools
2011.02.15
2010.07.13
Command line based text replace
sed
-
sed 's/Mark Monre/Marc Monroe/' 1.txt > 2.txt
-
find ./* -type f -exec sed -i 's///g' {} \;
The “replace” command
- Syntax:
replace OLD-STRING NEW-STRING OUTPUT-FILE
- Example:
$ replace UNIX Linux newfile
- Example:
$ cat /etc/passwd | replace : '|'
- Partial support for regular expressions: \^ – matches start of line, and $ matches end of line.
- Example: replace all IP address 192.168.1.2 start of line:
$ replace \^192.168.1.2 192.168.5.10 newfile
- a bash script, ‘fixer.sh’
#!/bin/bash replace CHANGEFROM CHANGETO $1.tmp rm $1 mv $1.tmp $1
now run this command line:
$ grep CHANGEFROM |cut -d':' -f1 |xargs -n 1 fixer.sh
the results is that all files in the directory (or whatever you grep for) will be changed automagically.
just make sure the grep doesn’t include the fixer script itself, or it will die half-way through changing when execute permissions are reset!
Perl
- Perl Pie:
perl -p -i -e ’s/hello/goodbye/g’ textfile.txt
- http://www.debian-administration.org/articles/298 has a fine article and discussion on Perl Pie.
-
perl -p -i -e ’s/|00000000.00|/||/g’ myfile.txt
Sources:
- How do I replace text string in many files at once? – http://www.cyberciti.biz/tips/how-do-i-replace-text-string-in-many-files-at-once.html
Related: Regular expressions – https://eikonal.wordpress.com/2010/04/02/regular-expressions/ | Perl online – https://eikonal.wordpress.com/2010/02/15/perl-online/
2010.04.02
Regular expressions
Sites
- Regular Expressions specification – http://pubs.opengroup.org/onlinepubs/007908799/xbd/re.html
- at WikiPedia – http://en.wikipedia.org/wiki/Regular_expression
- Regular Expressions at Open Directory – http://www.dmoz.org/Computers/Programming/Languages/Regular_Expressions/
- Regular-Expressions.info site – http://www.regular-expressions.info/
- a table of Regular Expressions rules – http://www.3gwt.net/demo/regexps.html
- Regular expressions in Ruby – http://www.ruby-doc.org/docs/ProgrammingRuby/html/language.html#UJ; {Ruby}
- Regular Expressions Library – http://regexlib.com/
- JavaScript 5.6/VBScript Regular Expression Syntax – http://www.regextester.com/jssyntax.html; {VBScript}
- PCRE Regular Expression Pattern Syntax Refference (PHP preg*) – http://www.regextester.com/pregsyntax.html‘ {PHP}
- POSIX 1003.2 Regular Expression Pattern Syntax Refference (PHP ereg*) – http://www.regextester.com/eregsyntax.html; {PHP}
- java.util.regex – http://download.oracle.com/javase/1.5.0/docs/api/java/util/regex/Pattern.html; {Java}
- A Tao of Regular Expressions – http://www.cs.colorado.edu/~schenkc/UNIX_Regular_Expressions.pdf
- Steven Levithan’s blog “Flagrant Badassery” – A JavaScript and regular expression centric blog – http://blog.stevenlevithan.com/; {JavaScript}
- “Regex Gury” blog by Jan Goyvaerts – http://www.regexguru.com/
- Lesson: Regular Expressions (at Java Tuturials) – http://download.oracle.com/javase/tutorial/essential/regex/index.html; {Java}
- PerlRE module – http://perldoc.perl.org/perlre.html; {Perl}
- “Microsoft Beefs Up VBScript with Regular Expressions” – http://msdn.microsoft.com/en-us/library/ms974570.aspx; {VBScript}
- “.NET Framework Regular Expressions” – http://msdn.microsoft.com/en-us/library/hs600312.aspx; {.Net}
- Regular Expressions and Other Pattern Matching – http://billposer.org/Linguistics/Computation/Resources.html#patterns
- “Structural Regular Expressions” by Rob Pike – http://doc.cat-v.org/bell_labs/structural_regexps/
- “Regular Expressions” at MDN Dev Center – https://developer.mozilla.org/en/Core_JavaScript_1.5_Guide/Regular_Expressions; {JavaScript}
- “Regular Expression HOWTO” by A.M. Kuchling – http://www.amk.ca/python/howto/regex/
Tools
Standalone tools:
- RegexBuddy [COMMERCIAL] by Jan Goyvaerts of Just Great Software (http://www.just-great-software.com/): http://www.regexbuddy.com/ | Trial – http://www.regexbuddy.com/RegexBuddyCookbook.exe
- PowerGREP [COMMERCIAL] by Just Great Software (http://www.just-great-software.com/): http://www.powergrep.com/ | Demo/trial – http://download.jgsoft.com/powergrep/SetupPowerGREPDemo.exe
- RegexMagic [COMMERCIAL] by Just Great Software (http://www.just-great-software.com/): http://www.regexmagic.com/ | Demo/trial – http://download.jgsoft.com/magic/SetupRegexMagicDemo.exe
- JavaScript Regex Syntax Highlighter by Steven Levithan – http://stevenlevithan.com/regex/syntaxhighlighter/
- XRegexp, a JavaScript library by Steven Levithan – http://xregexp.com/
- Expresso (at UltraPico) – http://www.ultrapico.com/Expresso.htm
- Wingrep [COMMERCIAL] – http://wingrep.com/
- RegexRenamer – http://regexrenamer.sourceforge.net/
- RegexRenamer is a powerful but simple user-friendly graphical tool used to batch rename files using regular expressions. A regular expression (or regex) is simply text that contains special characters that together defines a pattern that can be used to match text, in this case filenames. Using the power of regexes it becomes easy to apply complex transformations to large groups of files that otherwise would not be possible with plain match & replace.
Online testers:
- RegexPal by Steven Levithan – http://regexpal.com/ — a JavaScript regular expression tester.
- XRegExp by Steven Levithan – http://blog.stevenlevithan.com/archives/xregexp-1-0
- Regular expression tool – http://regex.larsolavtorvik.com/
- NRegex – http://www.nregex.com/nregex/default.aspx
- NRegex bookmarklet – Drag this to your browser’s Bookmarks Toolbar
javascript:var%20r%20=%20window.prompt('Enter%20regular%20expression'); location.href='http://www.nregex.com/nregex/default.aspx?regex='%20+%20encodeURIComponent(r);
- NRegex bookmarklet – Drag this to your browser’s Bookmarks Toolbar
- Rubular – a Ruby regular expression editor – http://www.rubular.com/
- Regexp Editor – Java applet by Serbei Evdokimov – http://www.myregexp.com/ | same with a signed JAR – http://www.myregexp.com/signedJar.html | Source code (“Regex Util”) at Sourceforge – http://sourceforge.net/projects/regex-util/develop
- “My Regex Tester” – http://www.myregextester.com/
- REGex TESTER (ver. 1.5.3) by Venimus – http://www.regextester.com/
- Regex Tester 2.0 alpha – http://www.regextester.com/index2.html
- Regular Expression Test Page by FileFormat.Info – http://www.fileformat.info/tool/regex.htm
- Oscar Steele’s reAnimator – http://osteele.com/tools/reanimator/
- Visualizing Regular Expressions – http://osteele.com/archives/2006/02/reanimator
Books
- “Regular Expressions Cookbook” by Jan Goyvaerts & Steven Levithan (O’Reilly; 2009.05; ISBN-10: 9780596520687; ISBN-13: 978-0596520687) – http://www.amazon.com/dp/0596520689/
- Book page at O’Reilly – http://oreilly.com/catalog/9780596520694
- Book’s home page – http://www.regular-expressions-cookbook.com/
- “Regular Expressions for Regular Programmers” (book review; Coding Horror; 2009.06.08) – http://www.codinghorror.com/blog/2009/06/regular-expressions-for-regular-programmers.html
- “Mastering Regular Expressions” by Jeffrey E.F. Friedl (O’Reilly; 2006.08.15; ISBN-10: 9780596528126; ISBN-13: 978-0596528126) – http://www.amazon.com/Mastering-Regular-Expressions-Jeffrey-Friedl/dp/0596528124/ | home at O’Reilly – http://oreilly.com/catalog/9781565922570/
- “Regular Expression Pocket Reference: Regular Expressions for Perl, Ruby, PHP, Python, C, Java and .NET (Pocket Reference)” by Tony Stubblebine (O’Reilly; 2007.07.25; ISBN-10: 9780596514273; ISBN-13: 978-0596514273) – http://www.amazon.com/Regular-Expression-Pocket-Reference-Expressions/dp/0596514271/
- “Sams Teach Yourself Regular Expressions in 10 Minutes” by Ben Forta (Sams; 2004.03.05; ISBN-10: 0672325667; ISBN-13: 978-0672325663) – http://www.amazon.com/Teach-Yourself-Regular-Expressions-Minutes/dp/0672325667/
Tidbits
Sources: The above links.
- [abc] – A single character: a, b or c
- [^abc] – Any single character but a, b, or c
- [a-z] – Any single character in the range a-z
- [a-zA-Z] – Any single character in the range a-z or A-Z
- ^ – Start of line
- $ – End of line
- \A – Start of string
- \z – End of string
- . – Any single character
- \s – Any whitespace character
- \S – Any non-whitespace character
- \d – Any digit
- \D – Any non-digit
- \w – Any word character (letter, number, underscore)
- \W – Any non-word character
- \b – Any word boundary character
- (…) – Capture everything enclosed
- (a|b) – a or b
- a? – Zero or one of a
- a* – Zero or more of a
- a+ – One or more of a
- a{3} – Exactly 3 of a
- a{3,} – 3 or more of a
- a{3,6} – Between 3 and 6 of a
- ^\s[ \t]*$ – Match a blank line
- \d{2}-\d{5} – Validate an ID number consisting of 2 digits, a hyphen, and another 5 digits
Special common strings:
- Personal Name: ^[\w\.\’]{2,}([\s][\w\.\’]{2,})+$
- Username: ^[\w\d\_\.]{4,}$
- Password at least 6 symbols: ^.{6,}$
- Password or empty input: ^.{6,}$|^$
- email: ^[\_]*([a-z0-9]+(\.|\_*)?)+@([a-z][a-z0-9\-]+(\.|\-*\.))+[a-z]{2,6}$
- Email address: \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b[A-z0-9_.%+-]+@[A-z0-9_.%+-]+\.[A-z]{2,4}
- US phone: \W?\d{3}\W?\d{3}\W?\d{4}
- US Phone number: ^\+?[\d\s]{3,}$
- US Phone with code: ^\+?[\d\s]+\(?[\d\s]{10,}$
- URL: \W?\d{3}\W?\d{3}\W?\d{4}\b\w+://(\w|-|\.|/)+(/|\b)
- US Social Security Number (SSN): \d{3}-\d{2}-\d{4}
- US ZIP: \d{5}(-\d{4})?
- IP (v4) address: \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b
- IP (v4) address: \b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b
- IP (v4) address: ^(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5])\.(\d|[1-9]\d|1\d\d|2[0-4]\d|25[0-5]){3}$
- IP (v4) address: \b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b
- IP (v4) address: \b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b
- IP (v6) address:
- MAC address: ^([0-9a-fA-F][0-9a-fA-F]:){5}([0-9a-fA-F][0-9a-fA-F])$
- Positive Integers: ^\d+$
- Negative Integers: ^-\d+$
- Integer: ^-{0,1}\d+$
- Positive Number: ^\d*\.{0,1}\d+$
- Negative Number: ^-\d*\.{0,1}\d+$
- Positive Number or Negative Number: ^-{0,1}\d*\.{0,1}\d+$
- Floating point number: [-+]?([0-9]*\.[0-9]+|[0-9]+)
- Floating point number: [-+]?(?:\b[0-9]+(?:\.[0-9]*)?|\.[0-9]+\b)(?:[eE][-+]?[0-9]+\b)?
- Roman number: ^(?i:(?=[MDCLXVI])((M{0,3})((C[DM])|(D?C{0,3}))?((X[LC])|(L?XX{0,2})|L)?((I[VX])|(V?(II{0,2}))|V)?))$
- Domain Name: ^([a-zA-Z0-9]([a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,6}$
- Domain Name: ^([a-z][a-z0-9\-]+(\.|\-*\.))+[a-z]{2,6}$
- Windows File Name: (?i)^(?!^(PRN|AUX|CLOCK\$|NUL|CON|COM\d|LPT\d|\..*)(\..+)?$)[^\\\./:\*\?\”\|][^\\/:\*\?\”\|]{0,254}$
- Date in format yyyy-MM-dd: (19|20)\d\d([- /.])(0[1-9]|1[012])\2(0[1-9]|[12][0-9]|3[01])
- Date (dd mm yyyy, d/m/yyyy, etc.): ^([1-9]|0[1-9]|[12][0-9]|3[01])\D([1-9]|0[1-9]|1[012])\D(19[0-9][0-9]|20[0-9][0-9])$
- Year 1900-2099: ^(19|20)[\d]{2,2}$
Related (here at this blog):
Command line based text replace – https://eikonal.wordpress.com/2010/07/13/command-line-based-text-replace/ |
Perl online – https://eikonal.wordpress.com/2010/02/15/perl-online/