xargs: How To Control and Use Command Line Arguments

参考：

http://www.cyberciti.biz/faq/linux-unix-bsd-xargs-construct-argument-lists-utility/

http://linux.101hacks.com/linux-commands/xargs-command-examples/

http://www.computerhope.com/unix/xargs.htm

http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html

http://unixhelp.ed.ac.uk/CGI/man-cgi?xargs

 

XARGS(1)							      XARGS(1)

NAME

       xargs - build and execute command lines from standard input

SYNOPSIS

       xargs  [-0prtx]	[-E  eof-str] [-e[eof-str]] [--eof[=eof-str]] [--null]
       [-d delimiter] [--delimiter delimiter]  [-I  replace-str]  [-i[replace-
       str]]	[--replace[=replace-str]]   [-l[max-lines]]   [-L   max-lines]
       [--max-lines[=max-lines]] [-n max-args] [--max-args=max-args] [-s  max-
       chars]  [--max-chars=max-chars]	[-P max-procs] [--max-procs=max-procs]
       [--interactive]	    [--verbose]	     [--exit]	   [--no-run-if-empty]
       [--arg-file=file] [--version] [--help] [command [initial-arguments]]

DESCRIPTION

       This manual page documents the GNU version of xargs.  xargs reads items
       from the standard input, delimited by blanks (which  can	 be  protected
       with  double or single quotes or a backslash) or newlines, and executes
       the command (default is /bin/echo) one or more times with any  initial-
       arguments  followed  by items read from standard input.	Blank lines on
       the standard input are ignored.

       Because Unix filenames can contain blanks and  newlines,	 this  default
       behaviour is often problematic; filenames containing blanks and/or new-
       lines are incorrectly processed by xargs.  In these  situations	it  is
       better  to  use	the  '-0' option, which prevents such problems.	  When
       using this option you will need to ensure that the program  which  pro-
       duces  the  input  for xargs also uses a null character as a separator.
       If that program is GNU find for example, the '-print0' option does this
       for you.

       If any invocation of the command exits with a status of 255, xargs will
       stop immediately without reading any further input.  An	error  message
       is issued on stderr when this happens.

  OPTIONS
       --arg-file=file, -a file
	      Read items from file instead of standard input.  If you use this
	      option, stdin remains unchanged when commands are	 run.	Other-
	      wise, stdin is redirected from /dev/null.

       --null, -0
	      Input  items  are	 terminated  by a null character instead of by
	      whitespace, and the quotes and backslash are not special	(every
	      character is taken literally).  Disables the end of file string,
	      which is treated like any other  argument.   Useful  when	 input
	      items  might  contain  white space, quote marks, or backslashes.
	      The GNU find -print0 option produces  input  suitable  for  this
	      mode.

       --delimiter=delim, -d delim
	      Input  items  are terminated by the specified character.	Quotes
	      and backslash are not special; every character in the  input  is
	      taken  literally.	  Disables  the	 end-of-file  string, which is
	      treated like any other argument.	This  can  be  used  when  the
	      input consists of simply newline-separated items, although it is
	      almost always better to design  your  program  to	 use  '--null'
	      where this is possible.  The specified delimiter may be a single
	      character, a C-style character escape such as \n, or an octal or
	      hexadecimal escape code.	Octal and hexadecimal escape codes are
	      understood as for the printf command.   Multibyte characters are
	      not supported.

       -Eeof-str
	      Set  the	end  of	 file  string  to eof-str.  If the end of file
	      string occurs as a line of input,	 the  rest  of	the  input  is
	      ignored.	If neither -E nor -e is used, no end of file string is
	      used.

       --eof[=eof-str], -e[eof-str]
	      This option is a synonym for the '-E' option.  Use '-E' instead,
	      because it is POSIX compliant while this option is not.  If eof-
	      str is omitted, there is no end of file string.  If  neither  -E
	      nor -e is used, no end of file string is used.

       --help Print a summary of the options to xargs and exit.

       -I replace-str
	      Replace occurrences of replace-str in the initial-arguments with
	      names read from standard input.  Also, unquoted  blanks  do  not
	      terminate	 input	items;	instead	 the  separator is the newline
	      character.  Implies -x and -L 1.

       --replace[=replace-str], -i[replace-str]
	      This option is a synonym for  -Ireplace-str  if  replace-str  is
	      specified,  and  for -I{} otherwise.  This option is deprecated;
	      use -I instead.

       -L max-lines
	      Use at most max-lines nonblank input  lines  per	command	 line.
	      Trailing blanks cause an input line to be logically continued on
	      the next input line.  Implies -x.

       --max-lines[=max-lines], -l[max-lines]
	      Synonym for the -L option.  Unlike -L, the max-lines argument is
	      optional.	  If  max-args	is  not specified, it defaults to one.
	      The -l option is deprecated since the POSIX  standard  specifies
	      -L instead.

       --max-args=max-args, -n max-args
	      Use  at  most  max-args  arguments per command line.  Fewer than
	      max-args arguments will be used if the size (see the -s  option)
	      is  exceeded, unless the -x option is given, in which case xargs
	      will exit.

       --interactive, -p
	      Prompt the user about whether to run each command line and  read
	      a	 line  from  the  terminal.   Only run the command line if the
	      response starts with 'y' or 'Y'.	Implies -t.

       --no-run-if-empty, -r
	      If the standard input does not contain any nonblanks, do not run
	      the command.  Normally, the command is run once even if there is
	      no input.	 This option is a GNU extension.

       --max-chars=max-chars, -s max-chars
	      Use at most max-chars characters per command line, including the
	      command  and  initial-arguments and the terminating nulls at the
	      ends of the argument strings.  The default is 131072 characters,
	      not  including  the size of the environment variables (which are
	      provided for separately so that it doesn't matter if your	 envi-
	      ronment  variables take up more than 131072 bytes).  The operat-
	      ing system places limits on the values  that  you	 can  usefully
	      specify,	and  if	 you exceed these a warning message is printed
	      and the value actually used is set to the appropriate  upper  or
	      lower limit.

       --verbose, -t
	      Print  the command line on the standard error output before exe-
	      cuting it.

       --version
	      Print the version number of xargs and exit.

       --exit, -x
	      Exit if the size (see the -s option) is exceeded.

       --max-procs=max-procs, -P max-procs
	      Run up to max-procs processes at a time; the default is  1.   If
	      max-procs	 is 0, xargs will run as many processes as possible at
	      a time.  Use the -n option with -P; otherwise chances  are  that
	      only one exec will be done.

EXAMPLES

       find /tmp -name core -type f -print | xargs /bin/rm -f

       Find  files  named core in or below the directory /tmp and delete them.
       Note that this will work incorrectly if there are  any  filenames  con-
       taining newlines or spaces.

       find /tmp -name core -type f -print0 | xargs -0 /bin/rm -f

       Find  files  named core in or below the directory /tmp and delete them,
       processing filenames in such a way that file or	directory  names  con-
       taining spaces or newlines are correctly handled.

       cut -d: -f1 < /etc/passwd | sort | xargs echo

       Generates a compact listing of all the users on the system.

 

xargs: How To Control and Use Command Line Arguments

Iam trying to use xargs command using shell pipes and not able to understand how to control and use command line arguments. For example I'd like to find out all *.c file located in 100s of sub-directories and move them to another directory called ~/old.src. How do I use command line args with xargs to achieve the same?

xargs command is designed to construct argument lists and invoke other utility. xargs reads items from the standard input or pipes, delimited by blanks or newlines, and executes the command one or more times with any initial-arguments followed by items read from standard input. Blank lines on the standard input are ignored.

xargs is more safer and easy to use

xargs functionality can be achived using the backquote feature of shell. But, it offers more options. It can deal with blanks or special characters in file names easily. It is often used with find, grep and other commands.

xargs examples

For example following example will print 1 2 3 4 using xargs (echo command is default)
$ echo 1 2 3 4 | xargs echo
OR
$ echo 1 2 3 4 | xargs
You can force xargs to use at most max-args arguments per command line. For example following will use first two argument per command:
$ echo 1 2 3 4 | xargs -n 2
Find all .bak files in or below the current directory and delete them.
$ find . -name "*.bak" -type f -print | xargs /bin/rm -f

{} as the argument list marker

{} is the default argument list marker. You need to use {} this with various command which take more than two arguments at a time. For example mv command need to know the file name. The following will find all .bak files in or below the current directory and move them to ~/.old.files directory:
$ find . -name "*.bak" -print0 | xargs -0 -I {} mv {} ~/old.files
You can rename {} to something else. In the following example {} is renamed as file. This is more readable as compare to previous example:
$ find . -name "*.bak" -print0 | xargs -0 -I file mv file ~/old.files
Where,

1. -0 If there are blank spaces or characters (including newlines) many commands will not work. This option take cares of file names with blank space.
2. -I Replace occurrences of replace-str in the initial-arguments with names read from standard input. Also, unquoted blanks do not terminate input items; instead the separator is the newline character.

Dealing file names with blank spaces and newline

The following will work incorrectly if there are any filenames containing newlines or spaces (it will find out all .mp3 file located in current directory and play them using mplayer):
$ find . -iname "*.mp3" -print | xargs mplayer
To get rid of this problem use -0 option:
$ find . -iname "*.mp3" -print0 | xargs -0 -I mp3file mplayer mp3file
To find out all *.c file located in 100s of subdirectories and move them to another directory called ~/old.src, use:
$ find /path/to/dir -iname "*.c" -print0 | xargs -0 -I file mv file ~/old.src

Avoiding errors and resource hungry problems with xargs and find combo

To copy all media files to another location called /bakup/iscsi, you can use cp as follows:
$ cp -r -v -p /share/media/mp3/ /backup/iscsi/mp3
However, cp command may fail if an error occurs such as if the number of files is too large for the cp command to handle. xargs in combination with find can handle such operation nicely. xargs is more resource efficient and will not halt with an error:

$ find /share/media/mp3/ -type f -name "*.mp3" -print0 | xargs -0 -r -I file cp -v -p file --target-directory=/bakup/iscsi/mp3

Please note that all of the above commands are tested with GNU/xargs version. BSD and UNIX xargs command may not have options such as -r. Please refer to your local xargs man page for further info:
man xargs

 

Hack 22. Xargs Command Examples

by RAMESH

xargs is a very powerful command that takes output of a command and pass it as argument of another command. Following are some practical examples on how to use xargs effectively.

Xargs Example 1:

When you are trying to delete too many files using rm, you may get error message: /bin/rm Argument list too long – Linux. Use xargs to avoid this problem.

# find ~ -name ‘*.log’ -print0 | xargs -0 rm -f

Xargs Example 2:

Get a list of all the *.conf file under /etc/. There are different ways to get the same result. Following example is only to demonstrate the use of xargs. The output of the find command in this example is passed to the ls –l one by one using xargs.

# find /etc -name "*.conf" | xargs ls –l

Xargs Example 3:

If you have a file with list of URLs that you would like to download, you can use xargs as shown below.

# cat url-list.txt | xargs wget –c

Xargs Example 4:

Find out all the jpg images and archive it.

# find / -name *.jpg -type f -print | xargs tar -cvzf images.tar.gz

Xargs Example 5:

Copy all the images to an external hard-drive.

# ls *.jpg | xargs -n1 -i cp {} /external-hard-drive/directory

Things you (probably) didn’t know about xargs

26 June 2011 – Bangalore, India

If you’ve spent any amount of time at a Unix command line you’ve probably already seen xargs. In case you haven’t, xargs is a command used to execute commands based on arguments from standard input.

Common use cases

I often see xargs used in combination with find in order to do something with the list of files returned by find.

Pedantic note: As people have correctly pointed out on Twitter and on Hacker News, find is a very powerful command and it has built in flags such as -execand -delete that you can often use instead of piping to xargs. However people either don’t know about the options to find, forget how to invoke -exec with it’s archaic syntax, or prefer the simplicity of xargs. There are also performance implications to the various choices. I should write a follow up post on find.

Contrived examples warning: I needed something simple examples that would not detract from the topic. This is the best I could do given the time I had.Patches are welcome :)

Recursively find all Python files and count the number of lines
find . -name '*.py' | xargs wc -l

Recursively find all Emacs backup files and remove them
find . -name '*~' | xargs rm

Recursively find all Python files and search them for the word ‘import’
find . -name '*.py' | xargs grep 'import'

Handling files or folders with spaces in the name

One problem with the above examples is that it does not correctly handle files or directories with a space in the name. This is because xargs by default will split on any white-space character. A quick solution to this is to tell find to delimit results with NUL (\0) characters (by supplying -print0 to find), and to tell xargs to split the input on NUL characters as well (-0).

Remove backup files recursively even if they contain spaces
find . -name '*~' -print0 | xargs -0 rm

Security note: filenames can often contain more than just spaces.

Placement of the arguments

In the examples above xargs reads all non-white-space elements from standard input and concatenates them into the given command line before executing it. This alone is very useful in many circumstances. Sometimes however you might want to insert the arguments into the middle of a command. The -I flag to xargs takes a string that will be replaced with the supplied input before the command is executed. A common choice is %.

Move all backup files somewhere else
find . -name '*~' -print 0 | xargs -0 -I % cp % ~/backups

Maximum command length

Sometimes the list of arguments piped to xargs would cause the resulting command line to exceed the maximum length allowed by the system. You can find this limit with

getconf ARG_MAX

In order to avoid hitting the system limit, xargs has its own limit to the maximum length of the resulting command. If the supplied arguments would cause the invoked command to exceed this built in limit, xargs will split the input and invoke the command repeatedly. This limit defaults to 4096, which can be significantly lower than ARG_MAX on modern systems. You can override xargs’s limit with the -s flag. This will be particularly important when you are dealing with a large source tree.

Operating on subset of arguments at a time

You might be dealing with commands that can only accept 1 or maybe 2 arguments at a time. For example the diff command operates on two files at a time. The -n flag to xargs specifies how many arguments at a time to supply to the given command. The command will be invoked repeatedly until all input is exhausted. Note that on the last invocation you might get less than the desired number of arguments if there is insufficient input. Let’s simply use xargs to break up the input into 2 arguments per line

$ echo {0..9} | xargs -n 2

0 1
2 3
4 5
6 7
8 9

In addition to running based on a specified number of arguments at time you can also invoke a command for each line of input at a time with -L 1. You can of course use an arbitrary number of lines a time, but 1 is most common. Here is how you might diff every git commit against its parent.

git log --format="%H %P" | xargs -L 1 git diff

Executing commands in parallel

You might be using xargs to invoke a compute intensive command for every line of input. Wouldn’t it be nice if xargs allowed you to take advantage of the multiple cores in your machine? That’s what -P is for. It allows xargs to invoke the specified command multiple times in parallel. You might use this for example to run multiple ffmpeg encodes in parallel. However I’m just going to show you yet another contrived example.

Parallel sleep

$ time echo {1..5} | xargs -n 1 -P 5 sleep

real    0m5.013s
user    0m0.003s
sys     0m0.014s

Sequential sleep

$ time echo {1..5} | xargs -n 1 sleep

real    0m15.022s
user    0m0.004s
sys     0m0.015s

If you are interested in using xargs for parallel computation also consider GNUparallel. xargs has the advantage of being installed by default on most systems, and easily available on BSD and OS X, but parallel has some really nice features.

If you found this post useful mention it on Twitter and follow me.