Streaming data is a common data processing problem. But luckily, the Bash
xargs command, a versatile tool, can help process and manipulate streaming data more efficiently.
In this tutorial, you will learn how to use the Bash
xargs command through practical examples.
Stay tuned and take control of command behaviors with the Bash
This tutorial houses hands-on demonstrations, and you will need a Linux system with the Bash shell installed to follow along. This tutorial uses Ubuntu 20.04, but any recent Linux distribution should work.
Finding and Removing Files with the Bash
xargs command is a Linux utility used to build and execute commands from standard input. This command reads input from stdin, parses into arguments, and executes the specified command one or more times with the parsed arguments.
The general syntax for the
xargs command is as below:
xargs <options> <command>
The most common use case for the
xargs command is to find and remove files based on specific criteria. For example, you can use the
xargs command to find all files older than a specified number of days and delete them.
Run the below command to
find all files older than 30 days (
-mtime +30) and pipe the output to the Bash
xargs command. The
xargs command then executes the
rm command to forcefully (
-f) delete each file without prompting for confirmation.
-print0 option below prints the filenames to stdout, separated by a null character, since the
xargs command expects its input to be null-delimited rather than newline-delimited.
find -mtime +30 -print0 | xargs -0 rm -f
This command does not provide output, but you will verify the deletion is successful in the following step.
💡 The Bash
xargscommand is often used with pipes to process data that would otherwise be too long for a single command.
Now, run the command below to list (
ls) all contents of the working directory, sorted by time (
t) in reverse order (
All the files older than 30 days have been deleted, as shown below. Only new files remain in the current directory.
Finding and Archiving Files
Another everyday use case for the
xargs command is to find all files based on a particular pattern and execute a command on each. Suppose you wish to find all text files and compress them to an archive (ZIP file), which lets you save some disk space. If so, the Bash
xargs command is up to the task.
Run the below command to
find all text files (
*.txt) and pipe the output to the
xargs command, which executes the tar command to archive the files to
-0 option below tells
xargs to treat its input as null-delimited instead of the default newline-delimited, so filenames with spaces in them are handled correctly.
find -name "*.txt" -print0 | xargs -0 tar -czvf archive-txt.tar.gz
Now, run the following command to list all files containing the name
ls -la archive-txt.tar.gz
Below, you can verify the archive (
archive-txt.tar.gz) exists and that the compression is a success.
Running Multiple Commands with Bash
Perhaps a file needs to be processed, and you wish to parallelize the processing by running multiple commands simultaneously.
For example, instead of running one
echo command for each filename to insert to the text file, as shown below, let the
xargs do the trick. The
xargs command supports running multiple commands on its input.
Suppose you have to create directories for multiple files. If so, you just need a list of filenames and run an
1. Run the below command to create a text file named
bash-xargs-demo.txt containing a list of filenames, one per line.
# Creates a bash-xargs-demo.txt containing a list of filenames, one per line.
cat > bash-xargs-demo.txt << EOF
# Viewing the bash-xargs-demo.txt content
2. Next, run the command below to read the filenames from the text file and create directories for each (
sh -c 'echo %; mkdir %').
-I % option tells
xargs to replace
% with the input it reads from stdin, which is
cat bash-xargs-demo.txt | xargs -I % sh -c 'echo %; mkdir %'
💡 Perhaps you want to see what happens in the background as you run the command. If so, append the
-toption to the
xargscommand to get a verbose output, as shown below.
cat bash-xargs-demo.txt | xargs -t -I % sh -c 'echo %; mkdir %'
3. Lastly, run the command below to list (
ls) all contents of the working directory in a long list (
You can verify that the directories were created successfully by running the
ls command below.
In the output below, you can see three new directories (One, Two, and Three), one for each input line in the text file.
Enabling Prompting Before Execution
By default, the Bash
xargs command executes specified commands for all input without asking for confirmation. This behavior can be dangerous if you are not careful, such as when deleting files with the
For example, the
rm command below deletes all files in the current directory recursively without a confirmation prompt. You could delete many important files if you accidentally ran this command in the wrong directory.
rm -rf *
💡 Perhaps you have mistakenly deleted important files. Luckily, Linux lets you recover deleted files.
To avoid untoward actions, append the
-p option to enable prompting before execution.
Run the below command to
find all files (
*) in the working directory and delete (
rm) them recursively (
-rf) with a confirmation prompt (
find * | xargs -p rm -rf
Notice below that even though you used the force (
-f) option in the
rm command, you will still get a prompt since the
-p option is appended to the
xargs command. The prompt below lets you verify that the command will do what you expect before confirming.
Type Y and press Enter to confirm the action. Or type N and press Enter if you made a mistake to cancel the command, and no harm will be done.
Reading Files Using the
So far, you have been reading input from stdin. But did you know that
xargs can also read input from files? Yes! This feature can be useful if you have a large amount of data that needs to be processed and you do not want to keep the data all in your memory.
Run the below command to read and print input (
-a) from the
foo.txt file to stdout.
💡 Note that for
xargsto read the input from files, you must append the
-aoption before any other options.
xargs -a foo.txt
Filtering Search Result with
grep command is handy when searching texts in files. But one drawback is that the process may take a long time if you search through many files. The good news is that
xargs can parallelize the grep command, speeding up the search significantly.
Run the following command to
find all text files (
*) in the working directory, filter, and print (
grep) any files that contain the
find . -name "*.txt" | xargs grep 'Examples'
As you can see below, two files contain the Examples string.
Streaming input from stdin is a powerful feature of the Bash
xargs command, but that is not all. In this tutorial, you have learned how to read input from files using the
-a option. In addition, you touched on using
xargs with other commands, such as
grep, to speed up your workflows.
At this point, you should be comfortable with using the Bash
xargs command. Experiment with different ways of using
xargs to find the method that works best for you!
Now, why not try running programs in parallel using xargs?