Streaming data is a common data processing problem. But luckily, the Bash xargs
command, a versatile tool, can help process and manipulate streaming data more efficiently.
In this tutorial, you will learn how to use the Bash xargs
command through practical examples.
Stay tuned and take control of command behaviors with the Bash xargs
command!
Prerequisite
This tutorial houses hands-on demonstrations, and you will need a Linux system with the Bash shell installed to follow along. This tutorial uses Ubuntu 20.04, but any recent Linux distribution should work.
Finding and Removing Files with the Bash xargs
Command
The Bash xargs
command is a Linux utility used to build and execute commands from standard input. This command reads input from stdin, parses into arguments, and executes the specified command one or more times with the parsed arguments.
The general syntax for the xargs
command is as below:
xargs <options> <command>
The most common use case for the xargs
command is to find and remove files based on specific criteria. For example, you can use the xargs
command to find all files older than a specified number of days and delete them.
Run the below command to find
all files older than 30 days (-mtime +30
) and pipe the output to the Bash xargs
command. The xargs
command then executes the rm
command to forcefully (-f
) delete each file without prompting for confirmation.
The -print0
option below prints the filenames to stdout, separated by a null character, since the xargs
command expects its input to be null-delimited rather than newline-delimited.
find -mtime +30 -print0 | xargs -0 rm -f
This command does not provide output, but you will verify the deletion is successful in the following step.
💡 The Bash
xargs
command is often used with pipes to process data that would otherwise be too long for a single command.
Now, run the command below to list (ls
) all contents of the working directory, sorted by time (t
) in reverse order (r
).
ls -ltr
All the files older than 30 days have been deleted, as shown below. Only new files remain in the current directory.
Finding and Archiving Files
Another everyday use case for the xargs
command is to find all files based on a particular pattern and execute a command on each. Suppose you wish to find all text files and compress them to an archive (ZIP file), which lets you save some disk space. If so, the Bash xargs
command is up to the task.
Run the below command to find
all text files (*.txt
) and pipe the output to the xargs
command, which executes the tar command to archive the files to *archive-txt.tar.gz*
.
The -0
option below tells xargs
to treat its input as null-delimited instead of the default newline-delimited, so filenames with spaces in them are handled correctly.
find -name "*.txt" -print0 | xargs -0 tar -czvf archive-txt.tar.gz
Now, run the following command to list all files containing the name archive-txt.tar.gz
.
ls -la archive-txt.tar.gz
Below, you can verify the archive (archive-txt.tar.gz
) exists and that the compression is a success.
Running Multiple Commands with Bash xargs
Perhaps a file needs to be processed, and you wish to parallelize the processing by running multiple commands simultaneously.
For example, instead of running one echo
command for each filename to insert to the text file, as shown below, let the xargs
do the trick. The xargs
command supports running multiple commands on its input.
Suppose you have to create directories for multiple files. If so, you just need a list of filenames and run an xargs
command.
1. Run the below command to create a text file named bash-xargs-demo.txt
containing a list of filenames, one per line.
# Creates a bash-xargs-demo.txt containing a list of filenames, one per line.
cat > bash-xargs-demo.txt << EOF
One
Two
Three
EOF
# Viewing the bash-xargs-demo.txt content
cat bash-xargs-demo.txt
2. Next, run the command below to read the filenames from the text file and create directories for each (sh -c 'echo %; mkdir %'
).
The -I %
option tells xargs
to replace %
with the input it reads from stdin, which is *bash-xargs-demo.txt*
.
cat bash-xargs-demo.txt | xargs -I % sh -c 'echo %; mkdir %'
💡 Perhaps you want to see what happens in the background as you run the command. If so, append the
-t
option to thexargs
command to get a verbose output, as shown below.cat bash-xargs-demo.txt | xargs -t -I % sh -c 'echo %; mkdir %'
3. Lastly, run the command below to list (ls
) all contents of the working directory in a long list (-la
).
ls -la
You can verify that the directories were created successfully by running the ls
command below.
In the output below, you can see three new directories (One, Two, and Three), one for each input line in the text file.
Enabling Prompting Before Execution
By default, the Bash xargs
command executes specified commands for all input without asking for confirmation. This behavior can be dangerous if you are not careful, such as when deleting files with the rm
command.
For example, the rm
command below deletes all files in the current directory recursively without a confirmation prompt. You could delete many important files if you accidentally ran this command in the wrong directory.
rm -rf *
💡 Perhaps you have mistakenly deleted important files. Luckily, Linux lets you recover deleted files.
To avoid untoward actions, append the -p
option to enable prompting before execution.
Run the below command to find
all files (*
) in the working directory and delete (rm
) them recursively (-rf
) with a confirmation prompt (-p
).
find * | xargs -p rm -rf
Notice below that even though you used the force (-f
) option in the rm
command, you will still get a prompt since the -p
option is appended to the xargs
command. The prompt below lets you verify that the command will do what you expect before confirming.
Type Y and press Enter to confirm the action. Or type N and press Enter if you made a mistake to cancel the command, and no harm will be done.
Reading Files Using the -a
Option
So far, you have been reading input from stdin. But did you know that xargs
can also read input from files? Yes! This feature can be useful if you have a large amount of data that needs to be processed and you do not want to keep the data all in your memory.
Run the below command to read and print input (-a
) from the foo.txt
file to stdout.
💡 Note that for
xargs
to read the input from files, you must append the-a
option before any other options.
xargs -a foo.txt
Filtering Search Result with xargs
and grep
The grep
command is handy when searching texts in files. But one drawback is that the process may take a long time if you search through many files. The good news is that xargs
can parallelize the grep command, speeding up the search significantly.
Run the following command to find
all text files (*
) in the working directory, filter, and print (grep
) any files that contain the Examples
string.
find . -name "*.txt" | xargs grep 'Examples'
As you can see below, two files contain the Examples string.
Conclusion
Streaming input from stdin is a powerful feature of the Bash xargs
command, but that is not all. In this tutorial, you have learned how to read input from files using the -a
option. In addition, you touched on using xargs
with other commands, such as grep
, to speed up your workflows.
At this point, you should be comfortable with using the Bash xargs
command. Experiment with different ways of using xargs
to find the method that works best for you!
Now, why not try running programs in parallel using xargs?