I‘m a new Linux user. We have a backup server and most our developers sftp or ftp to upload files. I just need to backup of last four weeks. If I try to list or remove directories I get an error cannot remove directory. I’m getting “Argument list too long” message. How do I solve this problem when executing commands in the bash/ksh/csh/tcsh/zsh shell?
Each command under Linux and UNIX-like systems accepts a parameter commonly known as command arguments (or args). For example, the command ‘cd /etc’ has one command line arguments, namely, /etc. Some command can accept more than two arguments and act on supplied args. For example, consider the following cp command:
cp /etc/file1 /etc/file2 /etc/file3 /mnt/pen[donotprint][/donotprint]
The cp command has total four command line arguments. The shell can hold a maximum of 131072 bytes for command line arguments. If you try to pass more than that number you will greeted with an error that read as follows:
Argument list too long
Remember each character requires one byte of storage.
Finding out maximum length of the command line argument
The maximum length of the command line argument is limited to *128k* within the Linux kernel. To find your limits, type:
getconf ARG_MAX |
Sample outputs:
2621440
The limit affects the execve() kernel function, which is utilized by all other exec() functions (execl, execlp, execle, etc).
Option #1: How to get rid of “Argument list too long error” problem while using rm, ls, cp, mv or any other shell commands
The best way to deal with this problem is combination of the find and xargs commands. The syntax is:
find /dir/to/search -type f -name 'file-pattern' -print0 | xargs -0 -n 10 command-name find /dir/to/search -type f -iname "*.EXT" -print0 | xargs -0 command-name find /dir/to/search -type f -iname "*.EXT" -command |
Just list files when you get “Argument list too long” error
Use any one of the following syntax to list only files
find /dir/to/search -type f -iname "*.EXT" -ls find /nas01/data/ -type f -iname "*.doc" -ls |
OR the following portable syntax:
find /nas01/data/ -name '*.doc' -exec ls {} + ## GNU/Find ## find /nas01/data/ -type f -name '*.doc' -exec ls -l {} + |
To avoid recursion use the following GNU find option:
find /nas01/data/ -name '*.doc' -maxdepth 1 -exec ls -l {} + |
Just delete files when “Argument list too long” error
The syntax is as follows:
# GNU/find and BSD/find may not work on older *nix find /dir/to/search -type f -iname "*.EXT" -delete find ~/Downloads/ -type f -iname "*.avi" -delete find ~/Projects/ -type f -iname "*.pdf" -delete ## use xargs and rm command ## find ~/Downloads/ -name '*.avi' -print0 | xargs -0 -n 10 rm -f |
More portable syntax:
find ~/Projects/ -name '*.pdf' -exec rm -vf {} + |
To avoid recursion use the following GNU find option:
## GNU find and may work with bsd find too ## find ~/Projects/ -name '*.pdf' -maxdepth 1 -exec rm -f {} + find ~/Downloads/ -type f -iname '*.avi' -print0 | xargs -0 -n 10 rm -f |
You can also use GNU parallel command:
## man parallel for more ## find ~/Downloads/ -type f -iname '*.avi' -print0 | parallel -X -0 rm -vf |
Where,
- -name "*.pdf" : Search pattern.
- -type f : Only match files.
- -print 0 : Avoid nasty surprises when dealing with weird file names with blank spaces, ‘, “, , tab, and newline characters. The -print0 primary in conjunction with xargs -0 as an effective alternative to dealing with weird filenames.
- -maxdepth 1 : Descend at most one directory levels below the command line arguments i.e avoid recursion.
- -exec rm -f {} + OR -exec ls {} + : Run rm or ls commands on as many as files possible.
Option #2: Iterate with the shell to avoid arg list too long error
The syntax is as follows and it is very slow and may not work at all:
for f in *; do command "$i"; done ## delete all pdf files ## for f in *.pdf; do /bin/rm -rf "$f"; done ############################################################################ ### Delete all *.mp4 and *.avi files using pure Bash solution using arrays ### Based on http://mywiki.wooledge.org/BashFAQ/095 ############################################################################ files=($HOME/Downloads/*.avi $HOME/Downloads/*.mp4) for ((i=0; i<${#files[*]}; i+=100)); do /bin/rm -vf "${files[@]:i:100}" done |
Command summary to to avoid the limit in a shell
## deals with weird filenames too ## find /dir/to/search/ -name "file-pattern-search" -exec command {} + ## find and xargs based solution ## find /dir/to/search/ -name "file-pattern-search" -print0 | xargs -0 command ## For gnu parallel fans ## find /dir/to/search/ -name 'file-pattern-search' -print0 | parallel -X -0 command ## older and not recommended solution; but kept here for the historical reasons only ## for i in *; do command "$i"; done |
See man pages for more info: mv(1)
