Argument list too long

Posted on | In

This issue was exposed when running postgres /bin/psql with a super long input (>130KB), the error was:

1
fork/exec /bin/psql: argument list too long

In Linux, the arugment total combined size is usually managed by kernal limit, a system defined constant, it can vary by system:

1
2
# Byte: 2097152 => 2MB by default
getconf ARG_MAX

While ARG_MAX defines the total combined size of all arguments and environment variables passed to a new process, MAX_ARG_STRLEN defines the maximum length of a single argument or environment variable string.

MAX_ARG_STRLEN is typically defined in the kernel headers, specifically in linux/binfmts.h. For many Linux systems, it’s defined as (PAGE_SIZE * 32)

1
2
# MAX_ARG_STRLEN for single argument limit, unit: Byte: 131072 => 128KB
echo "$(( $(getconf PAGE_SIZE) * 32 ))"

How to Reproduce The Error

It is simple to trigger the limit error, for example:

1
2
3
4
5
6
7
8
9
# create a random file with 129KB size
dd if=/dev/urandom of=/tmp/test bs=1K count=129

# run it
psql -h 127.0.0.1 -U cloudsqladmin -c "$(cat /tmp/test)"

# get error:
bash: warning: command substitution: ignored null byte in input
bash: /bin/psql: Argument list too long

How to Bypass

The possible solution to bypass the limit is to use pipe or filename as the input for the binary, for example:

1
2
3
4
5
6
7
8
9
10
11
# Create a 500KB file as fake query (OS limit 128KB):
postgres@a-878062546719-vm-a9bdb3589547ab41:/$ dd if=/dev/urandom of=/tmp/test bs=1K count=500
500+0 records in
500+0 records out
512000 bytes (512 kB, 500 KiB) copied, 0.00356667 s, 144 MB/s

# filename
postgres@a-878062546719-vm-a9bdb3589547ab41:/$ psql -h 127.0.0.1 -U cloudsqladmin -f /tmp/test

# pipe
postgres@a-878062546719-vm-a9bdb3589547ab41:/$ cat /tmp/test | psql -h 127.0.0.1 -U cloudsqladmin
0%