Linux Make Big Files

Posted on | Edited on | In

最近在学习Linux storage的过程中，研究了一下几个创建大文件的命令，特别是它们的区别. 可以参考这个提问Quickly create a large file on a Linux system.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
cd /tmp
# slow
dd if=/dev/zero of=./ddfile bs=1G count=3
# fastest
truncate -s 3G ./trunfile
# fast
fallocate -l 3G ./fallfile

# sync, depends on you needs
# if no sync, files may be still in memory
sync

# ls -ltrh
# in logical view, they are all 3GB
-rw-r--r--. 1 root root 3.0G Dec 30 07:01 ddfile
-rw-r--r--. 1 root root 3.0G Dec 30 07:02 trunfile
-rw-r--r--. 1 root root 3.0G Dec 30 07:02 fallfile

# check physical storage
# truncate space almost does not count 
# because of sparse
df -h .

# time test
# several seconds
time /bin/cp ./ddfile ~
time /bin/cp ./fallfile ~

# near instant
time /bin/cp ./trunfile ~

注意这里sync之后，file cache是仍然存在的，如果要彻底drop file cache, run echo 3 > /proc/sys/vm/drop_caches

所以说，在需要实打实disk allocated的场景中，不要使用truncate, 比如测试network, IO performance. dd最耗时，占用CPU 以及IO最多，fallocate比dd稍微好些，创建大文件比较快，也确实占用了空间。

其实dd也可以创建sparse file, with seek option:

1
2
3
4
dd if=/dev/zero of=sparse_file bs=1 count=0 seek=3G
# Block is 0
stat sparse_file
Size: 3221225472	Blocks: 0          IO Block: 4096   regular file

dd can be configured write to disk rather as well:

1
dd if=/dev/urandom of=/dev/sdb1 bs=1M count=2048

You can also shrink a file with many empty blocks with fallocate command:

1
2
3
4
5
6
7
8
9
10
dd if=/dev/zero of=./empty bs=1G count=3
# see the block number before and after change
stat ./empty
Size: 3221225472	Blocks: 6291544    IO Block: 4096   regular file

# deallocate empty block
fallocate -d ./empty

stat ./empty
Size: 3221225472	Blocks: 0          IO Block: 4096   regular file
0%