sed Command Daily Work Summary

Normally we use a interactive text editor, like vim. The command sed is one of the most commonly used command line editor in Linux world. sed stands for the stream editor, it uses the roles supplied to edit a stream of data on the fly.

Note: Be careful that sed will break the softlink and create a file with the same name! for example:

ln -s /tmp/source.txt /tmp/link.txt
sed -i -e "s#aaa#bbb#" /tmp/link.txt

then the softlink is gone, a new file named link.txt is created instead. so use readlink first to get resolved symbolic links, then use sed on it.

First you need to understand how sed works with text:

1. Reads one data line at a time from the input
2. Matches that data with the supplied editor commands
3. Changes data in the stream as specified in the commands
4. Outputs the new data to STDOUT

2个对于输出很有帮助的flags, 可以加在其他功能后面:

# -n: quiet
# /p: print the edited line
# 这样组合就只会输出改动的部分
sed -n -e 's/root/toor/p' file 

Substitution

I have a text file sedtxt:

Using Java print hello world and hello tree.
Using Java print hello world and hello tree.
Using Java print hello world and hello tree.
Using Java print hello world and hello tree.

If I want to substite all hello with hi in-place:

sed -i -e 's/hello/hi/g' sedtxt

-i: does in-place substitution in file sedtxt, create backup file automatically by using -i.bak -e: followed by commands s/x/y/flags: substitute option, / is the delimiter, can be other chars; g represents that replace in all occurrences (global).

Note that if not set g, it will replace first occurrence in each line, what if I want to replace the first occurrence in a file, the workaround could be limit the scaning range:

sed '0,/Apple/{s/Apple/Banana/}' input_filename

the explain see here

If I want to substitute the second hello with goodbye in each line:

sed -e 's/hello/goodbye/2' sedtxt

Using Java print hello world and goodbye tree.
Using Java print hello world and goodbye tree.
Using Java print hello world and goodbye tree.
Using Java print hello world and goodbye tree.

If I want to substitute hello with hi and Java with Python:

sed -e 's/hello/hi/g' -e 's/Java/Pyhton/g' sedtxt
# or
sed -e 's/hello/hi/g; s/Java/Python/g' sedtxt

Using Pyhton print hi world and hi tree.
Using Pyhton print hi world and hi tree.
Using Pyhton print hi world and hi tree.
Using Pyhton print hi world and hi tree.

If I want to print only the matching lines, convenient for debugging:

sed -n -e 's/hello/hi/gp' sedtxt

Using Java print hi world and hi tree.
Using Java print hi world and hi tree.
Using Java print hi world and hi tree.
Using Java print hi world and hi tree.

-n: quiet output p: substitute flag to print matching line

Note, combine with grep to debug is good

Using Address

The sed editor assigns the first line in the text stream as line number 1 and continues sequentially for each new line.

only replace 2rd line:

sed -e '2s/hello/hi/g' sedtxt

Using Java print hello world and hello tree.
Using Java print hi world and hi tree.
Using Java print hello world and hello tree.
Using Java print hello world and hello tree.

range substitution, '1,$s/hello/hi/g' means from top to bottom.

sed -e '2,3s/hello/hi/g' sedtxt

Using Java print hello world and hello tree.
Using Java print hi world and hi tree.
Using Java print hi world and hi tree.
Using Java print hello world and hello tree.

can also use text pattern to filter lines, this will apply one line contains print word.

sed -e '/print/s#and#or#' sedtxt

Using Java print hello world or hello tree.
Using Java print hello world or hello tree.
Using Java print hello world or hello tree.
Using Java print hello world or hello tree.

Deletion

Delete consecutive lines after match

I have a text file seddel:

The 1st line is 1
The 2rd line is 2
The 3rd line is 3
The 4th line is 4

Delete line 2 to end:

sed -e '2,$d' seddel

The 1st line is 1

can also use pattern matching, delete 3rd

sed -e '/3rd/d' seddel

The 1st line is 1
The 2rd line is 2
The 4th line is 4

You can combine 2 address syntax, this will start from first line and until match 3rd, replace 3rd in the range with NAN:

sed -e '1,/3rd/{s/3rd/NAN/g}' seddel

The 1st line is 1
The 2rd line is 2
The NAN line is 3
The 4th line is 4

Delete commented and empty line

sed -e '/^#/d; /^$/d' <file>

Insertion and Appending

The insert command (i) adds a new line before the specified line. The append command (a) adds a new line after the specified line.

I have a text file sedins:

The 1st line is 1
The 2rd line is 2
The 3rd line is 3
The 4th line is 4

Insert at first line:

sed -e '1iNew line coming!' sedins

New line coming!
The 1st line is 1
The 2rd line is 2
The 3rd line is 3
The 4th line is 4

Append at 3rd line:

sed -e '2aNew line coming!' sedins
# using regexp
sed -e '/The 2rd line is 2/a New line coming!' sedins

The 1st line is 1
The 2rd line is 2
New line coming!
The 3rd line is 3
The 4th line is 4

Insert with white spaces

For example, When developing non-root, I want to add runAsUser: 1000 right after securityContext: with correct alignment:

sed -i -e '/securityContext/a\         runAsUser: 1000' xxx.yml

Only to escape the first space. sed can automatically recognize the rest of the spaces.

...
       securityContext:
         runAsUser: 1000
         privileged: false
...

Changing

The change command allows you to change the contents of an entire line of text in the data stream.

I have a text file sedch:

The 1st line is 1
The 2rd line is 2
The 3rd line is 3
The 4th line is 4

change the second line:

sed -e '2cNONE' sedch
## or
sed -e '/2rd/cNONE' sedch

The 1st line is 1
NONE
The 3rd line is 3
The 4th line is 4

Transforming chars

The transform command (y) is the only sed editor command that operates on a single character.

I have a text file sedtrans:

The 1st line is 1
The 2rd line is 2
The 3rd line is 3
The 4th line is 4

The transform command performs a one-to-one mapping of the inchars and the outchars values.

sed -e 'y/1234/5678/' sedtrans

The 5st line is 5
The 6rd line is 6
The 7rd line is 7
The 8th line is 8

The transform command is a global command; that is, it performs the transformation on any character found in the text line automatically, without regard to the occurrence. You can’t limit the transformation to a specific occurrence of the character.

sed files in directory and subdirectories recursively

Actually we can find all files by find then exec sed, see this post:

find <dir> -type f -name "*sh" -exec sed -i -e "s|${old}|${new}|g" {} \;

Note that -type f is necessary, otherwise will pass directory name to sed.