I want to offline install some rpms for an application, I put all dependencies for that application in a dedicated directory. The problem is it will cause conflicts with the old installed ones, I also want to keep old existing rpms because they may needed by other packages. For example, I offline install bind-utils use command:
1
yum --disablerepo=* install -y ./bind-utils/*.rpm
Error output:
1 2 3 4 5 6 7 8 9 10 11
... Error: Package: 1:openssl-1.0.2k-12.el7.x86_64 (@anaconda/7.5) Requires: openssl-libs(x86-64) = 1:1.0.2k-12.el7 Removing: 1:openssl-libs-1.0.2k-12.el7.x86_64 (@anaconda/7.5) openssl-libs(x86-64) = 1:1.0.2k-12.el7 Updated By: 1:openssl-libs-1.0.2k-16.el7.x86_64 (/openssl-libs-1.0.2k-16.el7.x86_64) openssl-libs(x86-64) = 1:1.0.2k-16.el7 ... You could try using --skip-broken to work around the problem ** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows: mokutil-15-1.el7.x86_64 is a duplicate with mokutil-12-1.el7.x86_64
This error shows that yum try to update old rpm with new one but this breaks the dependency chain. Option --skip-broken won’t work here, it will skip the dependency-problem rpm which include exactly what I need:
1 2
# skipped bind-utils.x86_64 32:9.9.4-73.el7_6
Then I try to use:
1
rpm -ivh ./bind-utils/*.rpm
still bad with conflicts:
1 2 3
... file /usr/lib64/openssl/engines/libcapi.so from install of openssl-libs-1:1.0.2k-16.el7.x86_64 conflicts with file from package openssl-libs-1:1.0.2k-12.el7.x86_64 ...
Solution
After doing research I find some rpm options may help:
This upgrades or installs the package currently installed to a newer version. This is the same as install, except all other version(s) of the package are removed after the new package is installed.
This will upgrade packages, but only ones for which an earlier version is installed. ... --force Same as using --replacepkgs, --replacefiles, and --oldpackage. --replacepkgs Install the packages even if some of them are already installed on this system. --replacefiles Install the packages even if they replace files from other, already installed, packages. --oldpackage Allow an upgrade to replace a newer package with an older one.
Let’s add --force flag and try again, this works and the old rpms are still there:
warning: 416b2856f8dbb6f07a50a46018fee8596479ebc0eaeec069c26bedfa29033315-kubeadm-1.13.2-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY error: Failed dependencies: cri-tools >= 1.11.0 is needed by kubeadm-1.13.2-0.x86_64 kubectl >= 1.6.0 is needed by kubeadm-1.13.2-0.x86_64 kubelet >= 1.6.0 is needed by kubeadm-1.13.2-0.x86_64 kubernetes-cni >= 0.6.0 is needed by kubeadm-1.13.2-0.x86_64
This upgrades or installs the package currently installed to a newer version. This is the same as install, except all other version(s) of the package are removed after the new package is installed.
This will upgrade packages, but only ones for which an earlier version is installed.
Both rpm -Fvh and rpm -Uvh will perform the same task but the diff is rpm -Uvh is also same as rpm -ivh, you can use any of them I mean rpm -ivh or rpm -Uvh for installing the package.
But for upgrading installed package you can use any of rpm -Fvh or rpm -Uvh.
rpm -Fvh is used for upgrading the existing package (installed package).
rpm -Uvh is used for installing the package and upgrading the package both.
For example, upgrade ansible from 2.4.6.0 to 2.7.8:
awk is its own programming language itself and contains a lot of really good tools, enables a programmer to write tiny but effective programs in the form of statements that define text patterns that are to be searched for in each line of a document and the action that is to be taken when a match is found within a line.
## FS is used to specify delimiter to parse line, by default awk use space BEGIN { FS=":" ; print"User Name:"} ## $3 > 999 is the condition match ## NR is internal variable of awk $3 > 999 {print NR, $0; count++ } END {print"Total Lines: " NR " Count Lines: " count}
Let’s see more examples, actually sed may perform the same task but awk is more readable.
1 2 3
## set "," as delimiter, $1 to uppercase, $2 to lowercase ## toupper and tolower is awk internal functions awk -F","'{print toupper($1), tolower($2), $3}' <file>
lastlog.awk file to show non-root user login statistics
1 2 3 4 5 6 7 8 9 10 11 12 13
## exclude if match these: !(/Never logged in/ || /^Username/ || /^root/) { cnt++ ## line fields == 8 if (NF == 8) printf"%8s %2s %3s %4s\n", $1, $5, $4, $8 else printf"%8s %2s %3s %4s\n", $1, $6, $5, $9 } END { print"===============================" print"Total # of user processed: " cnt }
I need to clean yum pending or unfinished transactions before our installer start to work, otherwise the yum update or yum install may fail. But where are these pending transactions from? Sometimes the machine is down or unexpected thing happens, the yum installation process failed.
Problem
you may see error like this:
1 2 3
There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them. The program yum-complete-transaction is found in the yum-utils package.
Solution
According to the prompt, we need first install yum-utils
1
yum install -y yum-utils
yum-complete-transaction is a program which finds incomplete or aborted yum transactions on a system and attempts to complete them. It looks at the transaction-all* and transaction-done* files which can normally be found in /var/lib/yum if a yum transaction aborted in the middle of execution.
If it finds more than one unfinished transaction it will attempt to complete the most recent one first. You can run it more than once to clean up all unfinished transactions.
Then just issue the following command to do a cleanup:
1
yum-complete-transaction --cleanup-only
You can also check how many pending transactions exist:
Now, let’s practice what we have learned from Offline package Installation I. For example, I want to install docker and kubeadm etc offline in the target machine.
Docker
Note: here we only download actual dependencies need for installation, not all rpms if we use --installroot option
I want to install Docker 18.06.3 (currently kubeadm now properly recognizes Docker 18.09.0 and newer, but still treats 18.06 as the default supported version). You should perform below steps on a machine that hasn’t installed docker yet.
Note: install from a package, rpms list in this link are not complete, they are in top level but can be used to upgrade version.
The contents of /var/lib/docker/, including images, containers, volumes, and networks, are preserved. The Docker CE package is now called docker-ce.
Set up docker repository
Before you install Docker CE for the first time on a new host machine, you need to set up the Docker repository. Afterward, you can install and update Docker from the repository.
Install a specific version by its fully qualified package name, which is the package name (docker-ce) plus the version string (2nd column) starting at the first colon (:), up to the first hyphen, separated by a hyphen (-). For example, docker-ce-18.06.3.ce.
When I was working on the upgrade DS k8s installer issue, I ran into the problem that I need to install ansible, docker and kubeadm offline. In the production environment, we may not have internet access, that means we need to prepare the rpms and dependencies needed and create a self-contained installer.
Download missing rpms without installing
Note: This method is (by-design) sensitive to the existence of already-installed packages. It will only download missing dependencies you need for that particular box, not all rpms.
For example, I want to get missing rpms for vim editor, reside them in /root/vim folder
1 2
mkdir -p /root/vim yum install --downloadonly --downloaddir=/root/vim vim
List the target folder:
Another way is using yumdownloader that is from yum-utils. The difference is if the package is already installed completely, yumdownloader will download the outermost level rpm but --downloadonly will do nothing.
1
yum install -y yum-utils
1
yumdownloader --resolve --destdir=/root/vim vim
Download all rpms without installing
yum & yumdownloader
Usually what we really want is to resolve all dependencies and download them, even though some required rpms have already installed in box, yumdownloader or yum --downloadonly with --installroot option is the solution.
Keep in mind that yumdownloader will use your yum database when resolving dependencies.
For example if you download bash, which needs glibc, it will resolve glibc and skip it, since it is installed. If you want to download all dependencies, use a different installroot instead.
This is what I need for a self-contained offline installer.
Let’s check how many vim related rpms are here, way too many then what we get from the first section.
1 2
ls -ltr /root/vim | wc -l 57
repotrack
This method can also resolve and download all dependencies, repotrack is from yum-utils, it will down all the dependencies for any architecture by default.
if you check /root/vim, there are some i686 rpms, once you delete them and count again, 57 the same as we use yumdownloader above.
Note: actually repotrack has -a option to specify arch, but I am not able to use it, when I specify x86_64, it still downloads i686.
Install local rpms
Now the problem is how to install these rpms in correct order, install them one by one is obviously infeasible, the method that can resolve their dependencies and install automatically is welcome, both command like:
may work but it’s not a good way, you may encounter rpm version upgrade issue and duplicate problem. Now from my knowledge create a local yum repository is clean and elegant, please refer my blog Set up and Use Local Yum Repository.
This blog reformats and builds on top of this stackoverflowtopic.
Big thanks to rahmu and people contributed.
Problem
Let’s say the command conky stopped responding on my desktop, and I want to
kill it manually. I know a little bit of Unix, so I know that what I need to do
is execute the command kill <PID>. In order to retrieve the PID, I can use
ps or top or whatever tool my Unix distribution has given me. But how can I
do this in one command?
DISCLAIMER: This command only works in certain cases. Don’t copy/paste it in
your terminal and start using it, it could kill processes unsuspectingly.
Rather learn how to build it.
How it works
ps aux
This command will output the list of running processes and some info about them.
The interesting info is that it’ll output the PID of each process in its 2nd
column. Here’s an extract from the output of the command on my box:
As you can see in step 2, the command ps outputs the grep conky process in its
list (it’s a running process after all). In order to filter it, I can run
grep -v grep. The option -v tells grep to match all the lines excluding the
ones containing the pattern.
Now that I have isolated my target process. I want to retrieve its PID. In other
words I want to retrieve the 2nd word of the output. Lucky for me, most (all?)
modern unices will provide some version of awk, a scripting language that does
wonders with tabular data. Our task becomes as easy as print $2.
I have the PID. All I need is to pass it to kill. To do this, I will use
xargs.
xargs kill will read from the input (in our case from the pipe), form a command
consisting of kill <items> (<items> are whatever it read from the input),
and then execute the command created. In our case it will execute kill 1948.
Mission accomplished.
Final words
Note that depending on what version of unix you’re using, certain programs may
behave a little differently (for example, ps might output the PID in column
$3). If something seems wrong or different, read your vendor’s documentation
(or better, the man pages). Also be careful as long pipes can be dangerous.
Don’t make any assumptions especially when using commands like kill or rm.
For example, if there was another user named ‘conky’ (or ‘Aconkyous’) my command
may kill all his running processes too!
More information please see man bash, it has comprehensive information.
This blog collects the commonly used code snippets based on my daily work, also do summary from related stackoverflow topics.
set builtin
Usually I use set -x for debugging purpose, today I see a new statement set -ex. What is this and what is set in Bash? 后来又知道了很多，见awesome list中的bash tutoral.
The Set Builtin, in short, set allows you to change the values of shell options and set the positional parameters, or to display the names and values of shell variables.
set -e, causes the shell to exit if any subcommand or pipeline returns a non-zero status. This tells bash that it should exit the script if any statement returns a non-true return value. The benefit of using -e is that it prevents errors snowballing into serious issues when they could have been caught earlier.
readlink -f $0 will follow every symlink in every component of the given name recursively and get the canonical path. A single file existing on a system can have many different paths that refer to it, but only one canonical path, canonical gives a unique absolute path for a given file. That means even though you call a script in it’s current directory, readlink -f $0 will give you the absolute path!
dirname $0 cut the script name to get the calling path, the path is relative not absolute.
run script in it’s driectory
Sometimes we want to run script in it’s folder by ./xxx.sh. we can check that:
1 2 3 4 5
SCRIPT_PATH=$(dirname$0) if [[ "X""${SCRIPT_PATH}" != "X." ]]; then LogMsg "###### ERROR: Please run this script in it's directory!" exit 1 fi
create tmp file to store log
Create a temporary file or directory, this temp file is owned and grouped by the current user. Aside from the obvious step of setting proper permissions for files exposed to all users of the system, it is important to give temporary files nonpredictable filenames, for example:
1 2 3 4
# $$: current PID OUT_FILE=/tmp/$(basename$0).$$.$RANDOM$RANDOM # or OUT_FILE=$(mktemp /tmp/log.$$.XXXXXXXXX)
For regular use, it may be more wise to avoid /tmp and create a /tmp under its home.
it will randomly generate 6 characters to replace XXXXXX. You may need to delete the tmp file when script exits, for example, use trap:
1 2 3 4 5 6 7 8
function exitHook { rm -f $OUT_FILE rm -f ${OUT_FILE}.yml rm -f ${OUT_FILE}.out rm -f ${OUT_FILE}.err } ## must put at beginning of script trap exitHook EXIT
The test command, it can be written as[] or test expression, [[ ]] is modern format, it supports regular expression =~ for string. which one if preferred: test is traditional (and part of the POSIX specification for standard shells, which are often used for system startup scripts), whereas [[ ]] is specific to bash (and a few other modern shells). It’s important to know how to use test since it is widely used, but [[ ]] is clearly more useful and is easier to code, so it is preferred for modern scripts.
1 2 3 4
## don't double quote regexp if [[ "$name" =~ colou?r ]]; then echo"..." fi
INT=-3 if [ -z "$INT" ]; then echo"INT is empty." >&2 exit 1 fi if [ "$INT" -lt 0 ]; then echo"INT is negative." else echo"INT is positive." fi if [ $((INT % 2)) -eq 0 ]; then echo"INT is even." else echo"INT is odd." fi
# or using (()) if ((1)); thenecho"It is true."; fi if ((0)); thenecho"It is true."; fi
# 需要注意(()) 中的变量不再需要expansion symbol $了，直接用变量名 declare -i day=30 if (( day > 0 || day < 31 )); then echo"day is good" fi
if [[ "$INT" -ge "$MIN_VAL" && "$INT" -le "$MAX_VAL" ]] # same as test if [ "$INT" -ge "$MIN_VAL" -a "$INT" -le "$MAX_VAL" ] # note in test need escape if [[ ! ("$INT" -ge "$MIN_VAL" && "$INT" -le "$MAX_VAL") ]] if [ ! \( "$INT" -ge "$MIN_VAL" -a "$INT" -le "$MAX_VAL" \) ]
Since all expressions and operators used by test are treated as command arguments by the shell (unlike [[ ]] and (( )) ), characters that have special meaning to bash, such as <, >, (, and ), must be quoted or escaped.
一种是外部使用的, provided by bash, for example: [[ ]] && [[ ]] || [[ ]], [[ ! xxx ]]. They obey short circuit rule.
Tips: 对于简单的if-condition, 可以替换为形如:
1 2 3 4 5 6 7
# chaining commands [ -r ~/.profile ] && . ~/.profile cat ~/.profile && echo"this is profile" || echo"failed to read profile" test -f "$FILE" && source"$_" || echo"$_ does not exist" >& 2 [ ! -r "$FILE" ] && { echo"$FILE is not readable" ; exit 1 } # parameters expansion 甚至都不需要if-condition ${var:="hello"}
select loop
The select loop provides an easy way to create a numbered menu from which users can select options. It is useful when you need to ask the user to choose one or more items from a list of choices.
Note that this loop was introduced in ksh and has been adapted into bash. It is not available in sh.
# PS3 is designed for select command PS3="Enter your choice (must be a number): " select DRINK in tea cofee water juice appe all none do # After a match is found, no further matches are attempted. # don't need the double quote # the pattern match is the same as pathname expansion # for example: ???) [[:alpha:]]) *.txt) case$DRINKin tea | cofee | water | all) echo"Go to canteen" break ;; juice|appe) echo"Available at home" break ;; none) break ;; # match anything at last *) echo"ERROR: Invalid selection" ;; esac done
When select you can use index number or literal, if no break, it will loop forever.
If want case to match more than one terms, use ;;& instead of ;; at end of each case. The addition of the ;;& syntax allows case to continue to the next test rather than simply terminating.
input password and confirm
Must not show password user input:
1 2 3 4 5 6 7 8 9 10 11 12
echo"****************************************************************" echo"Please input the password:" echo"****************************************************************" whiletrue; do read -s -p "PASSWORD: " PASSWORD echo read -s -p "CONFIRM: " PASSWORD_CONFIRM echo [ ${#PASSWORD} -lt 6 ] && echo"The length of password at least 6, please try again" && continue [ "${PASSWORD}" = "${PASSWORD_CONFIRM}" ] && break echo"Passwords do not match please try again..." done
[[ "X$P1" = "X" ]] && exit 1 [[ "X$P2" = "X" ]] && exit 1
Note there are 2 shift in one case, after each shift, $# minus 1.
function
The function refers to passed arguments by their position (not by name), that is $1, $2, and so forth. $0 is the name of the script itself.
1 2 3 4 5 6 7 8 9
functionexample() { ## local var prevent var leaking to shell local first=$1 local second=$2 ## return code is similar to exit code but this is return ## will break the rest execution return <return code> }
Need to call your function after it is declared.
1 2 3 4 5
example "p1""p2"
args #0 is <absolute path to script itself> args #1 is p1 args #2 is p2
Show functions:
1 2 3 4 5 6
## list all function names declare -F ## show definition declare -f [function name] ## clear a function unset -f <function name>
Export functions, to make it available to subshells, similarly to export variables:
1 2
## -xf: export a function declare -xf <function name>
Actually, this style [INFO] [2019-10-11 15:59:26-0081] ... it better.
check last command result
1 2 3 4 5 6 7
echo_success_failure() { if [ $? -eq 0 ]; then LogMsg "###### INFO: Success..." else LogMsg "###### INFO: Failure..." fi }
run as root
1 2 3 4 5
effective_uid=`id -u` 2>/dev/null if [ $effective_uid -ne 0 ]; then LogMsg "###### ERROR: Please run this script as root or sudo" exit 1 fi
IFS and read array
The default value of IFS contains a space, a tab, and a newline character.
Convert string to array with specific delimiter, for example:
1 2 3 4 5 6 7
string="item1:item2:item3" # <<<: is here string, the same as here doc but shorter single string OLD_IFS=$IFS IFS=':'read -a array <<< "${string}" # or using process substitution IFS=':'read -a array < <(echo"${string}") IFS=$OLD_IFS
This version has no globbing problem, the delimiter is set in $IFS (here is space), variables quoted. Don’t forget to do sanity check after converting.
Actually if the string use spaces as delimiter, we can loop items directly:
1 2 3 4 5
string="item1 item2 item3" for i in${string} do echo${i} done
loop array
break and continue can be used on loop. 还要注意当do 写在下一行的时候，do前面不需要;.
1 2 3 4 5
declare -a array=("element1""element2""element3") for i in"${array[@]}" do echo"${i}" done
declare or typeset are an explicit way of declaring variable in shell scripts.
In BASH it is safer to quote the variable using "" for the cases when $i may contain white spaces or shell expandable characters.
If you want to use index of array element
1 2 3 4 5 6 7 8 9
# get length of an array arraylength=${#array[@]}
# use for loop to read all values and indexes for (( i=0; i<${arraylength}; i++ )) do ## ${array[$i]} 这里注意，先解析的$i echo$i" / "${arraylength}" : "${array[$i]} done
If we use declare to define a integer variable:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
declare -i x=10 while (( x > 0 )) do echo$x ## no need to use 'let x=x-1' ## because x is type integer x=x-1 done
# true loop 3 种写法 whiletrue | while ((1)) | while : do ## pass done
Until loop continues until it receives a zero exit status.
1 2 3 4 5 6
count = 1
until [[ "$count" -gt 5 ]]; do echo"$count" count=$((count + 1)) done
In ZSH shell, you can use foreach loop:
1 2 3 4
## () is a must foreach item (`ls /tmp`) echo$item end
# read 3 fields a line, line by line from distros.txt file # note that < is placed after done, it is the input for loop whileread distro version release; do printf"Distro: %s\tVersion: %s\tReleased: %s\n" \ "$distro" \ "$version" \ "$release" # no need cat here done < distros.txt # or done < <(cat distros.txt)
# can also pipeline input to a loop # while and read is running on subshell sort -k 1,1 -k 2n distros.txt | whileread distro version release; do printf"Distro: %s\tVersion: %s\tReleased: %s\n" \ "$distro" \ "$version" \ "$release" done
# using process substitution # list last 3 lines of dir whileread attr links owner group size datetime filename; do cat << EOF Filename: $filename Size: $size EOF done < <(ls -ltrh | tail -n +2)
Read can read from keyboard input or file or pipeline: read [-options] [variables...]. If no variable name is supplied, the shell variable $REPLY contains the line of data. If read receives fewer than the expected number, the extra variables are empty, while an excessive amount of input results in the final variable containing all of the extra input.
1 2 3 4 5 6 7 8 9 10 11 12 13
# pass parameters to read command # must stick to this format echo"admin 123456" | ./script.sh
# receive code snippet in script.sh # ${username} ===> admin # ${password} ===> 123456 echo -n "Please enter username -> " read username echo -n "Please enter an password -> " # -s: silent read -s password
Other options:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
# -p: prompt read -p "Enter one or more values > " echo"REPLY = '$REPLY'"
# -e: pair with -i # -i: default valut passed to read read -e -p "What is your user name? " -i $USER echo"REPLY = '$REPLY'"
setup ssh password-less
Idempotence：
1 2 3 4 5 6
ssh-keyscan -H ${remote} >> ~/.ssh/known_hosts sshpass -p "<password>" ssh-copy-id -i ~/.ssh/id_rsa.pub root@${remote} if [[ $? -ne 0 ]]; then LogMsg "######ERROR: Something went wrong with ssh-copy-id. Check for incorrect credentials ... " exit 1 fi
recursive call
1 2 3 4 5 6 7 8
example() { <execute sth> if [[ $? -ne 0 ]]; then LogMsg "######ERROR: Something went wrong… " example fi }
tee command
tee command reads the standard input and writes it to both the standard output and one or more files, -a flag used to append output to existing file, if no -a, tee will create the file if not exist.
1 2 3 4 5
LogMsg() { logMsg="$@" echo"["`date +"%Y/%m/%d %r"`"]"${logMsg} | tee -a logs/ds_${stage}_${timeStamp}.log }
#!/usr/bin/env bash # this script will do sth after reboot # in /root/completeme.sh # then restore /etc/profile #################################################
echo"Warning! This script is going to reboot now to complate the procedure" echo"After reboot, login as root to perform the final steps" echo"Press Ctrl-C now to stop this script in case you don\'t want to reboot"
chmod +x /root/completeme.sh cp /etc/profile /etc/profile.bak ## after reboot /etc/profile will be executed so /root/completeme.sh echo /root/completeme.sh >> /etc/profile reboot
## to increase CPU load ## dd if=/dev/zero of=/dev/null ## or use stress command!
whilesleep 60 do ## to remove header of ps output, append `=` or user --no-headers flag ## CPU$ 0.0 will be in part if CPU$ > 0.0 REC=`ps -eo pcpu= -o pid= -o comm= | sort -k1 -n -r | head -1` USAGE=`echo$REC | awk '{print $1}'` ## truncate decimal part USAGE=${USAGE%.*} PID=`echo$REC | awk '{print $2}'` PNAME=`echo$REC | awk '{print $3}'`
# Only if we have a high CPU load on one process, run a check within 7 seconds # In this check, we should monitor if the process is still that active # If that's the case, root gets a message
## man test if [ $USAGE -gt 80 ] then USAGE1=$USAGE PID1=$PID PNAME1=$PNAME sleep 7 REC=`ps --no-headers -eo pcpu,pid -o comm= | sort -k1 -n -r | head -1` USAGE2=`echo$REC | awk '{print $1}'` USAGE2=${USAGE2%.*} PID2=`echo$REC | awk '{print $2}'` PNAME2=`echo$REC | awk '{print $3}'` # Now we have variables with the old process information and with the # new information
[ $USAGE2 -gt 80 ] && [ $PID1 = $PID2 ] && mail -s "CPU load of $PNAME is above 80%" root@blah.com < . fi done
I want to introduce you Git LFS, it is a command line extension and specification for managing large files with Git. LFS is great for large, changing files, there is basically a text pointer to the large file archived some place else.
Usually we store large file or object in artifactory, for example: jFrog, Nexus, etc.
Install Git LFS
Note: you need to install Git LFS if you git pull from a remote repository that has it
For example, I am working on a RHEL machine.
First go to source page, follow the installation guide to install:
This will create a yum repos for git-lfs:
1
yum install -y git-lfs
you can see git-lfs is installed in your machine:
Once downloaded and installed, set up Git LFS and its respective hooks by running:
1
git lfs install
Note: You’ll need to run this in your repository directory, once per repository.
Track Large File
Select the file types you’d like Git LFS to manage (or directly edit your .gitattributes). You can configure additional file extensions at anytime.
1
git lfs track "*.tar.gz"
Note: run this track command at the top level of your repository, then you need to git add .gitattributes file
Manage Large File
Then, just do normal git add and git commit to manage your large size file.
This article used to walk you through some commonly tar usages , based on a real life scenario.
################################################################
# Date Description
# 05/29/2019 vim tar files
# 05/29/2019 extract single file to another directory
# 05/28/2019 extract file to another directory
# 05/23/2019 extract single file from archive
# 04/21/2019 untar keep owner and permission
# 02/27/2019 untar to specified folder
# 02/22/2019 list tar content
# 02/21/2019 tar exclude
# 02/20/2019 untar multiple files
# 02/19/2019 tar multiple files
#
################################################################
Sometimes I see people use -czf but sometimes czf, dash or not to pass flags?
Historical and compatible reason, no dash version is probably more portable.
tar is one of those ancient commands from the days when option syntax hadn’t been standardized. Because all useful invocations of tar require specifying an operation before providing any file name, most tar implementations interpret their first argument as an option even if it doesn’t begin with a -. Most current implementations accept a -.
注意, 这里的例子大多是用的old option style for compatibility. For example czf this set of letters must be the first to appear on the command line, after the tar program name and some white space; old options cannot appear anywhere else.
Basic operation: tar multiple files into example.tar.gz
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
## use -C to go to target directory ## target directory: the directory which contains file1/2/3 tar czf example.tar.gz -C <target directory> file1 file2 file3
## tar a directory as whole ## target directory: <folder name>'s parent folder ## untar 结果是<folder name>这个文件夹 tar czf example.tar.gz -C <target directory> <folder name>
For example, if you want to tar things inside a folder foler1 but excluding some files:
1 2 3 4 5 6
## 注意最后的`.` 目标必须放最后 ## target directory: 进入 cd folder1 tar czf folder1.tar.gz --exclude="folder1.tar.gz" --exclude='file1' --exclude='file2' * ## if you want to have hidden files tar czf folder1.tar.gz --exclude="folder1.tar.gz" --exclude='file1' --exclude='file2' * .file3 .file4
If you don’t exclude folder1.tar.gz, it will tar itself again.
02/22/2019
List tar.gz file content, flag z is used to distinguish tar and tar.gz
1 2
tar tvf target.tar tar ztvf target.tar.gz
02/27/2019
If you don’t specify target folder, untar will put things in current directory, use -C option to specify it. For example, I want to untar source.tar.gz to /etc/yum.repos.d/ folder:
1
tar zxf /tmp/source.tar.gz -C /etc/yum.repos.d/
For -C option, in c and r mode, this changes the directory before adding the following files. In x mode, change directories after opening the archive but before extracting entries from the archive.
04/21/2019
When unpacking, consider using p option to perserve file permissions. Use this in extract mode to override your umask and get the exact permissions specified in the archive… The p option is the default when working as the superuser, it will get what it has. If you are a regular user, add p to keep permissions.
1
tar zxpf target.tar.gz
It seems umask ignores execute bit? When I untar the file with rwxrwxrwx permission inside by regular user with umask 0002, the final permission is rwxrwxr-x.
if you want to keep owner as well:
1
tar --same-owner -zxpf target.tar.gz
Note that there is a - before zxpf.
05/23/2019
Extract specific files from tarball to current directory:
1
tar xzf target.tar.gz file1 file2
Note that no leading / in the path (it uses relative path in tar file!), you can use tar xtvf target.tar.gz to check the path.
05/28/2019
tar by default extracts file to current directory, if you want to place the untar files to another directory, run:
1
tar zxf target.tar.gz -C /target/directory
Note that the target directory has to exist before running that command.
05/29/2019
If you want to extact files to another directory:
1 2
## file1 and file2 put at end tar xzf target.tar.gz -C /target/directory file1 file2
11/12/2020
Latest VIM support edit on tar file:
1 2
## then select file in dashboard, edit and save normally vim source.tar.gz