Miscellanea 2019

Posted on Edited on In

This series contains something that is too short to be a blog, so put them all here, chronologically.

02/25/2019

  • hostPath in PersistentVolume is the mount path in host machine. mountPath in containers field is the mount path inside the container.

02/27/2019

  • Docker uses /var/lib/docker to store your images, containers, and local named volumes. Deleting this can result in data loss and possibly stop the engine from running. The overlay2 subdirectory specifically contains the various filesystem layers for images and containers.

  • Vim readonly mode, can open the same file in multiple windows:

    1
    vim -R file

02/28/2019

  • reboot machine rightnow, -r means reboot, for example:

    1
    shutdown -r now

    If you execute remotely, use ssh example.com to test if it bring up.

  • Jenkins: the exit code of last command of the Jenkin’s Execute Shell build step is what determines the success/failure, now it’s better to wrap the code snippet as a script and execute it. Need to do more search on it.

03/02/2019

  • For ls command: If no operands are given, the contents of the current directory are displayed. If more than one operand is given, non-directory operands are displayed first; directory and non-directory operands are sorted separately and in lexicographical order.

    I use this feature with tail command to pick latest package, for example:

    1
    ls | grep ansible-* | tail -1

  • ansible has log_path setting in ~/.ansible.cfg file, for example:

    1
    2
    [defaults]
    log_path = /ibm-test/DS-Kube-Installer/logs/ds_installer_20190301_1645.log

03/04/2019

  • Gluster file system with RedHat?

  • when run systemctl start docker, these directories are created: /var/lib/docker, /run/docker, etc/docker.

03/05/2019

  • Find Red Hat or CentOS version:
    1
    2
    3
    cat /etc/os-release

    Red Hat Enterprise Linux Server release 7.6 (Maipo)

03/13/2019

  • I see people sometimes use /bin/cp, /bin/rm in script, why they don’t use cp or rm directly? The answer is cp or rm may be an alias in target machine! For example:
    1
    2
    3
    alias cp='cp -i'
    alias mv='mv -i'
    alias rm='rm -i'
    So when use cp -f source target it will still prompt you the overwrite confirm if target and source are the same. /bin/cp -f source target is correct way to go.

03/16/2019

These are from Ansible: Up and Running, 2nd Edition book:

  • Gunicorn: a Python WSGI HTTP Server for UNIX.

  • Markdown table generator

  • Let’s Encrypt: a free, automated, and open Certificate Authority.

  • Celery: a distributed task queue.

  • RabbitMQ: open source message broker

  • A staging environment (stage) is a nearly exact replica of a production environment for software testing. Staging environments are made to test codes, builds, and updates to ensure quality under a production-like environment before application deployment.

03/17/2019

These are from Ansible: Up and Running, 2nd Edition book:

  • Mezzanine: similar in spirit to WordPress. Mezzanine is built on top of Django, the free Python-based framework for writing web applications.

  • Fabric: a Python-based tool that helps automate running tasks via SSH.

  • SQLite is serveless database: Most SQL database engines are implemented as a separate server process. Programs that want to access the database communicate with the server using some kind of interprocess communication (typically TCP/IP) to send requests to the server and to receive back results. SQLite does not work this way. With SQLite, the process that wants to access the database reads and writes directly from the database files on disk. There is no intermediary server process.

03/18/2019

  • Today after Fyre maintenance, one of my VM cannot resolve hostname, when I run
    1
    ping google.com
    it hangs, also nslookup doesn’t work as well. Let’s check /etc/resolv.conf file, it is good:
    1
    2
    3
    4
    ; generated by /usr/sbin/dhclient-script
    search fyre.ibm.com. svl.ibm.com.
    nameserver 172.16.200.52
    nameserver 172.16.200.50
    Then reboot VM again it works, sometimes Fyre generates weird problem.

03/22/2019

03/23/2019

03/25/2019

  • ELECTRON: Build cross platform desktop apps with JavaScript, HTML, and CSS

03/27/2019

  • This is from a issue I encountered: when we setup a NFS server as storage in K8s cluster with the /etc/exports file, we need to restrict the clients who is able to access the NFS mount instead of something like:

    1
    /data *(rw,insecure,async,no_root_squash)

    Correct way is to specify which NFS client can access:

    1
    2
    3
    /data example1.com(rw,insecure,async,no_root_squash)
    /data example1.com(rw,insecure,async,no_root_squash)
    /data example1.com(rw,insecure,async,no_root_squash)

    Here in ansible template, it uses lookup:

    1
    2
    3
    for host in {{ lookup('env','nfsclienthosts') }}; do
      echo "{{ dfsDataDir }} "$host"(rw,insecure,async,no_root_squash)" >> /etc/exports
    done

    I want to say be careful with the space in exports file, from RHEL NFS exports

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    Important

    The format of the /etc/exports file is very precise, particularly in 
    regards to use of the space character. Remember to always separate exported
    file systems from hosts and hosts from one another with a space character. 
    However, there should be no other space characters in the file except on 
    comment lines.

    For example, the following two lines do not mean the same thing:

    /home bob.example.com(rw)
    /home bob.example.com (rw)

    The first line allows only users from bob.example.com read and write access 
    to the /home directory. The second line allows users from bob.example.com 
    to mount the directory as read-only (the default), while the rest of the 
    world can mount it read/write.

03/28/2019

  • ps aux command will not show full outputs and the lines are truncated, if you are in a lightweight Linux distributions like BusyBox, you can try:
    1
    ps aux | cat
    otherwise try:
    1
    2
    ps auxw
    ps auxww

03/29/2019

  • open terminal run vimtutor, haha.

03/30/2019

  • see memory usage

    1
    free -h

  • clean swap space

    1
    swapoff -a && swapon -a

  • Nagios: open source Industry Standard In IT Infrastructure Monitoring

  • HAProxy:The Reliable, High Performance TCP/HTTP Load Balancer

  • /bin/false is a system command that is used anytime you need to pass a command to a program that should do nothing more than exit with an error. It’s the companion to /bin/true. Both of these are very old and standard POSIX utilities and neither produce any output by definition. true is sometimes used for a shell script that should loop indefinitely, like:

    1
    2
    3
    4
    5
    6
    7
    8
    while true; do
        ...
        # Waste time
        if [ $wasted_time -gt 100000 ]; then
            exit 0
        fi
        ...
    done

    /usr/sbin/nologin is specifically designed to replace a shell and produces output complaining you can’t log-in. Before it existed, it was common to use /bin/false for dummy users, but could be confusing since the user doesn’t know why they’re kicked off.

04/01/2019

  • Sometimes when I login to a user home, the prompt is like:
    1
    bash-4.2$
    instead of
    1
    [demo@myk8s1 ~]$
    the reason is .bash_history .bash_logout .bash_profile .bashrc under /home/demo are missing!

04/02/2019

  • docker bind mounts and volume, in our application, we use mount type in docker run command:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    docker inspect <container name>

    "Mounts": [
              {
                  "Type": "bind",
                  "Source": "/opt/builds",
                  "Destination": "/opt/builds",
                  "Mode": "",
                  "RW": true,
                  "Propagation": "rprivate"
              },
    ...

    I use busybox does a bind mount test, in the Dockerfile, I create a /tmp/bb folder and put test.txt file in it

    1
    2
    3
    4
    5
    6
    7
    8
    9
    FROM busybox
    MAINTAINER chengdol@ibm.com

    COPY ./hello.txt /
    RUN sh -c "mkdir /tmp/bb" && \
      touch /tmp/bb/test.txt && \
      sh -c "echo '123' > /tmp/bb/test.txt"

    CMD tail -f /hello.txt

    After build image and run as:

    1
    docker run -d --name mybb -v /tmp/bb:/tmp/bb mybb:v1

    Then I get into the container, the test.txt is missing. But if we first put something in the host /tmp/bb folder, it will show up inside container.

  • occasionally find a online drawing tool, draw.io, but this cloud service may not approved by company use.

04/03/2019

  • echo -n will disable the newline, echo -e will enable the escape, so if you run
    1
    echo -e "\n"
    it will output a newline

04/04/2019

  • Linux capability, how to know what capabilities a process required to work properly?

  • Linux user and group with permission problem

  • docker commit, commit what

  • docker save, docker export difference

  • suid rws bit field for file and t sometimes, link

  • usermod, specify or change home directory

  • useradd/userdel, groupadd/groupdel usage, passwd add password for user

  • sudo run process will be root USER

  • cp -p, don’t change permission, owner and timestamps of the file

04/09/2019

  • find files owned by particular user
    1
    find . -user "xxx"

###04/10/2019

  • check the parent of the process, show PPID (parent id) in ps command:

    1
    ps -efj

  • setuid only set when owner is root, and other user with permission to run the file will be the owner of that process:

    1
    -rwsr-xr-x 1 root  root  62 Apr 10 15:40 hang.sh

    if I’m demo user to run it, the process is owned by demo

    1
    demo     32403  0.0  0.0 113176  1388 pts/1    S+   10:13   0:00 /bin/bash ./hang.sh

    There is also setgid concept.

04/11/2019

  • copy with hidden files and directories

04/21/2019

  • global user start file: /etc/bashrc, the umask is inside it.

    Note that umask uses subtraction.

  • tools which preserve permissions apply the appropriate mode and ignore umask: cp -p, tar -p.

04/24/2019

  • I find sometimes I use grep -r XXX . cannot find the pattern in files in current and subdirectories. The reason is -r flag will not process symbolic link except it’s on the command link. you can use:
    1
    grep -Rn XXX .
    -R will follow symbolic links -n will show line number for each matched result -i make it case-insensitive -F used looking for fixed string to save time
    1
    grep -Rn --include "*.txt" XXX .
    if you know the pattern of the file, you can specify that using --include, you can also mention using --exclude option.

05/10/2019

  • scp from linux to windows machine

    1
    scp ~/Downloads/PXSmokeTest_outputs.dsx Administrator@indraniwindows1.fyre.ibm.com:

    the file will be put in C:\Users\Administrator> folder in windows.

  • How to scp files from remote host to container in k8s? inside the container, install openssh-clients

    1
    sudo yum install openssh-clients

    then just like normal:

    1
    sudo scp root@mycentctl1:/GitRepos/cognitive-designer-api/DSNexus_Build/Docker_Scripts/Kubernetes_11.7.DS/buildengine/opt/IBM/InformationServer/initScripts/* .

05/20/2019

  • Coordinated Universal Time (UTC) is 7 hours ahead of Pacific Time.

06/12/2019

  • grep exclude pattern use -v:
    1
    docker images | grep -E "xmeta|services|engine|compute" | grep -v "mycluster" | awk '{print $1}'
    this will exclude results have mycluster.

06/19/2019

  • workaround when you cannot find rpm or package to install in linux, download the binary and put it in working PATH, for example, to use jq, download binaries from here. Add executable bit and move to /usr/bin.

07/01/2019

  • check directory current used size:
    1
    2
    3
    # -s: total
    # -h: human
    du -sh <path to directory>
    if you want to know the partition size associated with this directory, for example /var/lib/docker, use
    1
    df -h /var

07/02/2019

  • change file or directory time stamp to 1969-12-31 16:00

    1
    touch -a -m -t 196912311600 xx.txt

    -a: change the access time of a file. By default it will take the current system time and update the atime field. -m: change the modification time of a file. -t: explicitly specify the time

    Check status by stat command:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    stat xx.txt

    File: `xx.txt'
    Size: 3         	Blocks: 8          IO Block: 4096   regular file
    Device: 801h/2049d	Inode: 394283      Links: 1
    Access: (0644/-rw-r--r--)  Uid: ( 1000/lakshmanan)   Gid: ( 1000/lakshmanan)
    Access: 2038-01-18 12:05:09.000000000 +0530
    Modify: 2038-01-18 12:05:09.000000000 +0530
    Change: 2012-10-19 00:40:58.763514502 +0530

    reference and update the time stamp of file a.txt from the time stamp of b.txt file

    1
    touch a.txt -r b.txt

    Change time stamp recursively

    1
    find . -type f -exec touch -a -m -t 196912311600 {} +

  • Convert format from DOS to UNIX: If you open a file via vim and see there are many ^M:

    1
    2
    3
    4
    5
    NOTICE^M
    ^M
    This document includes License Information documents below for multiple Programs. Each License Information document identifies the Program(s) to which it applies. Only those License Information documents for the Program(s) for which Licensee has acquired entitlements apply.^M
    ^M
    ^M

    This is because it’s DOS format:

    1
    2
    3
    file LA_en.ORIG

    LA_en.ORIG: ASCII text, with very long lines, with CRLF, LF line terminators

    how to convert to UNIX?

    1
    2
    yum install -y dos2unix
    dos2unix <file name>

07/12/2019

  • Previously I use
1
ls -ltr --block-size=M

to see the human readable size for each file, actually use

1
ls -ltrh

is enough!

07/15/2019

  • new tech word linting: the process of running a program that will analyse code for potential errors. For example, PHPLint, JSLint.

07/24/2019

  • if run script using sudo, for example:
    1
    sudo script.sh
    then every command in scipt is executed with sudo.

08/05/2019

  • find file owned by a particular user or group
    1
    find <path> -user <dsadm> -group <dstage>
    find file by case-insensitive name and use -ls format
    1
    find <path> -iname <name> -type f -ls
  • find particular files and change the chown or chmod
    1
    find <path> -user <dsadm> -group <dstage> -exec chmod 755 {} /;
    explain: chmod 755 {} \; specifies the command that will be executed by find for each file. {} is replaced by the file path, and the semicolon(;) denotes the end of the command (escaped, otherwise it would be interpreted by the shell instead of find).

08/09/2019

  • if var is not set, use default value 123456:
    1
    var=${var:-"123456"}

08/28/2019

  • docker commit will not apply chmod 777 / in new image, the permission mode of / directory is still original. Not sure why.

09/05/2019

  • curl can be used to verbose request in detail, to check the RESTful API content.

11/13/2019

  • uniq command, used to deduplicates

11/30/2019

  • https://distrowatch.com/ linux forum, contains lots of different linux distribtions and release informations.