Python3 Scripting

Posted on Edited on In

Why Python scripting? Easy to learn and write, interactively, powerful built-in data types and object oriented. Included packages to support huge range of tasks.

The demos of this blog: https://github.com/chengdol/python-scripting

Common Modules for Scripting

  • math: Trig functions, sqrt, logarithms, exponentials
  • pickle: Serialise / de-serialize objects for persistent storage
  • random: Generate random numbers with various distributions
  • re: Regular expression pattern matching and substitution
  • string: Comprehensive string formatting
  • configparser: Configuration file parser
  • bz2: gzip, read and write compressed files
  • tarfile: Read and write tar archives
  • datetime: Represent and manipulate dates and times
  • logging: Log message generator with various backends
  • argparse: Parser for command-line options
  • optparse: Parse command-line arguments
  • click: command-line argument toolkit, decorator
  • os: Interface to operating system services
  • sys: Access argv, stdin, stdout, etc.
  • socket: Python binding for the traditional BSD socket API
  • http: Modules for client and server side http, and cookies
  • shutil: Copy / remove files and directory trees
  • glob: Shell-style wildcard expansion
  • xml: Processing of XML data
  • hashlib: common interface to many hash functions
  • signal: single handling
  • subprocess: execute command by spawn new processes, connect to their input/output/error pipes, and obtain their return codes
  • shlex: parsing unix shell commands, for example, split long arguments
  • smtplib: email handling
  • threading: threading operations
  • timeit: measure executing time
  • pyyaml: parse yaml file
  • requests: simple http library
  • retrying: retrying flaky function
  • python-terraform: terraform wrapper

Work Environment

REPL

REPL: the interactive console. 这是最基本的一个python interactive shell, can be used for testing purpose.

IPython

ipython: the enhanced python interpreter, can run shell commands + REPL, make alias for arbitrary shell commands and with TAB completion.

How to install: yum install -y ipython3 Then in terminal, run ipython3

可以直接在ipython中运行比如ls -ltr, cd 之类的命令，这些都属于magic function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
## off magic
automagic
## then you need % prefix to run shell commands
%ls -ltr

## open magic again
%automagic
## no need % prefix
ls -ltr

## helper
ls?

## list env variables
whos

Create alias:

1
2
3
## create alias 'findsymlinks'
alias findsymlinks ls -l /etc/ | grep '^l'
%findsymlinks

Run and edit files in IPython:

1
2
3
run script.py
## default using vi or vim, $EDITOR
edit script.py

对于不能直接运行的shell commads, use shell escape with prefix !, similar to vim feature:

1
2
3
4
## can store the output to a variable
out = !df
for line in out.grep("tmpfs"):
    print(line)

IDLE

How to install: yum install -y idle3 To run idle3, you need desktop environment.

Managing File System

找能实现Bash中功能的函数就行。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
## walk around file system, wrap around Linux system call
import os

os.getcwd()
## cd /tmp
os.chdir("/tmp")
## current level dir list
os.listdir("/")
## recursively to subdir, will return a tuple
## (current dir name, child dirs, child files)
os.walk("/tmp")
os.stat("/etc/hosts")

os.mkdir(path [, mode])
os.rename(src, dst)
os.remove(file)
## remove empty dir
os.rmdir(dir)

## 0o644, prefixed with '0o'
os.chown(file, mode)
os.chmod(file, uid, gid)
## hardlink
os.link(src, dst)
## softlink
os.symlink(src, dst)

## high-level file operations
import shutil as st
st.copy(src, dst)
## also copy attr
st.copy2(src, dst)
## mv
st.move(src, dst)
## rm -f
st.rmtree(path)
st.copytree(src, dst, ignore=None)
## which
st.which("java")

st.make_archive(basename, format)
st.unpack_archive(filename, extrac_dir, format)

Interacting with Linux System

1
2
3
4
5
6
import sys

## print command line parameters
for arg in sys.argv[:]:
    print(f"{arg}", end = ' ')
print()

To parse parameters, use optparse module, see example in git repo 1_optparse-demo.py. Besides optparse and argparse from the standard library, click module is a good alternative.

To get env varaible:

1
2
3
4
5
6
import os

## set default value when empty response
os.getenv("EDITOR", "/usr/bin/vim")
os.getenv("HOME")
os.environ.get("HOME")

这节的git repo例子很有意思5_signal-primes-v5.py, 6_timeout.py, 用signal handler 去改变条件变量的值，从而改变运行逻辑。之前一直在BASH中用trap去做终止前的处理。Linux has 2 signals set aside for user: SIGUSR1, SIGUSR2.

Executing Commands

Run external commands, for example, call other shell commands or executables by subprocess, similar to linux ().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
import subprocess
import os

## 这是最简单的调用
## this will not disturb current env
env = os.environ.copy()
## the run method will call Popen under the nood
cmd = ["ls", "-ltr"]
## if does not set stdout/stderr, then command print result to console
process = subprocess.run(cmd, 
               env=env, 
               stdout=subprocess.PIPE, 
               stderr=subprocess.PIPE)


process.returncode
process.stdout.decode("utf-8")
process.stderr.decode("utf-8")


## 如果要运行和shell有关的命名且需要用到shell的特性 such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory.
## If shell is True, it is recommended to pass args as a string rather than as a sequence.
env = os.environ.copy()
cmd = "ls -ltr ~ | grep -i download"
process = subprocess.run(cmd, 
               shell=True,
               env=env, 
               stdout=subprocess.PIPE, 
               stderr=subprocess.PIPE)

## the same as
cmd = ["/bin/sh", "-c", "--", "ls -ltr ~ | grep -i download"]
process = subprocess.run(cmd,
               env=env, 
               stdout=subprocess.PIPE, 
               stderr=subprocess.PIPE)

process.returncode
process.stdout.decode("utf-8")
process.stderr.decode("utf-8")



## 高级操作 Popen, run 在背后调用的它
env = os.environ.copy()
## the same as export KUBECONFIG=clusters.yaml
env['KUBECONFIG'] = "clusters.yaml"
## this kubectl will refer KUBECONFIG env variable above
cmd = ["kubectl", "get", "sts"]
process = subprocess.Popen(cmd, 
                 env=env, 
                 stdout=subprocess.PIPE, 
                 stderr=subprocess.PIPE)

## out, err is byte type
out, err = process.communicate()
## conver to string
out = out.decode("utf-8")
err = err.decode("utf-8")

if process.returncode != 0:
  raise subprocess.CalledProcessError(process.returncode, cmd)

关于python concurrency专门的总结: Python Concurrency

String Manipulation

Besides basic operation, it talks about datetime:

1
2
3
4
5
6
7
8
9
10
11
12
import datetime

## depends on your local time zone
rightnow = datetime.datetime.now()
utc_rightnow = datetime.datetime.utcnow()

rightnow.month
rightnow.hour

## lots of % format
rightnow.strftime("Today is %A")
## datetime.timedelta, to add and decrease time slot

然后讲了re module. 可以参考这节的git code.

Processing Text, Logging

For long running background service, logging is a must, we can log events into: file, syslog or systemd journal.

Logger has different handlers, for example: StreamHandler(stdout, stderr), FileHanlder, watchFileHandler, SysLogHandler, SockerHandler, JournalHandler, etc.

Logging levels: NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL.