Python Packages and Virtual Environment

Posted on | Edited on | In

There is a better alternative, please see <<VSC Developing inside a Container>>.

To use and manage third-party libraries without messing up python environment, organizing different project that has its own unique dependencies:

  • pip: package management
  • virtualenv: project and dependencies
  • virtualenvwrapper: making venv more convenient

Does not talk the package that is the form of __init__.py under a folder, we are talking python distribution package.

Pip

Best practice:

  1. always work inside a virtual environment, keep things nice and clean.
  2. be careful using pip with sudo when install packages, otherwise the installation is system-wide.

Mac’s pre-installed Python is not meant for development, you can use Homebrew to install or download Python from python.org, that will go along with pip. For Linux, adhere to system manager to install pip or python. In Mac, try check if pip is there and it’s version.

To install pip(2/3) on Linux:

1
2
3
4
5
6
7
8
9
10
11
# search pip2 or pip3 package
sudo yum search python* | grep pip
# this will also install libtirpc
# python3, python3-libs, and python3-setuptools
sudo yum install python3-pip

pip3 -V
pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)

pip2 -V
pip 9.0.3 from /usr/lib/python2.7/site-packages (python 2.7)

pip commonly use commands:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# local or global config info
# you will see the package repo
pip3 config [debug, edit, get, list, set, unset]

# search
pip3 search <package name>
# download package in current dir
pip3 download <package name>
# will auto install other dependencies
pip3 install <package name>

# list packages installed
pip3 list
# show outdate packages
pip3 list -o

# uninstall
# will not uninstall its dependencies
pip3 uninstall <package name>

# show package info
# you will see the location where the package is installed
# and its source code url
pip3 show <package name>

# seek help
pip3 help

pip is actually fetching packages from Python package index (or your own package repo) https://pypi.org/

How to work:

  1. search key work directly.
  2. go to Browse projects -> Operating system -> Linux, then select other classifier (but this is still hard to search what is exactly needed).
  3. check development status, select package in production/stable version.

Pip install from specified repo:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# use additional repo
pip install --extra-index-url '<repo url>' vault-client==0.0.4

# or set by
# pip config set <ket> <value>
global.extra-index-url='<repo url>'
global.timeout='10'
global.trusted-host='registry.corp.xxx.com'
# then run
pip install vault-client==0.0.4

# 或者创建一个~/.pip/pip.conf 文件
[global]
timeout=10
trusted-host = egistry.corp.xxx.com
extra-index-url = <repo url>
# 然后
pip install --no-cache-dir vault-client==0.0.4

Virtual Environment

Combining with virtualenvwrapper is good, recommended.

Install virtualenv:

1
2
3
4
5
6
# install system-widely
# preferred way
# -m: run module
sudo python3 -m pip install virtualenv
# or
sudo pip3 install virtualenv

Create virtualenv:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
mkdir ~/virtualenvs && cd ~/virtualenvs

# create a virtual env called rates in python3
virtualenv -p python3 rates_py3
virtualenv -p python3.8.4 rates_py3
# python2 based
virtualenv -p python2 rates_py2

# activate
cd rates_py3
# after this you will see a prefix 
# 一旦激活，不管在其他任何地方，都是这个环境！
. ./bin/activate
# check
python -V
pip -V
# you will only see less packages installed
pip list

# then start your work in your project folder...

# deactivate
deactivate

Other similar tool, this venv may pre-installed or need to pip install globally:

1
2
# python >= 3.3, may popular in furture
python3 -m venv <virtual env name>

Syncing packages with colleagues, put this requirement file in version control to share and update:

1
2
3
4
5
6
7
8
9
# fist activate the virtual environment
# package list
python -m pip freeze > requirements.txt
# the condition can be ==, !=, >=, <=

# create another virtual environment with the same python verion like yours
# activeate this new environment
# then run
python -m pip install -r requirements.txt

You can specify version in pip install:

1
2
3
4
5
6
7
8
pip install flask==1.0.0
pip install 'Django<2.0'
# upgrade to latest version
pip install -U flask

# upgrade pip
# take care not to overwrite system pip using sudo
pip install -U pip

How to manage the project and virtual environment? Separating project with virtual environment! 放在不同的文件夹中，使用时激活就行了，一般一个venv对应一个project, 但如果要测试多个不同的环境，也可以多个venvs map to one project.

1
2
3
4
5
6
7
--dev
|  |-----my_game
|  |-----my_website
|
--virtual environment
   |-----my_game
   |-----my_website

Real-world example, when develop flask framework, use setup.py with editable pip to install packages in virtual environment, so you can edit the flask source code and it will reflect in real-time: When would the -e, --editable option be useful with pip install?

1
2
3
4
5
git clone https://github.com/pallets/flask

# activate virtual environment
# go to root level of flask directory
python -m pip install -e .

Now have developing env for flask.

You can also see tox.ini file in flask git repo, it is used for testing against different python versions in different virtual environments.

Virtualenvwrapper

A user-friently wrapper around virtualenv， easy creation and activation, bind projects to virtualenvs.

Setup:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# install system-widely
sudo python3 -m pip install virtualenvwrapper
sudo pip3 install virtualenvwrapper


# get path
which virtualenvwrapper.sh
/usr/local/bin/virtualenvwrapper.sh

# add below lines to ~/.bashrc
# point virtualenvwrapper to pyhton3 explicitly
# the path could be /usr/local/bin/python3, check the config
export python3=/usr/local/bin/python3
export VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python3
# the example path:
source /Library/Frameworks/Python.framework/Versions/3.10/bin/virtualenvwrapper.sh
# if you don't want to use default virtual env home
# use absolute path
export WORKON_HOME="/home/<user>/virtualenvs"
# set the project homes, when use mkproject will create project folder here
# use absolute path
export PROJECT_HOME="/home/<user>/dev"

Operations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# show virtual environments list
workon

# enter or switch virtual environment
workon <venv name>

# will create both venv and project
# if the project is bound with venv, use workon will auto switch to project folder
mkproject <pro name>
mkproject -p python3 <pro name>
mkproject -p python2 <pro name>

# create a venv only
mkvirtualenv <venv name>

# for old project does not bind with venv
# activate venv and go to old project folder, run below to bind them
setvirtualenvproject 

# remove a venv
# you need to manually remove project folder is you want
rmvirtualenv <venv name>

# deactivate venv
deactivate

Other Future Tools

New projects:

0%