python 集成cython && push 测试pip 仓库
昨天创建了一个简单的python 集成cython 的项目 master 但是有几个问题
目前的构建时基于make 同时需要本地执行，为了方便基于pip 的安装，做了如下调整
项目准备
项目使用venv 管理环境，初始化命令 python3 -m venv .
- 项目结构
├── cli
│ ├── __init__.py
│ ├── app.pyx
│ └── ext
│ ├── Makefile
│ ├── add.c
│ └── add.h
├── pyvenv.cfg
└── setup.py- 代码说明
cli 包含了代码以及cython 包装c 调用的代码， cli/ext 包含了一个c 静态库的代码（简单add）同时使用make 配置了一个简单的构建
cli/ini.py
import click
# 导入cython 暴露的包
import add_app
@click.command()
@click.option("--scale", default=1, help="Number to scale.")
@click.option("--pod", prompt="pod name",
help="The Pod counts.")
def apply(scale, pod):
"""Simple program that scale pod."""
# 调用c 代码
results = add_app.py_add(scale,10)
print("pod scale with counts",pod,results)
if __name__ == '__main__':
apply()cli/app.pyx： cython 包装c 代码
cdef extern from "./ext/add.h":
int add(int first,int second)
def py_add(first: int,second: int) -> int:
return add(first,second)ext/add.h: c 方法的头文件
int add(int first,int second);ext/add.c c 方法的实现
#include "add.h"
// 一个简单的add 方法
int add(int first,int second){
return first+second;
}setup.py: 这个是核心，为了方便直接pip 安装的时候进行构建，添加了cmdclass
import setuptools
from distutils.extension import Extension
from Cython.Distutils import build_ext
from Cython.Build import cythonize
from distutils import extension as distutils_extension
from distutils.command import build as distutils_build
from distutils.command import build_ext as distutils_build_ext
import subprocess
with open("README.md", "r") as fh:
long_description = fh.read()
# cython extension 配置，添加依赖以及构建的包名称
add_extension = Extension(
name="add_app",
sources=["cli/app.pyx"],
libraries=["add"],
library_dirs=["cli/ext"],
include_dirs=["cli/ext"]
)
class build_ext_Library(distutils_build_ext.build_ext):
def run(self):
command = "cd cli/ext && make"
process = subprocess.Popen(command, shell=True)
process.wait()
distutils_build_ext.build_ext.run(self)
setuptools.setup(
name="dalongrong_cythoncli",
version="0.0.15",
author="dalongrong",
# 配置pip 包包含的文件，方便安装的时候进行代码构建
package_data={
'cli': ['*.pyx',"ext/add.c","ext/add.h","ext/Makefile"]
},
author_email="1141591465@qq.com",
description="a simple cli project",
long_description=long_description,
install_requires=['click',"Cython==0.29.7"],
ext_modules= cythonize([add_extension]),
long_description_content_type="text/markdown",
# 包名称
packages = [
"cli"
],
# 自定义的构建任务
cmdclass ={
"build_ext":build_ext_Library
},
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
project_urls={
'Documentation': 'https://github.com/rongfengliang/cython-c-pip-demo.git',
'Say Thanks!': 'https://github.com/rongfengliang/cython-c-pip-demo.git',
'Source': 'https://github.com/rongfengliang/cython-c-pip-demo.git',
'Tracker': 'https://github.com/rongfengliang/cython-c-pip-demo.git',
},
entry_points={
'console_scripts': [
'podcli=cli:apply',
],
}
)push 私服
- 安装依赖
python -m pip install --user --upgrade setuptools wheel- 构建source 包
python setup.py sdist效果
running sdist
running egg_info
writing dalongrong_cythoncli.egg-info/PKG-INFO
writing dependency_links to dalongrong_cythoncli.egg-info/dependency_links.txt
writing entry points to dalongrong_cythoncli.egg-info/entry_points.txt
writing requirements to dalongrong_cythoncli.egg-info/requires.txt
writing top-level names to dalongrong_cythoncli.egg-info/top_level.txt
reading manifest file 'dalongrong_cythoncli.egg-info/SOURCES.txt'
writing manifest file 'dalongrong_cythoncli.egg-info/SOURCES.txt'
running check
warning: check: missing required meta-data: url
creating dalongrong_cythoncli-0.0.15
creating dalongrong_cythoncli-0.0.15/cli
creating dalongrong_cythoncli-0.0.15/cli/ext
creating dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying files to dalongrong_cythoncli-0.0.15...
copying README.md -> dalongrong_cythoncli-0.0.15
copying setup.py -> dalongrong_cythoncli-0.0.15
copying cli/__init__.py -> dalongrong_cythoncli-0.0.15/cli
copying cli/app.c -> dalongrong_cythoncli-0.0.15/cli
copying cli/app.pyx -> dalongrong_cythoncli-0.0.15/cli
copying cli/ext/Makefile -> dalongrong_cythoncli-0.0.15/cli/ext
copying cli/ext/add.c -> dalongrong_cythoncli-0.0.15/cli/ext
copying cli/ext/add.h -> dalongrong_cythoncli-0.0.15/cli/ext
copying dalongrong_cythoncli.egg-info/PKG-INFO -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying dalongrong_cythoncli.egg-info/SOURCES.txt -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying dalongrong_cythoncli.egg-info/dependency_links.txt -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying dalongrong_cythoncli.egg-info/entry_points.txt -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying dalongrong_cythoncli.egg-info/requires.txt -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying dalongrong_cythoncli.egg-info/top_level.txt -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
Writing dalongrong_cythoncli-0.0.15/setup.cfg
Creating tar archive
removing 'dalongrong_cythoncli-0.0.15' (and everything under it)- push test pip 仓库
注意需要先创建账户,同时需要安装twine ，这个工具可以全局安装，按照提示输入账户即可
twine upload --repository-url https://test.pypi.org/legacy/ dist/*- 效果
安装测试包
为了方便，使用centos 机器，同时也使用了venv
- 安装系统环境
yum install -y python36 python36-devel
实际上使用默认自带的也可以，但是为了使用venv 安装了3版本- 初始话环境
python3 -m venv appdemo- 安装几个依赖
主要是项目使用的，click 以及cython
cd appdemo && source bin/activate
pip install click cython- 安装包
pip install -i https://test.pypi.org/simple/ dalongrong-cythoncli
Looking in indexes: https://test.pypi.org/simple/
Collecting dalongrong-cythoncli
Downloading https://test-files.pythonhosted.org/packages/27/e2/56f135f3ee72d487fd073132d75195a2dd7a3c9122c53d87209640af554a/dalongrong_cythoncli-0.0.15.tar.gz
Requirement already satisfied: click in ./lib/python3.6/site-packages (from dalongrong-cythoncli) (7.0)
Requirement already satisfied: Cython==0.29.7 in ./lib/python3.6/site-packages (from dalongrong-cythoncli) (0.29.7)
Installing collected packages: dalongrong-cythoncli
Running setup.py install for dalongrong-cythoncli ... done
Successfully installed dalongrong-cythoncli-0.0.15- 使用
podcli --pod demoapp --scale 4
pod scale with counts demoapp 14- pip 包目录结构
ls lib/python3.6/site-packages/
add_app.cpython-36m-x86_64-linux-gnu.so cython.py __pycache__
cli dalongrong_cythoncli-0.0.15-py3.6.egg-info pyximport
click easy_install.py setuptools
Click-7.0.dist-info pip setuptools-39.0.1.dist-info
Cython pip-19.0.3.dist-info
Cython-0.29.7.dist-info pkg_resources说明
代码比较简单，主要是setup.py 配置extension 以及添加自定义build task，同时需要注意应该使用源码的打包模式
参考资料
https://medium.com/@shamir.stav_83310/making-your-c-library-callable-from-python-by-wrapping-it-with-cython-b09db35012a3
https://github.com/stavshamir/cython-c-wrapper/
https://cython.readthedocs.io/en/latest/src/tutorial/external.html
https://cython.readthedocs.io/en/latest/src/tutorial/clibraries.html
http://pages.cs.wisc.edu/~yezheng/post/cython/
https://github.com/rongfengliang/cython-c-pip-demo/tree/local_source
