解密Python C++库打包到wheel的正确方法
如果你用CPython写了一个扩展，然后要打包到wheel中发布，应该如何操作？你搜索网络，不管英文还是中文，得到的都是一知半解的答案。根据官方的粗浅文档，你可能可以很快完成一个wheel包，但和真正的wheel包差了十万八千里。这里主要考虑两个问题：1.包的结构，2.依赖库如何打包。
学习资源
因为涉及C/C++代码，那么最好的学习资源就是opencv-python的源码。理想情况下，做出来的包应该可以通过pip命令在任意平台安装。
Scikit-build
那么opencv-python是如何来为不同平台编译wheel的？通过源码可以发现，它用到了scikit-build，通过CMake来编译C/C++代码。我们可以直接运行GitHub上的示例工程来体验下。
这个工程很简单，setup.py里只写了一个包名：
from skbuild import setup
setup(
name="hello-cpp",
version="1.2.3",
description="a minimal example package (cpp version)",
author='The scikit-build team',
license="MIT",
packages=['hello'],
python_requires=">=3.7",
)
其它的都交给CMakeLists.txt去完成：
cmake_minimum_required(VERSION 3.4...3.22)
project(hello)
find_package(PythonExtensions REQUIRED)
add_library(_hello MODULE hello/_hello.cxx)
python_extension_module(_hello)
install(TARGETS _hello LIBRARY DESTINATION hello)
pyproject.toml中配置了编译环境。
[build-system]
requires = [
"setuptools>=42",
"scikit-build>=0.13",
"cmake>=3.18",
"ninja",
]
build-backend = "setuptools.build_meta"
[tool.cibuildwheel]
manylinux-x86_64-image = "manylinux2014"
manylinux-aarch64-image = "manylinux2014"
skip = ["pp*", "*-win32", "*-manylinux_i686", "*-musllinux_*"]
[tool.cibuildwheel.windows]
archs = ["AMD64"]
[tool.cibuildwheel.linux]
repair-wheel-command = "auditwheel repair -w {dest_dir} {wheel} --plat manylinux2014_x86_64"
archs = ["x86_64"]
[tool.cibuildwheel.macos]
archs = ["x86_64"]
repair-wheel-command = [
"delocate-listdeps {wheel}",
"delocate-wheel --require-archs {delocate_archs} -w {dest_dir} {wheel}",
]
我们运行pip wheel .就可以得到一个*.whl包。
现在用解压软件打开这个包来看下。我们看到根目录中包含了两个文件夹，一个是*dist-info，一个是你定义的package。在package中可以找到编译出来的库文件以及__init__.py文件。这就是正确的wheel包结构。如果你按照官方教程直接用python setup.py bdist_wheel打包一个C Extension工程，你会发现，编译出来的库是打包在根目录的。这有什么问题？安装之后打开安装目录Lib\site-packages，你会发现库是直接拷贝到这个目录下的，污染环境。
CMake配置
当你的C/C++代码还依赖别的库，那么就要考虑库的链接和打包问题。Windows上很简单，所有的库都放在同一个目录下即可，但Linux和macOS就需要设置相对路径。注意，设置的方法是不一样的。方法如下：
if(CMAKE_HOST_UNIX)
if(CMAKE_HOST_APPLE)
SET(CMAKE_CXX_FLAGS "-std=c++11 -O3 -Wl,-rpath,@loader_path")
SET(CMAKE_INSTALL_RPATH "@loader_path")
else()
SET(CMAKE_CXX_FLAGS "-std=c++11 -O3 -Wl,-rpath=$ORIGIN")
SET(CMAKE_INSTALL_RPATH "$ORIGIN")
endif()
SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
endif()
其它的链接，安装这里省略，稍后直接看源码。
setuptools
那么不用CMake能不能做到一样的打包效果呢？用setuptools是完全可以实现的。
我们在setup.py中要做这样几件事情：1.配置编译参数。2.拷贝依赖库。
配置编译参数
首先判断系统和CPU架构，然后添加对应的编译参数。
dbr_lib_dir = ''
dbr_include = ''
dbr_lib_name = 'DynamsoftBarcodeReader'
if sys.platform == "linux" or sys.platform == "linux2":
# linux
if platform.uname()[4] == 'AMD64' or platform.uname()[4] == 'x86_64':
dbr_lib_dir = 'lib/linux'
elif platform.uname()[4] == 'aarch64':
dbr_lib_dir = 'lib/aarch64'
else:
dbr_lib_dir = 'lib/arm32'
elif sys.platform == "darwin":
# OS X
dbr_lib_dir = 'lib/macos'
pass
elif sys.platform == "win32":
# Windows
dbr_lib_name = 'DBRx64'
dbr_lib_dir = 'lib/win'
if sys.platform == "linux" or sys.platform == "linux2":
ext_args = dict(
library_dirs = [dbr_lib_dir],
extra_compile_args = ['-std=c++11'],
extra_link_args = ["-Wl,-rpath=$ORIGIN"],
libraries = [dbr_lib_name],
include_dirs=['include']
)
elif sys.platform == "darwin":
ext_args = dict(
library_dirs = [dbr_lib_dir],
extra_compile_args = ['-std=c++11'],
extra_link_args = ["-Wl,-rpath,@loader_path"],
libraries = [dbr_lib_name],
include_dirs=['include']
)
if sys.platform == "linux" or sys.platform == "linux2" or sys.platform == "darwin":
module_barcodeQrSDK = Extension('barcodeQrSDK', ['src/barcodeQrSDK.cpp'], **ext_args)
else:
module_barcodeQrSDK = Extension('barcodeQrSDK',
sources = ['src/barcodeQrSDK.cpp'],
include_dirs=['include'], library_dirs=[dbr_lib_dir], libraries=[dbr_lib_name])
编译并拷贝依赖库
我们设置自定义函数来实现python setup.py build:
def copylibs(src, dst):
if os.path.isdir(src):
filelist = os.listdir(src)
for file in filelist:
libpath = os.path.join(src, file)
shutil.copy2(libpath, dst)
else:
shutil.copy2(src, dst)
class CustomBuildExt(build_ext.build_ext):
def run(self):
build_ext.build_ext.run(self)
dst = os.path.join(self.build_lib, "barcodeQrSDK")
copylibs(dbr_lib_dir, dst)
filelist = os.listdir(self.build_lib)
for file in filelist:
filePath = os.path.join(self.build_lib, file)
if not os.path.isdir(file):
copylibs(filePath, dst)
# delete file for wheel package
os.remove(filePath)
setup (name = 'barcode-qr-code-sdk',
...
cmdclass={
'build_ext': CustomBuildExt,},
)
当执行build_ext.build_ext.run(self)的时候，触发了默认的编译。这个时候会生成我们需要的Python库。然后把我们用到的库都拷贝到输出目录中。在执行打包命令的时候，build命令是首先被触发的，然后会把输出目录中的所有文件都打包到wheel里。这样就实现了和 scikit-build一样的效果。
repair wheel
Linux做出来的包，还需要通过auditwheel的repair命令重新生成一个支持manylinux的wheel包。
auditwheel repair *.whl --plat manylinux2014_x86_64"
如果不做这个处理，生成的Linux包是不能上传pypi的。
上传Pypi
现在在GitHub Action创建一个自动化编译打包发布流程：
name: Build and upload to PyPI
on: [push, pull_request]
jobs:
build_wheels:
name: Build wheels on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-20.04, windows-2019, macos-10.15]
steps:
- uses: actions/checkout@v2
- name: Build wheels
uses: pypa/cibuildwheel@v2.6.1
- uses: actions/upload-artifact@v2
with:
path: ./wheelhouse/*.whl
build_sdist:
name: Build source distribution
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Build sdist
run: pipx run build --sdist
- uses: actions/upload-artifact@v2
with:
path: dist/*.tar.gz
upload_pypi:
needs: [build_wheels, build_sdist]
runs-on: ubuntu-latest
# upload to PyPI on every tag starting with 'v'
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
# alternatively, to publish when a GitHub Release is created, use the following rule:
# if: github.event_name == 'release' && github.event.action == 'published'
steps:
- uses: actions/download-artifact@v2
with:
name: artifact
path: dist
- uses: pypa/gh-action-pypi-publish@v1.4.2
with:
user: __token__
password: ${{ secrets.pypi_password }}
skip_existing: true
各种主流python对应的包就做出来了。
