构建运行在容器里的分布式编译工具icecream

文档说明：只记录关键地方;
试验环境： linux debian 11
目标：构建能用于并行编译的容器环境

Icecream 是由 SUSE 创建的、基于 distcc 的并行编译工具
分布式编译工具（充分利用多台计算机的编译能力）

构建脚本

#!/bin/bash
set -eux
day=$(date "+%Y%m%d%H%M")
day=$(date -u +"%Y%m%dT%H%MZ")
export DOCKER_BUILDKIT=1
 
image="registry.cn-beijing.aliyuncs.com/jingjingxyk-public/app:icecream-debian-11-$day"
 
PROXY_URL=${2:+'http://192.168.3.26:8015'}
 
# 要使用代理，需要传递两个参数
# sh build-docker.sh --proxy 1
 
docker build -t $image   -f Dockerfile .  --progress=plain  --build-arg PROXY_URL=$PROXY_URL  # --no-cache
 
docker push $image
 
 

Dockerfile

 
FROM alpine:latest
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apk/repositories
RUN apk update
RUN apk add git curl
 
ARG  PROXY_URL
ENV  http_proxy=$PROXY_URL
ENV  https_proxy=$PROXY_URL
 
RUN git clone --depth=1 --progress https://github.com/JPEWdev/icecream-sundae.git
RUN git clone --depth=1 --progress https://github.com/icecc/icecream.git
ENV  http_proxy=''
ENV  https_proxy=''
 
 
FROM debian:11 AS builder
 
RUN test ! -f /etc/apt/source.list.save && cp  /etc/apt/sources.list /etc/apt/sources.list.save
RUN  sed -i "s@security.ubuntu.com@mirrors.ustc.edu.cn@g" /etc/apt/sources.list 
RUN  sed -i "s@archive.ubuntu.com@mirrors.ustc.edu.cn@g" /etc/apt/sources.list
 
RUN  apt update -y && apt install -y ca-certificates
ENV DEBIAN_FRONTEND=noninteractive
ENV TZ=Etc/UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
 
RUN  apt install -y  libcap-ng-dev liblzo2-dev libzstd-dev libarchive-dev asciidoc automake \
libtool gcc g++ clang \
libcap-ng-dev pkg-config \
make tini  libssl-dev ca-certificates  make cmake gcc g++ zip \
python3 python3-pip python3-dev wget ninja-build
 
RUN  apt-get install -y  g++ libcap-ng-dev libglib2.0-dev libicecc-dev liblzo2-dev libncursesw5-dev meson ninja-build
 
COPY --from=0 /icecream /icecream
COPY --from=0 /icecream-sundae /icecream-sundae
 
WORKDIR /icecream
RUN ./autogen.sh &&  ./configure --prefix=/opt/icecream &&   make -j `grep "processor" /proc/cpuinfo | sort -u | wc -l` &&  make install
ENV PATH=/opt/icecream/libexec/icecc/bin:/opt/icecream/bin:/opt/icecream/sbin:$PATH
EXPOSE 10245 8765 8766
 
WORKDIR /icecream-sundae
RUN mkdir builddir
WORKDIR /icecream-sundae/builddir
RUN meson .. --buildtype release
RUN ninja  && ninja install
WORKDIR /
RUN apt autoclean && apt clean && rm -rf  /icecream /icecream-sundae
ENTRYPOINT ["tini", "--"]
# CMD icecc-scheduler -u nobody -v
# CMD ['/bin/sh','-c','icecc-scheduler -u nobody']
 
 
#CMD在容器运行的时候提供一些命令及参数，用法如下：
#CMD ["executable","param1","param2"] (exec form, this is the preferred form)
#CMD ["param1","param2"] (as default parameters to ENTRYPOINT)
#CMD command param1 param2 (shell form)
#第一种用法：运行一个可执行的文件并提供参数。
#第二种用法：为ENTRYPOINT指定参数。
#第三种用法(shell form)：是以”/bin/sh -c”的方法执行的命令。
 
 
 

配置简述

# 调度节点
/opt/icecream/sbin/icecc-scheduler -d -u nobody
# 监控命令
monitor:  icecream-sundae
# 必要的端口
- 10245:10245/tcp 
- 8765:8765/tcp 
- 8766:8766/tcp 
- 8765:8765/udp
 
# 执行任务节点
/opt/icecream/sbin/iceccd -u nobody -d -s 192.168.3.32:8765
 

端口说明

 
守护进程计算机上的 TCP/10245（必需）
用于调度程序计算机的 TCP/8765（必需）
TCP/8766 用于调度程序的 telnet 接口（可选）
UDP/8765 用于广播以查找调度程序（可选）
 

例子: 调度节点

version: "3"
services:
    icecream-scheduler:
        image:   registry-vpc.cn-beijing.aliyuncs.com/jingjingxyk-public/app:icecream-debian-11-20220524T1607Z
        ports:
        - "10245:10245/tcp"
        - "8765:8765/tcp"
        - "8766:8766/tcp"
        - "8765:8765/udp"
        hostname: icecream-ubuntu-scheduler-01
        command:
            - /bin/sh
            - -c
            - |
                icecc-scheduler -d -u nobody
                tail -f /dev/null
 

例子：任务节点

version: "3"
services:
    icecream-worker:
        image:   registry-vpc.cn-beijing.aliyuncs.com/jingjingxyk-public/app:icecream-debian-11-20220524T1607Z
        ports:
        - "8765:8765/tcp"
        - "8766:8766/tcp"
        - "8765:8765/udp"
        hostname: icecream-ubuntu-worker-01  # 172.23.24.221 调度节点IP
        command:
            - /bin/sh
            - -c
            - |
                /opt/icecream/sbin/iceccd -u nobody -d -s 172.23.24.221:8765
                tail -f /dev/null
 

参考文档

1. icecream
2. icecream-sundae
3. shell之变量默认值
4. Dockerfile
5. Compose file
6. 自建拉取registry.k8s.io、k8s.gcr.io、gcr.io、quay.io、ghcr.io 容器镜像的服务
7. 分布式编译原理
8. ue5 Unreal Swarm分布式渲染 ，特别容易理解分布式编译