自定义 OpenShift s2i 镜像与模板——OracleJDK8
本文目标
由于 OpenShift 官方提供的镜像与模板（OpenJDK8）不完全满足业务需要：
-
不包含飞行记录功能。只有 OpenJDK11 以上才被 Oracle 开源
-
生成堆 dump 很大很慢（公司项目 JVM 堆最小16G，最大32G），
导出-压缩-传输较长（半小时以上），对业务停顿时间过长 -
镜像内置的获取性能指标的命令很少，常规网络状态查询命令如
netstat、ping等都没有 -
镜像内部s2i流程编写得过于复杂，镜像构建过程缓存多且未移除，导致最终镜像普遍体积过大（350M的业务应用，打出镜像有 1.1G…）
为了方便定位性能问题，笔者仔细阅读了 OpenShift v3.11 版本的 s2i 构建流程、镜像自定义等相关文档，定制出了自己的模板与镜像，目前已经在生产环境上验证通过，在这里记录一二。如果有帮到你，点个关注和赞再走吧~
本文主体思路是使用 s2i工具创建基础的构建调试环境，修改Dockerfile与构建脚本生成镜像，再推送镜像和模板到上篇文章OpenShift 本地开发环境配置(基于 Minishift)的环境中，用一个demo测试下最终效果。
自定义 s2i 镜像部分
安装 s2i 工具
s2i 工具官方Release Page：https://github.com/openshift/source-to-image/releases
#下载s2i工具
wget https://github.com/openshift/source-to-image/releases/download/v1.3.1/source-to-image-v1.3.1-a5a77147-linux-amd64.tar.gz
#解压s2i工具到/usr/bin
sudo tar zxf source-to-image-v1.3.1-a5a77147-linux-amd64.tar.gz -C /usr/bin
#查看s2i版本
s2i version
初始化 s2i 目录结构
笔者是在
~/WorkSpace/openshift目录下开始操作的，提前说下免得读者操作有误。
#生成s2i脚本目录结构，create参数为创建，第3参数为镜像名，第3参数为生成目录名
s2i create oracle-jdk-8 oracle-jdk-8-catelog
#进入生成的目录中
cd oracle-jdk-8-catelog
#查看基本目录结构
tree -L 3
.
├── Dockerfile #镜像Dockerfile
├── Makefile #Makefile构建脚本
├── README.md
├── s2i
│ └── bin #s2i脚本目录
│ ├── assemble #装配脚本
│ ├── run #运行脚本
│ ├── save-artifacts #可选，保存制品脚本，增量构建时用
│ └── usage #使用说明脚本
└── test
├── run #测试运行脚本
└── test-app #构建的源码（本例中可以是jar包）
└── index.html
这里笔者用不到 s2i/bin/save-artifacts，因为不需要增量构建，删除之; test/test-app/index.html 也用不到，删除之。
rm -rf s2i/bin/save-artifacts
rm -rf test/test-app/index.html
下载并解压 OracleJDK8安装包，适度精简
由于笔者已经将jdk8安装包下载好了，下载 OracleJDK 付费版本以前的版本，参考我的博客，这里就直接从 ~/Downloads 解压到 ~/WorkSpace/openshift/oracle-jdk-8-catelog 下
#解压jdk安装包
tar zxvf ~/Downloads/jdk-8u181-linux-x64.tar.gz -C ~/WorkSpace/openshift/oracle-jdk-8-catelog
#进入jdk解压目录
cd /home/hellxz/WorkSpace/openshift/oracle-jdk-8-catelog/jdk1.8.0_181
#适度精简，删除源码包，能小26M
rm -rf javafx-src.zip src.zip
#回到模板初始化的目录
cd ..
修改 Dockerfile
vim Dockerfile，把默认的都删除掉，添加下边的内容：
FROM debian:buster
ENV BUILDER_VERSION 1.0
# Set labels used in OpenShift to describe the builder image
LABEL description="Source To Image (S2I) image for Hellxz Providing Oracle JDK 8" \
maintainer="Hellxz Zhang <hellxz@foxmail.com>" \
io.k8s.description="Platform for building and running Java applications with Oracle JDK 8" \
io.k8s.display-name="Java Applications" \
io.openshift.expose-services="8080:http" \
io.openshift.tags="builder,java" \
io.openshift.s2i.scripts-url="image:///usr/libexec/s2i" \
io.openshift.s2i.destination="/tmp"
# Copy Oracle JDK8
COPY ./jdk1.8.0_181 /usr/lib/jvm
# Copy scripts to s2i build path, must same as `io.openshift.s2i.scripts-url` label specified.
COPY ./s2i/bin/ /usr/libexec/s2i
# Environments
ENV JAVA_HOME="/usr/lib/jvm" \
PATH="/usr/lib/jvm/bin:${PATH}" \
TZ="Asia/Shanghai"
# Create oraclejdk user with home dir /deployments
# Grant user exec privilege
# Set apt mirror and install some utils then clean
# Generate UTF-8 locales
RUN mkdir /deployments && \
useradd -M -d /deployments -u 1001 -s /bin/bash oraclejdk8 && \
chown oraclejdk8 /deployments && chmod 777 /deployments && \
bash -c "echo -e \"deb http://mirrors.163.com/debian/ buster main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-updates main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-backports main non-free contrib\ndeb http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\" > /etc/apt/sources.list" && \
apt-get update -y && apt-get install -y net-tools locales procps && \
apt-get clean all -y && rm -rf /var/lib/apt/lists/* && \
sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
# To avoid build image error tips set LC env after locale.gen
ENV LC_ALL="en_US.UTF-8"
# Switch user
USER 1001
# Switch WorkDir
WORKDIR /deployments
EXPOSE 8080 8443
# Set the default CMD for the image
CMD ["/usr/libexec/s2i/usage"]
上述的Dockerfile做了什么呢？
- 基于非常稳定的 Debian10
- 为镜像打Label标签，s2i 构建时会读取这些标签
- 复制OracleJDK目录及s2i脚本
- 配置JDK环境变量、UTF-8字符集、指定中国时区
- 创建部署目录、创建运行用户、更换国内源、安装必要收集性能指标工具（netstat和ps）
- 指定工作目录、运行用户、默认暴露的端口、默认的
CMD入口
完成Dockerfile的修改还没完，我们还需要修改 s2i 构建与运行脚本，以适应刚刚修改的镜像配置
修改 s2i 脚本
s2i脚本在 s2i/bin 目录下，我们需要修改 assemble 脚本来改变组装流程，修改 run 脚本以更换启动命令。
修改assemble
vim s2i/bin/assemble
#!/bin/bash -e
#
# S2I assemble script for the 'oracle-jdk-8' image.
# The 'assemble' script builds your application source so that it is ready to run.
#
# For more information refer to the documentation:
# https://github.com/openshift/source-to-image/blob/master/docs/builder_image.md
#
# Prevent some horror bugs. example: rm -rf ${SOME_UNEXISTS_VARIABLE}/*
set -e
# If the 'oracle-jdk-8' assemble script is executed with the '-h' flag, print the usage.
if [[ "$1" == "-h" ]]; then
exec /usr/libexec/s2i/usage
fi
echo "---> Moving Artifact from source..."
mv /tmp/src/*.jar /deployments/
echo "---> Clearing tmp dir..."
rm /tmp/src/.git -rf
echo "<--- Build Success..."
以上脚本主要做了两件事，一个是从 /tmp/src 下将 jar 包复制到部署目录，在之前的文章中介绍过 s2i 的工作流程，在构建时会从 Git 上拉取制品或源码到 /tmp/src 下；另一个是删除 /tmp/src/.git 目录，原因是 Git 的 .git 目录会同样保存压缩后的制品或源码文件，删除以减小镜像体积。
修改run
vim s2i/bin/run
#!/bin/bash -e
#
# S2I run script for the 'oracle-jdk-8' image.
# The run script executes the server that runs your application.
#
# For more information see the documentation:
# https://github.com/openshift/source-to-image/blob/master/docs/builder_image.md
#
exec java ${JVM_OPTS} -Djava.security.egd=file:/dev/./urandom -jar /deployments/*.jar
run 脚本只作启动功能，为了方便传 JVM 调优参数，定制了 JVM_OPTS 环境变量供使用。参数中的随机数配置能提高服务生成随机数的能力，提高启动速度。
修改 Makefile
修改Makefile的作用是简化构建镜像的命令，笔者这里修改了镜像名以及添加了推送私有镜像仓库的参数，这点上 npm 有点类似于 Makefile。
IMAGE_NAME = 192.168.99.1:5000/oracle-jdk-8
.PHONY: test
test:
docker build -t $(IMAGE_NAME)-candidate .
IMAGE_NAME=$(IMAGE_NAME)-candidate test/run
.PHONY: push
push:
docker build -t $(IMAGE_NAME) .
docker push $(IMAGE_NAME)
想了解更多 Makefile 写法，可以参考
左耳朵耗子陈皓 写的《跟我一起写Makefile》
编写测试脚本
test/run 这个测试脚本相当于完成了一次 s2i 到部署的过程，先准备测试镜像，再运行s2i的构建过程增量构建（将测试制品程序与镜像进行组装，容器内部调用了 assemble 脚本），然后启动容器，运行定时测试脚本访问容器端口地址，通过就算成功了。
由于默认测试脚本中测试服务是否可用是访问地址判断返回值是否200状态码，我这边提供个springboot的demo 部署后访问 /test 端点返回结果 ，首先构建下这个demo:
#克隆测试demo
git clone https://github.com/hellxz/cicd-demo.git test/test-app
#手动构建
cd test/test-app
mvn package -DskipTests
#复制jar包并删除本文无关的内容
mv target/*.jar .
rm -rf *.sh *file *.xml *.md src target .git *.yaml .gitignore .mvn
#回到Makefile所在目录
cd ../..
修改 test/run 脚本，调整探测间隔与访问的接口地址，本demo中需修改116行为5，以及120和121行访问路径添加 /test
执行测试
make test
我们能看到 springboot 的 banner 下有行字 Starting CicdDemoApplication v0.0.1 using Java 1.8.0_181 on f770c60456f4 with PID 1 (/deployments/cicd-demo-0.0.1.jar started by oraclejdk8 in /deployments)，这可以证明测试的demo的确是以 oraclejdk8 用户运行成功的，而且 pid 为 1。再往下看也没有检测报错的信息，说明测试通过，可以使用 Minishift测试了。
推送镜像到仓库中
#执行Makefile中定义的构建推送命令
make push
hellxz@debian:~/WorkSpace/openshift/oracle-jdk-8-catelog$ make push
docker build -t 192.168.99.1:5000/oracle-jdk-8 .
Sending build context to Docker daemon 389.2MB
Step 1/12 : FROM debian:buster
---> 2b6f409b1d24
Step 2/12 : ENV BUILDER_VERSION 1.0
---> Using cache
---> 46de72742080
Step 3/12 : LABEL description="Source To Image (S2I) image for Hellxz Providing Oracle JDK 8" maintainer="Hellxz Zhang <hellxz@foxmail.com>" io.k8s.description="Platform for building and running Java applications with Oracle JDK 8" io.k8s.display-name="Java Applications" io.openshift.expose-services="8080:http" io.openshift.tags="builder,java" io.openshift.s2i.scripts-url="image:///usr/libexec/s2i" io.openshift.s2i.destination="/tmp"
---> Using cache
---> fcbadb0d0d39
Step 4/12 : COPY ./jdk1.8.0_181 /usr/lib/jvm
---> Using cache
---> ebb551cc9d4c
Step 5/12 : COPY ./s2i/bin/ /usr/libexec/s2i
---> Using cache
---> 5f08421b9527
Step 6/12 : ENV JAVA_HOME="/usr/lib/jvm" PATH="/usr/lib/jvm/bin:${PATH}" TZ="Asia/Shanghai"
---> Using cache
---> 3c663a5370c8
Step 7/12 : RUN mkdir /deployments && useradd -M -d /deployments -u 1001 -s /bin/bash oraclejdk8 && chown oraclejdk8 /deployments && chmod 777 /deployments && bash -c "echo -e \"deb http://mirrors.163.com/debian/ buster main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-updates main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-backports main non-free contrib\ndeb http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\" > /etc/apt/sources.list" && apt-get update -y && apt-get install -y net-tools locales procps && apt-get clean all -y && rm -rf /var/lib/apt/lists/* && sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
---> Using cache
---> 5e9c181bb69b
Step 8/12 : ENV LC_ALL="en_US.UTF-8"
---> Using cache
---> 60091bb63284
Step 9/12 : USER 1001
---> Using cache
---> f8e6a2e96a8a
Step 10/12 : WORKDIR /deployments
---> Using cache
---> 66f6925e3ab6
Step 11/12 : EXPOSE 8080 8443
---> Using cache
---> bda814465820
Step 12/12 : CMD ["/usr/libexec/s2i/usage"]
---> Using cache
---> f30388d3181c
Successfully built f30388d3181c
Successfully tagged 192.168.99.1:5000/oracle-jdk-8:latest
docker push 192.168.99.1:5000/oracle-jdk-8
Using default tag: latest
The push refers to repository [192.168.99.1:5000/oracle-jdk-8]
16970671df83: Pushed
97a51654d23f: Pushed
16fd63c1b14c: Pushed
d6a325d281f2: Pushed
latest: digest: sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79 size: 1160
导入镜像到 OpenShift 内部镜像仓库中
这里Minishift导入镜像到内部仓库和OpenShift是一致的，说是OpenShift也可以。
#导入需要oc命令行工具，minishift默认已经提供了，但未加入到Path中，可以执行如下命令临时把oc加入Path
eval (minishift oc-env) #如果报错，则只执行括号内命令，根据提示执行相应命令
oc import-image 192.168.99.1:5000/oracle-jdk-8:latest --confirm --insecure
hellxz@debian:~/WorkSpace/openshift$ oc import-image 192.168.99.1:5000/oracle-jdk-8:latest --confirm --insecure
imagestream.image.openshift.io/oracle-jdk-8 imported
Name: oracle-jdk-8
Namespace: myproject
Created: 1 second ago
Labels: <none>
Annotations: openshift.io/image.dockerRepositoryCheck=2021-11-21T14:24:57Z
Docker Pull Spec: 172.30.1.1:5000/myproject/oracle-jdk-8
Image Lookup: local=false
Unique Images: 1
Tags: 1
latest
tagged from 192.168.99.1:5000/oracle-jdk-8:latest
will use insecure HTTPS or HTTP connections
* 192.168.99.1:5000/oracle-jdk-8@sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79
1 second ago
Image Name: oracle-jdk-8:latest
Docker Image: 192.168.99.1:5000/oracle-jdk-8@sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79
Name: sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79
Created: 1 second ago
Annotations: image.openshift.io/dockerLayersOrder=ascending
Image Size: 221.8MB in 4 layers
Layers: 50.44MB sha256:07471e81507f7cf1100827f10c60c3c0422d1222430e34e527d97ec72b14a193
161.8MB sha256:98b418031cdbc51e52d88a2632516069e753d5293ec440dae46163a45880492a
899B sha256:61addb0f8207e85f3a734299eda8b8afae987076de0cf67b60f33c9c9846f6b6
9.598MB sha256:09daea12eb7b9cd76b4629e22d566a6646eb6e19ea3c92706c7bc41bf0285384
Image Created: 30 minutes ago
Author: <none>
Arch: amd64
Command: /usr/libexec/s2i/usage
Working Dir: /deployments
User: 1001
Exposes Ports: 8080/tcp, 8443/tcp
Docker Labels: description=Source To Image (S2I) image for Hellxz Providing Oracle JDK 8
io.k8s.description=Platform for building and running Java applications with Oracle JDK 8
io.k8s.display-name=Java Applications
io.openshift.expose-services=8080:http
io.openshift.s2i.destination=/tmp
io.openshift.s2i.scripts-url=image:///usr/libexec/s2i
io.openshift.tags=builder,java
maintainer=Hellxz Zhang <hellxz@foxmail.com>
Environment: PATH=/usr/lib/jvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
BUILDER_VERSION=1.0
JAVA_HOME=/usr/lib/jvm
TZ=Asia/Shanghai
LC_ALL=en_US.UTF-8
如果报错，优先考虑是否登录了oc，
oc login -u developer，执行后再尝试导入。
自定义s2i模板部分
定义模板文件
这里直接提供模板内容，基于 openjdk8的模板修改而来的：
oraclejdk-8-basic-s2i.yaml
apiVersion: template.openshift.io/v1
kind: Template
message: A new java application has been created in your project.
metadata:
annotations:
description: An Java application using Oracle JDK 8.
iconClass: icon-rh-openjdk
openshift.io/display-name: OracleJDK 8 by Hellxz
openshift.io/provider-display-name: Hellxz Zhang <hellxz001@foxmail.com>.
tags: java
template.openshift.io/long-description: This template defines resources needed to develop Oracle JDK 8 Java based application.
template.openshift.io/support-url: hellxz001@foxmail.com
version: 0.0.1
labels:
template: oraclejdk-8-basic-s2i
xpaas: 0.0.1
name: oraclejdk-8-basic-s2i
objects:
- apiVersion: v1
kind: Service
metadata:
annotations:
description: The application's http port.
labels:
application: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
spec:
ports:
- port: 8080
targetPort: 8080
selector:
deploymentConfig: ${APPLICATION_NAME}
- apiVersion: v1
id: ${APPLICATION_NAME}-http
kind: Route
metadata:
annotations:
description: Route for application's http service.
labels:
application: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
spec:
host: ${HOSTNAME_HTTP}
to:
name: ${APPLICATION_NAME}
- apiVersion: v1
kind: ImageStream
metadata:
labels:
application: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
- apiVersion: v1
kind: BuildConfig
metadata:
labels:
application: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
spec:
output:
to:
kind: ImageStreamTag
name: ${APPLICATION_NAME}:latest
source:
contextDir: ""
git:
ref: ${SOURCE_REPOSITORY_REF}
uri: ${SOURCE_REPOSITORY_URL}
type: Git
strategy:
sourceStrategy:
forcePull: true
from:
kind: ImageStreamTag
name: oracle-jdk-8:latest
namespace: ${IMAGE_STREAM_NAMESPACE}
type: Source
triggers:
- imageChange: {}
type: ImageChange
- type: ConfigChange
- apiVersion: v1
kind: DeploymentConfig
metadata:
labels:
application: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
spec:
replicas: 1
selector:
deploymentConfig: ${APPLICATION_NAME}
strategy:
type: Recreate
template:
metadata:
labels:
application: ${APPLICATION_NAME}
deploymentConfig: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
spec:
containers:
- env: []
image: ${APPLICATION_NAME}
imagePullPolicy: Always
name: ${APPLICATION_NAME}
ports:
- containerPort: 8778
name: jolokia
protocol: TCP
- containerPort: 8080
name: http
protocol: TCP
- containerPort: 8443
name: https
protocol: TCP
terminationGracePeriodSeconds: 75
triggers:
- imageChangeParams:
automatic: true
containerNames:
- ${APPLICATION_NAME}
from:
kind: ImageStreamTag
name: ${APPLICATION_NAME}:latest
type: ImageChange
- type: ConfigChange
parameters:
- description: The name for the application.
displayName: Application Name
name: APPLICATION_NAME
required: true
value: oraclejdk8-app
- description: Git source URI for application
displayName: Git Repository URL
name: SOURCE_REPOSITORY_URL
required: true
- description: Git branch/tag reference
displayName: Git Reference
name: SOURCE_REPOSITORY_REF
value: master
- description: 'Custom hostname for http service route. Leave blank for default hostname,
e.g.: <application-name>-<project>.<default-domain-suffix>'
displayName: Custom http Route Hostname
name: HOSTNAME_HTTP
- description: Namespace in which the ImageStreams for Red Hat Middleware images are
installed. These ImageStreams are normally installed in the openshift namespace.
You should only need to modify this if you've installed the ImageStreams in a
different namespace/project.
displayName: ImageStream Namespace
name: IMAGE_STREAM_NAMESPACE
required: true
value: myproject
模板的结构简单说明：
- 定义格式基于k8s，是红帽扩展的 Template API类型
- 源数据 - 记录模板的内容，会在打开模板的UI上显示
- objects - 各种构建到部署过程的所有对象定义
- Service - k8s原生有的服务对象
- Route - OpenShift独有的路由对象
- ImageStream - OpenShift独有的镜像流对象
- BuildConfig - OpenShift独有的构建配置对象
- DeploymentConfig - OpenShift独有的部署配置对象，生成k8s中的Deployment对象
- parameters - 可修改参数，或者变量，供objects中各种对象引用，从而生成相关联的一系列对象
导入模板到 OpenShift
oc apply -f oraclejdk-8-basic-s2i.yaml
hellxz@debian:~/WorkSpace/openshift$ oc apply -f=oraclejdk-8-basic-s2i.yaml
template.template.openshift.io/oraclejdk-8-basic-s2i created
使用自定义模板和镜像部署 Java 程序
预备工作
由于OpenShift是由Git仓库拉取制品或源码进行构建的，所以需要把测试程序（上文中用到的cicd-demo-0.0.1.jar）上传到 Git 版本控制中，这里延用上篇文章中 Minishift 开发环境中配置的 Gitea（需要预先创建个账号和仓库，这里创建的仓库名为demo，用户名为hellxz）。
#进入测试程序jar所在目录
cd ~/WorkSpace/openshift/oracle-jdk-8-catelog/test/test-app
#初始化当前目录为git仓库
git init
#添加远程仓库地址（按实际操作来，变通一些）
git remote add origin git remote add origin http://localhost:3000/hellxz/demo.git
#设置用户名和email
git config --local user.name hellxz
git config --local user.email hellxz001@foxmail.com
#提交
git add .
git commit -m "init"
#推送远程仓库
git push --set-upstream origin master
如下图，已经推送完毕。
使用自定义模板部署程序
模板中引用的是自定义镜像的名称，命名空间也是myproject（Minishift默认的，生产环境可以改成需要的名称再上传，调整可多次导入）
首次部署流程比较麻烦，这里分多个动图录制
创建应用
Gitea 配置的对外地址配置得有点问题，不过问题不大……
创建拉取Git的secret
需要告知s2i构建脚本 Git 仓库的用户和密码，这样才能拉得下来制品，仅首次创建应用时需要创建。
修改Build配置yaml，添加 Git 仓库 secret 引用
执行构建，查看构建日志
查看测试程序运行状态
如图，容器运行已经是 Ready: true 状态，说明启动状态正常。
使用域名访问服务看看
由于没有外部的dns，域名可以修改 /etc/hosts 文件代替
测试通过，自定义镜像和模板都能正常工作。
写作不易，如果本文对您有所帮助，就点个关注点个赞再走呗~ 我是 Hellxz，我们下次再见。
本文同步发布以下两个地址，未经许可禁止转载。
