02、Kubernetes 入门（上）-重新认识Docker

为什么使用容器？

1.上线流程繁琐

开发->测试->申请资源->审批->部署->测试等环节

2.资源利用率低

普遍服务器利用率低，造成过多浪费

3.扩容/缩容不及时

业务高峰期扩容流程繁琐，上线不及时

4.服务器环境臃肿

服务器越来越臃肿，对维护、迁移带来困难

Docker设计目标：

提供简单的应用程序打包工具
开发人员和运维人员职责逻辑分离
多环境保持一致性

Kubernetes设计目标：

集中管理所有容器
资源编排
资源调度
弹性伸缩
资源隔离

容器 vs 虚拟机

优点：
1、环境治理
2、提高服务器资源利用率
3、快速搭建新技术环境，不用学习复杂的部署环境
4、轻量级
5、虚拟化满足不了一些业务场景
6、完美构建微服务部署环境
7、一次构建，多地方部署
8、快速部署、迁移、回滚，不依赖底层环境
9、高度保持多个环境一致性

缺点：
1、安全性不如VM
2、隔离性不如VM
3、大规模不易管理，K8s应用而生
4、不易有状态应用部署
5、排查问题比较难
6、不支持Windows

原理
cgroups 资源限制:比如CPU/内存
namespace 资源隔离：进程、文件系统、用户等
ufs 联合文件系统：镜像增量式存储，提高磁盘利用率

Docker 基本使用

curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

http://mirrors.aliyun.com/repo/Centos-7.repo


安装docker注意事项：
1、替换国内yum源
2、服务器时区和时间保持国内
3、selinux和firewalld关闭


sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker


docker inspect image nginx

view

CentOS7.x安装Docker
# 安装依赖包
yum install -y yum-utils device-mapper-persistent-data lvm2  # 添加Docker软件包源
yum-config-manager \
--add-repo \  https://download.docker.com/linux/centos/docker-ce.repo
# 安装Docker CE
yum install -y docker-ce
# 启动Docker服务并设置开机启动 systemctl start docker
systemctl enable docker

官方文档：https://docs.docker.com
阿里云源：http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

理解容器镜像

镜像是什么？

一个分层存储的文件
一个软件的环境
一个镜像可以创建N个容器
一种标准化的交付
一个不包含Linux内核而又精简的Linux操作系统
镜像不是一个单一的文件，而是有多层构成。我们可以通过docker history <ID/NAME> 查看镜像中各层内容及大小，每层对应着Dockerfile中的一条指令。Docker镜像默认存储在/var/lib/docker/\<storage-driver\>中。

镜像从哪里来？
Docker Hub是由Docker公司负责维护的公共注册中心，包含大量的容器镜像，Docker工具默认从这个公共镜像库下载镜像。地址：https://hub.docker.com

配置镜像加速器：https://www.daocloud.io/mirror
curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io

如图，容器其实是在镜像的最上面加了一层读写层，在运行容器里文件改动时，会先从镜像里要写的文件复制到容器自己的文件系统中（读写层）。

如果容器删除了，最上面的读写层也就删除了，改动也就丢失了。所以无论多少个容器共享一个镜像，所做的写操作都是从镜像的文件系统中复制过来操作的，并不会修改镜像的源文件，这种方式提高磁盘利用率。

若想持久化这些改动，可以通过docker commit 将容器保存成一个新镜像。

一个镜像创建多个容器
镜像增量式存储
创建的容器里面修改不会影响到镜像

管理镜像常用命令表

编写Dockerfile的技巧：
1、如果追求镜像更小，选择alpine
2、运行的Shell命令尽可能写到一个RUN里面，减少镜像层
3、清理部署时产生留的缓存或者文件

FROM centos:7
RUN yum install -y gcc gcc-c++ make openssl-devel pcre-devel
RUN ./configure --prefix=/usr/local/nginx     --with-http_ssl_module --with-http_stub_status_module && \
make -j 4 && \
make install

创建应用容器并做资源限制

创建容器常用选项

容器资源限制参数表

示例：

内存限额：
允许容器最多使用500M内存和100M的Swap，并禁用 OOM Killer：
docker run -d --name nginx03 --memory="500m" --memory-swap="600m" --oom-kill-disable nginx
CPU限额：
允许容器最多使用一个半的CPU：
docker run -d --name nginx04 --cpus="1.5" nginx  允许容器最多使用50%的CPU：
docker run -d --name nginx05 --cpus=".5" nginx

管理容器常用命令表

创建一个容器：
1、设置容器名为hello
2、暴露88端口让外部访问
3、传入一个test=123456变量
4、设置开机启动

docker run -d --name hello -e test=123456  -p 88:80 --restart=always  nginx


docker run [OPTIONS] IMAGE [COMMAND] [ARG...]


docker exec -it nginx bash
docker run -it centos bash


nginx -g daemon off;


资源限制：内存、CPU、硬盘、网络


需要持久化的数据：
1、日志，一般用于方便日志采集和故障排查
2、配置文件，比如nginx配置文件
3、业务数据，比如mysql，网站程序
4、临时缓存数据，比如nginx-proxy-cache

-v 数据卷名称或者源目录:容器目标

bind mounts注意点:
1、宿主机文件或者目录必须存在才能成功挂载
2、宿主机文件或者目录覆盖容器中内容


镜像分类：
1、基础镜像，例如centos(yum)、ubuntu(apt)、alpine(apk)
2、环境镜像，例如php、jdk、python
3、项目镜像，打包好的可部署镜像

制作镜像：
1、选择一个符合线上操作系统的基础镜像
2、用基础镜像创建一个容器，手动在容器里面跑一边你要部署的应用
3、确认你启动这个应用的前台运行命令


源码安装：
0. 安装依赖包，例如gcc、make
1、./configure
2、make
3、make install





PHP镜像：
1、PHP环境，可以运行认可php脚本
2、PHP-FPM

java -jar xxx.jar

docker-compose 单机容器编排工具

view code

持久化容器中应用程序数据

Docker提供三种方式将数据从宿主机挂载到容器中：

volumes：Docker管理宿主机文件系统的一部分（/var/lib/docker/volumes）。保存数据的最佳方式。
bind mounts：将宿主机上的任意位置的文件或者目录挂载到容器中。
tmpfs：挂载存储在主机系统的内存中，而不会写入主机的文件系统。如果不希望将数据持久存储在任何位置，可以使用 tmpfs，同时避免写入容器可写层提高性能。

Dockerfile 构建常见基础镜像

[root@mysql dockerfile]# tree
.
├── java
│    └── Dockerfile
├── nginx
│    ├── Dockerfile
│    ├── nginx-1.15.5.tar.gz
│    └── nginx.conf
├── php
│    ├── Dockerfile
│    ├── php-5.6.36.tar.gz
│    ├── php-fpm.conf
│    └── php.ini
└── tomcat
    ├── apache-tomcat-8.5.43.tar.gz
    └── Dockerfile

View Code

构建Nginx基础镜像

FROM centos:7
LABEL maintainer www.ctnrs.com
RUN yum install -y gcc gcc-c++ make \
    openssl-devel pcre-devel gd-devel \
    iproute net-tools telnet wget curl && \
    yum clean all && \
    rm -rf /var/cache/yum/*

COPY nginx-1.15.5.tar.gz /
RUN tar zxf nginx-1.15.5.tar.gz && \
    cd nginx-1.15.5 && \
    ./configure --prefix=/usr/local/nginx \
    --with-http_ssl_module \
    --with-http_stub_status_module && \
    make -j 4 && make install && \
    rm -rf /usr/local/nginx/html/* && \
    echo "ok" >> /usr/local/nginx/html/status.html && \
    cd / && rm -rf nginx* && \
    ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

ENV PATH $PATH:/usr/local/nginx/sbin
COPY nginx.conf /usr/local/nginx/conf/nginx.conf
WORKDIR /usr/local/nginx
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

View Code

构建PHP基础镜像

FROM centos:7
MAINTAINER www.ctnrs.com
RUN yum install epel-release -y && \
    yum install -y gcc gcc-c++ make gd-devel libxml2-devel \
    libcurl-devel libjpeg-devel libpng-devel openssl-devel \
    libmcrypt-devel libxslt-devel libtidy-devel autoconf \
    iproute net-tools telnet wget curl && \
    yum clean all && \
    rm -rf /var/cache/yum/*

COPY php-5.6.36.tar.gz /
RUN tar zxf php-5.6.36.tar.gz && \
    cd php-5.6.36 && \
    ./configure --prefix=/usr/local/php \
    --with-config-file-path=/usr/local/php/etc \
    --enable-fpm --enable-opcache \
    --with-mysql --with-mysqli --with-pdo-mysql \
    --with-openssl --with-zlib --with-curl --with-gd \
    --with-jpeg-dir --with-png-dir --with-freetype-dir \
    --enable-mbstring --with-mcrypt --enable-hash && \
    make -j 4 && make install && \
    cp php.ini-production /usr/local/php/etc/php.ini && \
    cp sapi/fpm/php-fpm.conf /usr/local/php/etc/php-fpm.conf && \
    sed -i "90a \daemonize = no" /usr/local/php/etc/php-fpm.conf && \
    mkdir /usr/local/php/log && \
    cd / && rm -rf php* && \
    ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

ENV PATH $PATH:/usr/local/php/sbin
COPY php.ini /usr/local/php/etc/
COPY php-fpm.conf /usr/local/php/etc/
WORKDIR /usr/local/php
EXPOSE 9000
CMD ["php-fpm"]

View Code

;;;;;;;;;;;;;;;;;;;;;
; FPM Configuration ;
;;;;;;;;;;;;;;;;;;;;;

; All relative paths in this configuration file are relative to PHP's install
; prefix (/usr/local/php). This prefix can be dynamically changed by using the
; '-p' argument from the command line.

; Include one or more files. If glob(3) exists, it is used to include a bunch of
; files from a glob(3) pattern. This directive can be used everywhere in the
; file.
; Relative path can also be used. They will be prefixed by:
;  - the global prefix if it's been set (-p argument)
;  - /usr/local/php otherwise
;include=etc/fpm.d/*.conf

;;;;;;;;;;;;;;;;;;
; Global Options ;
;;;;;;;;;;;;;;;;;;

[global]
; Pid file
; Note: the default prefix is /usr/local/php/var
; Default Value: none
pid = /var/run/php-fpm.pid

; Error log file
; If it's set to "syslog", log is sent to syslogd instead of being written
; in a local file.
; Note: the default prefix is /usr/local/php/var
; Default Value: log/php-fpm.log
error_log = /usr/local/php/log/php-fpm.log

; syslog_facility is used to specify what type of program is logging the
; message. This lets syslogd specify that messages from different facilities
; will be handled differently.
; See syslog(3) for possible values (ex daemon equiv LOG_DAEMON)
; Default Value: daemon
;syslog.facility = daemon

; syslog_ident is prepended to every message. If you have multiple FPM
; instances running on the same server, you can change the default value
; which must suit common needs.
; Default Value: php-fpm
;syslog.ident = php-fpm

; Log level
; Possible Values: alert, error, warning, notice, debug
; Default Value: notice
log_level = warning

; If this number of child processes exit with SIGSEGV or SIGBUS within the time
; interval set by emergency_restart_interval then FPM will restart. A value
; of '0' means 'Off'.
; Default Value: 0
;emergency_restart_threshold = 0

; Interval of time used by emergency_restart_interval to determine when
; a graceful restart will be initiated.  This can be useful to work around
; accidental corruptions in an accelerator's shared memory.
; Available Units: s(econds), m(inutes), h(ours), or d(ays)
; Default Unit: seconds
; Default Value: 0
emergency_restart_interval = 24h

; Time limit for child processes to wait for a reaction on signals from master.
; Available units: s(econds), m(inutes), h(ours), or d(ays)
; Default Unit: seconds
; Default Value: 0
process_control_timeout = 5s

; The maximum number of processes FPM will fork. This has been design to control
; the global number of processes when using dynamic PM within a lot of pools.
; Use it with caution.
; Note: A value of 0 indicates no limit
; Default Value: 0
; process.max = 128

; Specify the nice(2) priority to apply to the master process (only if set)
; The value can vary from -19 (highest priority) to 20 (lower priority)
; Note: - It will only work if the FPM master process is launched as root
;       - The pool process will inherit the master process priority
;         unless it specified otherwise
; Default Value: no set
; process.priority = -19

; Send FPM to background. Set to 'no' to keep FPM in foreground for debugging.
; Default Value: yes
daemonize = no

; Set open file descriptor rlimit for the master process.
; Default Value: system defined value
rlimit_files = 10240

; Set max core size rlimit for the master process.
; Possible Values: 'unlimited' or an integer greater or equal to 0
; Default Value: system defined value
;rlimit_core = 0

; Specify the event mechanism FPM will use. The following is available:
; - select     (any POSIX os)
; - poll       (any POSIX os)
; - epoll      (linux >= 2.5.44)
; - kqueue     (FreeBSD >= 4.1, OpenBSD >= 2.9, NetBSD >= 2.0)
; - /dev/poll  (Solaris >= 7)
; - port       (Solaris >= 10)
; Default Value: not set (auto detection)
;events.mechanism = epoll

; When FPM is build with systemd integration, specify the interval,
; in second, between health report notification to systemd.
; Set to 0 to disable.
; Available Units: s(econds), m(inutes), h(ours)
; Default Unit: seconds
; Default value: 10
;systemd_interval = 10

;;;;;;;;;;;;;;;;;;;;
; Pool Definitions ;
;;;;;;;;;;;;;;;;;;;;

; Multiple pools of child processes may be started with different listening
; ports and different management options.  The name of the pool will be
; used in logs and stats. There is no limitation on the number of pools which
; FPM can handle. Your system will tell you anyway :)

; Start a new pool named 'www'.
; the variable $pool can we used in any directive and will be replaced by the
; pool name ('www' here)
[www]

; Per pool prefix
; It only applies on the following directives:
; - 'access.log'
; - 'slowlog'
; - 'listen' (unixsocket)
; - 'chroot'
; - 'chdir'
; - 'php_values'
; - 'php_admin_values'
; When not set, the global prefix (or /usr/local/php) applies instead.
; Note: This directive can also be relative to the global prefix.
; Default Value: none
;prefix = /path/to/pools/$pool

; Unix user/group of processes
; Note: The user is mandatory. If the group is not set, the default user's group
;       will be used.
user = nobody
group = nobody

; The address on which to accept FastCGI requests.
; Valid syntaxes are:
;   'ip.add.re.ss:port'    - to listen on a TCP socket to a specific IPv4 address on
;                            a specific port;
;   '[ip:6:addr:ess]:port' - to listen on a TCP socket to a specific IPv6 address on
;                            a specific port;
;   'port'                 - to listen on a TCP socket to all IPv4 addresses on a
;                            specific port;
;   '[::]:port'            - to listen on a TCP socket to all addresses
;                            (IPv6 and IPv4-mapped) on a specific port;
;   '/path/to/unix/socket' - to listen on a unix socket.
; Note: This value is mandatory.
;listen = 127.0.0.1:9000
listen = 0.0.0.0:9000

; Set listen(2) backlog.
; Default Value: 65535 (-1 on FreeBSD and OpenBSD)
;listen.backlog = 65535

; Set permissions for unix socket, if one is used. In Linux, read/write
; permissions must be set in order to allow connections from a web server. Many
; BSD-derived systems allow connections regardless of permissions.
; Default Values: user and group are set as the running user
;                 mode is set to 0660
listen.owner = nobody
listen.group = nobody
;listen.mode = 0660
; When POSIX Access Control Lists are supported you can set them using
; these options, value is a comma separated list of user/group names.
; When set, listen.owner and listen.group are ignored
;listen.acl_users =
;listen.acl_groups =

; List of addresses (IPv4/IPv6) of FastCGI clients which are allowed to connect.
; Equivalent to the FCGI_WEB_SERVER_ADDRS environment variable in the original
; PHP FCGI (5.2.2+). Makes sense only with a tcp listening socket. Each address
; must be separated by a comma. If this value is left blank, connections will be
; accepted from any ip address.
; Default Value: any
; listen.allowed_clients = 127.0.0.1

; Specify the nice(2) priority to apply to the pool processes (only if set)
; The value can vary from -19 (highest priority) to 20 (lower priority)
; Note: - It will only work if the FPM master process is launched as root
;       - The pool processes will inherit the master process priority
;         unless it specified otherwise
; Default Value: no set
; process.priority = -19

; Set the process dumpable flag (PR_SET_DUMPABLE prctl) even if the process user
; or group is differrent than the master process user. It allows to create process
; core dump and ptrace the process for the pool user.
; Default Value: no
; process.dumpable = yes

; Choose how the process manager will control the number of child processes.
; Possible Values:
;   static  - a fixed number (pm.max_children) of child processes;
;   dynamic - the number of child processes are set dynamically based on the
;             following directives. With this process management, there will be
;             always at least 1 children.
;             pm.max_children      - the maximum number of children that can
;                                    be alive at the same time.
;             pm.start_servers     - the number of children created on startup.
;             pm.min_spare_servers - the minimum number of children in 'idle'
;                                    state (waiting to process). If the number
;                                    of 'idle' processes is less than this
;                                    number then some children will be created.
;             pm.max_spare_servers - the maximum number of children in 'idle'
;                                    state (waiting to process). If the number
;                                    of 'idle' processes is greater than this
;                                    number then some children will be killed.
;  ondemand - no children are created at startup. Children will be forked when
;             new requests will connect. The following parameter are used:
;             pm.max_children           - the maximum number of children that
;                                         can be alive at the same time.
;             pm.process_idle_timeout   - The number of seconds after which
;                                         an idle process will be killed.
; Note: This value is mandatory.
pm = dynamic

; The number of child processes to be created when pm is set to 'static' and the
; maximum number of child processes when pm is set to 'dynamic' or 'ondemand'.
; This value sets the limit on the number of simultaneous requests that will be
; served. Equivalent to the ApacheMaxClients directive with mpm_prefork.
; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP
; CGI. The below defaults are based on a server without much resources. Don't
; forget to tweak pm.* to fit your needs.
; Note: Used when pm is set to 'static', 'dynamic' or 'ondemand'
; Note: This value is mandatory.
pm.max_children = 200

; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 50

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 50

; The desired maximum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.max_spare_servers = 100

; The number of seconds after which an idle process will be killed.
; Note: Used only when pm is set to 'ondemand'
; Default Value: 10s
;pm.process_idle_timeout = 10s;

; The number of requests each child process should execute before respawning.
; This can be useful to work around memory leaks in 3rd party libraries. For
; endless request processing specify '0'. Equivalent to PHP_FCGI_MAX_REQUESTS.
; Default Value: 0
pm.max_requests = 51200

; The URI to view the FPM status page. If this value is not set, no URI will be
; recognized as a status page. It shows the following informations:
;   pool                 - the name of the pool;
;   process manager      - static, dynamic or ondemand;
;   start time           - the date and time FPM has started;
;   start since          - number of seconds since FPM has started;
;   accepted conn        - the number of request accepted by the pool;
;   listen queue         - the number of request in the queue of pending
;                          connections (see backlog in listen(2));
;   max listen queue     - the maximum number of requests in the queue
;                          of pending connections since FPM has started;
;   listen queue len     - the size of the socket queue of pending connections;
;   idle processes       - the number of idle processes;
;   active processes     - the number of active processes;
;   total processes      - the number of idle + active processes;
;   max active processes - the maximum number of active processes since FPM
;                          has started;
;   max children reached - number of times, the process limit has been reached,
;                          when pm tries to start more children (works only for
;                          pm 'dynamic' and 'ondemand');
; Value are updated in real time.
; Example output:
;   pool:                 www
;   process manager:      static
;   start time:           01/Jul/2011:17:53:49 +0200
;   start since:          62636
;   accepted conn:        190460
;   listen queue:         0
;   max listen queue:     1
;   listen queue len:     42
;   idle processes:       4
;   active processes:     11
;   total processes:      15
;   max active processes: 12
;   max children reached: 0
;
; By default the status page output is formatted as text/plain. Passing either
; 'html', 'xml' or 'json' in the query string will return the corresponding
; output syntax. Example:
;   http://www.foo.bar/status
;   http://www.foo.bar/status?json
;   http://www.foo.bar/status?html
;   http://www.foo.bar/status?xml
;
; By default the status page only outputs short status. Passing 'full' in the
; query string will also return status for each pool process.
; Example:
;   http://www.foo.bar/status?full
;   http://www.foo.bar/status?json&full
;   http://www.foo.bar/status?html&full
;   http://www.foo.bar/status?xml&full
; The Full status returns for each process:
;   pid                  - the PID of the process;
;   state                - the state of the process (Idle, Running, ...);
;   start time           - the date and time the process has started;
;   start since          - the number of seconds since the process has started;
;   requests             - the number of requests the process has served;
;   request duration     - the duration in µs of the requests;
;   request method       - the request method (GET, POST, ...);
;   request URI          - the request URI with the query string;
;   content length       - the content length of the request (only with POST);
;   user                 - the user (PHP_AUTH_USER) (or '-' if not set);
;   script               - the main script called (or '-' if not set);
;   last request cpu     - the %cpu the last request consumed
;                          it's always 0 if the process is not in Idle state
;                          because CPU calculation is done when the request
;                          processing has terminated;
;   last request memory  - the max amount of memory the last request consumed
;                          it's always 0 if the process is not in Idle state
;                          because memory calculation is done when the request
;                          processing has terminated;
; If the process is in Idle state, then informations are related to the
; last request the process has served. Otherwise informations are related to
; the current request being served.
; Example output:
;   ************************
;   pid:                  31330
;   state:                Running
;   start time:           01/Jul/2011:17:53:49 +0200
;   start since:          63087
;   requests:             12808
;   request duration:     1250261
;   request method:       GET
;   request URI:          /test_mem.php?N=10000
;   content length:       0
;   user:                 -
;   script:               /home/fat/web/docs/php/test_mem.php
;   last request cpu:     0.00
;   last request memory:  0
;
; Note: There is a real-time FPM status monitoring sample web page available
;       It's available in: /usr/local/php/share/php/fpm/status.html
;
; Note: The value must start with a leading slash (/). The value can be
;       anything, but it may not be a good idea to use the .php extension or it
;       may conflict with a real PHP file.
; Default Value: not set
pm.status_path = /status

; The ping URI to call the monitoring page of FPM. If this value is not set, no
; URI will be recognized as a ping page. This could be used to test from outside
; that FPM is alive and responding, or to
; - create a graph of FPM availability (rrd or such);
; - remove a server from a group if it is not responding (load balancing);
; - trigger alerts for the operating team (24/7).
; Note: The value must start with a leading slash (/). The value can be
;       anything, but it may not be a good idea to use the .php extension or it
;       may conflict with a real PHP file.
; Default Value: not set
;ping.path = /ping

; This directive may be used to customize the response of a ping request. The
; response is formatted as text/plain with a 200 response code.
; Default Value: pong
;ping.response = pong

; The access log file
; Default: not set
;access.log = log/$pool.access.log

; The access log format.
; The following syntax is allowed
;  %%: the '%' character
;  %C: %CPU used by the request
;      it can accept the following format:
;      - %{user}C for user CPU only
;      - %{system}C for system CPU only
;      - %{total}C  for user + system CPU (default)
;  %d: time taken to serve the request
;      it can accept the following format:
;      - %{seconds}d (default)
;      - %{miliseconds}d
;      - %{mili}d
;      - %{microseconds}d
;      - %{micro}d
;  %e: an environment variable (same as $_ENV or $_SERVER)
;      it must be associated with embraces to specify the name of the env
;      variable. Some exemples:
;      - server specifics like: %{REQUEST_METHOD}e or %{SERVER_PROTOCOL}e
;      - HTTP headers like: %{HTTP_HOST}e or %{HTTP_USER_AGENT}e
;  %f: script filename
;  %l: content-length of the request (for POST request only)
;  %m: request method
;  %M: peak of memory allocated by PHP
;      it can accept the following format:
;      - %{bytes}M (default)
;      - %{kilobytes}M
;      - %{kilo}M
;      - %{megabytes}M
;      - %{mega}M
;  %n: pool name
;  %o: output header
;      it must be associated with embraces to specify the name of the header:
;      - %{Content-Type}o
;      - %{X-Powered-By}o
;      - %{Transfert-Encoding}o
;      - ....
;  %p: PID of the child that serviced the request
;  %P: PID of the parent of the child that serviced the request
;  %q: the query string
;  %Q: the '?' character if query string exists
;  %r: the request URI (without the query string, see %q and %Q)
;  %R: remote IP address
;  %s: status (response code)
;  %t: server time the request was received
;      it can accept a strftime(3) format:
;      %d/%b/%Y:%H:%M:%S %z (default)
;  %T: time the log has been written (the request has finished)
;      it can accept a strftime(3) format:
;      %d/%b/%Y:%H:%M:%S %z (default)
;  %u: remote user
;
; Default: "%R - %u %t \"%m %r\" %s"
;access.format = "%R - %u %t \"%m %r%Q%q\" %s %f %{mili}d %{kilo}M %C%%"

; The log file for slow requests
; Default Value: not set
; Note: slowlog is mandatory if request_slowlog_timeout is set
slowlog = log/$pool.log.slow

; The timeout for serving a single request after which a PHP backtrace will be
; dumped to the 'slowlog' file. A value of '0s' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
request_slowlog_timeout = 10

; The timeout for serving a single request after which the worker process will
; be killed. This option should be used when the 'max_execution_time' ini option
; does not stop script execution for some reason. A value of '0' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
request_terminate_timeout = 600

; Set open file descriptor rlimit.
; Default Value: system defined value
rlimit_files = 10240

; Set max core size rlimit.
; Possible Values: 'unlimited' or an integer greater or equal to 0
; Default Value: system defined value
;rlimit_core = 0

; Chroot to this directory at the start. This value must be defined as an
; absolute path. When this value is not set, chroot is not used.
; Note: you can prefix with '$prefix' to chroot to the pool prefix or one
; of its subdirectories. If the pool prefix is not set, the global prefix
; will be used instead.
; Note: chrooting is a great security feature and should be used whenever
;       possible. However, all PHP paths will be relative to the chroot
;       (error_log, sessions.save_path, ...).
; Default Value: not set
;chroot =

; Chdir to this directory at the start.
; Note: relative path can be used.
; Default Value: current directory or / when chroot
;chdir = /var/www

; Redirect worker stdout and stderr into main error log. If not set, stdout and
; stderr will be redirected to /dev/null according to FastCGI specs.
; Note: on highloaded environement, this can cause some delay in the page
; process time (several ms).
; Default Value: no
;catch_workers_output = yes

; Clear environment in FPM workers
; Prevents arbitrary environment variables from reaching FPM worker processes
; by clearing the environment in workers before env vars specified in this
; pool configuration are added.
; Setting to "no" will make all environment variables available to PHP code
; via getenv(), $_ENV and $_SERVER.
; Default Value: yes
;clear_env = no

; Limits the extensions of the main script FPM will allow to parse. This can
; prevent configuration mistakes on the web server side. You should only limit
; FPM to .php extensions to prevent malicious users to use other extensions to
; exectute php code.
; Note: set an empty value to allow all extensions.
; Default Value: .php
;security.limit_extensions = .php .php3 .php4 .php5

; Pass environment variables like LD_LIBRARY_PATH. All $VARIABLEs are taken from
; the current environment.
; Default Value: clean env
;env[HOSTNAME] = $HOSTNAME
;env[PATH] = /usr/local/bin:/usr/bin:/bin
;env[TMP] = /tmp
;env[TMPDIR] = /tmp
;env[TEMP] = /tmp

; Additional php.ini defines, specific to this pool of workers. These settings
; overwrite the values previously defined in the php.ini. The directives are the
; same as the PHP SAPI:
;   php_value/php_flag             - you can set classic ini defines which can
;                                    be overwritten from PHP call 'ini_set'.
;   php_admin_value/php_admin_flag - these directives won't be overwritten by
;                                     PHP call 'ini_set'
; For php_*flag, valid values are on, off, 1, 0, true, false, yes or no.

; Defining 'extension' will load the corresponding shared extension from
; extension_dir. Defining 'disable_functions' or 'disable_classes' will not
; overwrite previously defined php.ini values, but will append the new value
; instead.

; Note: path INI options can be relative and will be expanded with the prefix
; (pool, global or /usr/local/php)

; Default Value: nothing is defined by default except the values in php.ini and
;                specified at startup with the -d argument
;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f www@my.domain.com
;php_flag[display_errors] = off
;php_admin_value[error_log] = /var/log/fpm-php.www.log
;php_admin_flag[log_errors] = on
;php_admin_value[memory_limit] = 32M

php-fpm.conf

构建Tomcat基础镜像

FROM centos:7
MAINTAINER www.ctnrs.com

ENV VERSION=8.5.43

RUN yum install java-1.8.0-openjdk wget curl unzip iproute net-tools -y && \
    yum clean all && \
    rm -rf /var/cache/yum/*

COPY apache-tomcat-${VERSION}.tar.gz /
RUN tar zxf apache-tomcat-${VERSION}.tar.gz && \
    mv apache-tomcat-${VERSION} /usr/local/tomcat && \
    rm -rf apache-tomcat-${VERSION}.tar.gz /usr/local/tomcat/webapps/* && \
    mkdir /usr/local/tomcat/webapps/test && \
    echo "ok" > /usr/local/tomcat/webapps/test/status.html && \
    sed -i '1a JAVA_OPTS="-Djava.security.egd=file:/dev/./urandom"' /usr/local/tomcat/bin/catalina.sh && \
    ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

ENV PATH $PATH:/usr/local/tomcat/bin

WORKDIR /usr/local/tomcat

EXPOSE 8080
CMD ["catalina.sh", "run"]

View Code

构建Java基础镜像

FROM java:8-jdk-alpine
LABEL maintainer www.ctnrs.com
ENV JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF8 -Duser.timezone=GMT+08"
RUN  apk add -U tzdata && \
     ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
COPY ./target/eureka-service.jar ./
EXPOSE 8888
CMD java -jar $JAVA_OPTS /eureka-service.jar

View Code

企业级 Harbor 镜像仓库

Harbor 是由VMWare公司开源的容器镜像仓库。事实上，Harbor是在Docker Registry上进行了相应的企业级扩展，从而获得了更加广泛的应用，这些新的企业级特性包括：管理用户界面，基于角色的访问控制，AD/LDAP集成以及审计日志等，足以满足基本企业需求。官方地址：https://vmware.github.io/harbor/cn/

1、安装docker与docker-compose  依赖Python环境
wget http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum install docker-ce -y
systemctl start docker
systemctl enable docker
2、解压离线包部署
tar zxvf harbor-offline-installer-v1.9.1.tgz
cd harbor
vi harbor.yml
hostname: 10.0.0.70
./prepare
./install.sh

在Jenkins主机配置Docker可信任

由于habor未配置https，还需要在docker配置可信任。

# cat /etc/docker/daemon.json
{"registry-mirrors": ["http://f1361db2.m.daocloud.io"],
  "insecure-registries": ["10.0.0.0.70"]
}
# systemctl restart docker

[root@mysql harbor]# docker-compose ps
      Name                     Command                  State                 Ports
---------------------------------------------------------------------------------------------
harbor-core         /harbor/harbor_core              Up (healthy)
harbor-db           /docker-entrypoint.sh            Up (healthy)   5432/tcp
harbor-jobservice   /harbor/harbor_jobservice  ...   Up (healthy)
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp
redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp
registry            /entrypoint.sh /etc/regist ...   Up (healthy)   5000/tcp
registryctl         /harbor/start.sh                 Up (healthy)

查看启动情况

admin 默认密码：Harbor12345

创建用户

终端登录 Harbor

# 报错
[root@mysql harbor]# docker login 10.0.0.70
Username: admin
Password:
Error response from daemon: Get http://10.0.0.70/v2/: dial tcp 10.0.0.70:80: connect: connection refused
# 重启
docker-compose down -v
docker-compose up -d

配置上传镜像

1、配置http镜像仓库可信任

[root@mysql harbor]# cat /etc/docker/daemon.json
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],
  "insecure-registries": ["10.0.0.70"]
}
systemctl restart docker

2、打标签
docker tag tomcat:v1 10.0.0.70/library/tomcat:v1

3、上传
 for i in {nginx,php};do docker push 10.0.0.70/library/${i}:v1;done

4、下载
# docker pull 10.0.0.70/library/nginx:v1

上传完毕后查看

基于 Docker 构建企业 Jenkins CI平台

持续集成（Continuous Integration，CI）：代码合并、构建、部署、测试都在一起，不断地执行这个过程，并对结果反馈。
持续部署（Continuous Deployment，CD）：部署到测试环境、预生产环境、生产环境。
持续交付（Continuous Delivery，CD）：将最终产品发布到生产环境，给用户使用。

高效的CI/CD环境可以获得：

•及时发现问题
•大幅度减少故障率
•加快迭代速度
•减少时间成本

CI 工作流程

CI流程：
1、拉取代码
2、代码编译（java项目），产出war包
3、打包项目镜像并推送到镜像仓库
4、部署镜像测试

部署gitlab

docker run -d \
  --name gitlab \
  -p 8443:443 \
  -p 9999:80 \
  -p 9998:22 \
  -v $PWD/config:/etc/gitlab \
  -v $PWD/logs:/var/log/gitlab \
  -v $PWD/data:/var/opt/gitlab \
  -v /etc/localtime:/etc/localtime \
  lizhenliang/gitlab-ce-zh:latest
  gitlab/gitlab-ce:latest

访问地址：http://IP:9999

初次会先设置管理员密码，然后登陆，默认管理员用户名root，密码就是刚设置的。

创建项目，提交测试代码

进入后先创建项目，提交代码，以便后面测试。

git remote add pb http://10.0.0.70:9999/root/java-demo.git

[root@mysql tomcat-java-demo-master]# cat .git/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "pb"]
    url = http://10.0.0.70:9999/root/java-demo.git
    fetch = +refs/heads/*:refs/remotes/pb/*



git config --global user.email "fadewalk@163.com"
git config --global user.name "fadewalk"



[root@mysql tomcat-java-demo-master]# git push pb master
Username for 'http://10.0.0.70:9999': root
Password for 'http://root@10.0.0.70:9999':
Counting objects: 179, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (166/166), done.
Writing objects: 100% (179/179), 1.12 MiB | 0 bytes/s, done.
Total 179 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), done.
To http://10.0.0.70:9999/root/java-demo.git
 * [new branch]      master -> master

代码push

准备JDK和Maven环境

解压相应的包到宿主机的目录

tar zxvf jdk-8u45-linux-x64.tar.gz
mv jdk1.8.0_45 /usr/local/jdk
tar zxf apache-maven-3.5.0-bin.tar.gz
mv apache-maven-3.5.0 /usr/local/maven

启动Jenkins容器

docker run -d --name jenkins -p 8099:8080 -p 50000:50000 -u root  \
   -v /opt/jenkins_home:/var/jenkins_home \
   -v /var/run/docker.sock:/var/run/docker.sock   \
   -v /usr/bin/docker:/usr/bin/docker \
   -v /usr/local/maven:/usr/local/maven \
   -v /usr/local/jdk:/usr/local/jdk \
   -v /etc/localtime:/etc/localtime \
   --name jenkins jenkins/jenkins:lts

将宿主机中的环境目录直接挂载到Jenkins容器的目录中，实现利用宿主机的环境构建性能

[root@mysql tools]# docker exec -it jenkins bash
root@d413a8199d28:/# cat /var/jenkins_home/secrets/initialAdminPassword
813c4d8fc29f45c7b898e73a79f03283

使用/root/.ssh中私钥访问gitlab。更加的方便安全。在Jenkins中配置私钥访问gitlab代码仓库。

安装插件

系统管理-->插件管理-->Installed

搜索git/pipeline，点击安装。在线安装不行，可以离线安装，或者在初始化Jenkins的页面就把插件都安装好，只是要多等几分钟。

插件页面这里配置了以后，一直是报错，没有用

修改国内源：
cd /opt/jenkins_home/updates
sed -i 's/http:\/\/updates.jenkins-ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json && \
sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json
然后重启jenkins容器生效。

or 修改国内源

添加参数化构建

This project is parameterized -> String Parameter

Name：Branch # 变量名，下面脚本中调用

Default Value：master # 默认分支

Description：发布的代码分支 # 描述

Pipeline脚本

#!/usr/bin/env groovy

def registry = "10.0.0.70"
def project = "dev"
def app_name = "java-demo"
def image_name = "${registry}/${project}/${app_name}:${Branch}-${BUILD_NUMBER}"
def git_address = "http://root:wyl606,git@10.0.0.70:9999/root/java-demo.git"
def docker_registry_auth = "6041d04f-2b8f-4275-9dff-4c528a28c610"
def git_auth = "'05562e63-b2bf-4e0b-bb2c-2513b10b599b"

pipeline {
    agent any
    stages {
        stage('拉取代码'){
            steps {
              checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
            }
        }

        stage('代码编译'){
           steps {
             sh """
                JAVA_HOME=/usr/local/jdk
                PATH=$JAVA_HOME/bin:/usr/local/maven/bin:$PATH
                mvn clean package -Dmaven.test.skip=true
                """
           }
        }

        stage('构建镜像'){
           steps {
                withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
                sh """
                  echo '
                    FROM ${registry}/library/tomcat:v1
                    LABEL maitainer lizhenliang
                    RUN rm -rf /usr/local/tomcat/webapps/*
                    ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
                  ' > Dockerfile
                  docker build -t ${image_name} .
                  docker login -u ${username} -p '${password}' ${registry}
                  docker push ${image_name}
                """
                }
           }
        }

        stage('部署到Docker'){
           steps {
              sh """
              REPOSITORY=${image_name}
              docker rm -f tomcat-java-demo |true
              docker container run -d --name tomcat-java-demo -p 88:8080 ${image_name}
              """
            }
        }
    }
}

Pipeline script

添加凭据

这里Jenkins 中 gitlab的用户名和密码生成的凭据id 通过http无法正常连接，
只好将用户名和密码写到了连接中的方式解决

1、添加拉取git代码凭据，并获取id替换到上面git_auth变量值。
2、添加拉取harbor镜像凭据，并获取id替换到上面docker_registry_auth变量值。

凭据的id要与scripts 里面的变量一致

点击进行构建

构建的时候
maven建议采用国内的源，这样更快一些。
在配置文件中添加

maven 镜像添加
[root@k8s-m1 conf]# grep ali settings.xml
      <name>aliyun maven</name>
      <url>https://maven.aliyun.com/repository/public</url>

流水线构建完成后，在docker的镜像仓库Harbor 里面查看push 上来的镜像

部署成功，在部署应用端查看

当修改代码，或者提交新的分支。后gitlab仓库的master会更新。

后面又触发新的构建

Prometheus+Grafana 监控 Docker

cAdvisor （Container Advisor）：用于收集正在运行的容器资源使用和性能信息。 https://github.com/google/cadvisor
Prometheus（普罗米修斯）：容器监控系统。 https://prometheus.io https://github.com/prometheus
Grafana：是一个开源的度量分析和可视化系统。 https://grafana.com/grafana/download https://grafana.com/dashboards/193（监控Docker主机模板）

被监控端执行 10.0.0.70

Docker部署cAdvisor：
docker run -d \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \  google/cadvisor:latest

监控端执行 10.0.0.23

Docker部署Grafana：
docker run -d \
--name=grafana \
-p 3000:3000 \
grafana/grafana

Docker部署Prometheus：
docker run -d \
--name=prometheus \
-p 9090:9090 \
-v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml \  prom/prometheus

[root@k8s-m1 tmp]# cat prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['10.0.0.70:8080']

cat prometheus.yml