配置 Containerd 在 harbor 私有仓库拉取镜像

官方文档地址：https://github.com/containerd/cri/blob/master/docs/registry.md

严格来说，这个具体可分为两部分
1.在k8s中使用Containerd，从 harbor 私有仓库拉取镜像
2.单独使用Containerd进行配置，从 harbor 私有仓库拉取镜像 并运行容器

docker-compose安装harbor：https://www.cnblogs.com/sanduzxcvbnm/p/16370495.html

参考文章地址：
https://www.cnblogs.com/rancherlabs/p/14324469.html
https://blog.weiyigeek.top/2021/6-30-581.html

官方文档内容

# Configure Image Registry

This document describes the method to configure the image registry for `containerd` for use with the `cri` plugin.

NOTE: The configuration syntax used in this doc is in version 2 which is the
recommended since `containerd` 1.3. If your configuration is still in version 1,
you can replace `"io.containerd.grpc.v1.cri"` with `cri`.

## Configure Registry Endpoint

With containerd, `docker.io` is the default image registry. You can also set up other image registries similar to docker.

To configure image registries create/modify the `/etc/containerd/config.toml` as follows:

```toml
# Config file is parsed as version 1 by default.
# To use the long form of plugin names set "version = 2"
# explicitly use v2 config format
version = 2

[plugin."io.containerd.grpc.v1.cri".registry.mirrors]
  [plugin."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["https://registry-1.docker.io"]
  [plugin."io.containerd.grpc.v1.cri".registry.mirrors."test.https-registry.io"]
    endpoint = ["https://HostIP1:Port1"]
  [plugin."io.containerd.grpc.v1.cri".registry.mirrors."test.http-registry.io"]
    endpoint = ["http://HostIP2:Port2"]
  # wildcard matching is supported but not required.
  [plugin."io.containerd.grpc.v1.cri".registry.mirrors."*"]
    endpoint = ["https://HostIP3:Port3"]
```

The default configuration can be generated by `containerd config default > /etc/containerd/config.toml`.

The endpoint is a list that can contain multiple image registry URLs split by commas. When pulling an image
from a registry, containerd will try these endpoint URLs one by one, and use the first working one. Please note
that if the default registry endpoint is not already specified in the endpoint list, it will be automatically
tried at the end with scheme `https` and path `v2`, e.g. `https://gcr.io/v2` for `gcr.io`.

As an example, for the image `gcr.io/library/busybox:latest`, the endpoints are:

* `gcr.io` is configured: endpoints for `gcr.io` + default endpoint `https://gcr.io/v2`.
* `*` is configured, and `gcr.io` is not: endpoints for `*` + default
  endpoint `https://gcr.io/v2`.
* None of above is configured: default endpoint `https://gcr.io/v2`.

After modify this config, you need restart the `containerd` service.

## Configure Registry TLS Communication

`cri` plugin also supports configuring TLS settings when communicating with a registry.

To configure the TLS settings for a specific registry, create/modify the `/etc/containerd/config.toml` as follows:

```toml
# explicitly use v2 config format
version = 2

# The registry host has to be a domain name or IP. Port number is also
# needed if the default HTTPS or HTTP port is not used.
[plugin."io.containerd.grpc.v1.cri".registry.configs."my.custom.registry".tls]
    ca_file   = "ca.pem"
    cert_file = "cert.pem"
    key_file  = "key.pem"
```

In the config example shown above, TLS mutual authentication will be used for communications with the registry endpoint located at <https://my.custom.registry>.
`ca_file` is file name of the certificate authority (CA) certificate used to authenticate the x509 certificate/key pair specified by the files respectively pointed to by `cert_file` and `key_file`.

`cert_file` and `key_file` are not needed when TLS mutual authentication is unused.

```toml
# explicitly use v2 config format
version = 2

[plugin."io.containerd.grpc.v1.cri".registry.configs."my.custom.registry".tls]
    ca_file   = "ca.pem"
```

To skip the registry certificate verification:

```toml
# explicitly use v2 config format
version = 2

[plugin."io.containerd.grpc.v1.cri".registry.configs."my.custom.registry".tls]
  insecure_skip_verify = true
```

## Configure Registry Credentials

`cri` plugin also supports docker like registry credential config.

To configure a credential for a specific registry, create/modify the
`/etc/containerd/config.toml` as follows:

```toml
# explicitly use v2 config format
version = 2

# The registry host has to be a domain name or IP. Port number is also
# needed if the default HTTPS or HTTP port is not used.
[plugin."io.containerd.grpc.v1.cri".registry.configs."gcr.io".auth]
  username = ""
  password = ""
  auth = ""
  identitytoken = ""
```

The meaning of each field is the same with the corresponding field in `.docker/config.json`.

Please note that auth config passed by CRI takes precedence over this config.
The registry credential in this config will only be used when auth config is
not specified by Kubernetes via CRI.

After modifying this config, you need to restart the `containerd` service.

### Configure Registry Credentials Example - GCR with Service Account Key Authentication

If you don't already have Google Container Registry (GCR) set-up then you need to do the following steps:

* Create a Google Cloud Platform (GCP) account and project if not already created (see [GCP getting started](https://cloud.google.com/gcp/getting-started))
* Enable GCR for your project (see [Quickstart for Container Registry](https://cloud.google.com/container-registry/docs/quickstart))
* For authentication to GCR: Create [service account and JSON key](https://cloud.google.com/container-registry/docs/advanced-authentication#json-key)
* The JSON key file needs to be downloaded to your system from the GCP console
* For access to the GCR storage: Add service account to the GCR storage bucket with storage admin access rights (see [Granting permissions](https://cloud.google.com/container-registry/docs/access-control#grant-bucket))

Refer to [Pushing and pulling images](https://cloud.google.com/container-registry/docs/pushing-and-pulling) for detailed information on the above steps.

> Note: The JSON key file is a multi-line file and it can be cumbersome to use the contents as a key outside of the file. It is worthwhile generating a single line format output of the file. One way of doing this is using the `jq` tool as follows: `jq -c . key.json`

It is beneficial to first confirm that from your terminal you can authenticate with your GCR and have access to the storage before hooking it into containerd. This can be verified by performing a login to your GCR and
pushing an image to it as follows:

```console
docker login -u _json_key -p "$(cat key.json)" gcr.io

docker pull busybox

docker tag busybox gcr.io/your-gcp-project-id/busybox

docker push gcr.io/your-gcp-project-id/busybox

docker logout gcr.io
```

Now that you know you can access your GCR from your terminal, it is now time to try out containerd.

Edit the containerd config (default location is at `/etc/containerd/config.toml`)
to add your JSON key for `gcr.io` domain image pull
requests:

```toml
version = 2

[plugins."io.containerd.grpc.v1.cri".registry]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["https://registry-1.docker.io"]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
      endpoint = ["https://gcr.io"]
  [plugins."io.containerd.grpc.v1.cri".registry.configs]
    [plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io".auth]
      username = "_json_key"
      password = 'paste output from jq'
```

> Note: `username` of `_json_key` signifies that JSON key authentication will be used.

Restart containerd:

```console
service containerd restart
```

Pull an image from your GCR with `crictl`:

```console
$ sudo crictl pull gcr.io/your-gcp-project-id/busybox

DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '3s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gcr.io/your-gcr-instance-id/busybox,},Auth:nil,SandboxConfig:nil,}
DEBU[0001] PullImageResponse: &PullImageResponse{ImageRef:sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42,}
Image is up to date for sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42
```

在k8s中使用Containerd，从 harbor 私有仓库拉取镜像

k8s-1.20发布之后，不再使用docker作为底层容器运行时，而是默认使用Container Runtime Interface(CRI)。因此原来在docker中配置的个人仓库环境不再起作用，导致k8s配置pods时拉取镜像失败。

1.Containerd生成默认配置文件

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

2.修改配置文件

vim /etc/containerd/config.toml  # 添加如下信息

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        # 如下这些仓库可以作为公共仓库使用
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
            endpoint = ["https://docker.mirrors.ustc.edu.cn","http://hub-mirror.c.163.com"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
            endpoint = ["https://gcr.mirrors.ustc.edu.cn"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
            endpoint = ["https://gcr.mirrors.ustc.edu.cn/google-containers/"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"]
            endpoint = ["https://quay.mirrors.ustc.edu.cn"]
        # 内部私有仓库配置
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."www.myharbor.com"]
            endpoint = ["https://www.myharbor.com/"]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        # 内部私有仓库认证信息
        [plugins."io.containerd.grpc.v1.cri".registry.configs."www.myharbor.com"] # 这行不确定要不要写上
          [plugins."io.containerd.grpc.v1.cri".registry.configs."www.myharbor.com".tls]
            insecure_skip_verify = false # 是否跳过证书认证
            ca_file = "/etc/containerd/www.myharbor.com/ca.crt" # CA 证书 
          [plugins."io.containerd.grpc.v1.cri".registry.configs."www.myharbor.com".auth]
            username = "test" # 在harbor里单独创建的用户，授权访问指定项目
            password = "Test123456"

注意：
2.1 配置文件中有个默认的sandbox_image = "k8s.gcr.io/pause:3.2",因为网络原因，理论上这个镜像是无法拉取的，但是配置了国内公共仓库：k8s.gcr.io (只是名称而已，实际是从endpoint地址中拉取镜像),因此可以拉取这个镜像。若是配置文件中没有配置国内公共仓库;k8s.gcr.io，则需要手动修改sandbox_image的值，确保这个镜像可以拉取

2.2 如果镜像仓库配置了双向认证，那么需要为 containerd 配置 ssl 证书用于 镜像仓库对 containerd 做认证。

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        # 内部私有仓库认证信息
        [plugins."io.containerd.grpc.v1.cri".registry.configs."www.myharbor.com"] # 这行不确定要不要写上
          [plugins."io.containerd.grpc.v1.cri".registry.configs."www.myharbor.com".tls]
            insecure_skip_verify = false # 是否跳过证书认证
            ca_file = "/etc/containerd/www.myharbor.com/ca.crt" # CA 证书
            cert_file = "/etc/containerd/www.myharbor.com/www.myharbor.com.crt" # harbor 证书
            key_file = "/etc/containerd/www.myharbor.com/www.myharbor.com.key" # harbor 私钥 
          [plugins."io.containerd.grpc.v1.cri".registry.configs."www.myharbor.com".auth]
            username = "test" # 在harbor里单独创建的用户，授权访问指定项目
            password = "Test123456

2.3 Containerd 与 docker 都有默认仓库，均为 docker.io 。如果配置中未指定 mirror 为 docker.io，containerd 后会自动加载 docker.io 配置。与 docker 不同的是，containerd 可以修改 docker.io 对应的 endpoint（默认为 https://registry-1.docker.io ） ，而 docker 无法修改。

Docker 中可以通过 registry-mirrors 设置镜像加速地址。如果 pull 的镜像不带仓库地址（项目名+镜像名:tag），则会从默认镜像仓库去拉取镜像。如果配置了镜像加速地址，会先访问镜像加速仓库，如果没有返回数据，再访问默认的镜像仓库。

Containerd 目前没有直接配置镜像加速的功能，但 containerd 中可以修改 docker.io 对应的 endpoint，所以可以通过修改 endpoint 来实现镜像加速下载。因为 endpoint 是轮询访问，所以可以给 docker.io 配置多个仓库地址来实现 加速地址+默认仓库地址。

如上就是上文配置那些公共仓库的缘由。

3.重载 systemd 的 daemon守护进程并重启containerd.service服务

systemctl daemon-reload && systemctl restart containerd.service

4.在k8s集群节点上执行如下命令进行测试

crictl pull www.myharbor.com/library/nginx:v1.0

5.在对应的namespace中创建secret

kubectl create secret docker-registry www.myharbor.com --docker-server=https://www.myharbor.com --docker-username=test --docker-password=Test123456 --docker-email=info@foo.com -n <namespace>

6.在pod/deployment中设置imagePullSecrets

apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
  - name: private-reg-container
    image: <your-private-image>
  imagePullSecrets:
  - name: www.myharbor.com

单独使用Containerd进行配置，从 harbor 私有仓库拉取镜像 并运行容器

1.Containerd生成默认配置文件

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

2.修改配置文件

vim /etc/containerd/config.toml  # 添加如下信息

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        # 如下这些仓库可以作为公共仓库使用
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
            endpoint = ["https://docker.mirrors.ustc.edu.cn","http://hub-mirror.c.163.com"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
            endpoint = ["https://gcr.mirrors.ustc.edu.cn"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
            endpoint = ["https://gcr.mirrors.ustc.edu.cn/google-containers/"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"]
            endpoint = ["https://quay.mirrors.ustc.edu.cn"]

注意：跟上一步相比，没有添加harbor私有仓库信息。这是因为就算添加了，在使用 ctl命令 进行手动拉取镜像此时会报如下错误（巨坑-经过无数次失败测试，原本以为是CA证书签发的harbor证书问题），即使你在config.toml中配置insecure_skip_verify为true也是不行的。

# ctr images pull www.myharbor.com/mytest/busybox:v0.2
INFO[0000] trying next host     error="failed to do request: Head \"https://www.myharbor.com/v2/mytest/busybox/manifests/v0.2\": x509: certificate signed by unknown authority" host=www.myharbor.com
ctr: failed to resolve reference "www.myharbor.com/mytest/busybox:v0.2": failed to do request: Head "https://www.myharbor.com/v2/mytest/busybox/manifests/v0.2": x509: certificate signed by unknown authority

3.手动拉取镜像

# 解决办法1.指定 -k 参数跳过证书校验。
$ ctr images pull --user test:Test123456 -k www.myharbor.com/mytest/busybox:v0.2

# 解决办法2.指定CA证书、Harbor相关证书文件路径。
$ ctr images pull --user test:Test123456 --tlscacert ca.crt www.myharbor.com/mytest/busybox:v0.2