Centos7上使用jenkins执行pipeline在容器内部拉取环境运行代码的各种坑

环境

Centos 7上部署了三个服务
bitbucket 192.168.1.156:7900
jenkins 192.168.1.156:8080
docker服务

原理
在Dockerfile中定义pipeline scripts，然后拉取docker hub的镜像作为运行环境，创建容器，在容器内部拉取bitbucket代码，然后在容器中运行bitbucket代码，最后销毁容器

坑一：忘了bitbucket的登录密码

解决办法
这个直接去bitbucket官网：lockout recovery process

Steps for Linux:

  1. Edit the \bin_start-webapp.sh file and add the "-Datlassian.recovery.password=temporarypassword" value to the JVM_SUPPORT_RECOMMENDED_ARGS property. The property value must be non-blank, and should look like this when you've done that:
JVM_SUPPORT_RECOMMENDED_ARGS=-Datlassian.recovery.password=temporarypassword

Here we are using "temporarypassword", but you should use your own value.
2. Start Bitbucket manually by running \bin\start-bitbucket.sh.
3. Log in using the 'recovery_admin' username and the temporary password specified in Step 1.
4. Repair your configuration. We strongly recommend that you do not perform other actions while Bitbucket is in recovery mode.
5. Confirm your ability to log in with your usual admin profile.
6. Shut down Bitbucket, remove the atlassian.recovery.password argument from _start-webapp.sh, and restart Bitbucket as usual.

坑二：java.io.IOException: error=2, 没有那个文件或目录

Started by user admin
java.io.IOException: error=2, 没有那个文件或目录
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
Caused: java.io.IOException: Cannot run program "" (in directory "/var/lib/jenkins/caches/git-05cd0ebc51f1a5a76f140726c1184714"): error=2, 没有那个文件或目录
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at hudson.Proc$LocalProc.<init>(Proc.java:254)
	at hudson.Proc$LocalProc.<init>(Proc.java:223)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:997)
	at hudson.Launcher$ProcStarter.start(Launcher.java:509)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2729)
**Caused: hudson.plugins.git.GitException: Error performing git command:  init /var/lib/jenkins/caches/git-05cd0ebc51f1a5a76f140726c1184714**
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2748)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2662)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2658)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1981)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1047)
Caused: hudson.plugins.git.GitException: Could not init /var/lib/jenkins/caches/git-05cd0ebc51f1a5a76f140726c1184714
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1049)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.init(CliGitAPIImpl.java:355)
	at hudson.plugins.git.GitAPI.init(GitAPI.java:218)
	at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:336)
	at jenkins.scm.api.SCMFileSystem$Builder.build(SCMFileSystem.java:584)
	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:219)
	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:191)
	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:174)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:118)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:70)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:311)
	at hudson.model.ResourceController.execute(ResourceController.java:101)
	at hudson.model.Executor.run(Executor.java:442)
Finished: FAILURE

原因分析
这个是因为没有centos7上git没有找到，可以在pipeline使用如下的脚本看一下git能不能找到，和jenkins的环境变量的路径

sh '''
which git
echo $PATH

可以看到控制台打印的信息，git并没找到，从 /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin可以看出，jenkins去这些路径下去搜索git，查看了一下我本地的git安装目录是：/usr/local/git/bin/git，也有可能是环境变量配置的不对，但是我在centos7中git --version也没问题

Started by user admin
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/locust_test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Hello)
[Pipeline] sh
+ which git
**which: no git in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)**
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

解决办法
卸载旧的git，使用如下命令重装git和配置环境变量

cd git-2.22.0
make prefix=/usr/local/git all
make prefix=/usr/local/git install

echo "export PATH=$PATH:/usr/local/git/bin"  >> /etc/bashrc
source /etc/bashrc

坑三：stderr: No RSA host key is known for [192.168.1.156]:7999 and you have requested strict checking.

原来以为是strict checking，其实这只是一个警告信息，真正的原因是无法读取远程仓库

Started by user admin
hudson.plugins.git.GitException: Command "/usr/local/git/bin/git fetch --tags --force --progress --prune -- origin +refs/heads/master:refs/remotes/origin/master" returned status code 128:
stdout: 
stderr: No RSA host key is known for [192.168.1.156]:7999 and you have requested strict checking.
Host key verification failed.
fatal: 无法读取远程仓库。

请确认您有正确的访问权限并且仓库存在。

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2736)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2111)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:87)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:623)
	at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:367)
	at jenkins.scm.api.SCMFileSystem$Builder.build(SCMFileSystem.java:584)
	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:219)
	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:191)
	at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:174)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:118)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:70)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:311)
	at hudson.model.ResourceController.execute(ResourceController.java:101)
	at hudson.model.Executor.run(Executor.java:442)
Finished: FAILURE

解决办法
在Manage Jenkins --> Configure Global Security 下有一个配置，这里的选项选择为No verification

除此之外，往上有人说在~/.ssh/config可以配置，这可能是本地git pull的时候禁止strict checking，但这个问题不是主要的

[root@k8s-master git-2.22.0]# cd ~/.ssh/
[root@k8s-master .ssh]# ls
config  id_rsa  id_rsa.pub  known_hosts
[root@k8s-master .ssh]#  cat config
#Host *
#  StrictHostKeyChecking no
[root@k8s-master .ssh]# cat known_hosts 
192.168.1.166 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDWZg5QCJJg49sumc5K0myAaRU9AyDiV/FlhYhDaNcagsDsQ778qri3EYajIPKoCnI6FfkbNm3C2aW5sQyPxh7k=
192.168.1.176 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAAFwv2SOO2JoWxmsiJkGPt6pdqLVDSknFv7jiyRnTndQYgAsOu7DvvD3eyfu2pMfjfLPFMgXqbvInTFeiBZ7sk=
[192.168.1.156]:7999 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYxoQy9wMWxQ5SqKwUg7zvRSPXvf33H4sITLT1HYr2KnDP/y/jBVcw6WXMIWjDimfAUi7X/U/WlYRpNo9PxHRYBXXeGXpiLITI0ibhCRJBJkkHeMQy0LpN87bXEk3dep3lfZsZtz6xGKEWcswXj1Zzgb01A86uezBpEQ8aFBInUr/3N4k8WI/1hVVXPNf8ubhilVqwHJuKxWlpWb+LrA2JQtJYVQ5jnGUOmXy83JPol7b9/Fqidfih0QvRZ2hfZQ2ixeu2cwuqorL5G8cnRESRTSo69wRdy2/47ksCEraJgenGow5WQh5AjB5chYIHGjsP5kHROROC0OLidXOCqwtV

坑四：fatal: 无法读取远程仓库 请确认您有正确的访问权限并且仓库存在

原因分析
就是上面的第二个问题，这个是添加凭据的时候，我是直接拷贝centos7上的id_rsa私钥，但是试了很多次，每次拷贝都是无法读取远程仓库

解决办法
直接把id_rsa文件上传到mac上，使用pbcopy < id_rsa，将内容拷贝到粘贴板上，再复制进去
但是使用ssh key-gen -t rsa之前，记得使用git config --global设置下用户名和邮箱

坑五：Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

具体的console输出信息如下：

[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker inspect -f . docker.io/beck123/locust:v1

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/containers/docker.io/beck123/locust:v1/json: dial unix /var/run/docker.sock: connect: permission denied
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker pull docker.io/beck123/locust:v1
Warning: failed to get default registry endpoint from daemon (Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/info: dial unix /var/run/docker.sock: connect: permission denied). Using system default: https://index.docker.io/v1/
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.38/images/create?fromImage=beck123%2Flocust&tag=v1: dial unix /var/run/docker.sock: connect: permission denied

原因分析
这个是因为作为jenkins用户执行的时候，没有加入docker组，没有docker组的权限

解决办法

sudo usermod -a -G docker jenkins

坑六：ls: cannot access '.': Operation not permitted

一开始的时候，想通过pipline对比下容器拉取代码前后文件的变化，因此写了下面的脚本

stages {
        stage("Git checkout") {
            steps {
                // sh '''
                //     if [ -d locust_test ]
                //     then
                //       rm -f locust_test
                //     fi
                //    '''
                sh '''
                echo "拉取代码前容器内部的文件:"
                ls -l
                '''
                git credentialsId: 'admin', url: 'ssh://git@192.168.1.156:7999/aut/locust_test.git'
            }
        }
            
        stage('Test') {
            steps {
                sh '''
                    echo "拉取代码后容器内部的文件:"
                    ls -l
                    locust -f src/locustfile.py --headless -u 100 -r 5 -t 120s -H http://192.168.1.2:8099
                   '''
            }
        }
    }

没想到运行后，console就报错了

+ echo 拉取代码前容器内部的文件:
拉取代码前容器内部的文件:
+ ls -al
ls: cannot access '.': Operation not permitted
ls: cannot access '..': Operation not permitted
ls: cannot access '.git': Operation not permitted
ls: cannot access '.gitignore': Operation not permitted
ls: cannot access 'Dockerfile': Operation not permitted
ls: cannot access 'README.md': Operation not permitted
ls: cannot access 'requirements.txt': Operation not permitted
ls: cannot access 'src': Operation not permitted
ls: cannot access 'Jenkinsfile': Operation not permitted
total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
d????????? ? ? ? ?            ? .git
-????????? ? ? ? ?            ? .gitignore
-????????? ? ? ? ?            ? Dockerfile
-????????? ? ? ? ?            ? Jenkinsfile
-????????? ? ? ? ?            ? README.md
-????????? ? ? ? ?            ? requirements.txt
d????????? ? ? ? ?            ? src

在centos7上自己创建了一个容器，同样也无法在容器内部执行```ls x命令

[root@k8s-master workspace]# docker exec -it locust /bin/bash
root@2fe8815e89eb:/env/locust_test# ls
requirements.txt
root@2fe8815e89eb:/env/locust_test# cd ..
root@2fe8815e89eb:/env# ls
locust_test
root@2fe8815e89eb:/env# ls locust_test/
ls: cannot access 'locust_test/': Operation not permitted
root@2fe8815e89eb:/env/locust_test# ls -l
ls: cannot access 'requirements.txt': Operation not permitted
total 0
-????????? ? ? ? ?            ? requirements.txt

原因分析
查看了一下centos7上的docker版本是18.06.1，本地mac上的版本是20.10.21，而本地mac执行jenkins pipline是没有任何问题的，新的包没有这个问题。不过后面找到原因是docker run创建容器的时候没有加参数--privileged=true，不加该参数，只是外部的一个普通用户权限，加了该参数，才真正拥有root权限

解决办法
在centos7上重新加上该参数，重新生成了一个容器，容器内部执行ls相关命令终于不报错了。但是我还没想好怎么在pipline中加这个参数

[root@k8s-master workspace]# docker run -it -d --name locust --privileged=true beck123/locust:v1 
1889fbd8872fb08899cf4c3e4432ad745d7d9f1f777ffe8a17d239b218fa26df
[root@k8s-master workspace]# docker exec -it locust /bin/bash
root@1889fbd8872f:/env/locust_test# ls
requirements.txt
root@1889fbd8872f:/env/locust_test# cd ..
root@1889fbd8872f:/env# ls -l
total 0
drwxr-xr-x 1 root root 30 Dec 23 16:01 locust_test
root@1889fbd8872f:/env# ls -l locust_test/
total 4
-rw-r--r-- 1 root root 1045 Dec 23 13:49 requirements.txt

---

看官方文档[Pipeline Syntax](https://www.jenkins.io/doc/book/pipeline/syntax/#agent)，发现这里docker run后面的参数可以使用```args```来定义
![](https://img2023.cnblogs.com/blog/1186367/202212/1186367-20221224223213728-1091573876.png)
于是Jenkinsfile可以这样定义
```linux
agent {
        docker { 
            image 'docker.io/beck123/locust:v1'
            args '--privileged=true' 
        }
    }

运行之后可以在console中看到生效了

posted @ 2022-12-24 19:58  cnhkzyy  阅读(880)  评论(0)  编辑  收藏  举报