1. what is kubeflow pipeline for tfx pipeline ？
kubeflow pipeline is an ochetrator of tfx pipeline, which runs on a kubernetes cluster.
LocalDagRuner is an orchetrator of tfx pipeline, which runs local.
# run a tfx pipeline usging LocalGagRunner
tfx.orchestration.LocalDagRunner().run(
_create_schema_pipeline(
pipeline_name=SCHEMA_PIPELINE_NAME,
pipeline_root=SCHEMA_PIPELINE_ROOT,
data_root=DATA_ROOT,
schema_path=SCHEMA_PATH,
metadata_path=SCHEMA_METADATA_PATH,
module_file=_trainer_module_file,
serving_model_dir=SERVING_MODEL_DIR,
)
)
# run a tfx pipeline using KubeflowDagRunner
tfx.orchestration.experimental.KubeflowDagRunner().run(
_create_schema_pipeline(
pipeline_name=SCHEMA_PIPELINE_NAME,
pipeline_root=SCHEMA_PIPELINE_ROOT,
data_root=DATA_ROOT,
schema_path=SCHEMA_PATH,
metadata_path=SCHEMA_METADATA_PATH,
module_file=_trainer_module_file,
serving_model_dir=SERVING_MODEL_DIR,
)
)
2. steps of running a tfx pipeline using kubeflow pipeline
2.1 generate file pipeline.yaml (namely definition file of kubeflow pipeline):
tfx.orchestration.experimental.KubeflowDagRunner().run(
_create_schema_pipeline(
pipeline_name=SCHEMA_PIPELINE_NAME,
pipeline_root=SCHEMA_PIPELINE_ROOT,
data_root=DATA_ROOT,
schema_path=SCHEMA_PATH,
metadata_path=SCHEMA_METADATA_PATH,
module_file=_trainer_module_file,
serving_model_dir=SERVING_MODEL_DIR,
)
)
Constructs a pipeline definition YAML file based on the TFX logical pipeline. [1]
Args
output_dir An optional output directory into which to output the pipeline definition files. Defaults to the current working directory.
output_filename An optional output file name for the pipeline definition file. Defaults to pipeline_name.tar.gz when compiling a TFX pipeline. Currently supports .tar.gz, .tgz, .zip, .yaml, .yml formats. See https://github.com/kubeflow/pipelines/blob/181de66cf9fa87bcd0fe9291926790c400140783/sdk/python/kfp/compiler/compiler.py#L851 for format restriction.
2.2 change image registry in file pipline.yaml, due to that gcr.io is not accessible in china.
# in file pipeline.yaml
# raw image generated by tfx.orchestration.experimental.KubeflowDagRunner().run(),
# it equals to hub.docker.com/tensorflow/tfx:1.14.0, which is not accessible in china.
#image: tensorflow/tfx:1.14.0
# replacement image
# docker.nju.edu.cn has not tfx:1.14.0 temporally, si3nce it's the latest version,
# and docker.nju.edu.cn has not pulled it yet,
# so use tfx:1.13.0.
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
Attention:
- The size of image tensorflow/tfx:1.13.0 is about 30G, and its blobs (namely gzip) is about 9G, it's better to pull (namely download) it before hand.
imagePullPolicy: Nevermeans never to pull image when running the container, other options areAlways,IfNotPresent. imagePullPolicy: Neverneeds the image exits on each node which is possible to assign the pod to, or will raise error:
'Warning ErrImageNeverPull 2m36s (x10 over 4m16s) kubelet Container image "docker.nju.edu.cn/tensorflow/tfx:1.13.0" is not present with pull policy of Never',
when scheduling the pod (namely assigning the pod to one node in the kubernetes cluster).
Or, settingnodeAffinityto the node who has the image for the pod:
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- maye-inspiron-5547
- containerd has namespaces for images, the default namespace is "default", the namespace of images pulled by containers in a kubernetes cluster is "k8s.io". If needed image not in namespace "k8s.io", kubernetes can not see it.
crictl is container runtime interface cli of kubernetes,
crictl镜像的namespace就一个，k8s.io。因此也是默认拉取镜像的namespace。
如果通过ctr拉取镜像时如果不指定放在k8s.io空间下，crictl是无法读取到本地的该镜像的。
ctr是containerd自带的命令行工具。一共有三个命名空间default,k8s.io 和moby。默认default。
nerdctl is docker-compatible cli of containerd.
ctr image ls : list images in namespace "default" .
拉取镜像到k8s.io命名空间:
nerdctl pull nginx:latest --namespace k8s.io
查看k8s.io下的镜像:
sudo nerdctl images --namespace k8s.io
Attention:
nerdctl image list --namespace k8s.io
No image shown
This is due that images in k8s.io are pulled by containers in kubernetes cluster up to now, and the containers are run as root, so these images are pulled by root, other uses can not see them.
copy an image from one namespace to another namespace:
ctr -n default images export busybox-1.28.tar.gz docker.io/library/busybox:1.28
ctr -n k8s.io image import busybox-1.28.tar.gz
# or,
nerdctl save -o busybox.tar.gz busybox:latest --namespace default
nerdctl load -i busybox.tar.gz --namespace k8s.io
delete an image:
nerdctl rmi tf_std_server:v1 --namespace k8s.io
# or,
ctr -n k8s.io image rm tf_std_server:v1
Where tf_std_server:v1 is the tag of the image to be removed.
2.3 mount pathes which need to be accessed by all components of the tfx pipeline, due to that one component one pod, and each container has standalone file system, including pipeline_root, input_base of ExampleGen, module_path of Transform and Trainer , artifact_uri of SchemaImpoter .
# in file pipeline.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: detect-anomolies-on-wafer-tfdv-schema-
#name: detect-anomolies-on-wafer-tfdv-schema-maye
annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.0, pipelines.kubeflow.org/pipeline_compilation_time: '2024-01-07T22:16:36.438482',
pipelines.kubeflow.org/pipeline_spec: '{"description": "Constructs a Kubeflow
pipeline.", "inputs": [{"default": "pipelines/detect_anomolies_on_wafer_tfdv_schema",
"name": "pipeline-root"}], "name": "detect_anomolies_on_wafer_tfdv_schema"}'}
labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.0}
spec:
entrypoint: detect-anomolies-on-wafer-tfdv-schema
nodeAffinity: #### putting nodeAffinity in workflow spec means
required: #### all component of the workflow use this nodeAffinity.
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- maye-inspiron-5547
volumes:
- name: wafer-data #### define volume for input_base
hostPath:
path: /home/maye/trainEvalData
type: Directory
- name: transform-module #### define volume for module_path of Transform
hostPath:
path: /home/maye/maye_temp/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
type: File
- name: trainer-module #### define volume for module_path of Trainer
hostPath:
path: /home/maye/maye_temp/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
type: File
- name: schema-path #### define volume for artifact_uri of SchemaImporter
hostPath:
path: /home/maye/maye_temp/detect_anomalies_in_wafer_schema
type: Directory
- name: tfx-pv #### define volume for pipeline-root
persistentVolumeClaim:
claimName: tfx-pv-claim
templates:
- name: detect-anomolies-on-wafer-tfdv-schema
inputs:
parameters:
- {name: pipeline-root} #### reference argument pipeline-root of workflow
dag:
tasks:
- name: importexamplegen
template: importexamplegen
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'} #### real argument for
#### for pipeline-root
volumes:
- name: wafer-data #### reference volume for input_base
- name: tfx-pv #### reference volume for pipeline_root
- name: pusher
template: pusher
dependencies: [trainer]
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'}
volumes:
- name: tfx-pv
- name: schema-importer
template: schema-importer
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'}
volumes:
- name: tfx-pv
- name: schema-path
- name: statisticsgen
template: statisticsgen
dependencies: [importexamplegen]
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'}
volumes:
- name: tfx-pv
- name: trainer
template: trainer
dependencies: [importexamplegen, transform]
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'}
volumes:
- name: trainer-module
- name: tfx-pv
- name: transform
template: transform
dependencies: [importexamplegen, schema-importer]
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'}
volumes:
- name: transform-module
- name: tfx-pv
- name: schema-path
- name: importexamplegen
container:
args:
- --pipeline_root #### pass pipeline_root as argument of container shell cmd
- '{{inputs.parameters.pipeline-root}}'
- --kubeflow_metadata_config
- |-
{
"grpc_config": {
"grpc_service_host": {
"environment_variable": "METADATA_GRPC_SERVICE_HOST"
},
"grpc_service_port": {
"environment_variable": "METADATA_GRPC_SERVICE_PORT"
}
}
}
- --node_id
- ImportExampleGen
- --tfx_ir
- |-
{
"pipelineInfo": {
"id": "detect_anomolies_on_wafer_tfdv_schema"
},
"nodes": [
{
"pipelineNode": {
"nodeInfo": {
"type": {
"name": "tfx.components.example_gen.import_example_gen.component.ImportExampleGen"
},
"id": "ImportExampleGen"
},
"contexts": {
"contexts": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen"
}
}
}
]
},
"outputs": {
"outputs": {
"examples": {
"artifactSpec": {
"type": {
"name": "Examples",
"properties": {
"split_names": "STRING",
"span": "INT",
"version": "INT"
},
"baseType": "DATASET"
}
}
}
}
},
"parameters": {
"parameters": {
"output_config": {
"fieldValue": {
"stringValue": "{}"
}
},
# splits pattern should match the file name of
# train/split data file inside input_base directory.
"input_config": {
"fieldValue": {
"stringValue": "{\n \"splits\": [\n {\n \"name\": \"train\",\n \"pattern\": \"train\"\n },\n {\n \"name\": \"eval\",\n \"pattern\": \"eval\"\n }\n ]\n}"
}
},
"output_data_format": {
"fieldValue": {
"intValue": "6"
}
},
# parameter input_base of ImportExampleGen the value is mounted path of volume input_base.
"input_base": {
"fieldValue": {
"stringValue": "/maye/trainEvalData"
}
},
"output_file_format": {
"fieldValue": {
"intValue": "5"
}
}
}
},
"downstreamNodes": [
"StatisticsGen",
"Trainer",
"Transform"
],
"executionOptions": {
"cachingOptions": {}
}
}
}
],
# parameter pipeline-root of ImportExampleGne, its value is passed via container shell cmd argument
# pipeline-root, is mounted path of volume pipeline_root.
"runtimeSpec": {
"pipelineRoot": {
"runtimeParameter": {
"name": "pipeline-root",
"type": "STRING",
"defaultValue": {
"stringValue": "pipelines/detect_anomolies_on_wafer_tfdv_schema"
}
}
},
"pipelineRunId": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
"executionMode": "SYNC",
"deploymentConfig": {
"@type": "type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig",
"executorSpecs": {
"ImportExampleGen": {
"@type": "type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec",
"pythonExecutorSpec": {
"classPath": "tfx.components.example_gen.import_example_gen.executor.Executor"
}
}
},
"customDriverSpecs": {
"ImportExampleGen": {
"@type": "type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec",
"classPath": "tfx.components.example_gen.driver.FileBasedDriver"
}
},
"metadataConnectionConfig": {
"@type": "type.googleapis.com/ml_metadata.ConnectionConfig",
"sqlite": {
"filenameUri": "metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db",
"connectionMode": "READWRITE_OPENCREATE"
}
}
}
}
- --metadata_ui_path
- /mlpipeline-ui-metadata.json
- --runtime_parameter
- pipeline-root=STRING:{{inputs.parameters.pipeline-root}}
command: [python, -m, tfx.orchestration.kubeflow.container_entrypoint]
...
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /maye/trainEvalData #### mount volume for input_path
name: wafer-data
- mountPath: /tfx/tfx_pv #### mount volume for pipeline_root
name: tfx-pv
inputs:
parameters:
- {name: pipeline-root} #### formal parameter of the container
outputs:
artifacts:
- {name: mlpipeline-ui-metadata, path: /mlpipeline-ui-metadata.json}
...
- name: pusher
container:
...
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /tfx/tfx_pv
name: tfx-pv
...
- name: schema-importer
container:
...
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /tfx/tfx_pv
name: tfx-pv
- mountPath: /tfx/pipelines/detect_anomalies_in_wafer_schema
name: schema-path
readOnly: True
...
- name: statisticsgen
container:
...
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /tfx/tfx_pv
name: tfx-pv
...
- name: trainer
container:
..
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /tfx/pipelines/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
name: trainer-module
readOnly: True
- mountPath: /tfx/tfx_pv
name: tfx-pv
...
- name: transform
container:
...
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /tfx/pipelines/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
name: transform-module
readOnly: True
- mountPath: /tfx/pipelines/detect_anomalies_in_wafer_schema
name: schema-path
readOnly: True
- mountPath: /tfx/tfx_pv
name: tfx-pv
...
arguments:
parameters: #### real argument pipeline-root of workflow
- {name: pipeline-root, value: /tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema}
serviceAccountName: pipeline-runner
Attention:
- In this example,
input_baseofExampleGen,module_pathofTransformandTrainer,artifact_uriofSchemaImpoterare mounted from hostPath, since here all tfx components are specified to run on one computer vianodeAffinityinside field of workflow spec. In case that tfx components run on different computers, theses directories need to be mounted from directories which can be accessed by all these computers, such as a nfs directory:
# in file pipeline.yaml
...
kind: Workflow
...
spec:
...
volumes:
- name: schema-path #### define volume for artifact_uri of SchemaImporter
#hostPath:
# path: #/home/maye/maye_temp/detect_anomalies_in_wafer_#schema
#type: Directory
nfs:
server: nfs-server-ip
path: /home/maye/nfs/tfx_schema_path
readOnly: true
...
Note:
- About How to install nfs and set a nfs shared directory, see reference «Install nfs (network file system) » https://www.cnblogs.com/zhenxia-jiuyou/p/17999749 .f
2.4 submit the file pipeline.yaml to ml-pipeline, via "upload_pipeline" on webpage of ml-pipeline-ui, or python sdk kfp.Client().upload_pipeline(<pipeline.yaml path>, <pipeline name>, <pipeline description>
After upload pipeline, the pipeline will be recorded in database "mlpipeline".
2.5 Create a Run on webpag of ml-pipline-ui, then the tfx pipeline will run on kubernetes cluster, one component one pod.
3. Error & Solution
[ERROR: Failed to pull image]
(base) maye@maye-Inspiron-5547:~$ kubectl describe pod detect-anomolies-on-wafer-tfdv-schema-ldvtw-1952722848 -n kubeflow
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 52m (x4 over 92m) kubelet Error: ImagePullBackOff
Warning Failed 13m (x9 over 92m) kubelet Error: ErrImagePull
Warning Failed 4m42s (x9 over 92m) kubelet Failed to pull image "tensorflow/tfx:1.14.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/tensorflow/tfx:1.14.0": failed to copy: httpReadSeeker: failed open: unexpected status code https://pft7f97f.mirror.aliyuncs.com/v2/tensorflow/tfx/blobs/sha256:f2cce533751060f702397991bc7f0acf6d691c898fe1c7cc25b3ece25a409879?ns=docker.io: 500 Internal Server Error - Server message: unknown: unknown error
Normal BackOff 4m17s (x13 over 92m) kubelet Back-off pulling image "tensorflow/tfx:1.14.0"
(base) maye@maye-Inspiron-5547:~$
[SOLUTION]
This is due to that docker.io is not accessible in china, replace it with its mirror website, such as: docker.nju.edu.cn , in file pipeline.yaml, namely replace "tensorflow/tfx:1.14.0" to "docker.nju.edu.cn/tensorflow/tfx:1.14.0" .
[ERROR: OSError: No files found based on the file pattern]
File "/usr/local/lib/python3.8/dist-packages/apache_beam/io/filebasedsource.py", line 190, in _validate
raise IOError('No files found based on the file pattern %s' % pattern)
OSError: No files found based on the file pattern /home/maye/tensorflow_models/pipelines/detect_anomolies_on_wafer_tfdv_schema/train_eval_data/train_data
time="2024-01-15T11:29:52.963Z" level=info msg="sub-process exited" argo=true error="<nil>"
time="2024-01-15T11:29:52.963Z" level=error msg="cannot save artifact /mlpipeline-ui-metadata.json" argo=true error="stat /mlpipeline-ui-metadata.json: no such file or directory"
Error: exit status 1
[SOLUTION]
This is due to that container has standalone file-system, and "/home/maye/tensorflow_models/pipelines" is path of host's file system, container can not see host's files system, need to mount host's file path to container's file path.
[ERROR:create run, no workflow created.]
After clicking "Start" on Create Run webpage of ml-pipeline-ui:
(base) maye@maye-Inspiron-5547:~$ kubectl get workflow --all-namespaces
No resources found
(base) maye@maye-Inspiron-5547:~$
[ANALYSIS]
- see log of pod ml-pipeline:
I0210 15:00:27.829437 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler starting
I0210 15:00:32.863338 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler finished
I0211 07:50:38.920161 7 pipeline_upload_server.go:93] Upload pipeline called
I0211 08:04:11.507556 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler starting
I0211 08:04:11.515234 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler finished
I0211 08:04:14.736846 7 interceptor.go:29] /api.PipelineService/GetPipelineV1 handler starting
I0211 08:04:14.740579 7 interceptor.go:37] /api.PipelineService/GetPipelineV1 handler finished
I0211 08:04:14.791440 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipeline handler starting
I0211 08:04:14.792236 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipeline handler finished
I0211 08:04:14.800037 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelineVersions handler starting
I0211 08:04:14.842042 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelineVersions handler finished
I0211 08:04:14.935314 7 interceptor.go:29] /api.PipelineService/ListPipelineVersionsV1 handler starting
I0211 08:04:14.937185 7 interceptor.go:37] /api.PipelineService/ListPipelineVersionsV1 handler finished
I0211 08:04:14.943242 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelineVersions handler starting
I0211 08:04:14.954441 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelineVersions handler finished
I0211 08:04:37.523278 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipeline handler starting
I0211 08:04:37.524221 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipeline handler finished
I0211 08:04:37.530014 7 interceptor.go:29] /api.PipelineService/GetPipelineVersionTemplate handler starting
I0211 08:04:37.530090 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler starting
I0211 08:04:37.532328 7 interceptor.go:37] /api.PipelineService/GetPipelineVersionTemplate handler finished
I0211 08:04:42.544322 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler finished
I0211 08:04:42.754095 7 interceptor.go:29] /api.PipelineService/GetPipelineV1 handler starting
I0211 08:04:42.757409 7 interceptor.go:37] /api.PipelineService/GetPipelineV1 handler finished
I0211 08:04:42.864104 7 interceptor.go:29] /api.PipelineService/GetPipelineVersionV1 handler starting
I0211 08:04:42.865472 7 interceptor.go:37] /api.PipelineService/GetPipelineVersionV1 handler finished
### On clicking "Start" on "Create Run" webpage:
I0211 08:04:59.667121 7 interceptor.go:29] /api.RunService/CreateRunV1 handler starting
I0211 08:05:02.843924 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
### E0211 08:05:08.749184 Cannot find reported workflow name="detect-anomolies-on-wafer-tfdv-schema-qxpkk" namespace="kubeflow" runId="1548e025-3d74-4036-a03f-ef79945f9e00" in run store. Deleting the workflow to avoid resource leaking.
E0211 08:05:08.749184 7 resource_manager.go:1150] Cannot find reported workflow name="detect-anomolies-on-wafer-tfdv-schema-qxpkk" namespace="kubeflow" runId="1548e025-3d74-4036-a03f-ef79945f9e00" in run store. Deleting the workflow to avoid resource leaking. This can be caused by installing two KFP instances that try to manage the same workflows or an unknown bug. If you encounter this, recommend reporting more details in https://github.com/kubeflow/pipelines/issues/6189
I0211 08:05:08.789328 7 interceptor.go:37] /api.RunService/CreateRunV1 handler finished
I0211 08:05:08.794044 7 error.go:278] ResourceNotFoundError: Run 1548e025-3d74-4036-a03f-ef79945f9e00 not found
github.com/kubeflow/pipelines/backend/src/common/util.NewResourceNotFoundError
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
github.com/kubeflow/pipelines/backend/src/apiserver/storage.(*RunStore).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/storage/run_store.go:239
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:558
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReportWorkflowResource
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:1131
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).reportWorkflow
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:65
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflowV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:81
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:338
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:340
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to fetch run 1548e025-3d74-4036-a03f-ef79945f9e00
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:560
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReportWorkflowResource
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:1131
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).reportWorkflow
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:65
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflowV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:81
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:338
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:340
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to report workflow name="detect-anomolies-on-wafer-tfdv-schema-qxpkk" namespace="kubeflow" runId="1548e025-3d74-4036-a03f-ef79945f9e00"
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReportWorkflowResource
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:1166
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).reportWorkflow
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:65
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflowV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:81
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:338
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:340
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to report workflow
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).reportWorkflow
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:67
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflowV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:81
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:338
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:340
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
/api.ReportService/ReportWorkflowV1 call failed
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:32
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:340
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
### On clicking "Runs" on left side of the webpage:
I0211 08:05:08.881806 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/ListRuns handler starting
I0211 08:05:08.891104 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/ListRuns handler finished
I0211 08:05:08.917534 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler starting
I0211 08:05:08.938525 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler finished
I0211 08:05:08.979992 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler starting
I0211 08:05:08.980784 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler finished
I0211 08:06:08.685483 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler starting
I0211 08:06:08.688713 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler finished
I0211 08:06:08.699087 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler starting
I0211 08:06:08.714444 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler finished
I0211 08:06:08.816790 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:06:08.818782 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:06:08.828492 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:06:08.829180 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:06:15.480025 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:06:15.482173 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:06:15.492709 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:06:15.493347 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:06:19.578019 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:06:19.580287 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:06:19.596495 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:06:19.597208 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:06:24.574346 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:06:24.577018 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:06:24.586912 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:06:24.587646 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:06:29.578200 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:06:29.580100 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:06:29.591175 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:06:29.591784 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:06:34.581855 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:06:34.584802 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:06:34.596516 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:06:34.597199 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:06:39.573334 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:06:39.576020 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:06:39.587745 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:06:39.588750 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:06:44.574375 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:06:44.581079 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:06:44.592028 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:06:44.592634 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:07:35.068133 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler starting
I0211 08:07:35.071218 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler finished
I0211 08:07:35.113244 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:07:35.116544 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:07:35.130555 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:07:35.131446 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:07:46.222548 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:07:46.225129 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:07:46.233948 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:07:46.235215 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:07:50.295396 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:07:50.297710 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:07:50.309575 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:07:50.310169 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:07:55.297243 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:07:55.299880 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:07:55.316767 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:07:55.317675 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:07:59.711753 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler starting
I0211 08:07:59.715019 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler finished
I0211 08:07:59.753949 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:07:59.756576 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:07:59.769327 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:07:59.771023 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:08:10.924382 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:08:10.926505 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:08:10.936227 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:08:10.936827 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:08:15.006137 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:08:15.008205 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:08:15.017049 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:08:15.017661 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:08:20.012851 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:08:20.014645 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:08:20.025384 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:08:20.026072 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:08:25.012148 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:08:25.014513 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:08:25.024777 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:08:25.025655 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:08:30.009940 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:08:30.011822 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:08:30.019926 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:08:30.020547 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:08:35.012673 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:08:35.014765 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:08:35.024761 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:08:35.025404 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:08:40.011791 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:08:40.013856 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:08:40.023315 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:08:40.023918 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:08:45.009456 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:08:45.015709 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:08:45.027559 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:08:45.028209 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:08:50.005127 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:08:50.007655 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:08:50.016394 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:08:50.017005 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:08:55.005754 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:08:55.008453 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:08:55.016297 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:08:55.016943 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:00.013682 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:00.015620 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:00.024596 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:00.025262 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:05.012559 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:05.014555 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:05.023454 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:05.024010 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:10.008322 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:10.010335 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:10.019012 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:10.019634 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:15.010714 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:15.012708 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:15.021501 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:15.022096 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:20.010758 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:20.013982 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:20.028516 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:20.029314 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:25.011728 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:25.014924 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:25.033016 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:25.034086 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:30.007174 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:30.009191 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:30.024978 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:30.025840 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:35.011652 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:35.014553 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:35.024499 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:35.025649 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:40.013638 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:40.016620 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:40.029630 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:40.030587 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:45.009147 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:45.012376 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:45.021967 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:45.022570 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:50.005323 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:50.007507 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:50.016239 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:50.017098 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:09:55.008478 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:09:55.010528 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:09:55.020128 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:09:55.020724 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:00.013206 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:00.015728 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:00.024392 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:00.025072 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:05.008808 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:05.010871 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:05.019633 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:05.020266 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:10.005736 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:10.007746 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:10.016476 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:10.017103 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:15.010163 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:15.012163 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:15.025800 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:15.026422 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:20.007285 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:20.009244 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:20.018670 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:20.019280 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:25.038173 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:25.040960 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:25.050922 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:25.051524 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:30.008650 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:30.010851 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:30.021009 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:30.021603 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:35.010618 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:35.012928 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:35.021677 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:35.022271 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:40.012957 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:40.014993 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:40.023395 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:40.024031 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:45.011166 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:45.013809 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:45.022219 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:45.027211 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:50.011816 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:50.015010 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:50.023455 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:50.024078 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:10:55.010045 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:10:55.012214 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:10:55.029462 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:10:55.030050 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:00.006817 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:00.008731 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:00.018078 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:00.018671 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:05.012335 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:05.014328 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:05.022462 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:05.023060 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:10.010455 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:10.012617 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:10.023147 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:10.023929 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:15.011551 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:15.013673 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:15.021883 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:15.022478 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:20.009049 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:20.011671 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:20.020585 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:20.021231 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:25.012589 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:25.014492 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:25.023495 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:25.024193 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:30.011408 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:30.013403 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:30.022205 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:30.023064 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:35.010932 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:35.013060 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:35.023026 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:35.023813 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:40.018513 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:40.022866 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:40.037644 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:40.038639 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:45.011170 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:45.013098 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:45.024105 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:45.024740 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:50.011199 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:50.015908 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:50.025039 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:50.025639 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:11:55.013320 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:11:55.015266 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:11:55.025985 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:11:55.026594 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:00.011326 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:00.013486 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:00.022309 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:00.022977 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:05.011436 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:05.013431 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:05.022279 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:05.022883 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:10.007010 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:10.009196 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:10.017504 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:10.018093 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:15.010948 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:15.013273 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:15.022737 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:15.023347 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:20.009327 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:20.011323 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:20.021631 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:20.022295 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:25.010061 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:25.012268 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:25.022237 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:25.022865 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:30.010306 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:30.012394 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:30.025452 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:30.026457 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:35.010149 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:35.012259 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:35.024073 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:35.024693 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:40.009804 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:40.013069 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:40.022344 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:40.022952 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:45.008353 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:45.010398 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:45.020890 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:45.021678 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:50.006828 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:50.014563 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:50.024353 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:50.025135 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:12:55.011197 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:12:55.013321 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:12:55.022923 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:12:55.023525 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:13:00.010059 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:13:00.011951 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:13:00.020691 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:13:00.021299 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:13:02.011607 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler starting
I0211 08:13:02.015317 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler finished
I0211 08:13:02.073162 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:13:02.076225 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:13:02.091079 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:13:02.091909 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:13:09.942266 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler starting
I0211 08:13:09.945902 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler finished
I0211 08:13:10.212251 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler starting
I0211 08:13:10.226807 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler finished
I0211 08:13:10.420824 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:13:10.423790 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:13:10.450797 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:13:10.452409 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:13:26.591014 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:13:26.594212 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:13:26.604286 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:13:26.605099 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:13:30.767909 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:13:30.771031 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:13:30.787624 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:13:30.788481 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:13:35.767412 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:13:35.770145 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:13:35.784416 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:13:35.785220 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:13:40.774513 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:13:40.777459 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:13:40.798940 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:13:40.799902 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:13:45.775203 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:13:45.778420 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:13:45.791219 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:13:45.792152 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:13:50.762186 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:13:50.765604 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:13:50.787382 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:13:50.788064 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:13:55.770334 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:13:55.773810 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:13:55.789208 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:13:55.790041 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:14:00.791504 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:14:00.794960 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:14:00.841359 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:14:00.844300 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
### On clicking "Terminate" on run detail webpage:
I0211 08:14:01.657124 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/TerminateRun handler starting
I0211 08:14:05.770117 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:14:05.832577 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:14:05.845022 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:14:05.846320 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:14:14.723500 7 error.go:278] workflows.argoproj.io "detect-anomolies-on-wafer-tfdv-schema-qxpkk" not found
Failed to terminate workflow detect-anomolies-on-wafer-tfdv-schema-qxpkk due to patching error
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:339
github.com/kubeflow/pipelines/backend/src/apiserver/resource.TerminateWorkflow.func1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:722
github.com/cenkalti/backoff.RetryNotify
/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:37
github.com/cenkalti/backoff.Retry
/go/pkg/mod/github.com/cenkalti/backoff@v2.2.1+incompatible/retry.go:24
github.com/kubeflow/pipelines/backend/src/apiserver/resource.TerminateWorkflow
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:725
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).TerminateRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:752
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).terminateRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:447
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).TerminateRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:612
github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._RunService_TerminateRun_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/run.pb.go:2837
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._RunService_TerminateRun_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/run.pb.go:2839
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to terminate workflow detect-anomolies-on-wafer-tfdv-schema-qxpkk due to patching error after multiple retries
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:339
github.com/kubeflow/pipelines/backend/src/apiserver/resource.TerminateWorkflow
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:727
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).TerminateRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:752
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).terminateRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:447
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).TerminateRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:612
github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._RunService_TerminateRun_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/run.pb.go:2837
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._RunService_TerminateRun_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/run.pb.go:2839
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
InternalServerError: Failed to terminate run 1548e025-3d74-4036-a03f-ef79945f9e00 due to error terminating its workflow
github.com/kubeflow/pipelines/backend/src/common/util.NewInternalServerError
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:144
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).TerminateRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:754
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).terminateRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:447
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).TerminateRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:612
github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._RunService_TerminateRun_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/run.pb.go:2837
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._RunService_TerminateRun_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/run.pb.go:2839
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to terminate a run
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).TerminateRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:614
github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._RunService_TerminateRun_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/run.pb.go:2837
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._RunService_TerminateRun_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/run.pb.go:2839
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
/kubeflow.pipelines.backend.api.v2beta1.RunService/TerminateRun call failed
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:32
github.com/kubeflow/pipelines/backend/api/v2beta1/go_client._RunService_TerminateRun_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v2beta1/go_client/run.pb.go:2839
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
I0211 08:14:14.854052 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:14:14.857956 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:14:14.890131 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:14:14.890733 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:16:13.300773 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler starting
I0211 08:16:13.310558 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler finished
I0211 08:16:13.410243 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:16:13.413409 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:16:13.436236 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:16:13.437530 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:16:34.574534 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:16:34.576972 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:16:34.587412 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:16:34.588082 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:16:38.898247 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:16:38.900779 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:16:38.910915 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:16:38.911757 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:16:43.903211 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:16:43.905887 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:16:43.924155 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:16:43.925002 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:16:48.900978 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:16:48.903723 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:16:48.921553 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:16:48.922407 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:16:53.900240 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:16:53.902276 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:16:53.911259 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:16:53.911890 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:16:58.899183 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:16:58.901188 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:16:58.911088 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:16:58.911691 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:03.899545 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:03.901582 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:03.913704 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:03.914351 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:08.895923 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:08.897920 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:08.906314 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:08.906934 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:13.901809 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:13.904426 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:13.914614 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:13.915278 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:18.905054 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:18.910722 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:18.922491 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:18.923267 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:23.947884 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:23.951421 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:23.973379 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:23.974091 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:28.906170 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:28.909255 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:28.918716 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:28.919336 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:33.905425 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:33.909046 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:33.941493 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:33.942158 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:38.901024 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:38.903073 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:38.912505 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:38.913204 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:43.900571 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:43.902471 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:43.910749 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:43.911381 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:48.904252 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:48.906934 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:48.921878 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:48.922548 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:53.898339 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:53.900315 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:53.908339 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:53.909025 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:17:58.903901 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:17:58.905700 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:17:58.913524 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:17:58.914097 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:18:03.902100 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:18:03.903848 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:18:03.912656 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:18:03.913311 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:18:08.900715 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:18:08.902647 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:18:08.911180 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:18:08.911807 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:18:13.901814 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
### I0211 08:18:24.013598 7 error.go:278] dial tcp: lookup mysql on 10.96.0.10:53: no such host, when /api.RunService/GetRunV1. This is due to tampory not good network, so can not connect to mysql.
I0211 08:18:24.013598 7 error.go:278] dial tcp: lookup mysql on 10.96.0.10:53: no such host
InternalServerError: Failed to get run: dial tcp: lookup mysql on 10.96.0.10:53: no such host
github.com/kubeflow/pipelines/backend/src/common/util.NewInternalServerError
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:144
github.com/kubeflow/pipelines/backend/src/apiserver/storage.(*RunStore).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/storage/run_store.go:230
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:558
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).getRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:164
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRunV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:178
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2184
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to fetch run 1548e025-3d74-4036-a03f-ef79945f9e00
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:560
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).getRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:164
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRunV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:178
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2184
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to get a v1beta1 run
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRunV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:180
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2184
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
/api.RunService/GetRunV1 call failed
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:32
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
I0211 08:18:24.022713 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:18:39.065927 7 error.go:278] dial tcp: lookup mysql on 10.96.0.10:53: no such host
InternalServerError: Failed to get run: dial tcp: lookup mysql on 10.96.0.10:53: no such host
github.com/kubeflow/pipelines/backend/src/common/util.NewInternalServerError
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:144
github.com/kubeflow/pipelines/backend/src/apiserver/storage.(*RunStore).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/storage/run_store.go:230
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:558
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).getRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:164
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRunV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:178
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2184
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to fetch run 1548e025-3d74-4036-a03f-ef79945f9e00
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:560
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).getRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:164
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRunV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:178
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2184
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to get a v1beta1 run
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRunV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:180
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2184
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
/api.RunService/GetRunV1 call failed
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:32
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
I0211 08:18:39.074108 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:18:39.080390 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:18:39.089111 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:18:39.091542 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:18:39.101869 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:18:39.104398 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:18:39.109421 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:18:39.110398 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:18:39.116283 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:18:39.118842 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:18:39.125643 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:18:39.126310 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:18:39.134620 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:18:39.135400 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:18:39.143445 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:18:39.151411 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:18:43.898436 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:18:43.900338 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:18:43.908796 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:18:43.909408 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:18:48.907853 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:18:48.910316 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:18:48.919987 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:18:48.920762 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:18:53.899675 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:18:53.901646 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:18:53.909790 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:18:53.910421 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:18:58.899644 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:18:58.901731 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:18:58.909818 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:18:58.910415 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:03.901324 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:03.903565 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:03.912625 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:03.913348 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:08.910225 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:08.913272 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:08.933230 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:08.933977 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:13.905154 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:13.907497 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:13.917786 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:13.918462 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:18.901613 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:18.903596 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:18.912786 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:18.913429 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:23.896356 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:23.898302 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:23.906252 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:23.906920 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:28.940577 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:28.943701 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:28.955967 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:28.956582 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:33.909749 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:33.912135 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:33.921634 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:33.922219 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:38.905412 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:38.907417 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:38.916799 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:38.917433 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:43.900410 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:43.902433 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:43.910262 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:43.910879 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:48.900237 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:48.902238 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:48.910384 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:48.910983 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:53.899335 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:53.901196 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:53.911143 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:53.911788 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:19:58.900176 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:19:58.902123 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:19:58.910123 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:19:58.910730 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:03.903340 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:03.905327 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:03.914499 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:03.915078 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:08.902251 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:08.904577 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:08.912517 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:08.913142 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:13.901004 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:13.902888 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:13.911495 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:13.912094 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:18.898891 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:18.904143 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:18.912254 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:18.912831 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:23.900750 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:23.902579 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:23.911557 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:23.912131 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:28.899088 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:28.901851 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:28.911961 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:28.912673 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:33.903302 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:33.906439 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:33.917844 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:33.918771 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:38.901432 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:38.904370 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:38.915354 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:38.915964 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:43.901574 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:43.908295 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:43.916609 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:43.917413 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:48.902748 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:48.906067 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:48.917921 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:48.918830 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:53.902361 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:53.904137 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:53.912515 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:53.913144 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:20:58.935533 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:20:58.938722 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:20:58.991927 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:20:58.992638 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:21:03.897774 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:21:03.899883 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:21:03.907636 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:21:03.908197 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:21:08.901370 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:21:08.905870 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:21:08.920357 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:21:08.920989 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:21:13.902275 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:21:13.904250 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:21:13.912497 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:21:13.913089 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:21:18.900226 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:21:18.902166 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:21:18.910332 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:21:18.910948 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:22:18.900879 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:22:18.902921 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:22:18.912189 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:22:18.912801 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:23:18.900000 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:23:23.909369 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:23:23.920374 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:23:23.921106 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:24:18.901661 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:24:18.903619 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:24:18.912085 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:24:18.912728 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:25:18.898994 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:25:18.901123 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:25:18.909605 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:25:18.910299 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:26:18.903349 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:26:18.911694 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:26:18.920376 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:26:18.921083 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:27:18.902460 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:27:18.904887 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:27:18.914403 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:27:18.915070 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:28:18.902139 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:28:18.904138 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:28:18.912687 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:28:18.917626 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:29:18.898883 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:29:18.901269 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:29:18.909679 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:29:18.910533 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:30:18.901977 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:30:18.904023 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:30:18.913423 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:30:18.914052 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:31:18.907025 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:31:18.914890 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:31:18.923782 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:31:18.924460 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:32:18.900491 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:32:18.902460 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:32:18.911095 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:32:18.911721 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:33:18.902002 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:33:18.903973 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:33:18.913365 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
### I0211 08:33:38.925416 7 error.go:278] dial tcp: lookup mysql on 10.96.0.10:53: no such host, when /api.ExperimentService/GetExperimentV1, this is due to temporary not good network, so can not connect to mysql, since this not happens at every /api.ExperimentService/GetExperimentV1 .
I0211 08:33:38.925416 7 error.go:278] dial tcp: lookup mysql on 10.96.0.10:53: no such host
InternalServerError: Failed to get experiment: dial tcp: lookup mysql on 10.96.0.10:53: no such host
github.com/kubeflow/pipelines/backend/src/common/util.NewInternalServerError
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:144
github.com/kubeflow/pipelines/backend/src/apiserver/storage.(*ExperimentStore).GetExperiment
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/storage/experiment_store.go:150
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).GetExperiment
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:170
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).getExperiment
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:159
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).GetExperimentV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:169
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ExperimentService_GetExperimentV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/experiment.pb.go:1074
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ExperimentService_GetExperimentV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/experiment.pb.go:1076
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to fetch v1beta1 experiment
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ExperimentServer).GetExperimentV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/experiment_server.go:171
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ExperimentService_GetExperimentV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/experiment.pb.go:1074
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ExperimentService_GetExperimentV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/experiment.pb.go:1076
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
/api.ExperimentService/GetExperimentV1 call failed
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:32
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ExperimentService_GetExperimentV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/experiment.pb.go:1076
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
I0211 08:34:18.913985 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:34:33.930041 7 error.go:278] dial tcp: lookup mysql on 10.96.0.10:53: no such host
InternalServerError: Failed to get run: dial tcp: lookup mysql on 10.96.0.10:53: no such host
github.com/kubeflow/pipelines/backend/src/common/util.NewInternalServerError
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:144
github.com/kubeflow/pipelines/backend/src/apiserver/storage.(*RunStore).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/storage/run_store.go:230
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:558
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).getRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:164
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRunV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:178
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2184
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to fetch run 1548e025-3d74-4036-a03f-ef79945f9e00
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:560
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).getRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:164
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRunV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:178
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2184
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to get a v1beta1 run
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*RunServer).GetRunV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/run_server.go:180
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2184
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
/api.RunService/GetRunV1 call failed
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:32
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._RunService_GetRunV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/run.pb.go:2186
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
I0211 08:35:18.902891 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:35:23.913583 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:35:23.924042 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:35:23.924619 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:36:18.902214 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:36:18.904216 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:36:18.912798 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:36:18.913386 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:37:06.988542 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler starting
I0211 08:37:06.992001 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler finished
I0211 08:37:07.051003 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:37:07.054541 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:37:07.119953 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:37:07.121282 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:37:18.195598 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:37:18.198627 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:37:18.209522 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:37:18.210246 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:37:22.287104 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:37:22.289791 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:37:22.299898 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:37:22.300866 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:37:27.282019 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:37:32.288368 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:37:32.298215 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:37:32.298853 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:44:40.945212 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler starting
I0211 08:44:45.955180 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler finished
I0211 08:44:45.976345 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:44:45.980056 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:44:46.035927 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:44:46.036513 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:44:52.127088 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:44:52.129005 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:44:52.141059 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:44:52.141700 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:45:53.823278 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler starting
I0211 08:45:53.826248 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler finished
I0211 08:45:53.892955 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:45:53.895678 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:45:53.908012 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:45:53.908893 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
### On clicking "Pipelines" on left side ot the webpage:
I0211 08:45:56.857968 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler starting
I0211 08:45:56.861553 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler finished
### On clicking "Runs" on left side of the webpage:
I0211 08:46:04.377036 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/ListRuns handler starting
I0211 08:46:04.383337 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/ListRuns handler finished
I0211 08:46:04.429033 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler starting
I0211 08:46:04.442850 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler finished
I0211 08:46:04.460512 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler starting
I0211 08:46:04.461582 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler finished
I0211 08:46:09.077478 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:09.079566 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:09.087968 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:09.088599 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:13.485126 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler starting
I0211 08:46:13.489461 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler finished
I0211 08:46:13.521435 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:13.523619 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:13.576188 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:13.577036 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:14.077428 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:14.080181 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:14.089772 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:14.090545 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:18.740877 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:18.742884 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:18.756820 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:18.757437 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:19.078228 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:19.080433 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:19.089668 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:19.090307 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:23.741964 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:23.743918 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:23.755415 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:23.756013 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:24.109710 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler starting
I0211 08:46:24.111909 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelines handler finished
I0211 08:46:24.178301 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:24.181961 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:24.285192 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:24.286515 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:26.218126 7 interceptor.go:29] /api.PipelineService/GetPipelineV1 handler starting
I0211 08:46:26.221738 7 interceptor.go:37] /api.PipelineService/GetPipelineV1 handler finished
I0211 08:46:26.241651 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipeline handler starting
I0211 08:46:26.242552 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipeline handler finished
I0211 08:46:26.261567 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelineVersions handler starting
I0211 08:46:26.275489 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelineVersions handler finished
I0211 08:46:26.328062 7 interceptor.go:29] /api.PipelineService/ListPipelineVersionsV1 handler starting
I0211 08:46:26.330541 7 interceptor.go:37] /api.PipelineService/ListPipelineVersionsV1 handler finished
I0211 08:46:26.336490 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelineVersions handler starting
I0211 08:46:26.346678 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/ListPipelineVersions handler finished
I0211 08:46:29.079035 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:29.084264 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:29.092932 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:29.093583 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:34.090287 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:34.093763 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:34.119963 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:34.120679 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:34.194817 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipeline handler starting
I0211 08:46:34.195666 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipeline handler finished
I0211 08:46:34.200808 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler starting
I0211 08:46:34.212849 7 interceptor.go:29] /api.PipelineService/GetPipelineVersionTemplate handler starting
I0211 08:46:34.219000 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler finished
I0211 08:46:34.221618 7 interceptor.go:37] /api.PipelineService/GetPipelineVersionTemplate handler finished
I0211 08:46:34.350644 7 interceptor.go:29] /api.PipelineService/GetPipelineV1 handler starting
I0211 08:46:34.353970 7 interceptor.go:37] /api.PipelineService/GetPipelineV1 handler finished
I0211 08:46:34.430933 7 interceptor.go:29] /api.PipelineService/GetPipelineVersionV1 handler starting
I0211 08:46:34.432165 7 interceptor.go:37] /api.PipelineService/GetPipelineVersionV1 handler finished
I0211 08:46:39.077939 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:39.080390 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:39.088426 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:39.089067 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:44.082375 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:44.084817 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:44.093656 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:44.094353 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
### retry creating run for the sampe pipeline, On clicking "Start" on "Create Run" webpage:
I0211 08:46:44.696273 7 interceptor.go:29] /api.RunService/CreateRunV1 handler starting
I0211 08:46:47.084088 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 08:46:49.167581 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
### E0211 08:46:51.546205 7 resource_manager.go:1150] Cannot find reported workflow name="detect-anomolies-on-wafer-tfdv-schema-pnpjf" namespace="kubeflow" runId="c6013cee-f684-4524-a2b5-719f134e46af" in run store. Deleting the workflow to avoid resource leaking. when /api.ReportService/ReportWorkflowV1 .
E0211 08:46:51.546205 7 resource_manager.go:1150] Cannot find reported workflow name="detect-anomolies-on-wafer-tfdv-schema-pnpjf" namespace="kubeflow" runId="c6013cee-f684-4524-a2b5-719f134e46af" in run store. Deleting the workflow to avoid resource leaking. This can be caused by installing two KFP instances that try to manage the same workflows or an unknown bug. If you encounter this, recommend reporting more details in https://github.com/kubeflow/pipelines/issues/6189
I0211 08:46:51.547952 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:51.587290 7 interceptor.go:37] /api.RunService/CreateRunV1 handler finished
I0211 08:46:51.633255 7 error.go:278] ResourceNotFoundError: Run c6013cee-f684-4524-a2b5-719f134e46af not found
github.com/kubeflow/pipelines/backend/src/common/util.NewResourceNotFoundError
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
github.com/kubeflow/pipelines/backend/src/apiserver/storage.(*RunStore).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/storage/run_store.go:239
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:558
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReportWorkflowResource
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:1131
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).reportWorkflow
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:65
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflowV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:81
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:338
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:340
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to fetch run c6013cee-f684-4524-a2b5-719f134e46af
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).GetRun
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:560
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReportWorkflowResource
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:1131
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).reportWorkflow
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:65
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflowV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:81
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:338
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:340
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to report workflow name="detect-anomolies-on-wafer-tfdv-schema-pnpjf" namespace="kubeflow" runId="c6013cee-f684-4524-a2b5-719f134e46af"
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
github.com/kubeflow/pipelines/backend/src/apiserver/resource.(*ResourceManager).ReportWorkflowResource
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/resource/resource_manager.go:1166
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).reportWorkflow
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:65
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflowV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:81
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:338
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:340
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
Failed to report workflow
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
github.com/kubeflow/pipelines/backend/src/common/util.Wrap
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).reportWorkflow
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:67
github.com/kubeflow/pipelines/backend/src/apiserver/server.(*ReportServer).ReportWorkflowV1
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/server/report_server.go:81
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler.func1
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:338
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:30
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:340
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
/api.ReportService/ReportWorkflowV1 call failed
github.com/kubeflow/pipelines/backend/src/common/util.(*UserError).wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
github.com/kubeflow/pipelines/backend/src/common/util.Wrapf
/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
main.apiServerInterceptor
/go/src/github.com/kubeflow/pipelines/backend/src/apiserver/interceptor.go:32
github.com/kubeflow/pipelines/backend/api/v1beta1/go_client._ReportService_ReportWorkflowV1_Handler
/go/src/github.com/kubeflow/pipelines/backend/api/v1beta1/go_client/report.pb.go:340
google.golang.org/grpc.(*Server).processUnaryRPC
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1282
google.golang.org/grpc.(*Server).handleStream
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:1616
google.golang.org/grpc.(*Server).serveStreams.func1.2
/go/pkg/mod/google.golang.org/grpc@v1.44.0/server.go:921
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598
I0211 08:46:51.695664 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/ListRuns handler starting
I0211 08:46:51.708136 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/ListRuns handler finished
I0211 08:46:51.747116 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:51.748115 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:51.763852 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler starting
I0211 08:46:51.781439 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler finished
I0211 08:46:51.846488 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler starting
I0211 08:46:51.867544 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler finished
I0211 08:46:51.989897 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler starting
I0211 08:46:51.990839 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler finished
I0211 08:46:52.168506 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler starting
I0211 08:46:52.169564 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler finished
I0211 08:46:52.658084 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 08:46:52.863926 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 08:46:54.078691 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:54.081337 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:54.090692 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:54.091265 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:46:59.084293 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:46:59.087144 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:46:59.105608 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:46:59.106445 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:04.076192 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:04.077995 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:04.086300 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:04.086905 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:09.076406 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:09.078292 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:09.086559 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:09.087157 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:14.076108 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:14.078093 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:14.086614 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:14.087238 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:19.076232 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:19.078144 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:19.086399 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:19.087092 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:24.078576 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:24.082079 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:24.091627 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:24.092212 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:29.081065 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:29.083315 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:29.092661 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:29.093293 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:34.079077 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:34.081960 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:34.090898 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:34.091517 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:39.083089 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:39.086475 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:39.104546 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:39.105550 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:44.078250 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:44.080057 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:44.090477 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:44.091091 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:49.078532 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:49.080804 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:49.089270 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:49.089910 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:54.076429 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:54.078441 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:54.086186 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:54.086783 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:47:59.078589 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:47:59.081867 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:47:59.092700 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:47:59.093962 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:04.076957 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:04.078919 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:04.088253 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:04.088861 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:09.076144 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:09.078167 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:09.087144 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:09.087735 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:14.078319 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:14.081547 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:14.091804 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:14.092405 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:19.081273 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:19.083620 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:19.093181 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:19.093906 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:24.078203 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:24.081466 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:24.095031 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:24.095645 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:29.080721 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:29.083509 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:29.092254 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:29.092859 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:34.079839 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:34.082579 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:34.099832 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:34.100608 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:39.076314 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:39.078352 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:39.086344 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:39.086918 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:44.079563 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:44.083031 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:44.091608 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:44.092241 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:49.079765 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:49.081652 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:49.090150 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:49.090747 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:54.075956 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:54.082829 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:54.090669 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:54.091376 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:48:59.076235 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:48:59.078223 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:48:59.086952 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:48:59.087561 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:04.079310 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:04.082145 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:04.095494 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:04.096350 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:09.076078 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:09.077959 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:09.086315 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:09.086888 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:14.079105 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:14.084861 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:14.092674 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:14.093293 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:19.076633 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:19.078602 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:19.086803 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:19.087424 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:24.076238 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:24.078117 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:24.086882 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:24.087558 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:29.080162 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:29.083036 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:29.095976 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:29.096801 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:34.078196 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:34.080962 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:34.094693 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:34.095568 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:39.078541 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:39.081479 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:39.091857 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:39.092478 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:44.078954 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:44.081802 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:44.089676 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:44.090293 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:49.077839 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:49.080491 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:49.090563 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:49.091183 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:54.081384 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:54.084155 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:54.095305 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:54.095925 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:49:59.079056 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:49:59.082658 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:49:59.091009 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:49:59.091708 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:04.077402 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:04.080519 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:04.089373 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:04.089987 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:09.078972 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:09.081861 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:09.089888 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:09.090488 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:14.078264 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:14.081587 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:14.089903 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:14.090469 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:19.079278 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:19.081958 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:19.097333 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:19.098152 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:24.077021 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:24.079885 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:24.093245 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:24.094022 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:29.080562 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:29.083279 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:29.095329 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:29.096304 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:34.078253 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:34.080976 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:34.095672 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:34.096517 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:39.079320 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:39.082700 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:39.103058 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:39.103849 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:44.079459 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:44.082511 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:44.098186 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:44.099279 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:49.086950 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:49.089385 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:49.097563 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:49.098155 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:54.078453 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:54.080504 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:54.089506 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:54.094525 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:50:59.078663 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:50:59.081017 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:50:59.088761 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:50:59.089374 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:04.078556 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:04.081669 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:04.089893 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:04.090499 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:09.080709 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:09.083566 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:09.093049 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:09.093649 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:14.078257 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:14.081047 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:14.089282 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:14.089856 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:19.078662 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:19.081811 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:19.091177 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:19.091771 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:24.080160 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:24.082419 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:24.093659 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:24.094300 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:29.079505 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:29.081568 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:29.090463 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:29.091055 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:34.078707 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:34.081718 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:34.090351 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:34.090949 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:39.082413 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:39.084799 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:39.092984 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:39.093580 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:44.077546 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:44.080092 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:44.089095 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:44.089662 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:49.076841 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:49.079656 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:49.088336 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:49.088964 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:54.081494 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:54.083928 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:54.092727 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:54.093362 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:51:59.078109 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:51:59.080176 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:51:59.098968 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:51:59.100613 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:04.076626 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:04.078397 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:04.087745 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:04.088343 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:09.078604 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:09.081065 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:09.091361 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:09.091938 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:14.080331 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:14.082298 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:14.090471 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:14.091295 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:19.078162 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:19.080747 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:19.090765 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:19.091357 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:24.077936 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:24.081132 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:24.090068 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:24.090643 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:29.079310 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:29.082234 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:29.096203 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:29.097114 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:34.076646 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:34.078609 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:34.086331 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:34.086902 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:39.080845 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:39.083371 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:39.091870 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:39.092502 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:44.078634 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:44.081151 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:44.091068 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:44.091684 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:49.079891 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:49.081904 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:49.089926 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:49.090514 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:54.078671 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:54.081757 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:54.093617 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:54.094253 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:52:59.078746 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:52:59.085210 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:52:59.093705 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:52:59.094369 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:04.081243 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:04.083634 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:04.091862 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:04.092520 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:09.076121 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:09.078118 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:09.088533 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:09.089258 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:14.078039 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:14.080631 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:14.089401 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:14.090018 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:19.078482 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:19.081411 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:19.090024 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:19.090620 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:24.077903 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:24.079823 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:24.088979 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:24.089528 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:29.078045 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:29.080987 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:29.089997 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:29.090642 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:34.078279 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:34.081171 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:34.093743 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:34.094333 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:39.081760 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:39.083665 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:39.092664 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:39.093266 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:44.080672 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:44.082506 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:44.090873 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:44.091496 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:49.078561 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:49.080646 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:49.090917 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:49.091635 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:54.078038 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:54.080777 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:54.088655 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:54.089267 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:53:59.077401 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:53:59.080428 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:53:59.089135 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:53:59.089712 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:04.077942 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:04.080846 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:04.089102 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:04.089764 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:09.078233 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:09.081613 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:09.089695 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:09.090363 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:14.079911 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:14.081880 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:14.090298 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:14.090861 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:19.078396 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:19.081373 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:19.089687 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:19.090368 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:24.078619 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:24.081435 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:24.092505 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:24.093117 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:29.078432 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:29.080595 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:29.103202 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:29.103822 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:34.078963 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:34.081765 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:34.089974 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:34.090523 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:39.077774 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:39.080664 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:39.093559 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:39.094480 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:44.077618 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:44.080011 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:44.091321 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:44.092388 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:49.081437 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:49.083804 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:49.091986 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:49.092587 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:54.076369 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:54.078966 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:54.089965 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:54.090817 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:54:59.078249 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:54:59.080061 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:54:59.088168 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:54:59.093638 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:04.081046 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:04.083812 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:04.092346 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:04.093026 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:09.078458 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:09.081357 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:09.093560 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:09.094144 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:14.078981 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:14.082357 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:14.090948 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:14.091596 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:19.078589 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:19.080710 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:19.088509 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:19.089145 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:24.082461 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:24.084855 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:24.093285 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:24.093890 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:29.078677 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:29.081916 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:29.092423 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:29.093036 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:34.079267 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:34.082023 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:34.091055 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:34.091680 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:39.079281 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:39.082276 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:39.090444 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:39.091018 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:44.075898 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:44.078066 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:44.087259 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:44.087871 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:49.079233 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:49.081167 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:49.090095 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:49.090673 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:54.076336 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:54.078492 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:54.101104 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:54.102422 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:55:59.082391 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:55:59.084520 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:55:59.094880 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:55:59.095607 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:04.078304 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:04.080752 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:04.088546 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:04.089236 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:09.078233 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:09.081800 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:09.091679 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:09.092252 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:14.077036 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:14.082515 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:14.100176 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:14.101090 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:19.080886 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:19.083317 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:19.091004 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:19.091599 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:24.077012 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:24.078865 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:24.087078 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:24.087654 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:29.079275 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:29.081389 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:29.091264 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:29.092013 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:34.077991 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:34.081336 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:34.091187 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:34.091794 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:39.078851 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:39.081127 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:39.092394 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:39.093078 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:44.078075 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:44.081722 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:44.090297 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:44.090962 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:49.078138 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:49.080943 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:49.090708 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:49.091389 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:54.078229 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:54.080838 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:54.089104 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:54.089704 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:56:59.078254 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:56:59.081001 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:56:59.088813 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:56:59.089478 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:04.077235 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:09.086917 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:09.097499 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:09.099994 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:09.100874 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:09.101832 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:09.111899 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:09.112593 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:14.077580 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:14.080036 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:14.087902 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:14.088477 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:19.075864 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:19.077800 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:19.088115 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:19.088724 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:24.075873 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:24.077803 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:24.085764 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:24.086384 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:29.080499 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:29.082823 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:29.091233 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:29.091840 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:34.078004 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:34.080334 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:34.090064 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:34.090758 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:39.078430 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:39.081120 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:39.090698 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:39.091320 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:44.078404 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:44.080548 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:44.091349 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:44.092089 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:49.078373 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:49.082433 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:49.092350 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:49.092958 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:54.080224 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:54.082250 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:54.090314 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:54.090895 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:57:59.076363 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:57:59.078249 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:57:59.086777 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:57:59.087363 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:04.081107 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:04.083215 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:04.093623 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:04.094286 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:09.081180 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:09.083559 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:09.094611 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:09.095220 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:14.076044 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:14.078302 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:14.096406 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:14.097554 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:19.078512 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:19.081293 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:19.089906 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:19.090532 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:24.080984 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:24.082721 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:24.090684 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:24.091292 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:29.081872 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:29.084170 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:29.093121 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:29.093681 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:34.077064 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:34.080248 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:34.089944 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:34.090673 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:39.076807 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:39.079424 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:39.088706 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:39.089684 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:44.078168 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:44.081445 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:44.089349 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:44.089951 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:49.078382 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:49.080286 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:49.089504 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:49.090096 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:54.078254 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:54.081319 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:54.092846 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:54.093499 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:58:59.080562 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:58:59.082682 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:58:59.094146 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:58:59.094768 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:04.078500 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:04.081428 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:04.089084 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:04.089715 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:09.077721 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:09.080744 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:09.089362 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:09.094008 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:14.078218 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:14.081134 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:14.088828 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:14.089465 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:19.076561 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:19.079424 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:19.090282 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:19.091268 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:24.077870 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:24.077870 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:24.080494 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:24.088532 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:24.089190 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:29.083442 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:29.085470 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:29.096601 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:29.097268 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:34.079923 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:34.082004 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:34.091209 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:34.091781 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:39.078541 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:39.081924 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:39.093072 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:39.093655 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:44.077402 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:44.079372 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:44.087115 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:44.087712 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:49.078734 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:49.081303 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:49.088599 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:49.089215 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:54.080823 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:54.082916 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:54.091570 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:54.092198 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 08:59:59.077639 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 08:59:59.080264 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 08:59:59.089714 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 08:59:59.090348 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:04.078579 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:04.081478 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:04.089839 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:04.090413 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:09.082464 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:09.084750 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:09.094214 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:09.094785 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:14.078655 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:14.081427 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:14.091080 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:14.091679 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:19.083683 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:19.086093 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:19.094339 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:19.094920 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:24.079009 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:24.080967 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:24.092197 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:24.092794 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:29.078123 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:29.080013 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:29.087655 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:29.088258 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:34.078050 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:34.081109 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:34.089964 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:34.090555 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:39.080086 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:39.082916 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:39.091579 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:39.092151 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:44.080555 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:44.083226 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:44.092001 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:44.092565 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:49.077835 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:49.080898 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:49.090664 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:49.091274 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:54.081314 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:54.083971 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:54.092509 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:54.093127 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:00:59.078848 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:00:59.081606 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:00:59.095414 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:00:59.096174 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:04.079722 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:04.082812 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:04.092802 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:04.093439 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:09.077576 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:09.080828 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:09.089172 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:09.089762 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:14.078169 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:14.086749 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:14.097497 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:14.098047 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:19.081616 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:19.083937 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:19.092506 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:19.093138 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:24.078495 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:24.081572 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:24.090329 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:24.090887 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:29.079054 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:29.081196 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:29.088834 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:29.089442 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:34.078145 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:34.080836 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:34.089492 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:34.090063 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:39.079451 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:39.082221 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:39.090843 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:39.091444 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:44.078420 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:44.081587 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:44.089716 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:44.090310 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:49.076004 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:49.078980 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:49.090159 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:49.090746 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:54.078045 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:54.081265 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:54.089581 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:54.090379 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:01:59.081629 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:01:59.083807 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:01:59.092349 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:01:59.092965 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:04.078583 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:04.080804 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:04.088759 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:04.089366 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:09.080408 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:09.082483 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:09.090290 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:09.090911 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:14.078078 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:14.080671 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:14.097353 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:14.098169 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:19.078411 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:19.080750 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:19.090879 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:19.091467 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:24.079903 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:24.082283 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:24.091324 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:24.091974 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:29.076314 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:29.079281 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:29.088508 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:29.089151 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:34.078288 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:34.080585 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:34.088800 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:34.089408 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:39.081088 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:39.087204 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:39.095547 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:39.096155 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:44.082100 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:44.084425 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:44.092645 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:44.093243 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:49.078714 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:49.080802 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:49.088713 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:49.089374 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:54.080754 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:54.083913 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:54.092349 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:54.093043 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:02:59.077362 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:02:59.079239 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:02:59.087027 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:02:59.087627 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:04.081459 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:04.084007 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:04.093850 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:04.094421 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:09.077949 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:09.080500 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:09.094737 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:09.095531 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:14.076636 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:14.080687 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:14.090237 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:19.097653 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:19.104864 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:19.106983 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:19.176153 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:19.176823 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:24.077783 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:24.089243 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:24.098965 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:24.099532 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:29.079370 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:29.082631 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:29.090872 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:29.091481 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:34.078345 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:34.080541 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:34.088099 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:34.088688 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:39.078101 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:39.081138 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:39.088883 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:39.089549 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:44.078051 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:44.079956 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:44.090584 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:44.091356 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:49.078867 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:49.081585 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:49.091231 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:49.091809 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:54.076612 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:54.079341 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:54.088100 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:54.088675 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:03:59.077107 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:03:59.079593 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:03:59.092131 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:03:59.093051 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:04.078060 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:04.080487 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:04.090329 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:04.090915 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:09.078117 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:09.081398 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:09.096251 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:09.096867 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:14.078034 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:14.080840 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:14.089928 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:14.090504 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:19.081352 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:19.083838 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:19.093988 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:19.094568 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:24.085307 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:24.087490 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:24.099137 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:24.099955 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:29.078791 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:29.080745 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:29.088355 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:29.089015 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:34.081975 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:34.083920 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:34.092388 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:34.093047 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:39.080035 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:39.082912 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:39.092627 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:39.093301 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:44.078602 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:44.081688 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:44.090127 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:44.090696 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:49.080066 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:49.082783 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:49.091041 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:49.091639 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:54.078759 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:54.081197 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:54.092259 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:54.092877 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:04:59.079916 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:04:59.081820 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:04:59.090139 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:04:59.090873 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:04.077406 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:04.079576 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:04.087743 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:04.088325 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:09.078174 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:09.081158 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:09.090103 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:09.090747 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:14.078491 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:14.080739 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:14.089113 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:14.089677 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:19.077527 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:19.079459 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:19.087268 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:19.087857 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:24.080981 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:24.088006 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:24.096015 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:24.096634 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:29.078066 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:29.081113 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:29.090704 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:29.091341 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:34.081658 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:34.084184 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:34.095933 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:34.096549 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:39.078387 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:39.081401 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:39.090380 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:39.091006 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:44.081329 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:44.084047 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:44.093038 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:44.093602 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:49.078142 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:49.080769 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:49.088898 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:49.089544 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:54.081099 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:54.083585 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:54.091906 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:54.092503 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:05:59.080010 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:05:59.081890 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:05:59.102789 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:05:59.103588 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:04.079498 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:04.081351 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:04.089552 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:04.090162 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:09.174663 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:09.177064 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:09.231351 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:09.231980 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:14.078584 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:14.081181 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:14.089372 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:14.089956 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:19.078542 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:19.082448 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:19.093591 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:19.094177 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:24.077890 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:24.079988 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:24.087834 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:24.088424 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:29.078141 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:29.081335 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:29.090562 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:29.091148 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:34.079107 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:34.081940 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:34.090865 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:34.091542 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:39.078225 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:39.080887 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:39.089870 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:39.090487 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:44.081595 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:44.084056 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:44.093038 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:44.093764 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:49.078472 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:49.081012 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:49.090190 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:49.090789 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:54.080556 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:54.082505 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:54.090504 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:54.091115 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:06:59.076155 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:06:59.078020 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:06:59.086144 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:06:59.086870 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:04.080759 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:04.082663 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:04.102162 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:04.103062 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:09.077403 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:09.080822 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:09.094948 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:09.095814 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:14.080889 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:14.082890 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:14.095236 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:14.096060 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:19.077231 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:19.080164 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:19.093557 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:19.094333 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:24.079802 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:24.082429 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:24.096264 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:24.100840 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:29.078146 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:29.081104 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:29.090884 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:29.091465 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:34.077891 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:34.080982 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:34.089054 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:34.089661 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:39.078353 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:39.080774 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:39.089605 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:39.090213 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:44.078356 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:44.080823 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:44.089320 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:44.089925 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:49.085834 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:49.088622 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:49.103252 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:49.103943 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:54.076355 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:54.081804 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:54.089376 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:54.089991 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:07:59.078166 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:07:59.081562 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:07:59.090076 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:07:59.090702 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:04.077482 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:04.079543 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:04.086868 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:04.087440 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:09.077770 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:09.080023 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:09.089463 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:09.090074 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:14.075987 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:14.077953 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:14.086961 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:14.087535 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:19.083115 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:19.085601 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:19.101666 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:19.102580 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:24.083617 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:24.086292 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:24.098376 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:24.099018 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:29.078589 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:29.081729 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:29.094667 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:29.095621 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:34.077286 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:34.079983 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:34.102642 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:34.103412 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:39.076079 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:39.078625 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:39.092988 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:39.093722 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:44.078035 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:44.081144 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:44.090696 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:44.091282 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:49.075905 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:49.077780 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:49.088272 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:49.089331 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:54.078690 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:54.081616 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:54.089418 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:54.090006 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:08:59.076658 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:08:59.078567 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:08:59.086683 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:08:59.087237 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:04.079791 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:04.081792 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:04.091766 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:04.092562 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:09.077182 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:09.079781 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:09.092737 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:09.093904 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:14.076892 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:14.079872 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:14.093182 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:14.094061 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:19.084336 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:19.087504 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:19.103276 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:19.104191 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:24.076642 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:24.080087 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:24.092967 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:24.093627 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:29.079509 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:29.086500 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:29.094751 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:29.095324 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:34.077877 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:34.080598 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:34.098258 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:34.099149 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:39.075783 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:39.077722 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:39.085606 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:39.086220 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:44.078481 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:44.081731 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:44.090605 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:44.091210 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:49.076311 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:49.078136 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:49.086023 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:49.086661 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:54.078621 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:54.081229 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:54.089851 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:54.090587 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:09:59.076021 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:09:59.077899 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:09:59.085222 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:09:59.085782 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:04.078707 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:04.081475 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:04.092328 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:04.092888 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:09.076744 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:09.079143 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:09.088820 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:09.089537 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:14.080049 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:14.082127 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:14.092535 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:14.093211 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:19.080392 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:19.082950 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:19.093726 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:19.094369 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:24.082769 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:24.085296 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:24.093663 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:24.094263 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:29.078043 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:29.080991 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:29.089513 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:29.090115 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:34.080745 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:34.082828 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:34.092872 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:34.093626 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:39.076668 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:39.078756 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:39.090351 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:39.091124 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:44.075857 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:44.078294 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:44.087934 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:44.088742 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:49.075727 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:49.077680 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:49.089249 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:49.090168 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:54.080729 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:54.083536 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:54.094577 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:54.095198 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:10:59.078577 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:10:59.082299 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:10:59.091055 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:10:59.091653 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:04.078286 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:04.080922 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:04.093625 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:04.094925 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:09.078340 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:09.081454 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:09.089769 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:09.090372 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:14.078210 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:14.080413 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:14.090189 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:14.090849 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:19.078938 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:19.082071 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:19.089868 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:19.090576 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:24.078579 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:24.080733 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:24.091902 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:24.092676 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:29.076241 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:29.078689 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:29.087956 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:34.082394 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:34.093720 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:34.095259 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:34.106167 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:34.106995 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:39.077176 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:39.079481 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:39.091763 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:39.092363 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:44.077459 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:44.079886 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:44.091420 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:44.092555 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:49.076573 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:49.079535 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:49.092446 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:49.093396 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:54.078706 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:54.081130 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:54.090216 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:54.090806 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:11:59.078844 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:11:59.081370 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:11:59.092743 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:11:59.093642 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:04.078335 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:04.081476 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:04.089911 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:04.090517 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:09.080616 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:09.082473 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:09.090968 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:09.091569 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:14.077133 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:14.079490 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:14.089468 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:14.090130 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:19.079017 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:19.081547 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:19.090279 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:19.090866 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:24.077862 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:24.080383 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:24.102515 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:24.103159 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:29.078648 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:29.080650 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:29.088383 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:29.088986 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:34.079573 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:34.082116 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:34.090495 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:34.091056 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:39.078526 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:39.081798 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:39.090554 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:39.091142 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:44.084123 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:44.086920 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:44.101583 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:44.102365 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:49.078212 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:49.081572 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:49.090019 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:49.090589 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:54.080571 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:54.083158 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:54.093117 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:54.093762 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:12:59.076311 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:12:59.078460 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:12:59.087322 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:12:59.088008 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:04.075835 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:04.077959 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:04.085573 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:04.086175 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:09.076495 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:09.078506 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:09.086710 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:09.087320 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:14.076719 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:14.079373 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:14.094715 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:14.095509 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:19.079056 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:19.082515 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:19.090696 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:19.091314 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:24.076024 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:24.078144 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:24.087417 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:24.088077 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:29.078424 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:29.080295 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:29.090975 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:29.091625 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:34.078615 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:34.081959 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:34.093516 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:34.097910 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:39.078937 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:39.081414 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:39.091037 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:39.091643 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:44.080491 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:44.082439 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:44.090797 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:44.091434 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:49.078278 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:49.081007 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:49.094225 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:49.095050 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:54.081044 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:54.083644 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:54.092519 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:54.093107 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:13:59.076890 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:13:59.079815 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:13:59.093787 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:13:59.094556 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:04.076188 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:04.081291 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:04.090675 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:04.091305 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:09.078281 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:09.081150 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:09.090350 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:09.091157 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:14.078273 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:14.081256 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:14.095262 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:14.096117 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:19.077320 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:19.079253 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:19.087210 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:19.087823 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:24.079230 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:24.081733 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:24.089894 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:24.090505 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:29.080721 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:29.083675 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:29.092517 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:29.093178 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:34.078053 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:34.080716 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:34.089335 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:34.089929 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:39.165795 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:39.168261 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:39.247198 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:39.248159 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:44.075939 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:44.077923 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:44.089125 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:44.089737 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:49.077183 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:49.079350 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:49.090602 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:49.091248 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:49.091248 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:54.077611 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:54.079771 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:54.088894 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:54.089500 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:14:59.078329 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:14:59.080206 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:14:59.088515 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:14:59.089178 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:04.076912 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:04.079817 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:04.088997 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:04.089578 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:09.078224 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:09.080940 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:09.090451 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:09.091106 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:14.080140 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:14.082136 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:14.090717 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:14.091291 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:19.077839 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:19.080751 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:19.093928 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:19.094536 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:24.076644 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:24.079945 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:24.091121 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:24.091811 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:29.077210 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:29.079657 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:29.099015 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:29.099783 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:34.077086 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:34.080364 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:34.089180 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:34.089738 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:39.077877 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:49.086795 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:49.100426 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:49.104058 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:49.107102 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:49.108028 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:49.117959 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:49.120585 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:49.149733 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:49.150718 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:49.158196 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:49.158989 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:54.078645 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:54.081376 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:54.090301 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:54.090948 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:15:59.078725 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:15:59.081827 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:15:59.091622 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:15:59.092274 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:04.076086 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:04.078115 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:04.086252 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:04.086862 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:09.077492 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:09.080238 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:09.095686 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:09.096711 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:14.080956 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:14.083100 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:14.094554 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:14.095248 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:19.080995 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:19.083416 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:19.092090 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:19.092728 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:24.078842 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:24.081522 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:24.090587 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:24.091185 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:29.078468 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:29.081537 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:29.089883 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:29.090489 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:34.078618 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:34.081366 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:34.090524 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:34.091185 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:39.078790 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:39.081304 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:39.093714 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:39.094331 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:44.077327 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:44.081121 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:44.092421 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:44.093052 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:49.077873 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:49.081256 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:49.091849 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:49.092629 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:54.080737 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:54.083571 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:54.097672 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:54.098589 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:16:59.077102 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:16:59.079808 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:16:59.107509 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:16:59.108216 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:04.076856 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:04.079145 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:04.090033 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:04.091134 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:09.078385 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:09.085768 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:09.099726 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:09.100503 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:14.084011 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:14.086899 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:14.108789 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:14.109697 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:19.075922 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:19.077753 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:19.085906 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:19.086858 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:24.080427 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:24.083215 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:24.095160 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:24.095928 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:29.076425 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:29.079138 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:29.092353 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:29.093196 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:34.077959 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:34.080957 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:34.091911 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:34.092673 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:39.078333 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:39.080507 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:39.088351 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:39.089013 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:44.078341 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:44.080981 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:44.089609 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:44.090176 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:49.076172 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:49.079130 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:49.094987 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:49.100025 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:54.080676 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:54.085150 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:54.097797 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:54.098398 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:17:59.077891 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:17:59.081520 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:17:59.089754 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:17:59.090314 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:04.080640 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:04.083451 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:04.104288 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:04.104939 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:09.076994 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:09.079452 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:09.093229 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:09.094039 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:14.078365 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:14.080825 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:14.088483 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:14.089140 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:19.079490 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:19.081891 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:19.089833 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:19.090437 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:24.076184 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:24.078201 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:24.085862 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:24.086482 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:29.076086 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:29.078026 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:29.085982 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:29.086587 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:34.078266 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:34.081152 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:34.092456 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:34.093112 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:39.080030 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:39.081846 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:39.089711 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:39.090272 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:44.083970 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:44.086985 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:44.100343 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:44.101266 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:49.077335 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:49.080065 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:49.091681 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:49.092472 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:54.076555 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:54.079243 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:54.087475 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:54.088084 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:18:59.075906 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:18:59.077832 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:18:59.086162 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:18:59.086799 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:04.078083 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:04.080237 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:04.094613 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:04.095540 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:09.077819 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:09.081052 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:09.089920 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:09.090512 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:14.078454 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:14.080515 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:14.087997 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:14.091632 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:19.077725 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:19.079561 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:19.089643 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:19.090291 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:24.077961 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:24.080459 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:24.089218 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:24.089774 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:29.076127 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:29.078118 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:29.087650 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:29.088449 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:34.077797 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:34.079859 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:34.088824 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:34.089546 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:39.076074 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:39.078515 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:39.088547 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:39.089269 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:44.077978 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:44.080151 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:44.089975 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:44.090599 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:49.076627 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:49.078909 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:49.090332 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:49.091271 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:54.080434 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:54.087631 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:54.095371 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:54.095966 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:19:59.077926 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:19:59.081217 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:19:59.092384 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:19:59.093053 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:04.076589 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:04.078481 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:04.086335 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:04.086902 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:09.079147 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:09.081670 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:09.089899 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:09.090478 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:14.077602 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:14.079998 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:14.088292 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:14.088942 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:19.076664 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:19.078578 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:19.086757 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:19.087768 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:24.081537 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:24.083735 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:24.092394 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:24.093054 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:29.076039 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:29.077919 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:29.086924 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:29.087522 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:34.078435 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:34.081461 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:34.092771 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:34.093596 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:39.081760 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:39.083910 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:39.091971 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:39.092535 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:44.080100 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:44.082590 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:44.094254 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:44.094830 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:49.076009 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:49.077854 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:49.085386 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:49.085976 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:54.077505 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:54.080289 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:54.088851 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:54.089471 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:20:59.080116 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:20:59.082001 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:20:59.089961 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:20:59.090538 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:04.079364 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:04.082290 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:04.092380 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:04.093012 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:09.081195 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:09.083365 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:09.091741 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:09.092341 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:14.078184 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:14.080116 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:14.121035 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:14.121873 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:19.078434 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:19.080781 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:19.088931 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:19.089540 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:24.160536 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:24.162875 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:24.216603 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:24.217492 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:29.077927 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:29.080375 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:29.088182 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:29.088828 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:34.080831 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:34.082888 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:34.095286 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:34.095949 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:39.078550 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:39.082268 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:39.090617 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:39.091201 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:44.078412 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:44.081118 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:44.090138 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:44.090750 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:49.077692 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:49.079751 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:49.088792 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:49.089429 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:54.079960 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:54.081949 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:54.090447 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:54.095740 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:21:59.076021 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:21:59.078124 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:21:59.085884 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:21:59.086477 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:04.077914 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:04.080600 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:04.089785 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:04.090362 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:09.078255 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:09.081771 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:09.091327 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:09.091883 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:14.080219 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:14.082032 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:14.090605 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:14.091172 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:19.079964 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:19.082266 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:19.093396 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:19.093978 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:24.080485 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:24.083096 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:24.091446 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:24.092001 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:29.081573 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:29.083901 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:29.093152 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:29.093993 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:34.078120 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:34.081189 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:34.089143 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:34.089770 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:39.078367 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:39.081391 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:39.090253 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:39.090825 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:44.077349 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:44.079323 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:44.088542 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:44.089250 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:49.079148 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:49.081973 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:49.091287 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:49.092105 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:54.080896 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:54.083387 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:54.093250 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:54.093898 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:22:59.078629 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:22:59.080569 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:22:59.088250 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:22:59.088900 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:04.076155 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:04.078112 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:04.088981 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:04.089638 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:09.079753 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:09.081743 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:09.091429 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:09.092004 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:14.079688 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:14.082003 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:14.092734 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:14.093460 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:19.078089 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:19.080409 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:19.088573 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:19.089183 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:34.254667 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:34.257081 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:34.272509 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:34.274667 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:34.282217 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:34.284320 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:34.305898 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:34.307124 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:34.313249 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:34.313853 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:34.342853 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:34.343467 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:39.078152 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:39.080585 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:39.090086 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:39.090654 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:44.078905 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:44.080881 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:44.089418 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:44.089991 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:49.078006 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:49.081071 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:49.092986 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:49.093599 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:54.077957 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:54.080400 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:54.088527 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:54.089175 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:23:59.079187 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:23:59.086192 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:23:59.094078 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:23:59.094892 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:04.080881 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:04.083961 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:04.093518 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:04.094104 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:09.077749 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:09.080268 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:09.089351 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:09.089942 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:14.078030 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:14.080943 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:14.089520 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:14.090121 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:19.079094 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:19.081769 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:19.090728 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:19.091363 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:24.077052 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:24.079068 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:24.087926 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:24.088677 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:29.078501 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:29.081798 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:29.089811 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:29.090476 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:34.080114 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:34.082240 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:34.093011 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:34.093589 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:39.078139 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:39.080215 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:39.088169 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:39.088761 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:44.078141 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:44.080999 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:44.090483 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:44.091089 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:49.076025 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:49.077924 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:49.085370 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:49.085982 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:54.076156 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:54.078037 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:54.085939 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:54.086561 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:24:59.078028 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:24:59.081000 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:24:59.089352 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:24:59.089930 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:04.078112 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:04.080223 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:04.087706 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:25:04.088340 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:09.078401 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:09.081435 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:09.090257 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:25:09.090883 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:14.081712 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:14.083934 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:14.092347 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:25:14.093087 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:19.077980 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:19.080582 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:19.093518 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:25:19.094439 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:24.075939 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:24.077744 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:24.086238 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:25:24.086821 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:29.080886 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:29.082929 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:29.091203 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:25:29.091770 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:34.080288 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:34.082230 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:34.091486 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:25:34.092045 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:39.078121 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:39.080437 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:39.087702 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:25:39.088300 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:44.078327 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:44.080517 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:44.092880 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:25:44.093501 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:49.078805 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:49.080698 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:49.090667 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:25:49.091341 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:54.078342 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:54.080577 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:54.088374 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:25:54.088978 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:25:59.079268 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:25:59.082281 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:25:59.090790 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:04.078155 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:04.099257 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:04.101560 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:04.117022 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:04.117946 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:09.078412 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:09.081038 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:09.091668 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:09.092257 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:14.080519 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:14.082620 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:14.090569 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:14.091177 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:19.078434 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:19.080932 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:19.089452 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:19.090079 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:24.076056 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:24.078017 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:24.089291 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:24.090128 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:29.078307 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:29.080480 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:29.088712 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:29.089373 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:34.081417 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:34.083828 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:34.092054 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:34.092652 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:39.081111 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:39.083590 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:39.091931 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:39.092497 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:44.078619 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:44.082134 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:44.091171 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:44.091753 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:49.076001 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:49.077977 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:49.089017 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:49.089600 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:54.076111 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:54.078359 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:54.087101 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:54.087821 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:26:59.078206 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:26:59.080682 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:26:59.090393 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:26:59.090971 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:04.078014 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:04.080986 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:04.089345 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:04.089961 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:09.079754 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:09.082452 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:09.090661 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:09.091259 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:14.077505 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:14.079345 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:14.088808 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:14.089432 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:19.078182 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:19.080775 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:19.089040 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:19.089653 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:24.080709 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:24.082761 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:24.091220 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:24.091802 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:29.078416 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:29.081258 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:29.089286 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:29.089872 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:34.079451 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:34.081432 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:34.092242 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:34.092814 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:39.076088 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:39.077940 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:39.086070 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:39.086649 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:44.076687 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:44.080372 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:44.089979 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:44.090699 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:46.637704 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler starting
I0211 09:27:46.641134 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/GetRun handler finished
I0211 09:27:46.741807 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:46.745570 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:46.755350 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:46.755984 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:49.077563 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:49.079282 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:49.088707 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:49.089388 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:54.075925 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:54.077978 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:54.085450 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:54.086022 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:27:59.076669 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:27:59.078603 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:27:59.086296 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:27:59.086921 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:28:02.087884 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:28:02.090673 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:28:02.107352 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:28:02.108065 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:28:04.079474 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:28:04.081760 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:28:04.092352 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:28:04.093283 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:28:09.076692 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:28:09.084834 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:28:09.096212 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:28:09.096830 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:28:14.078832 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:28:14.080663 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:28:14.088611 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:28:14.089214 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:28:19.077418 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:28:19.079160 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:28:19.088054 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:28:19.088648 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:28:24.080197 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:28:24.082981 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:28:24.090895 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:28:24.091521 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:28:29.077899 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:28:29.081170 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:28:29.089189 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:28:29.089784 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:28:34.078302 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:28:34.080983 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:28:34.089800 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:28:34.090394 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:28:39.078068 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:28:39.081061 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:28:39.090570 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:28:39.091334 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 09:28:44.078112 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 09:28:44.080782 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 09:28:44.088808 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 09:28:44.089429 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
### try Creating run the third time for the same pipeline, this time is ok, kubectl get workflow --all-namespaces --> displayed the created workflow.
I0211 10:11:11.566976 7 interceptor.go:29] /api.RunService/CreateRunV1 handler starting
I0211 10:11:14.076120 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:11:14.686817 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:11:16.154164 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:11:16.164032 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:11:16.164836 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:11:16.184592 7 interceptor.go:37] /api.RunService/CreateRunV1 handler finished
I0211 10:11:16.248378 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.RunService/ListRuns handler starting
I0211 10:11:16.264149 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.RunService/ListRuns handler finished
I0211 10:11:16.294409 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler starting
I0211 10:11:16.310108 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler finished
I0211 10:11:16.327412 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler starting
I0211 10:11:16.351689 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler finished
I0211 10:11:16.361945 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler starting
I0211 10:11:16.363782 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler finished
I0211 10:11:16.366676 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler starting
I0211 10:11:16.379811 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion handler finished
I0211 10:11:16.380029 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler starting
I0211 10:11:16.380839 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler finished
I0211 10:11:16.433660 7 interceptor.go:29] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler starting
I0211 10:11:16.434325 7 interceptor.go:37] /kubeflow.pipelines.backend.api.v2beta1.ExperimentService/GetExperiment handler finished
I0211 10:11:19.076253 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:11:19.078401 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:11:19.088458 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:11:19.089180 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:11:24.076004 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:11:25.037416 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:11:25.045634 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:11:25.046240 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:11:25.109744 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:11:27.748692 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:11:29.077857 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:11:32.463652 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:11:32.475512 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:11:32.476094 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:11:34.076256 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:11:34.078897 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:11:34.088342 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:11:34.088991 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:11:36.752264 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:11:38.010360 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:11:39.077175 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:11:39.079098 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:11:39.089151 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:11:39.090006 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:11:43.301155 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:11:44.079302 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:11:44.081410 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:11:44.091410 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:11:44.092183 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:11:47.214882 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:11:47.771313 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:11:49.076152 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:11:49.078194 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:11:49.087567 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:11:49.088211 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:11:54.076029 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:11:54.077894 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:11:54.086238 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:11:54.086814 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:11:59.078080 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:11:59.080049 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:11:59.088262 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:11:59.088869 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:04.077680 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:04.081034 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:04.090859 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:04.091520 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:09.077691 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:09.079978 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:09.091556 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:09.092168 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:14.078370 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:14.081387 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:14.089764 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:14.090359 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:14.805740 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:12:14.833269 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:12:19.077680 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:19.081120 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:19.098330 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:19.099364 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:24.078111 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:24.081257 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:24.090592 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:24.091229 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:29.078866 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:29.081221 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:29.090080 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:29.090660 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:34.078233 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:34.081412 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:34.089672 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:34.090277 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:39.081936 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:39.084203 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:39.092693 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:39.093298 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:44.075838 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:44.077751 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:44.086653 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:44.087221 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:45.015446 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:12:45.042340 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:12:49.081713 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:49.083875 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:49.096209 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:49.096846 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:54.078060 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:54.080557 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:54.088457 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:54.089116 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:12:59.076015 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:12:59.078039 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:12:59.088268 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:12:59.088873 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:04.078474 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:04.080564 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:04.088463 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:04.089144 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:09.078395 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:09.080390 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:09.089525 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:09.090099 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:14.076178 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:14.078162 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:14.085902 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:14.086545 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:15.457424 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:13:17.175400 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:13:19.077962 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:19.079963 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:19.090474 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:19.091146 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:24.078563 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:24.080964 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:24.109181 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:24.110010 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:29.077567 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:29.079427 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:29.087834 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:29.091863 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:34.078906 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:34.082132 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:34.091880 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:34.092488 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:39.080388 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:39.082205 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:39.090330 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:39.090967 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:44.078591 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:44.081812 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:44.089992 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:44.090580 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:46.619422 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:13:46.659869 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:13:49.077748 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:49.080470 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:49.089512 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:49.090126 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:54.078396 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:54.080873 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:54.089526 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:54.090151 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:13:59.081890 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:13:59.084473 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:13:59.092560 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:13:59.093211 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:04.077688 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:04.079745 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:04.087723 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:04.088300 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:09.077880 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:09.080890 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:09.089866 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:09.090462 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:14.048434 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:14:14.074624 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:14:14.076159 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:14.078277 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:14.088679 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:14.089393 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:19.077678 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:19.080340 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:19.089127 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:19.089700 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:24.076815 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:24.079015 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:24.088265 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:24.088865 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:29.079581 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:29.082284 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:29.103264 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:29.104076 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:34.076012 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:34.077942 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:34.085804 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:34.086379 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:39.076131 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:39.078136 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:39.085741 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:39.086410 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:44.078054 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:44.080653 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:44.089037 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:44.089678 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:46.113544 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:14:46.141136 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:14:49.076158 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:49.079044 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:49.090050 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:49.090658 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:54.078788 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:54.080736 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:54.091305 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:54.091901 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:14:59.079096 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:14:59.082256 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:14:59.090582 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:14:59.091224 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:04.082416 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:04.084890 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:04.094350 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:04.094960 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:09.080548 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:09.083070 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:09.092224 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:09.092817 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:14.079076 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:14.082340 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:14.091154 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:14.091740 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:17.369617 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:15:17.396388 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:15:19.077699 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:19.080060 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:19.087682 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:19.088303 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:24.077544 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:24.079412 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:24.087730 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:24.088303 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:29.077550 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:29.080663 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:29.092614 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:29.096837 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:34.080836 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:34.083064 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:34.094738 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:34.095382 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:39.080229 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:39.082095 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:39.090951 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:39.091615 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:44.078199 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:44.080229 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:44.088666 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:44.089301 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:44.522820 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 10:15:44.546582 7 interceptor.go:37] /api.ReportService/ReportWorkflowV1 handler finished
I0211 10:15:49.076451 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:49.078653 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:49.086171 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:49.086805 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:54.078285 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:54.081575 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:54.091184 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:54.091827 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
I0211 10:15:59.079810 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
I0211 10:15:59.081803 7 interceptor.go:37] /api.RunService/GetRunV1 handler finished
I0211 10:15:59.090088 7 interceptor.go:29] /api.ExperimentService/GetExperimentV1 handler starting
I0211 10:15:59.090701 7 interceptor.go:37] /api.ExperimentService/GetExperimentV1 handler finished
(base) maye@maye-Inspiron-5547:~$
- see log of pod workflow-conroller:
time="2024-02-11T08:46:27.537Z" level=info msg="Update leases 200"
time="2024-02-11T08:46:32.542Z" level=info msg="Get leases 200"
time="2024-02-11T08:46:32.605Z" level=info msg="Update leases 200"
time="2024-02-11T08:46:37.608Z" level=info msg="Get leases 200"
time="2024-02-11T08:46:37.629Z" level=info msg="Update leases 200"
time="2024-02-11T08:46:42.634Z" level=info msg="Get leases 200"
time="2024-02-11T08:46:42.652Z" level=info msg="Update leases 200"
### Preprocessing the second created run:
time="2024-02-11T08:46:44.833Z" level=info msg="Processing workflow" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf "
time="2024-02-11T08:46:44.833Z" level=info msg="Processing workflow" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:44.915Z" level=info msg="Get configmaps 404"
time="2024-02-11T08:46:44.916Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2024-02-11T08:46:44.916Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2024-02-11T08:46:44.916Z" level=info msg="Updated phase -> Running" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:44.916Z" level=info msg="DAG node detect-anomolies-on-wafer-tfdv-schema-pnpjf initialized Running" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:44.917Z" level=info msg="All of node detect-anomolies-on-wafer-tfdv-schema-pnpjf.importexamplegen dependencies [] completed" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:44.918Z" level=info msg="Pod node detect-anomolies-on-wafer-tfdv-schema-pnpjf-587264735 initialized Pending" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:44.985Z" level=info msg="Create events 201"
time="2024-02-11T08:46:45.099Z" level=info msg="Create pods 201"
W0211 08:46:45.100481 1 warnings.go:70] spec.containers[1].env[4].name: duplicate name "WORKFLOW_ID"
time="2024-02-11T08:46:45.102Z" level=info msg="Created pod: detect-anomolies-on-wafer-tfdv-schema-pnpjf.importexamplegen (detect-anomolies-on-wafer-tfdv-schema-pnpjf-587264735)" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.103Z" level=info msg="All of node detect-anomolies-on-wafer-tfdv-schema-pnpjf.schema-importer dependencies [] completed" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.104Z" level=info msg="Pod node detect-anomolies-on-wafer-tfdv-schema-pnpjf-28753274 initialized Pending" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.864Z" level=info msg="Create pods 201"
W0211 08:46:45.865156 1 warnings.go:70] spec.containers[1].env[4].name: duplicate name "WORKFLOW_ID"
time="2024-02-11T08:46:45.866Z" level=info msg="Created pod: detect-anomolies-on-wafer-tfdv-schema-pnpjf.schema-importer (detect-anomolies-on-wafer-tfdv-schema-pnpjf-28753274)" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.866Z" level=info msg="TaskSet Reconciliation" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.866Z" level=info msg=reconcileAgentPod namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.867Z" level=info msg="Workflow to be dehydrated" Workflow Size=45369
time="2024-02-11T08:46:46.122Z" level=info msg="Update workflows 200"
time="2024-02-11T08:46:46.125Z" level=info msg="Workflow update successful" namespace=kubeflow phase=Running resourceVersion=235624 workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:46.239Z" level=info msg="Create events 201"
time="2024-02-11T08:46:47.656Z" level=info msg="Get leases 200"
time="2024-02-11T08:46:47.676Z" level=info msg="Update leases 200"
time="2024-02-11T08:46:52.682Z" level=info msg="Get leases 200"
time="2024-02-11T08:46:52.889Z" level=info msg="Update leases 200"
time="2024-02-11T08:46:57.893Z" level=info msg="Get leases 200"
time="2024-02-11T08:46:57.913Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:02.916Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:02.997Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:08.000Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:08.014Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:13.018Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:13.060Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:14.155Z" level=info msg="List workflows 200"
time="2024-02-11T08:47:14.156Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=kubeflow
time="2024-02-11T08:47:18.063Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:18.083Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:23.088Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:23.118Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:28.121Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:28.231Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:33.237Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:33.275Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:38.279Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:38.287Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:40.083Z" level=info msg="Watch configmaps 200"
time="2024-02-11T08:47:43.294Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:43.344Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:48.348Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:48.398Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:53.404Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:53.424Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:58.427Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:58.447Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:03.452Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:03.482Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:08.486Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:08.505Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:13.117Z" level=info msg="Watch workflows 200"
time="2024-02-11T08:48:13.512Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:13.583Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:14.159Z" level=info msg="List workflows 200"
time="2024-02-11T08:48:14.159Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=kubeflow
time="2024-02-11T08:48:18.601Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:18.618Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:23.630Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:23.641Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:25.100Z" level=info msg="Watch configmaps 200"
time="2024-02-11T08:48:28.645Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:28.702Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:33.708Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:33.732Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:38.737Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:38.755Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:43.761Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:43.778Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:48.782Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:48.813Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:53.816Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:53.870Z" level=info msg="Update leases 200"
time="2024-02-11T08:48:58.873Z" level=info msg="Get leases 200"
time="2024-02-11T08:48:58.926Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:03.937Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:03.987Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:08.993Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:09.039Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:14.044Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:14.062Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:14.109Z" level=info msg="Watch cronworkflows 200"
time="2024-02-11T08:49:14.160Z" level=info msg="List workflows 200"
time="2024-02-11T08:49:14.160Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=kubeflow
time="2024-02-11T08:49:19.067Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:19.085Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:24.102Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:24.167Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:29.176Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:29.214Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:34.218Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:34.288Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:35.089Z" level=info msg="Alloc=6422 TotalAlloc=964006 Sys=24657 NumGC=775 Goroutines=163"
time="2024-02-11T08:49:39.292Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:39.311Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:44.317Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:44.379Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:49.382Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:49.413Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:54.420Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:54.473Z" level=info msg="Update leases 200"
time="2024-02-11T08:49:59.477Z" level=info msg="Get leases 200"
time="2024-02-11T08:49:59.503Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:04.507Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:04.549Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:09.554Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:09.583Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:14.158Z" level=info msg="List workflows 200"
time="2024-02-11T08:50:14.158Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=kubeflow
time="2024-02-11T08:50:14.591Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:14.629Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:19.632Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:19.683Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:21.114Z" level=info msg="Watch workflows 200"
time="2024-02-11T08:50:24.688Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:24.730Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:29.734Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:29.742Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:34.747Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:34.765Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:39.769Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:39.800Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:44.804Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:44.845Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:49.847Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:49.912Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:54.919Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:54.946Z" level=info msg="Update leases 200"
time="2024-02-11T08:50:59.951Z" level=info msg="Get leases 200"
time="2024-02-11T08:50:59.970Z" level=info msg="Update leases 200"
time="2024-02-11T08:51:04.976Z" level=info msg="Get leases 200"
time="2024-02-11T08:51:05.037Z" level=info msg="Update leases 200"
time="2024-02-11T08:51:10.075Z" level=info msg="Get leases 200"
time="2024-02-11T08:51:10.121Z" level=info msg="Update leases 200"
time="2024-02-11T08:51:14.165Z" level=info msg="List workflows 200"
time="2024-02-11T08:51:14.165Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=kubeflow
time="2024-02-11T08:51:15.127Z" level=info msg="Get leases 200"
time="2024-02-11T08:51:15.150Z" level=info msg="Update leases 200"
time="2024-02-11T08:51:17.074Z" level=info msg="Watch workflowtemplates 200"
time="2024-02-11T08:51:20.153Z" level=info msg="Get leases 200"
time="2024-02-11T08:51:20.162Z" level=info msg="Update leases 200"
time="2024-02-11T08:51:25.166Z" level=info msg="Get leases 200"
time="2024-02-11T08:51:25.240Z" level=info msg="Update leases 200"
time="2024-02-11T08:51:30.245Z" level=info msg="Get leases 200"
time="2024-02-11T08:51:30.275Z" level=info msg="Update leases 200"
time="2024-02-11T08:51:35.279Z" level=info msg="Get leases 200"
time="2024-02-11T08:51:35.320Z" level=info msg="Update leases 200"
time="2024-02-11T08:51:40.324Z" level=info msg="Get leases 200"
time="2024-02-11T08:51:40.365Z" level=info msg="Update leases 200"
time="2024-02-11T08:51:45.373Z" level=info msg="Get leases 200"
time="2024-02-11T08:51:45.421Z" level=info msg="Update leases 200"
time="2024-02-11T08:51:50.114Z" level=info msg="Watch workflowtaskresults 200"
time="2024-02-11T08:51:50.426Z" level=info msg="Get leases 200"
time="2024-02-11T08:51:50.445Z" level=info msg="Update leases 200"
time="2024-02-11T08:51:54.132Z" level=info msg="Watch pods 200"
time="2024-02-11T08:51:55.451Z" level=info msg="Get leases 200"
time="2024-02-11T08:51:55.501Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:00.505Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:00.591Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:05.597Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:05.636Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:10.640Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:10.671Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:14.157Z" level=info msg="List workflows 200"
time="2024-02-11T08:52:14.158Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=kubeflow
time="2024-02-11T08:52:15.677Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:15.704Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:20.707Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:20.749Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:25.069Z" level=info msg="Watch workflowtasksets 200"
time="2024-02-11T08:52:25.895Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:25.938Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:30.943Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:30.962Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:35.968Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:36.007Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:41.012Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:41.030Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:46.036Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:46.075Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:51.080Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:51.110Z" level=info msg="Update leases 200"
time="2024-02-11T08:52:56.116Z" level=info msg="Get leases 200"
time="2024-02-11T08:52:56.144Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:01.147Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:01.167Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:06.173Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:06.201Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:11.227Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:11.272Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:14.157Z" level=info msg="List workflows 200"
time="2024-02-11T08:53:14.158Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=kubeflow
time="2024-02-11T08:53:16.278Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:16.302Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:21.307Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:21.448Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:26.452Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:26.495Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:31.499Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:31.517Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:36.524Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:36.619Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:41.623Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:41.664Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:46.671Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:46.698Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:51.703Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:51.732Z" level=info msg="Update leases 200"
time="2024-02-11T08:53:56.739Z" level=info msg="Get leases 200"
time="2024-02-11T08:53:56.788Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:01.792Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:01.844Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:06.849Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:06.893Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:11.896Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:12.068Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:14.158Z" level=info msg="List workflows 200"
time="2024-02-11T08:54:14.159Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=kubeflow
time="2024-02-11T08:54:17.125Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:17.158Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:22.161Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:22.181Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:27.186Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:27.241Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:32.246Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:32.271Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:35.089Z" level=info msg="Alloc=6546 TotalAlloc=966948 Sys=24657 NumGC=777 Goroutines=163"
time="2024-02-11T08:54:37.277Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:37.305Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:42.338Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:42.361Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:47.367Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:47.406Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:52.418Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:52.476Z" level=info msg="Update leases 200"
time="2024-02-11T08:54:57.481Z" level=info msg="Get leases 200"
time="2024-02-11T08:54:57.530Z" level=info msg="Update leases 200"
time="2024-02-11T08:55:02.534Z" level=info msg="Get leases 200"
time="2024-02-11T08:55:02.575Z" level=info msg="Update leases 200"
time="2024-02-11T08:55:07.582Z" level=info msg="Get leases 200"
time="2024-02-11T08:55:07.636Z" level=info msg="Update leases 200"
time="2024-02-11T08:55:12.641Z" level=info msg="Get leases 200"
time="2024-02-11T08:55:12.665Z" level=info msg="Update leases 200"
time="2024-02-11T08:55:14.158Z" level=info msg="List workflows 200"
time="2024-02-11T08:55:14.158Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=kubeflow
time="2024-02-11T08:55:17.670Z" level=info msg="Get leases 200"
time="2024-02-11T08:55:17.699Z" level=info msg="Update leases 200"
time="2024-02-11T08:55:22.702Z" level=info msg="Get leases 200"
time="2024-02-11T08:55:22.722Z" level=info msg="Update leases 200"
time="2024-02-11T08:55:27.756Z" level=info msg="Get leases 200"
time="2024-02-11T08:55:27.814Z" level=info msg="Update leases 200"
time="2024-02-11T08:55:32.816Z" level=info msg="Get leases 200"
time="2024-02-11T08:55:32.835Z" level=info msg="Update leases 200"
time="2024-02-11T08:55:37.879Z" level=info msg="Get leases 200"
time="2024-02-11T08:55:37.902Z" level=info msg="Update leases 200"
time="2024-02-11T08:55:42.904Z" level=info msg="Get leases 200"
time="2024-02-11T08:55:42.991Z" level=info msg="Update leases 200"
### Preprocessing the third create run, which is ok:
time="2024-02-11T10:11:21.910Z" level=info msg="Processing workflow" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-hq49w
time="2024-02-11T10:11:21.910Z" level=info msg="Processing workflow" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-hq49w
time="2024-02-11T10:11:22.070Z" level=info msg="Task-result reconciliation" namespace=kubeflow numObjs=0 workflow=detect-anomolies-on-wafer-tfdv-schema-hq49w
time="2024-02-11T10:11:22.070Z" level=info msg="node changed" new.message="Unschedulable: 0/2 nodes are available: persistentvolumeclaim \"tfx-pv-claim\" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.." new.phase=Pending new.progress=0/1 nodeID=detect-anomolies-on-wafer-tfdv-schema-hq49w-2248169053 old.message= old.phase=Pending old.progress=0/1
time="2024-02-11T10:11:22.070Z" level=info msg="node changed" new.message="Unschedulable: 0/2 nodes are available: persistentvolumeclaim \"tfx-pv-claim\" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.." new.phase=Pending new.progress=0/1 nodeID=detect-anomolies-on-wafer-tfdv-schema-hq49w-218673478 old.message= old.phase=Pending old.progress=0/1
time="2024-02-11T10:11:22.073Z" level=info msg="TaskSet Reconciliation" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-hq49w
time="2024-02-11T10:11:22.073Z" level=info msg=reconcileAgentPod namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-hq49w
time="2024-02-11T10:11:22.073Z" level=info msg="Workflow to be dehydrated" Workflow Size=45948
time="2024-02-11T10:11:22.177Z" level=info msg="Update workflows 200"
time="2024-02-11T10:11:22.181Z" level=info msg="Workflow update successful" namespace=kubeflow phase=Running resourceVersion=243632 workflow=detect-anomolies-on-wafer-tfdv-schema-hq49w
time="2024-02-11T10:11:26.666Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:26.931Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:31.935Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:31.979Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:36.983Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:36.997Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:42.000Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:42.083Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:45.138Z" level=info msg="Watch workflows 200"
time="2024-02-11T10:11:47.087Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:47.131Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:52.188Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:52.198Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:56.125Z" level=info msg="Watch configmaps 200"
time="2024-02-11T10:11:57.202Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:57.232Z" level=info msg="Update leases 200"
time="2024-02-11T10:12:02.235Z" level=info msg="Get leases 200"
time="2024-02-11T10:12:02.288Z" level=info msg="Update leases 200"
time="2024-02-11T10:12:07.294Z" level=info msg="Get leases 200"
time="2024-02-11T10:12:07.383Z" level=info msg="Update leases 200"
time="2024-02-11T10:12:12.386Z" level=info msg="Get leases 200"
time="2024-02-11T10:12:12.529Z" level=info msg="Update leases 200"
time="2024-02-11T10:12:14.161Z" level=info msg="List workflows 200"
time="2024-02-11T10:12:14.161Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=kubeflow
time="2024-02-11T10:12:17.533Z" level=info msg="Get leases 200"
time="2024-02-11T10:12:17.556Z" level=info msg="Update leases 200"
(base) maye@maye-Inspiron-5547:~$
- see log of pod metadata-grpc-deployment:
(base) maye@maye-Inspiron-5547:~$ kubectl logs metadata-grpc-deployment-659594dfcb-g9jfl -n kubeflow
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0210 14:36:16.449616 1 metadata_store_server_main.cc:577] Server listening on 0.0.0.0:8080
E0211 08:06:44.637343 13 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:06:44.637894 13 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:06:49.608660 11 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:06:49.608848 11 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:09:00.206667 17 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:09:00.206913 17 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:10:25.120438 20 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:10:25.120702 20 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:10:30.112478 21 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:10:30.112686 21 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:11:10.053038 16 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:11:10.053236 16 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:11:15.044875 25 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:11:15.045130 25 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:11:25.109097 27 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:11:25.109256 27 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:13:40.919914 38 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:13:40.920224 38 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:14:15.845067 42 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:14:15.845237 42 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:17:24.143090 48 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:17:24.143290 48 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:17:43.945977 48 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:17:43.946336 48 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:17:44.086115 47 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:17:44.086366 47 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:17:53.927296 49 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:17:53.927531 49 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:18:54.158545 52 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:18:54.159018 52 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:18:59.182435 49 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:18:59.182756 49 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:19:09.033064 54 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:19:09.033257 54 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:20:04.018133 61 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:20:04.018342 61 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:20:36.170630 64 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:20:36.170981 64 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:21:19.007383 69 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:21:19.007689 69 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:25:48.097622 55 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:25:48.097944 55 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:31:04.045995 55 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:31:04.046341 55 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:31:34.012112 55 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:31:34.012413 55 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:37:30.496155 55 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:37:30.496717 55 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:45:02.423204 72 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:45:02.423542 72 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
### mysql can not be connected, at almost same time with ml-pipeline CreateRun->ReportWorkflow Error:
E0211 08:46:44.117887 73 mysql_metadata_source.cc:174] MySQL database was not initialized.
I0211 08:46:44.696273 7 interceptor.go:29] /api.RunService/CreateRunV1 handler starting
I0211 08:46:47.084088 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 08:46:49.167581 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
log of pod ml-pipeline:
E0211 08:46:51.546205 7 resource_manager.go:1150] Cannot find reported workflow name="detect-anomolies-on-wafer-tfdv-schema-pnpjf" namespace="kubeflow" runId="c6013cee-f684-4524-a2b5-719f134e46af" in run store. Deleting the workflow to avoid resource leaking. This can be caused by installing two KFP instances that try to manage the same workflows or an unknown bug. If you encounter this, recommend reporting more details in https://github.com/kubeflow/pipelines/issues/6189
E0211 08:46:44.117887 73 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:46:44.118113 73 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:46:49.366025 71 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:46:49.366206 71 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:47:06.305011 76 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:47:06.305274 76 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:47:34.486318 77 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:47:34.486477 77 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:48:31.376595 82 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:48:31.377065 82 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:49:13.297348 86 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:49:13.297716 86 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:49:36.340270 87 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:49:36.340474 87 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:49:54.107439 90 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:49:54.107637 90 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:50:51.305665 95 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:50:51.305857 95 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:51:44.193018 97 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:51:44.193215 97 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:52:04.176846 98 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:52:04.177146 98 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:52:14.117028 96 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:52:14.117452 96 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:52:14.161477 99 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:52:14.161850 99 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:52:49.210477 104 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:52:49.210666 104 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:53:49.111207 109 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:53:49.111433 109 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:53:49.205780 111 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:53:49.206022 111 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:53:54.217880 108 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:53:54.218108 108 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:55:19.182169 116 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:55:19.182380 116 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:55:34.221475 118 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:55:34.221766 118 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:57:19.214854 127 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:57:19.215050 127 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:57:29.107112 130 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:57:29.107344 130 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:58:14.124778 134 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:58:14.124981 134 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:58:29.197607 137 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:58:29.197846 137 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:58:34.128255 134 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:58:34.128471 134 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:58:34.183866 138 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:58:34.184051 138 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:58:59.236608 134 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:58:59.236991 134 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:59:36.455128 141 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:59:36.455446 141 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:59:39.206697 134 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:59:39.207163 134 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 08:59:59.108808 146 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 08:59:59.109122 146 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 09:00:44.189244 154 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 09:00:44.189496 154 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 09:01:31.146998 163 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 09:01:31.147287 163 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0211 09:02:44.193024 166 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0211 09:02:44.193275 166 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
(base) maye@maye-Inspiron-5547:~$
- see if the created run is recored in database:
Yes, the created runs are recorded in database "mlpipeline" table "run_details" .
# enter mysql pod
$ kubectl exec -it mysql-pod-cluster-ip -- bash
mysql> use mlpipeline;
mysql> SELECT * FROM run_details;
| 1548e025-3d74-4036-a03f-ef79945f9e00 | Run of maye wafer pipeline (bc769) | detect-anomolies-on-wafer-tfdv-schema-qxpkk | | | 51f06a9b-c05e-43ad-aaa9-9415406c3dcc | | AVAILABLE | pipeline-runner | 6a41cad2-291f-4135-99f0-45cccdeecd17 | 9a1ec59a-c868-4a34-a49d-a61d1a944ab0 | maye wafer pipeline | | {"kind":"Workflow","apiVersion":"argoproj.io/v1alpha1","metadata":{"generateName":"detect-anomolies-on-wafer-tfdv-schema-","creationTimestamp":null,"labels":{"pipelines.kubeflow.org/kfp_sdk_version":"1.8.0"},"annotations":{"pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline_compilation_time":"2024-01-07T22:16:36.438482","pipelines.kubeflow.org/pipeline_spec":"{\"description\": \"Constructs a Kubeflow pipeline.\", \"inputs\": [{\"default\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\", \"name\": \"pipeline-root\"}], \"name\": \"detect_anomolies_on_wafer_tfdv_schema\"}"}},"spec":{"templates":[{"name":"detect-anomolies-on-wafer-tfdv-schema","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{},"metadata":{},"dag":{"tasks":[{"name":"importexamplegen","template":"importexamplegen","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]}},{"name":"pusher","template":"pusher","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["trainer"]},{"name":"schema-importer","template":"schema-importer","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]}},{"name":"statisticsgen","template":"statisticsgen","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["importexamplegen"]},{"name":"trainer","template":"trainer","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"depenmysql>dencies":["importexamplegen","transform"]},{"name":"transform","template":"transform","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["importexamplegen","schema-importer"]}]}},{"name":"importexamplegen","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","ImportExampleGen","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.example_gen.import_example_gen.component.ImportExampleGen\"\n },\n \"id\": \"ImportExampleGen\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ]\n },\n \"outputs\": {\n \"outputs\": {\n \"examples\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Examples\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\",\n \"version\": \"INT\"\n },\n \"baseType\": \"DATASET\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"output_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{}\"\n }\n },\n \"input_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"splits\\\": [\\n {\\n \\\"name\\\": \\\"train\\\",\\n \\\"pattern\\\": \\\"train\\\"\\n },\\n {\\n \\\"name\\\": \\\"eval\\\",\\n \\\"pattern\\\": \\\"eval\\\"\\n }\\n ]\\n}\"\n }\n },\n \"output_data_format\": {\n \"fieldValue\": {\n \"intValue\": \"6\"\n }\n },\n \"input_base\": {\n \"fieldValue\": {\n \"stringValue\": \"/maye/trainEvalData\"\n }\n },\n \"output_file_format\": {\n \"fieldValue\": {\n \"intValue\": \"5\"\n }\n }\n }\n },\n \"downstreamNodes\": [\n \"StatisticsGen\",\n \"Trainer\",\n \"Transform\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"ImportExampleGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.example_gen.import_example_gen.executor.Executor\"\n }\n }\n },\n \"customDriverSpecs\": {\n \"ImportExampleGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.example_gen.driver.FileBasedDriver\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"wafer-data","mountPath":"/maye/trainEvalData"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"pusher","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Pusher","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.pusher.component.Pusher\",\n \"baseType\": \"DEPLOY\"\n },\n \"id\": \"Pusher\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Pusher\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"model\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"Trainer\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Trainer\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Model\",\n \"baseType\": \"MODEL\"\n }\n },\n \"outputKey\": \"model\"\n }\n ]\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"pushed_model\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"PushedModel\",\n \"baseType\": \"MODEL\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"null\"\n }\n },\n \"push_destination\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"filesystem\\\": {\\n \\\"base_directory\\\": \\\"serving_model/detect_anomolies_on_wafer_tfdv\\\"\\n }\\n}\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"Trainer\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Pusher\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.pusher.executor.Executor\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"schema-importer","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","schema_importer","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.dsl.components.common.importer.Importer\"\n },\n \"id\": \"schema_importer\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.schema_importer\"\n }\n }\n }\n ]\n },\n \"outputs\": {\n \"outputs\": {\n \"result\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"artifact_uri\": {\n \"fieldValue\": {\n \"stringValue\": \"/tfx/pipelines/detect_anomalies_in_wafer_schema/\"\n }\n },\n \"reimport\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"output_key\": {\n \"fieldValue\": {\n \"stringValue\": \"result\"\n }\n }\n }\n },\n \"downstreamNodes\": [\n \"Transform\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"},{"name":"schema-path","readOnly":true,"mountPath":"/tfx/pipelines/detect_anomalies_in_wafer_schema"}],"imagePullPolicy":"Never"}},{"name":"statisticsgen","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","StatisticsGen","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.statistics_gen.component.StatisticsGen\",\n \"baseType\": \"PROCESS\"\n },\n \"id\": \"StatisticsGen\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.StatisticsGen\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"statistics\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"span\": \"INT\",\n \"split_names\": \"STRING\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"exclude_splits\": {\n \"fieldValue\": {\n \"stringValue\": \"[]\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"StatisticsGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.statistics_gen.executor.Executor\"\n }\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"trainer","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Trainer","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.trainer.component.Trainer\",\n \"baseType\": \"TRAIN\"\n },\n \"id\": \"Trainer\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Trainer\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n },\n \"transform_graph\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"Transform\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Transform\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"TransformGraph\"\n }\n },\n \"outputKey\": \"transform_graph\"\n }\n ]\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"model\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Model\",\n \"baseType\": \"MODEL\"\n }\n }\n },\n \"model_run\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ModelRun\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"train_args\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"num_steps\\\": 21\\n}\"\n }\n },\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\\"epochs\\\": 50}\"\n }\n },\n \"eval_args\": {\n \"fieldValue\": {\n \"stringValue\": \"{}\"\n }\n },\n \"module_path\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomalies_in_wafer_trainer@/tfx/pipelines/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\",\n \"Transform\"\n ],\n \"downstreamNodes\": [\n \"Pusher\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Trainer\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.trainer.executor.GenericExecutor\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"trainer-module","readOnly":true,"mountPath":"/tfx/pipelines/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"transform","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Transform","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.transform.component.Transform\",\n \"baseType\": \"TRANSFORM\"\n },\n \"id\": \"Transform\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Transform\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"schema\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"schema_importer\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.schema_importer\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n },\n \"outputKey\": \"result\"\n }\n ],\n \"minCount\": 1\n },\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"post_transform_anomalies\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleAnomalies\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n }\n }\n }\n },\n \"updated_analyzer_cache\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"TransformCache\"\n }\n }\n },\n \"transform_graph\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"TransformGraph\"\n }\n }\n },\n \"post_transform_schema\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n },\n \"pre_transform_schema\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n },\n \"post_transform_stats\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n },\n \"pre_transform_stats\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"disable_statistics\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"force_tf_compat_v1\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"module_path\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomalies_in_wafer_trainer@/tfx/pipelines/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl\"\n }\n },\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"null\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\",\n \"schema_importer\"\n ],\n \"downstreamNodes\": [\n \"Trainer\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Transform\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.transform.executor.Executor\"\n }\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"transform-module","readOnly":true,"mountPath":"/tfx/pipelines/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl"},{"name":"schema-path","readOnly":true,"mountPath":"/tfx/pipelines/detect_anomalies_in_wafer_schema"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}}],"entrypoint":"detect-anomolies-on-wafer-tfdv-schema","arguments":{"parameters":[{"name":"pipeline-root","value":"/tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema"}]},"serviceAccountName":"pipeline-runner","volumes":[{"name":"wafer-data","hostPath":{"path":"/home/maye/trainEvalData","type":"Directory"}},{"name":"transform-module","hostPath":{"path":"/home/maye/maye_temp/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl","type":"File"}},{"name":"trainer-module","hostPath":{"path":"/home/maye/maye_temp/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl","type":"File"}},{"name":"schema-path","hostPath":{"path":"/home/maye/maye_temp/detect_anomalies_in_wafer_schema","type":"Directory"}},{"name":"tfx-pv","persistentVolumeClaim":{"claimName":"tfx-pv-claim"}}]},"status":{"startedAt":null,"finishedAt":null}} | [{"name":"pipeline-root","value":"/tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema"}] | | | 1707638699 | 1707638699 | 0 | Terminating | CANCELING | [{"UpdateTimeInSec":1707638699,"State":"PENDING"}] | | {"metadata":{"name":"detect-anomolies-on-wafer-tfdv-schema-qxpkk","generateName":"detect-anomolies-on-wafer-tfdv-schema-","namespace":"kubeflow","uid":"86bd51ab-76b9-4e6e-b62d-8bcbbbb3b0a8","resourceVersion":"231657","generation":1,"creationTimestamp":"2024-02-11T08:04:59Z","labels":{"pipeline/runid":"1548e025-3d74-4036-a03f-ef79945f9e00","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0"},"annotations":{"pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline_compilation_time":"2024-01-07T22:16:36.438482","pipelines.kubeflow.org/pipeline_spec":"{\"description\": \"Constructs a Kubeflow pipeline.\", \"inputs\": [{\"default\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\", \"name\": \"pipeline-root\"}], \"name\": \"detect_anomolies_on_wafer_tfdv_schema\"}","pipelines.kubeflow.org/run_name":"Run of maye wafer pipeline (bc769)"},"managedFields":[{"manager":"apiserver","operation":"Update","apiVersion":"argoproj.io/v1alpha1","time":"2024-02-11T08:04:59Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:pipelines.kubeflow.org/kfp_sdk_version":{},"f:pipelines.kubeflow.org/pipeline_compilation_time":{},"f:pipelines.kubeflow.org/pipeline_spec":{},"f:pipelines.kubeflow.org/run_name":{}},"f:generateName":{},"f:labels":{".":{},"f:pipeline/runid":{},"f:pipelines.kubeflow.org/kfp_sdk_version":{}}},"f:spec":{},"f:status":{}}}]},"spec":{"templates":[{"name":"detect-anomolies-on-wafer-tfdv-schema","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"pipelines.kubeflow.org/cache_enabled":"true"}},"dag":{"tasks":[{"name":"importexamplegen","template":"importexamplegen","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]}},{"name":"pusher","template":"pusher","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["trainer"]},{"name":"schema-importer","template":"schema-importer","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]}},{"name":"statisticsgen","template":"statisticsgen","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["importexamplegen"]}
,{"name":"trainer","template":"trainer","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["importexamplegen","transform"]},{"name":"transform","template":"transform","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["importexamplegen","schema-importer"]}]}},{"name":"importexamplegen","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","ImportExampleGen","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.example_gen.import_example_gen.component.ImportExampleGen\"\n },\n \"id\": \"ImportExampleGen\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ]\n },\n \"outputs\": {\n \"outputs\": {\n \"examples\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Examples\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\",\n \"version\": \"INT\"\n },\n \"baseType\": \"DATASET\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"output_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{}\"\n }\n },\n \"input_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"splits\\\": [\\n {\\n \\\"name\\\": \\\"train\\\",\\n \\\"pattern\\\": \\\"train\\\"\\n },\\n {\\n \\\"name\\\": \\\"eval\\\",\\n \\\"pattern\\\": \\\"eval\\\"\\n }\\n ]\\n}\"\n }\n },\n \"output_data_format\": {\n \"fieldValue\": {\n \"intValue\": \"6\"\n }\n },\n \"input_base\": {\n \"fieldValue\": {\n \"stringValue\": \"/maye/trainEvalData\"\n }\n },\n \"output_file_format\": {\n \"fieldValue\": {\n \"intValue\": \"5\"\n }\n }\n }\n },\n \"downstreamNodes\": [\n \"StatisticsGen\",\n \"Trainer\",\n \"Transform\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"ImportExampleGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.example_gen.import_example_gen.executor.Executor\"\n }\n }\n },\n \"customDriverSpecs\": {\n \"ImportExampleGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.example_gen.driver.FileBasedDriver\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"wafer-data","mountPath":"/maye/trainEvalData"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"pusher","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Pusher","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.pusher.component.Pusher\",\n \"baseType\": \"DEPLOY\"\n },\n \"id\": \"Pusher\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Pusher\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"model\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"Trainer\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Trainer\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Model\",\n \"baseType\": \"MODEL\"\n }\n },\n \"outputKey\": \"model\"\n }\n ]\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"pushed_model\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"PushedModel\",\n \"baseType\": \"MODEL\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"null\"\n }\n },\n \"push_destination\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"filesystem\\\": {\\n \\\"base_directory\\\": \\\"serving_model/detect_anomolies_on_wafer_tfdv\\\"\\n }\\n}\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"Trainer\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Pusher\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.pusher.executor.Executor\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"schema-importer","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","schema_importer","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.dsl.components.common.importer.Importer\"\n },\n \"id\": \"schema_importer\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.schema_importer\"\n }\n }\n }\n ]\n },\n \"outputs\": {\n \"outputs\": {\n \"result\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"artifact_uri\": {\n \"fieldValue\": {\n \"stringValue\": \"/tfx/pipelines/detect_anomalies_in_wafer_schema/\"\n }\n },\n \"reimport\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"output_key\": {\n \"fieldValue\": {\n \"stringValue\": \"result\"\n }\n }\n }\n },\n \"downstreamNodes\": [\n \"Transform\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"},{"name":"schema-path","readOnly":true,"mountPath":"/tfx/pipelines/detect_anomalies_in_wafer_schema"}],"imagePullPolicy":"Never"}},{"name":"statisticsgen","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","StatisticsGen","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.statistics_gen.component.StatisticsGen\",\n \"baseType\": \"PROCESS\"\n },\n \"id\": \"StatisticsGen\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.StatisticsGen\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"statistics\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"span\": \"INT\",\n \"split_names\": \"STRING\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"exclude_splits\": {\n \"fieldValue\": {\n \"stringValue\": \"[]\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"StatisticsGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.statistics_gen.executor.Executor\"\n }\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"trainer","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Trainer","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.trainer.component.Trainer\",\n \"baseType\": \"TRAIN\"\n },\n \"id\": \"Trainer\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Trainer\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n },\n \"transform_graph\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"Transform\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Transform\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"TransformGraph\"\n }\n },\n \"outputKey\": \"transform_graph\"\n }\n ]\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"model\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Model\",\n \"baseType\": \"MODEL\"\n }\n }\n },\n \"model_run\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ModelRun\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"train_args\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"num_steps\\\": 21\\n}\"\n }\n },\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\\"epochs\\\": 50}\"\n }\n },\n \"eval_args\": {\n \"fieldValue\": {\n \"stringValue\": \"{}\"\n }\n },\n \"module_path\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomalies_in_wafer_trainer@/tfx/pipelines/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\",\n \"Transform\"\n ],\n \"downstreamNodes\": [\n \"Pusher\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Trainer\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.trainer.executor.GenericExecutor\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"trainer-module","readOnly":true,"mountPath":"/tfx/pipelines/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"transform","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Transform","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.transform.component.Transform\",\n \"baseType\": \"TRANSFORM\"\n },\n \"id\": \"Transform\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Transform\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"schema\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"schema_importer\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.schema_importer\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n },\n \"outputKey\": \"result\"\n }\n ],\n \"minCount\": 1\n },\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"post_transform_anomalies\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleAnomalies\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n }\n }\n }\n },\n \"updated_analyzer_cache\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"TransformCache\"\n }\n }\n },\n \"transform_graph\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"TransformGraph\"\n }\n }\n },\n \"post_transform_schema\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n },\n \"pre_transform_schema\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n },\n \"post_transform_stats\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n },\n \"pre_transform_stats\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"disable_statistics\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"force_tf_compat_v1\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"module_path\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomalies_in_wafer_trainer@/tfx/pipelines/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl\"\n }\n },\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"null\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\",\n \"schema_importer\"\n ],\n \"downstreamNodes\": [\n \"Trainer\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Transform\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.transform.executor.Executor\"\n }\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"transform-module","readOnly":true,"mountPath":"/tfx/pipelines/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl"},{"name":"schema-path","readOnly":true,"mountPath":"/tfx/pipelines/detect_anomalies_in_wafer_schema"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}}],"entrypoint":"detect-anomolies-on-wafer-tfdv-schema","arguments":{"parameters":[{"name":"pipeline-root","value":"/tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema"}]},"serviceAccountName":"pipeline-runner","volumes":[{"name":"wafer-data","hostPath":{"path":"/home/maye/trainEvalData","type":"Directory"}},{"name":"transform-module","hostPath":{"path":"/home/maye/maye_temp/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl","type":"File"}},{"name":"trainer-module","hostPath":{"path":"/home/maye/maye_temp/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl","type":"File"}},{"name":"schema-path","hostPath":{"path":"/home/maye/maye_temp/detect_anomalies_in_wafer_schema","type":"Directory"}},{"name":"tfx-pv","persistentVolumeClaim":{"claimName":"tfx-pv-claim"}}],"podMetadata":{"labels":{"pipeline/runid":"1548e025-3d74-4036-a03f-ef79945f9e00"}}},"status":{"startedAt":null,"finishedAt":null}} | 0 | 0 |
#### sencond run record:
| c6013cee-f684-4524-a2b5-719f134e46af | Run of maye wafer pipeline (c6d86) | detect-anomolies-on-wafer-tfdv-schema-pnpjf | | | 51f06a9b-c05e-43ad-aaa9-9415406c3dcc | | AVAILABLE | pipeline-runner | 6a41cad2-291f-4135-99f0-45cccdeecd17 | 9a1ec59a-c868-4a34-a49d-a61d1a944ab0 | maye wafer pipeline | | {"kind":"Workflow","apiVersion":"argoproj.io/v1alpha1","metadata":{"generateName":"detect-anomolies-on-wafer-tfdv-schema-","creationTimestamp":null,"labels":{"pipelines.kubeflow.org/kfp_sdk_version":"1.8.0"},"annotations":{"pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline_compilation_time":"2024-01-07T22:16:36.438482","pipelines.kubeflow.org/pipeline_spec":"{\"description\": \"Constructs a Kubeflow pipeline.\", \"inputs\": [{\"default\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\", \"name\": \"pipeline-root\"}], \"name\": \"detect_anomolies_on_wafer_tfdv_schema\"}"}},"spec":{"templates":[{"name":"detect-anomolies-on-wafer-tfdv-schema","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{},"metadata":{},"dag":{"tasks":[{"name":"importexamplegen","template":"importexamplegen","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]}},{"name":"pusher","template":"pusher","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["trainer"]},{"name":"schema-importer","template":"schema-importer","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]}},{"name":"statisticsgen","template":"statisticsgen","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["importexamplegen"]},{"name":"trainer","template":"trainer","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["importexamplegen","transform"]},{"name":"transform","template":"transform","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["importexamplegen","schema-importer"]}]}},{"name":"importexamplegen","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","ImportExampleGen","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.example_gen.import_example_gen.component.ImportExampleGen\"\n },\n \"id\": \"ImportExampleGen\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ]\n },\n \"outputs\": {\n \"outputs\": {\n \"examples\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Examples\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\",\n \"version\": \"INT\"\n },\n \"baseType\": \"DATASET\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"output_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{}\"\n }\n },\n \"input_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"splits\\\": [\\n {\\n \\\"name\\\": \\\"train\\\",\\n \\\"pattern\\\": \\\"train\\\"\\n },\\n {\\n \\\"name\\\": \\\"eval\\\",\\n \\\"pattern\\\": \\\"eval\\\"\\n }\\n ]\\n}\"\n }\n },\n \"output_data_format\": {\n \"fieldValue\": {\n \"intValue\": \"6\"\n }\n },\n \"input_base\": {\n \"fieldValue\": {\n \"stringValue\": \"/maye/trainEvalData\"\n }\n },\n \"output_file_format\": {\n \"fieldValue\": {\n \"intValue\": \"5\"\n }\n }\n }\n },\n \"downstreamNodes\": [\n \"StatisticsGen\",\n \"Trainer\",\n \"Transform\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"ImportExampleGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.example_gen.import_example_gen.executor.Executor\"\n }\n }\n },\n \"customDriverSpecs\": {\n \"ImportExampleGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.example_gen.driver.FileBasedDriver\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"wafer-data","mountPath":"/maye/trainEvalData"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"pusher","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Pusher","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.pusher.component.Pusher\",\n \"baseType\": \"DEPLOY\"\n },\n \"id\": \"Pusher\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Pusher\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"model\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"Trainer\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Trainer\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Model\",\n \"baseType\": \"MODEL\"\n }\n },\n \"outputKey\": \"model\"\n }\n ]\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"pushed_model\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"PushedModel\",\n \"baseType\": \"MODEL\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"null\"\n }\n },\n \"push_destination\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"filesystem\\\": {\\n \\\"base_directory\\\": \\\"serving_model/detect_anomolies_on_wafer_tfdv\\\"\\n }\\n}\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"Trainer\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Pusher\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.pusher.executor.Executor\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"schema-importer","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","schema_importer","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.dsl.components.common.importer.Importer\"\n },\n \"id\": \"schema_importer\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.schema_importer\"\n }\n }\n }\n ]\n },\n \"outputs\": {\n \"outputs\": {\n \"result\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"artifact_uri\": {\n \"fieldValue\": {\n \"stringValue\": \"/tfx/pipelines/detect_anomalies_in_wafer_schema/\"\n }\n },\n \"reimport\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"output_key\": {\n \"fieldValue\": {\n \"stringValue\": \"result\"\n }\n }\n }\n },\n \"downstreamNodes\": [\n \"Transform\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"},{"name":"schema-path","readOnly":true,"mountPath":"/tfx/pipelines/detect_anomalies_in_wafer_schema"}],"imagePullPolicy":"Never"}},{"name":"statisticsgen","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","StatisticsGen","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.statistics_gen.component.StatisticsGen\",\n \"baseType\": \"PROCESS\"\n },\n \"id\": \"StatisticsGen\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.StatisticsGen\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"statistics\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"span\": \"INT\",\n \"split_names\": \"STRING\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"exclude_splits\": {\n \"fieldValue\": {\n \"stringValue\": \"[]\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"StatisticsGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.statistics_gen.executor.Executor\"\n }\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"trainer","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Trainer","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.trainer.component.Trainer\",\n \"baseType\": \"TRAIN\"\n },\n \"id\": \"Trainer\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Trainer\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n },\n \"transform_graph\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"Transform\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Transform\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"TransformGraph\"\n }\n },\n \"outputKey\": \"transform_graph\"\n }\n ]\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"model\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Model\",\n \"baseType\": \"MODEL\"\n }\n }\n },\n \"model_run\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ModelRun\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"train_args\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"num_steps\\\": 21\\n}\"\n }\n },\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\\"epochs\\\": 50}\"\n }\n },\n \"eval_args\": {\n \"fieldValue\": {\n \"stringValue\": \"{}\"\n }\n },\n \"module_path\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomalies_in_wafer_trainer@/tfx/pipelines/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\",\n \"Transform\"\n ],\n \"downstreamNodes\": [\n \"Pusher\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Trainer\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.trainer.executor.GenericExecutor\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"trainer-module","readOnly":true,"mountPath":"/tfx/pipelines/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"transform","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json"}]},"metadata":{"labels":{"add-pod-env":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Transform","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.transform.component.Transform\",\n \"baseType\": \"TRANSFORM\"\n },\n \"id\": \"Transform\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Transform\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"schema\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"schema_importer\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.schema_importer\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n },\n \"outputKey\": \"result\"\n }\n ],\n \"minCount\": 1\n },\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"post_transform_anomalies\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleAnomalies\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n }\n }\n }\n },\n \"updated_analyzer_cache\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"TransformCache\"\n }\n }\n },\n \"transform_graph\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"TransformGraph\"\n }\n }\n },\n \"post_transform_schema\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n },\n \"pre_transform_schema\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n },\n \"post_transform_stats\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n },\n \"pre_transform_stats\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"disable_statistics\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"force_tf_compat_v1\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"module_path\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomalies_in_wafer_trainer@/tfx/pipelines/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl\"\n }\n },\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"null\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\",\n \"schema_importer\"\n ],\n \"downstreamNodes\": [\n \"Trainer\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Transform\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.transform.executor.Executor\"\n }\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"transform-module","readOnly":true,"mountPath":"/tfx/pipelines/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl"},{"name":"schema-path","readOnly":true,"mountPath":"/tfx/pipelines/detect_anomalies_in_wafer_schema"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}}],"entrypoint":"detect-anomolies-on-wafer-tfdv-schema","arguments":{"parameters":[{"name":"pipeline-root","value":"/tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema"}]},"serviceAccountName":"pipeline-runner","volumes":[{"name":"wafer-data","hostPath":{"path":"/home/maye/trainEvalData","type":"Directory"}},{"name":"transform-module","hostPath":{"path":"/home/maye/maye_temp/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl","type":"File"}},{"name":"trainer-module","hostPath":{"path":"/home/maye/maye_temp/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl","type":"File"}},{"name":"schema-path","hostPath":{"path":"/home/maye/maye_temp/detect_anomalies_in_wafer_schema","type":"Directory"}},{"name":"tfx-pv","persistentVolumeClaim":{"claimName":"tfx-pv-claim"}}]},"status":{"startedAt":null,"finishedAt":null}} | [{"name":"pipeline-root","value":"/tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema"}] | | | 1707641204 | 1707641204 | 0 | Running | RUNNING | [{"UpdateTimeInSec":1707641204,"State":"PENDING"},{"UpdateTimeInSec":1707641212,"State":"RUNNING"}] | | {"metadata":{"name":"detect-anomolies-on-wafer-tfdv-schema-pnpjf","generateName":"detect-anomolies-on-wafer-tfdv-schema-","namespace":"kubeflow","uid":"ce065700-561f-49fa-b488-87fb69137d48","resourceVersion":"235624","generation":2,"creationTimestamp":"2024-02-11T08:46:44Z","labels":{"pipeline/runid":"c6013cee-f684-4524-a2b5-719f134e46af","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","workflows.argoproj.io/phase":"Running"},"annotations":{"pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline_compilation_time":"2024-01-07T22:16:36.438482","pipelines.kubeflow.org/pipeline_spec":"{\"description\": \"Constructs a Kubeflow pipeline.\", \"inputs\": [{\"default\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\", \"name\": \"pipeline-root\"}], \"name\": \"detect_anomolies_on_wafer_tfdv_schema\"}","pipelines.kubeflow.org/run_name":"Run of maye wafer pipeline (c6d86)","workflows.argoproj.io/pod-name-format":"v1"},"managedFields":[{"manager":"apiserver","operation":"Update","apiVersion":"argoproj.io/v1alpha1","time":"2024-02-11T08:46:44Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:pipelines.kubeflow.org/kfp_sdk_version":{},"f:pipelines.kubeflow.org/pipeline_compilation_time":{},"f:pipelines.kubeflow.org/pipeline_spec":{},"f:pipelines.kubeflow.org/run_name":{}},"f:generateName":{},"f:labels":{".":{},"f:pipeline/runid":{},"f:pipelines.kubeflow.org/kfp_sdk_version":{}}},"f:spec":{}}},{"manager":"workflow-controller","operation":"Update","apiVersion":"argoproj.io/v1alpha1","time":"2024-02-11T08:46:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:workflows.argoproj.io/pod-name-format":{}},"f:labels":{"f:workflows.argoproj.io/phase":{}}},"f:status":{}}}]},"spec":{"templates":[{"name":"detect-anomolies-on-wafer-tfdv-schema","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"pipelines.kubeflow.org/cache_enabled":"true"}},"dag":{"tasks":[{"name":"importexamplegen","template":"importexamplegen","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]}},{"name":"pusher","template":"pusher","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["trainer"]},{"name":"schema-importer","template":"schema-importer","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]}},{"name":"statisticsgen","template":"statisticsgen","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["importexamplegen"]},{"name":"trainer","template":"trainer","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["importexamplegen","transform"]},{"name":"transform","template":"transform","arguments":{"parameters":[{"name":"pipeline-root","value":"{{inputs.parameters.pipeline-root}}"}]},"dependencies":["importexamplegen","schema-importer"]}]}},{"name":"importexamplegen","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","ImportExampleGen","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.example_gen.import_example_gen.component.ImportExampleGen\"\n },\n \"id\": \"ImportExampleGen\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ]\n },\n \"outputs\": {\n \"outputs\": {\n \"examples\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Examples\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\",\n \"version\": \"INT\"\n },\n \"baseType\": \"DATASET\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"output_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{}\"\n }\n },\n \"input_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"splits\\\": [\\n {\\n \\\"name\\\": \\\"train\\\",\\n \\\"pattern\\\": \\\"train\\\"\\n },\\n {\\n \\\"name\\\": \\\"eval\\\",\\n \\\"pattern\\\": \\\"eval\\\"\\n }\\n ]\\n}\"\n }\n },\n \"output_data_format\": {\n \"fieldValue\": {\n \"intValue\": \"6\"\n }\n },\n \"input_base\": {\n \"fieldValue\": {\n \"stringValue\": \"/maye/trainEvalData\"\n }\n },\n \"output_file_format\": {\n \"fieldValue\": {\n \"intValue\": \"5\"\n }\n }\n }\n },\n \"downstreamNodes\": [\n \"StatisticsGen\",\n \"Trainer\",\n \"Transform\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"ImportExampleGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.example_gen.import_example_gen.executor.Executor\"\n }\n }\n },\n \"customDriverSpecs\": {\n \"ImportExampleGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.example_gen.driver.FileBasedDriver\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"wafer-data","mountPath":"/maye/trainEvalData"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"pusher","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Pusher","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.pusher.component.Pusher\",\n \"baseType\": \"DEPLOY\"\n },\n \"id\": \"Pusher\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Pusher\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"model\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"Trainer\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Trainer\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Model\",\n \"baseType\": \"MODEL\"\n }\n },\n \"outputKey\": \"model\"\n }\n ]\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"pushed_model\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"PushedModel\",\n \"baseType\": \"MODEL\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"null\"\n }\n },\n \"push_destination\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"filesystem\\\": {\\n \\\"base_directory\\\": \\\"serving_model/detect_anomolies_on_wafer_tfdv\\\"\\n }\\n}\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"Trainer\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Pusher\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.pusher.executor.Executor\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"schema-importer","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","schema_importer","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.dsl.components.common.importer.Importer\"\n },\n \"id\": \"schema_importer\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.schema_importer\"\n }\n }\n }\n ]\n },\n \"outputs\": {\n \"outputs\": {\n \"result\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"artifact_uri\": {\n \"fieldValue\": {\n \"stringValue\": \"/tfx/pipelines/detect_anomalies_in_wafer_schema/\"\n }\n },\n \"reimport\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"output_key\": {\n \"fieldValue\": {\n \"stringValue\": \"result\"\n }\n }\n }\n },\n \"downstreamNodes\": [\n \"Transform\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"},{"name":"schema-path","readOnly":true,"mountPath":"/tfx/pipelines/detect_anomalies_in_wafer_schema"}],"imagePullPolicy":"Never"}},{"name":"statisticsgen","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","StatisticsGen","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.statistics_gen.component.StatisticsGen\",\n \"baseType\": \"PROCESS\"\n },\n \"id\": \"StatisticsGen\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\":
\"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.StatisticsGen\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"statistics\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"span\": \"INT\",\n \"split_names\": \"STRING\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"exclude_splits\": {\n \"fieldValue\": {\n \"stringValue\": \"[]\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"StatisticsGen\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.statistics_gen.executor.Executor\"\n }\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"trainer","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Trainer","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.trainer.component.Trainer\",\n \"baseType\": \"TRAIN\"\n },\n \"id\": \"Trainer\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Trainer\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n },\n \"transform_graph\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"Transform\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Transform\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"TransformGraph\"\n }\n },\n \"outputKey\": \"transform_graph\"\n }\n ]\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"model\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Model\",\n \"baseType\": \"MODEL\"\n }\n }\n },\n \"model_run\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ModelRun\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"train_args\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\n \\\"num_steps\\\": 21\\n}\"\n }\n },\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"{\\\"epochs\\\": 50}\"\n }\n },\n \"eval_args\": {\n \"fieldValue\": {\n \"stringValue\": \"{}\"\n }\n },\n \"module_path\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomalies_in_wafer_trainer@/tfx/pipelines/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\",\n \"Transform\"\n ],\n \"downstreamNodes\": [\n \"Pusher\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Trainer\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec\",\n \"classPath\": \"tfx.components.trainer.executor.GenericExecutor\"\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"trainer-module","readOnly":true,"mountPath":"/tfx/pipelines/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}},{"name":"transform","inputs":{"parameters":[{"name":"pipeline-root"}]},"outputs":{"artifacts":[{"name":"mlpipeline-ui-metadata","path":"/mlpipeline-ui-metadata.json","optional":true}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"},"labels":{"add-pod-env":"true","pipelines.kubeflow.org/cache_enabled":"true","pipelines.kubeflow.org/enable_caching":"true","pipelines.kubeflow.org/kfp_sdk_version":"1.8.0","pipelines.kubeflow.org/pipeline-sdk-type":"tfx"}},"container":{"name":"","image":"docker.nju.edu.cn/tensorflow/tfx:1.13.0","command":["python","-m","tfx.orchestration.kubeflow.container_entrypoint"],"args":["--pipeline_root","{{inputs.parameters.pipeline-root}}","--kubeflow_metadata_config","{\n \"grpc_config\": {\n \"grpc_service_host\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_HOST\"\n },\n \"grpc_service_port\": {\n \"environment_variable\": \"METADATA_GRPC_SERVICE_PORT\"\n }\n }\n}","--node_id","Transform","--tfx_ir","{\n \"pipelineInfo\": {\n \"id\": \"detect_anomolies_on_wafer_tfdv_schema\"\n },\n \"nodes\": [\n {\n \"pipelineNode\": {\n \"nodeInfo\": {\n \"type\": {\n \"name\": \"tfx.components.transform.component.Transform\",\n \"baseType\": \"TRANSFORM\"\n },\n \"id\": \"Transform\"\n },\n \"contexts\": {\n \"contexts\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.Transform\"\n }\n }\n }\n ]\n },\n \"inputs\": {\n \"inputs\": {\n \"schema\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"schema_importer\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.schema_importer\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n },\n \"outputKey\": \"result\"\n }\n ],\n \"minCount\": 1\n },\n \"examples\": {\n \"channels\": [\n {\n \"producerNodeQuery\": {\n \"id\": \"ImportExampleGen\"\n },\n \"contextQueries\": [\n {\n \"type\": {\n \"name\": \"pipeline\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"pipeline_run\"\n },\n \"name\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n {\n \"type\": {\n \"name\": \"node\"\n },\n \"name\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen\"\n }\n }\n }\n ],\n \"artifactQuery\": {\n \"type\": {\n \"name\": \"Examples\",\n \"baseType\": \"DATASET\"\n }\n },\n \"outputKey\": \"examples\"\n }\n ],\n \"minCount\": 1\n }\n }\n },\n \"outputs\": {\n \"outputs\": {\n \"post_transform_anomalies\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleAnomalies\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n }\n }\n }\n },\n \"updated_analyzer_cache\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"TransformCache\"\n }\n }\n },\n \"transform_graph\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"TransformGraph\"\n }\n }\n },\n \"post_transform_schema\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n },\n \"pre_transform_schema\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"Schema\"\n }\n }\n },\n \"post_transform_stats\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n },\n \"pre_transform_stats\": {\n \"artifactSpec\": {\n \"type\": {\n \"name\": \"ExampleStatistics\",\n \"properties\": {\n \"split_names\": \"STRING\",\n \"span\": \"INT\"\n },\n \"baseType\": \"STATISTICS\"\n }\n }\n }\n }\n },\n \"parameters\": {\n \"parameters\": {\n \"disable_statistics\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"force_tf_compat_v1\": {\n \"fieldValue\": {\n \"intValue\": \"0\"\n }\n },\n \"module_path\": {\n \"fieldValue\": {\n \"stringValue\": \"detect_anomalies_in_wafer_trainer@/tfx/pipelines/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl\"\n }\n },\n \"custom_config\": {\n \"fieldValue\": {\n \"stringValue\": \"null\"\n }\n }\n }\n },\n \"upstreamNodes\": [\n \"ImportExampleGen\",\n \"schema_importer\"\n ],\n \"downstreamNodes\": [\n \"Trainer\"\n ],\n \"executionOptions\": {\n \"cachingOptions\": {}\n }\n }\n }\n ],\n \"runtimeSpec\": {\n \"pipelineRoot\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-root\",\n \"type\": \"STRING\",\n \"defaultValue\": {\n \"stringValue\": \"pipelines/detect_anomolies_on_wafer_tfdv_schema\"\n }\n }\n },\n \"pipelineRunId\": {\n \"runtimeParameter\": {\n \"name\": \"pipeline-run-id\",\n \"type\": \"STRING\"\n }\n }\n },\n \"executionMode\": \"SYNC\",\n \"deploymentConfig\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig\",\n \"executorSpecs\": {\n \"Transform\": {\n \"@type\": \"type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec\",\n \"pythonExecutorSpec\": {\n \"classPath\": \"tfx.components.transform.executor.Executor\"\n }\n }\n },\n \"metadataConnectionConfig\": {\n \"@type\": \"type.googleapis.com/ml_metadata.ConnectionConfig\",\n \"sqlite\": {\n \"filenameUri\": \"metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db\",\n \"connectionMode\": \"READWRITE_OPENCREATE\"\n }\n }\n }\n}","--metadata_ui_path","/mlpipeline-ui-metadata.json","--runtime_parameter","pipeline-root=STRING:{{inputs.parameters.pipeline-root}}"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},{"name":"KFP_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"WORKFLOW_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['workflows.argoproj.io/workflow']"}}},{"name":"KFP_RUN_ID","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipeline/runid']"}}},{"name":"ENABLE_CACHING","valueFrom":{"fieldRef":{"fieldPath":"metadata.labels['pipelines.kubeflow.org/enable_caching']"}}}],"resources":{},"volumeMounts":[{"name":"transform-module","readOnly":true,"mountPath":"/tfx/pipelines/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl"},{"name":"schema-path","readOnly":true,"mountPath":"/tfx/pipelines/detect_anomalies_in_wafer_schema"},{"name":"tfx-pv","mountPath":"/tfx/tfx_pv"}],"imagePullPolicy":"Never"}}],"entrypoint":"detect-anomolies-on-wafer-tfdv-schema","arguments":{"parameters":[{"name":"pipeline-root","value":"/tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema"}]},"serviceAccountName":"pipeline-runner","volumes":[{"name":"wafer-data","hostPath":{"path":"/home/maye/trainEvalData","type":"Directory"}},{"name":"transform-module","hostPath":{"path":"/home/maye/maye_temp/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl","type":"File"}},{"name":"trainer-module","hostPath":{"path":"/home/maye/maye_temp/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl","type":"File"}},{"name":"schema-path","hostPath":{"path":"/home/maye/maye_temp/detect_anomalies_in_wafer_schema","type":"Directory"}},{"name":"tfx-pv","persistentVolumeClaim":{"claimName":"tfx-pv-claim"}}],"podMetadata":{"labels":{"pipeline/runid":"c6013cee-f684-4524-a2b5-719f134e46af"}}},"status":{"phase":"Running","startedAt":"2024-02-11T08:46:44Z","finishedAt":null,"progress":"0/2","nodes":{"detect-anomolies-on-wafer-tfdv-schema-pnpjf":{"id":"detect-anomolies-on-wafer-tfdv-schema-pnpjf","name":"detect-anomolies-on-wafer-tfdv-schema-pnpjf","displayName":"detect-anomolies-on-wafer-tfdv-schema-pnpjf","type":"DAG","templateName":"detect-anomolies-on-wafer-tfdv-schema","templateScope":"local/detect-anomolies-on-wafer-tfdv-schema-pnpjf","phase":"Running","startedAt":"2024-02-11T08:46:44Z","finishedAt":null,"progress":"0/2","inputs":{"parameters":[{"name":"pipeline-root","value":"/tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema"}]},"children":["detect-anomolies-on-wafer-tfdv-schema-pnpjf-587264735","detect-anomolies-on-wafer-tfdv-schema-pnpjf-28753274"]},"detect-anomolies-on-wafer-tfdv-schema-pnpjf-28753274":{"id":"detect-anomolies-on-wafer-tfdv-schema-pnpjf-28753274","name":"detect-anomolies-on-wafer-tfdv-schema-pnpjf.schema-importer","displayName":"schema-importer","type":"Pod","templateName":"schema-importer","templateScope":"local/detect-anomolies-on-wafer-tfdv-schema-pnpjf","phase":"Pending","boundaryID":"detect-anomolies-on-wafer-tfdv-schema-pnpjf","startedAt":"2024-02-11T08:46:45Z","finishedAt":null,"progress":"0/1","inputs":{"parameters":[{"name":"pipeline-root","value":"/tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema"}]}},"detect-anomolies-on-wafer-tfdv-schema-pnpjf-587264735":{"id":"detect-anomolies-on-wafer-tfdv-schema-pnpjf-587264735","name":"detect-anomolies-on-wafer-tfdv-schema-pnpjf.importexamplegen","displayName":"importexamplegen","type":"Pod","templateName":"importexamplegen","templateScope":"local/detect-anomolies-on-wafer-tfdv-schema-pnpjf","phase":"Pending","boundaryID":"detect-anomolies-on-wafer-tfdv-schema-pnpjf","startedAt":"2024-02-11T08:46:44Z","finishedAt":null,"progress":"0/1","inputs":{"parameters":[{"name":"pipeline-root","value":"/tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema"}]}}},"artifactRepositoryRef":{"default":true,"artifactRepository":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"keyFormat":"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}"}}}}} | 0 | 0 |
mysql> select UUID from run_details;
+--------------------------------------+
| UUID |
+--------------------------------------+
| 1548e025-3d74-4036-a03f-ef79945f9e00 |
| c6013cee-f684-4524-a2b5-719f134e46af |
+--------------------------------------+
2 rows in set (0.00 sec)
mysql>
In summary, this error happens if:
"resource_manager.go:1150] Cannot find reported workflow name="detect-anomolies-on-wafer-tfdv-schema-qxpkk" namespace="kubeflow" runId="1548e025-3d74-4036-a03f-ef79945f9e00" in run store. Deleting the workflow to avoid resource leaking. "
when /api.RunService/CreateRunV1 -> /api.ReportService/ReportWorkflowV1 .
In this case, pod metadata-grpc-deployment can not connect to mysql sometimes due to temporary not good network, so pod ml-pipeline raises error:
“Cannot find reported workflow name="detect-anomolies-on-wafer-tfdv-schema-qxpkk" namespace="kubeflow" runId="1548e025-3d74-4036-a03f-ef79945f9e00" in run store.”， even though the run is recorded in database (namely run store). And on this condition ml-pipeline "Deleting the workflow to avoid resource leaking.", namely deletes the workflow resource on kubernetes cluster, this is why kubectl get workflow --all-namespaces --> no resource found.
Comparing log of ml-pipeline and workflow-controller, when
ml-pipeline "Deleting the workflow to avoid resource leaking.", there is no related info in log of workflow-controller, and in log of workflow-controller, the deteted workflow status is running, and on ml-pipeline-ui webpage, the workflow status is also running, components are pending. This is a little strange, It seems like that workflow-controller does not know about ml-pipeline "Deleting the workflow to avoid resource leaking." .
log of pod ml-pipeline:
I0211 08:46:44.696273 7 interceptor.go:29] /api.RunService/CreateRunV1 handler starting
I0211 08:46:47.084088 7 interceptor.go:29] /api.ReportService/ReportWorkflowV1 handler starting
I0211 08:46:49.167581 7 interceptor.go:29] /api.RunService/GetRunV1 handler starting
E0211 08:46:51.546205 7 resource_manager.go:1150] Cannot find reported workflow name="detect-anomolies-on-wafer-tfdv-schema-pnpjf" namespace="kubeflow" runId="c6013cee-f684-4524-a2b5-719f134e46af" in run store. Deleting the workflow to avoid resource leaking. This can be caused by installing two KFP instances that try to manage the same workflows or an unknown bug. If you encounter this, recommend reporting more details in https://github.com/kubeflow/pipelines/issues/6189
log of pod workflow-controller:
time="2024-02-11T08:46:44.833Z" level=info msg="Processing workflow" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:44.915Z" level=info msg="Get configmaps 404"
time="2024-02-11T08:46:44.916Z" level=warning msg="Non-transient error: configmaps "artifact-repositories" not found"
time="2024-02-11T08:46:44.916Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2024-02-11T08:46:44.916Z" level=info msg="Updated phase -> Running" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:44.916Z" level=info msg="DAG node detect-anomolies-on-wafer-tfdv-schema-pnpjf initialized Running" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:44.917Z" level=info msg="All of node detect-anomolies-on-wafer-tfdv-schema-pnpjf.importexamplegen dependencies [] completed" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:44.918Z" level=info msg="Pod node detect-anomolies-on-wafer-tfdv-schema-pnpjf-587264735 initialized Pending" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:44.985Z" level=info msg="Create events 201"
time="2024-02-11T08:46:45.099Z" level=info msg="Create pods 201"
W0211 08:46:45.100481 1 warnings.go:70] spec.containers[1].env[4].name: duplicate name "WORKFLOW_ID"
time="2024-02-11T08:46:45.102Z" level=info msg="Created pod: detect-anomolies-on-wafer-tfdv-schema-pnpjf.importexamplegen (detect-anomolies-on-wafer-tfdv-schema-pnpjf-587264735)" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.103Z" level=info msg="All of node detect-anomolies-on-wafer-tfdv-schema-pnpjf.schema-importer dependencies [] completed" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.104Z" level=info msg="Pod node detect-anomolies-on-wafer-tfdv-schema-pnpjf-28753274 initialized Pending" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.864Z" level=info msg="Create pods 201"
W0211 08:46:45.865156 1 warnings.go:70] spec.containers[1].env[4].name: duplicate name "WORKFLOW_ID"
time="2024-02-11T08:46:45.866Z" level=info msg="Created pod: detect-anomolies-on-wafer-tfdv-schema-pnpjf.schema-importer (detect-anomolies-on-wafer-tfdv-schema-pnpjf-28753274)" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.866Z" level=info msg="TaskSet Reconciliation" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.866Z" level=info msg=reconcileAgentPod namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:45.867Z" level=info msg="Workflow to be dehydrated" Workflow Size=45369
time="2024-02-11T08:46:46.122Z" level=info msg="Update workflows 200"
time="2024-02-11T08:46:46.125Z" level=info msg="Workflow update successful" namespace=kubeflow phase=Running resourceVersion=235624 workflow=detect-anomolies-on-wafer-tfdv-schema-pnpjf
time="2024-02-11T08:46:46.239Z" level=info msg="Create events 201"
time="2024-02-11T08:46:47.656Z" level=info msg="Get leases 200"
time="2024-02-11T08:46:47.676Z" level=info msg="Update leases 200"
time="2024-02-11T08:46:52.682Z" level=info msg="Get leases 200"
time="2024-02-11T08:46:52.889Z" level=info msg="Update leases 200"
time="2024-02-11T08:46:57.893Z" level=info msg="Get leases 200"
time="2024-02-11T08:46:57.913Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:02.916Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:02.997Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:08.000Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:08.014Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:13.018Z" level=info msg="Get leases 200"
time="2024-02-11T08:47:13.060Z" level=info msg="Update leases 200"
time="2024-02-11T08:47:14.155Z" level=info msg="List workflows 200"
time="2024-02-11T08:47:14.156Z" level=info msg=healthz age=5m0s err="
time="2024-02-11T10:11:21.910Z" level=info msg="Processing workflow" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-hq49w
time="2024-02-11T10:11:22.070Z" level=info msg="Task-result reconciliation" namespace=kubeflow numObjs=0 workflow=detect-anomolies-on-wafer-tfdv-schema-hq49w
time="2024-02-11T10:11:22.070Z" level=info msg="node changed" new.message="Unschedulable: 0/2 nodes are available: persistentvolumeclaim "tfx-pv-claim" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.." new.phase=Pending new.progress=0/1 nodeID=detect-anomolies-on-wafer-tfdv-schema-hq49w-2248169053 old.message= old.phase=Pending old.progress=0/1
time="2024-02-11T10:11:22.070Z" level=info msg="node changed" new.message="Unschedulable: 0/2 nodes are available: persistentvolumeclaim "tfx-pv-claim" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.." new.phase=Pending new.progress=0/1 nodeID=detect-anomolies-on-wafer-tfdv-schema-hq49w-218673478 old.message= old.phase=Pending old.progress=0/1
time="2024-02-11T10:11:22.073Z" level=info msg="TaskSet Reconciliation" namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-hq49w
time="2024-02-11T10:11:22.073Z" level=info msg=reconcileAgentPod namespace=kubeflow workflow=detect-anomolies-on-wafer-tfdv-schema-hq49w
time="2024-02-11T10:11:22.073Z" level=info msg="Workflow to be dehydrated" Workflow Size=45948
time="2024-02-11T10:11:22.177Z" level=info msg="Update workflows 200"
time="2024-02-11T10:11:22.181Z" level=info msg="Workflow update successful" namespace=kubeflow phase=Running resourceVersion=243632 workflow=detect-anomolies-on-wafer-tfdv-schema-hq49w
time="2024-02-11T10:11:26.666Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:26.931Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:31.935Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:31.979Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:36.983Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:36.997Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:42.000Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:42.083Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:45.138Z" level=info msg="Watch workflows 200"
time="2024-02-11T10:11:47.087Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:47.131Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:52.188Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:52.198Z" level=info msg="Update leases 200"
time="2024-02-11T10:11:56.125Z" level=info msg="Watch configmaps 200"
time="2024-02-11T10:11:57.202Z" level=info msg="Get leases 200"
time="2024-02-11T10:11:57.232Z" level=info msg="Update leases 200"
[SOLUTION]
It is due to temporary not good network, so metadata-grpc-deployment can not connect to mysql. Retry, then ok.
[ERROR: dial tcp: lookup minio-service.kubeflow on 10.96.0.10:53: no such host]
On ml-pipeline-ui webpage, component SchemaImporter:
This step is in Error state with this message: Error (exit code 1): failed to put file: Get "http://minio-service.kubeflow:9000/mlpipeline/?location=": dial tcp: lookup minio-service.kubeflow on 10.96.0.10:53: no such host
[SOLUTION]
This is due to temporary not good network, so minio can not be connected. retry, then ok. In this example, pod minio and pod component SchmaImporter run on the same computer.
__
4. complete pipeline.yaml example
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: detect-anomolies-on-wafer-tfdv-schema-
#name: detect-anomolies-on-wafer-tfdv-schema-maye
annotations: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.0, pipelines.kubeflow.org/pipeline_compilation_time: '2024-01-07T22:16:36.438482',
pipelines.kubeflow.org/pipeline_spec: '{"description": "Constructs a Kubeflow
pipeline.", "inputs": [{"default": "pipelines/detect_anomolies_on_wafer_tfdv_schema",
"name": "pipeline-root"}], "name": "detect_anomolies_on_wafer_tfdv_schema"}'}
labels: {pipelines.kubeflow.org/kfp_sdk_version: 1.8.0}
spec:
entrypoint: detect-anomolies-on-wafer-tfdv-schema
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- maye-inspiron-5547
volumes:
- name: wafer-data
hostPath:
path: /home/maye/trainEvalData
type: Directory
- name: transform-module
hostPath:
path: /home/maye/maye_temp/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
type: File
- name: trainer-module
hostPath:
path: /home/maye/maye_temp/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
type: File
- name: schema-path
hostPath:
path: /home/maye/maye_temp/detect_anomalies_in_wafer_schema
type: Directory
- name: tfx-pv
persistentVolumeClaim:
claimName: tfx-pv-claim
templates:
- name: detect-anomolies-on-wafer-tfdv-schema
inputs:
parameters:
- {name: pipeline-root}
dag:
tasks:
- name: importexamplegen
template: importexamplegen
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'}
volumes:
- name: wafer-data
- name: tfx-pv
- name: pusher
template: pusher
dependencies: [trainer]
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'}
volumes:
- name: tfx-pv
- name: schema-importer
template: schema-importer
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'}
volumes:
- name: tfx-pv
- name: schema-path
- name: statisticsgen
template: statisticsgen
dependencies: [importexamplegen]
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'}
#artifacts:
#- {name: import_example_gen_outputs, from: "{{tasks.importexamplegen.outputs.artifacts.import_example_gen_outputs}}"}
volumes:
- name: tfx-pv
- name: trainer
template: trainer
dependencies: [importexamplegen, transform]
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'}
volumes:
- name: trainer-module
- name: tfx-pv
- name: transform
template: transform
dependencies: [importexamplegen, schema-importer]
arguments:
parameters:
- {name: pipeline-root, value: '{{inputs.parameters.pipeline-root}}'}
volumes:
- name: transform-module
- name: tfx-pv
- name: schema-path
- name: importexamplegen
container:
args:
- --pipeline_root
- '{{inputs.parameters.pipeline-root}}'
- --kubeflow_metadata_config
- |-
{
"grpc_config": {
"grpc_service_host": {
"environment_variable": "METADATA_GRPC_SERVICE_HOST"
},
"grpc_service_port": {
"environment_variable": "METADATA_GRPC_SERVICE_PORT"
}
}
}
- --node_id
- ImportExampleGen
- --tfx_ir
- |-
{
"pipelineInfo": {
"id": "detect_anomolies_on_wafer_tfdv_schema"
},
"nodes": [
{
"pipelineNode": {
"nodeInfo": {
"type": {
"name": "tfx.components.example_gen.import_example_gen.component.ImportExampleGen"
},
"id": "ImportExampleGen"
},
"contexts": {
"contexts": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen"
}
}
}
]
},
"outputs": {
"outputs": {
"examples": {
"artifactSpec": {
"type": {
"name": "Examples",
"properties": {
"split_names": "STRING",
"span": "INT",
"version": "INT"
},
"baseType": "DATASET"
}
}
}
}
},
"parameters": {
"parameters": {
"output_config": {
"fieldValue": {
"stringValue": "{}"
}
},
"input_config": {
"fieldValue": {
"stringValue": "{\n \"splits\": [\n {\n \"name\": \"train\",\n \"pattern\": \"train\"\n },\n {\n \"name\": \"eval\",\n \"pattern\": \"eval\"\n }\n ]\n}"
}
},
"output_data_format": {
"fieldValue": {
"intValue": "6"
}
},
"input_base": {
"fieldValue": {
"stringValue": "/maye/trainEvalData"
}
},
"output_file_format": {
"fieldValue": {
"intValue": "5"
}
}
}
},
"downstreamNodes": [
"StatisticsGen",
"Trainer",
"Transform"
],
"executionOptions": {
"cachingOptions": {}
}
}
}
],
"runtimeSpec": {
"pipelineRoot": {
"runtimeParameter": {
"name": "pipeline-root",
"type": "STRING",
"defaultValue": {
"stringValue": "pipelines/detect_anomolies_on_wafer_tfdv_schema"
}
}
},
"pipelineRunId": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
"executionMode": "SYNC",
"deploymentConfig": {
"@type": "type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig",
"executorSpecs": {
"ImportExampleGen": {
"@type": "type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec",
"pythonExecutorSpec": {
"classPath": "tfx.components.example_gen.import_example_gen.executor.Executor"
}
}
},
"customDriverSpecs": {
"ImportExampleGen": {
"@type": "type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec",
"classPath": "tfx.components.example_gen.driver.FileBasedDriver"
}
},
"metadataConnectionConfig": {
"@type": "type.googleapis.com/ml_metadata.ConnectionConfig",
"sqlite": {
"filenameUri": "metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db",
"connectionMode": "READWRITE_OPENCREATE"
}
}
}
}
- --metadata_ui_path
- /mlpipeline-ui-metadata.json
- --runtime_parameter
- pipeline-root=STRING:{{inputs.parameters.pipeline-root}}
command: [python, -m, tfx.orchestration.kubeflow.container_entrypoint]
env:
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_POD_NAME
valueFrom:
fieldRef: {fieldPath: metadata.name}
- name: KFP_POD_UID
valueFrom:
fieldRef: {fieldPath: metadata.uid}
- name: KFP_NAMESPACE
valueFrom:
fieldRef: {fieldPath: metadata.namespace}
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_RUN_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipeline/runid'']'}
- name: ENABLE_CACHING
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipelines.kubeflow.org/enable_caching'']'}
envFrom:
- configMapRef: {name: metadata-grpc-configmap, optional: true}
#image: tensorflow/tfx:1.14.0
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /maye/trainEvalData
name: wafer-data
- mountPath: /tfx/tfx_pv
name: tfx-pv
inputs:
parameters:
- {name: pipeline-root}
outputs:
artifacts:
- {name: mlpipeline-ui-metadata, path: /mlpipeline-ui-metadata.json}
#- {name: import_example_gen_outputs, path: /tmp/pipelines}
metadata:
labels:
add-pod-env: "true"
pipelines.kubeflow.org/pipeline-sdk-type: tfx
pipelines.kubeflow.org/kfp_sdk_version: 1.8.0
pipelines.kubeflow.org/enable_caching: "true"
- name: pusher
container:
args:
- --pipeline_root
- '{{inputs.parameters.pipeline-root}}'
- --kubeflow_metadata_config
- |-
{
"grpc_config": {
"grpc_service_host": {
"environment_variable": "METADATA_GRPC_SERVICE_HOST"
},
"grpc_service_port": {
"environment_variable": "METADATA_GRPC_SERVICE_PORT"
}
}
}
- --node_id
- Pusher
- --tfx_ir
- |-
{
"pipelineInfo": {
"id": "detect_anomolies_on_wafer_tfdv_schema"
},
"nodes": [
{
"pipelineNode": {
"nodeInfo": {
"type": {
"name": "tfx.components.pusher.component.Pusher",
"baseType": "DEPLOY"
},
"id": "Pusher"
},
"contexts": {
"contexts": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.Pusher"
}
}
}
]
},
"inputs": {
"inputs": {
"model": {
"channels": [
{
"producerNodeQuery": {
"id": "Trainer"
},
"contextQueries": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.Trainer"
}
}
}
],
"artifactQuery": {
"type": {
"name": "Model",
"baseType": "MODEL"
}
},
"outputKey": "model"
}
]
}
}
},
"outputs": {
"outputs": {
"pushed_model": {
"artifactSpec": {
"type": {
"name": "PushedModel",
"baseType": "MODEL"
}
}
}
}
},
"parameters": {
"parameters": {
"custom_config": {
"fieldValue": {
"stringValue": "null"
}
},
"push_destination": {
"fieldValue": {
"stringValue": "{\n \"filesystem\": {\n \"base_directory\": \"serving_model/detect_anomolies_on_wafer_tfdv\"\n }\n}"
}
}
}
},
"upstreamNodes": [
"Trainer"
],
"executionOptions": {
"cachingOptions": {}
}
}
}
],
"runtimeSpec": {
"pipelineRoot": {
"runtimeParameter": {
"name": "pipeline-root",
"type": "STRING",
"defaultValue": {
"stringValue": "pipelines/detect_anomolies_on_wafer_tfdv_schema"
}
}
},
"pipelineRunId": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
"executionMode": "SYNC",
"deploymentConfig": {
"@type": "type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig",
"executorSpecs": {
"Pusher": {
"@type": "type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec",
"classPath": "tfx.components.pusher.executor.Executor"
}
},
"metadataConnectionConfig": {
"@type": "type.googleapis.com/ml_metadata.ConnectionConfig",
"sqlite": {
"filenameUri": "metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db",
"connectionMode": "READWRITE_OPENCREATE"
}
}
}
}
- --metadata_ui_path
- /mlpipeline-ui-metadata.json
- --runtime_parameter
- pipeline-root=STRING:{{inputs.parameters.pipeline-root}}
command: [python, -m, tfx.orchestration.kubeflow.container_entrypoint]
env:
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_POD_NAME
valueFrom:
fieldRef: {fieldPath: metadata.name}
- name: KFP_POD_UID
valueFrom:
fieldRef: {fieldPath: metadata.uid}
- name: KFP_NAMESPACE
valueFrom:
fieldRef: {fieldPath: metadata.namespace}
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_RUN_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipeline/runid'']'}
- name: ENABLE_CACHING
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipelines.kubeflow.org/enable_caching'']'}
envFrom:
- configMapRef: {name: metadata-grpc-configmap, optional: true}
#image: tensorflow/tfx:1.14.0
#image: dockerproxy.com/tensorflow/tfx:1.14.0
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /tfx/tfx_pv
name: tfx-pv
inputs:
parameters:
- {name: pipeline-root}
outputs:
artifacts:
- {name: mlpipeline-ui-metadata, path: /mlpipeline-ui-metadata.json}
metadata:
labels:
add-pod-env: "true"
pipelines.kubeflow.org/pipeline-sdk-type: tfx
pipelines.kubeflow.org/kfp_sdk_version: 1.8.0
pipelines.kubeflow.org/enable_caching: "true"
- name: schema-importer
container:
args:
- --pipeline_root
- '{{inputs.parameters.pipeline-root}}'
- --kubeflow_metadata_config
- |-
{
"grpc_config": {
"grpc_service_host": {
"environment_variable": "METADATA_GRPC_SERVICE_HOST"
},
"grpc_service_port": {
"environment_variable": "METADATA_GRPC_SERVICE_PORT"
}
}
}
- --node_id
- schema_importer
- --tfx_ir
- |-
{
"pipelineInfo": {
"id": "detect_anomolies_on_wafer_tfdv_schema"
},
"nodes": [
{
"pipelineNode": {
"nodeInfo": {
"type": {
"name": "tfx.dsl.components.common.importer.Importer"
},
"id": "schema_importer"
},
"contexts": {
"contexts": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.schema_importer"
}
}
}
]
},
"outputs": {
"outputs": {
"result": {
"artifactSpec": {
"type": {
"name": "Schema"
}
}
}
}
},
"parameters": {
"parameters": {
"artifact_uri": {
"fieldValue": {
"stringValue": "/tfx/pipelines/detect_anomalies_in_wafer_schema/"
}
},
"reimport": {
"fieldValue": {
"intValue": "0"
}
},
"output_key": {
"fieldValue": {
"stringValue": "result"
}
}
}
},
"downstreamNodes": [
"Transform"
],
"executionOptions": {
"cachingOptions": {}
}
}
}
],
"runtimeSpec": {
"pipelineRoot": {
"runtimeParameter": {
"name": "pipeline-root",
"type": "STRING",
"defaultValue": {
"stringValue": "pipelines/detect_anomolies_on_wafer_tfdv_schema"
}
}
},
"pipelineRunId": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
"executionMode": "SYNC",
"deploymentConfig": {
"@type": "type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig",
"metadataConnectionConfig": {
"@type": "type.googleapis.com/ml_metadata.ConnectionConfig",
"sqlite": {
"filenameUri": "metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db",
"connectionMode": "READWRITE_OPENCREATE"
}
}
}
}
- --metadata_ui_path
- /mlpipeline-ui-metadata.json
- --runtime_parameter
- pipeline-root=STRING:{{inputs.parameters.pipeline-root}}
command: [python, -m, tfx.orchestration.kubeflow.container_entrypoint]
env:
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_POD_NAME
valueFrom:
fieldRef: {fieldPath: metadata.name}
- name: KFP_POD_UID
valueFrom:
fieldRef: {fieldPath: metadata.uid}
- name: KFP_NAMESPACE
valueFrom:
fieldRef: {fieldPath: metadata.namespace}
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_RUN_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipeline/runid'']'}
- name: ENABLE_CACHING
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipelines.kubeflow.org/enable_caching'']'}
envFrom:
- configMapRef: {name: metadata-grpc-configmap, optional: true}
#image: tensorflow/tfx:1.14.0
#image: dockerproxy.com/tensorflow/tfx:1.14.0
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /tfx/tfx_pv
name: tfx-pv
- mountPath: /tfx/pipelines/detect_anomalies_in_wafer_schema
name: schema-path
readOnly: True
inputs:
parameters:
- {name: pipeline-root}
outputs:
artifacts:
- {name: mlpipeline-ui-metadata, path: /mlpipeline-ui-metadata.json}
metadata:
labels:
add-pod-env: "true"
pipelines.kubeflow.org/pipeline-sdk-type: tfx
pipelines.kubeflow.org/kfp_sdk_version: 1.8.0
pipelines.kubeflow.org/enable_caching: "true"
- name: statisticsgen
container:
args:
- --pipeline_root
- '{{inputs.parameters.pipeline-root}}'
- --kubeflow_metadata_config
- |-
{
"grpc_config": {
"grpc_service_host": {
"environment_variable": "METADATA_GRPC_SERVICE_HOST"
},
"grpc_service_port": {
"environment_variable": "METADATA_GRPC_SERVICE_PORT"
}
}
}
- --node_id
- StatisticsGen
- --tfx_ir
- |-
{
"pipelineInfo": {
"id": "detect_anomolies_on_wafer_tfdv_schema"
},
"nodes": [
{
"pipelineNode": {
"nodeInfo": {
"type": {
"name": "tfx.components.statistics_gen.component.StatisticsGen",
"baseType": "PROCESS"
},
"id": "StatisticsGen"
},
"contexts": {
"contexts": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.StatisticsGen"
}
}
}
]
},
"inputs": {
"inputs": {
"examples": {
"channels": [
{
"producerNodeQuery": {
"id": "ImportExampleGen"
},
"contextQueries": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen"
}
}
}
],
"artifactQuery": {
"type": {
"name": "Examples",
"baseType": "DATASET"
}
},
"outputKey": "examples"
}
],
"minCount": 1
}
}
},
"outputs": {
"outputs": {
"statistics": {
"artifactSpec": {
"type": {
"name": "ExampleStatistics",
"properties": {
"span": "INT",
"split_names": "STRING"
},
"baseType": "STATISTICS"
}
}
}
}
},
"parameters": {
"parameters": {
"exclude_splits": {
"fieldValue": {
"stringValue": "[]"
}
}
}
},
"upstreamNodes": [
"ImportExampleGen"
],
"executionOptions": {
"cachingOptions": {}
}
}
}
],
"runtimeSpec": {
"pipelineRoot": {
"runtimeParameter": {
"name": "pipeline-root",
"type": "STRING",
"defaultValue": {
"stringValue": "pipelines/detect_anomolies_on_wafer_tfdv_schema"
}
}
},
"pipelineRunId": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
"executionMode": "SYNC",
"deploymentConfig": {
"@type": "type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig",
"executorSpecs": {
"StatisticsGen": {
"@type": "type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec",
"pythonExecutorSpec": {
"classPath": "tfx.components.statistics_gen.executor.Executor"
}
}
},
"metadataConnectionConfig": {
"@type": "type.googleapis.com/ml_metadata.ConnectionConfig",
"sqlite": {
"filenameUri": "metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db",
"connectionMode": "READWRITE_OPENCREATE"
}
}
}
}
- --metadata_ui_path
- /mlpipeline-ui-metadata.json
- --runtime_parameter
- pipeline-root=STRING:{{inputs.parameters.pipeline-root}}
command: [python, -m, tfx.orchestration.kubeflow.container_entrypoint]
env:
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_POD_NAME
valueFrom:
fieldRef: {fieldPath: metadata.name}
- name: KFP_POD_UID
valueFrom:
fieldRef: {fieldPath: metadata.uid}
- name: KFP_NAMESPACE
valueFrom:
fieldRef: {fieldPath: metadata.namespace}
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_RUN_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipeline/runid'']'}
- name: ENABLE_CACHING
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipelines.kubeflow.org/enable_caching'']'}
envFrom:
- configMapRef: {name: metadata-grpc-configmap, optional: true}
#image: tensorflow/tfx:1.14.0
#image: dockerproxy.com/tensorflow/tfx:1.14.0
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /tfx/tfx_pv
name: tfx-pv
inputs:
parameters:
- {name: pipeline-root}
#artifacts:
#- {name: import_example_gen_outputs, path: /tmp/pipelines}
outputs:
artifacts:
- {name: mlpipeline-ui-metadata, path: /mlpipeline-ui-metadata.json}
#- {name: statistics_gen_outputs, path: /tmp/pipelines}
metadata:
labels:
add-pod-env: "true"
pipelines.kubeflow.org/pipeline-sdk-type: tfx
pipelines.kubeflow.org/kfp_sdk_version: 1.8.0
pipelines.kubeflow.org/enable_caching: "true"
- name: trainer
container:
args:
- --pipeline_root
- '{{inputs.parameters.pipeline-root}}'
- --kubeflow_metadata_config
- |-
{
"grpc_config": {
"grpc_service_host": {
"environment_variable": "METADATA_GRPC_SERVICE_HOST"
},
"grpc_service_port": {
"environment_variable": "METADATA_GRPC_SERVICE_PORT"
}
}
}
- --node_id
- Trainer
- --tfx_ir
- |-
{
"pipelineInfo": {
"id": "detect_anomolies_on_wafer_tfdv_schema"
},
"nodes": [
{
"pipelineNode": {
"nodeInfo": {
"type": {
"name": "tfx.components.trainer.component.Trainer",
"baseType": "TRAIN"
},
"id": "Trainer"
},
"contexts": {
"contexts": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.Trainer"
}
}
}
]
},
"inputs": {
"inputs": {
"examples": {
"channels": [
{
"producerNodeQuery": {
"id": "ImportExampleGen"
},
"contextQueries": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"copy an image from one namespace to another namespace:
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen"
}
}
}
],
"artifactQuery": {
"type": {
"name": "Examples",
"baseType": "DATASET"
}
},
"outputKey": "examples"
}
],
"minCount": 1
},
"transform_graph": {
"channels": [
{
"producerNodeQuery": {
"id": "Transform"
},
"contextQueries": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.Transform"
}
}
}
],
"artifactQuery": {
"type": {
"name": "TransformGraph"
}
},
"outputKey": "transform_graph"
}
]
}
}
},
"outputs": {
"outputs": {
"model": {
"artifactSpec": {
"type": {
"name": "Model",
"baseType": "MODEL"
}
}
},
"model_run": {
"artifactSpec": {
"type": {
"name": "ModelRun"
}
}
}
}
},
"parameters": {
"parameters": {
"train_args": {
"fieldValue": {
"stringValue": "{\n \"num_steps\": 21\n}"
}
},
"custom_config": {
"fieldValue": {
"stringValue": "{\"epochs\": 50}"
}
},
"eval_args": {
"fieldValue": {
"stringValue": "{}"
}
},
"module_path": {
"fieldValue": {
"stringValue": "detect_anomalies_in_wafer_trainer@/tfx/pipelines/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl"
}
}
}
},
"upstreamNodes": [
"ImportExampleGen",
"Transform"
],
"downstreamNodes": [
"Pusher"
],
"executionOptions": {
"cachingOptions": {}
}
}
}
],
"runtimeSpec": {
"pipelineRoot": {
"runtimeParameter": {
"name": "pipeline-root",
"type": "STRING",
"defaultValue": {
"stringValue": "pipelines/detect_anomolies_on_wafer_tfdv_schema"
}
}
},
"pipelineRunId": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
"executionMode": "SYNC",
"deploymentConfig": {
"@type": "type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig",
"executorSpecs": {
"Trainer": {
"@type": "type.googleapis.com/tfx.orchestration.executable_spec.PythonClassExecutableSpec",
"classPath": "tfx.components.trainer.executor.GenericExecutor"
}
},
"metadataConnectionConfig": {
"@type": "type.googleapis.com/ml_metadata.ConnectionConfig",
"sqlite": {
"filenameUri": "metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db",
"connectionMode": "READWRITE_OPENCREATE"
}
}
}
}
- --metadata_ui_path
- /mlpipeline-ui-metadata.json
- --runtime_parameter
- pipeline-root=STRING:{{inputs.parameters.pipeline-root}}
command: [python, -m, tfx.orchestration.kubeflow.container_entrypoint]
env:
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_POD_NAME
valueFrom:
fieldRef: {fieldPath: metadata.name}
- name: KFP_POD_UID
valueFrom:
fieldRef: {fieldPath: metadata.uid}
- name: KFP_NAMESPACE
valueFrom:
fieldRef: {fieldPath: metadata.namespace}
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_RUN_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipeline/runid'']'}
- name: ENABLE_CACHING
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipelines.kubeflow.org/enable_caching'']'}
envFrom:
- configMapRef: {name: metadata-grpc-configmap, optional: true}
#image: tensorflow/tfx:1.14.0
#image: dockerproxy.com/tensorflow/tfx:1.14.0
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /tfx/pipelines/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
name: trainer-module
readOnly: True
- mountPath: /tfx/tfx_pv
name: tfx-pv
inputs:
parameters:
- {name: pipeline-root}
outputs:
artifacts:
- {name: mlpipeline-ui-metadata, path: /mlpipeline-ui-metadata.json}
metadata:
labels:
add-pod-env: "true"
pipelines.kubeflow.org/pipeline-sdk-type: tfx
pipelines.kubeflow.org/kfp_sdk_version: 1.8.0
pipelines.kubeflow.org/enable_caching: "true"
- name: transform
container:
args:
- --pipeline_root
- '{{inputs.parameters.pipeline-root}}'
- --kubeflow_metadata_config
- |-
{
"grpc_config": {
"grpc_service_host": {
"environment_variable": "METADATA_GRPC_SERVICE_HOST"
},
"grpc_service_port": {
"environment_variable": "METADATA_GRPC_SERVICE_PORT"
}
}
}
- --node_id
- Transform
- --tfx_ir
- |-
{
"pipelineInfo": {
"id": "detect_anomolies_on_wafer_tfdv_schema"
},
"nodes": [
{
"pipelineNode": {
"nodeInfo": {
"type": {
"name": "tfx.components.transform.component.Transform",
"baseType": "TRANSFORM"
},
"id": "Transform"
},
"contexts": {
"contexts": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.Transform"
}
}
}
]
},
"inputs": {
"inputs": {
"schema": {
"channels": [
{
"producerNodeQuery": {
"id": "schema_importer"
},
"contextQueries": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.schema_importer"
}
}
}
],
"artifactQuery": {
"type": {
"name": "Schema"
}
},
"outputKey": "result"
}
],
"minCount": 1
},
"examples": {
"channels": [
{
"producerNodeQuery": {
"id": "ImportExampleGen"
},
"contextQueries": [
{
"type": {
"name": "pipeline"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema"
}
}
},
{
"type": {
"name": "pipeline_run"
},
"name": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
{
"type": {
"name": "node"
},
"name": {
"fieldValue": {
"stringValue": "detect_anomolies_on_wafer_tfdv_schema.ImportExampleGen"
}
}
}
],
"artifactQuery": {
"type": {
"name": "Examples",
"baseType": "DATASET"
}
},
"outputKey": "examples"
}
],
"minCount": 1
}
}
},
"outputs": {
"outputs": {
"post_transform_anomalies": {
"artifactSpec": {
"type": {
"name": "ExampleAnomalies",
"properties": {
"split_names": "STRING",
"span": "INT"
}
}
}
},
"updated_analyzer_cache": {
"artifactSpec": {
"type": {
"name": "TransformCache"
}
}
},
"transform_graph": {
"artifactSpec": {
"type": {
"name": "TransformGraph"
}
}
},
"post_transform_schema": {
"artifactSpec": {
"type": {
"name": "Schema"
}
}
},
"pre_transform_schema": {
"artifactSpec": {
"type": {
"name": "Schema"
}
}
},
"post_transform_stats": {
"artifactSpec": {
"type": {
"name": "ExampleStatistics",
"properties": {
"split_names": "STRING",
"span": "INT"
},
"baseType": "STATISTICS"
}
}
},
"pre_transform_stats": {
"artifactSpec": {
"type": {
"name": "ExampleStatistics",
"properties": {
"split_names": "STRING",
"span": "INT"
},
"baseType": "STATISTICS"
}
}
}
}
},
"parameters": {
"parameters": {
"disable_statistics": {
"fieldValue": {
"intValue": "0"
}
},
"force_tf_compat_v1": {
"fieldValue": {
"intValue": "0"
}
},
"module_path": {
"fieldValue": {
"stringValue": "detect_anomalies_in_wafer_trainer@/tfx/pipelines/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl"
}
},
"custom_config": {
"fieldValue": {
"stringValue": "null"
}
}
}
},
"upstreamNodes": [
"ImportExampleGen",copy an image from one namespace to another namespace:
"schema_importer"
],
"downstreamNodes": [
"Trainer"
],
"executionOptions": {
"cachingOptions": {}
}
}
}
],
"runtimeSpec": {
"pipelineRoot": {
"runtimeParameter": {
"name": "pipeline-root",
"type": "STRING",
"defaultValue": {
"stringValue": "pipelines/detect_anomolies_on_wafer_tfdv_schema"
}
}
},
"pipelineRunId": {
"runtimeParameter": {
"name": "pipeline-run-id",
"type": "STRING"
}
}
},
"executionMode": "SYNC",
"deploymentConfig": {
"@type": "type.googleapis.com/tfx.orchestration.IntermediateDeploymentConfig",
"executorSpecs": {
"Transform": {
"@type": "type.googleapis.com/tfx.orchestration.executable_spec.BeamExecutableSpec",
"pythonExecutorSpec": {
"classPath": "tfx.components.transform.executor.Executor"
}
}
},
"metadataConnectionConfig": {
"@type": "type.googleapis.com/ml_metadata.ConnectionConfig",
"sqlite": {
"filenameUri": "metadata/detect_anomolies_on_wafer_tfdv_schema/metadata.db",
"connectionMode": "READWRITE_OPENCREATE"
}
}
}
}
- --metadata_ui_path
- /mlpipeline-ui-metadata.json
- --runtime_parameter
- pipeline-root=STRING:{{inputs.parameters.pipeline-root}}
command: [python, -m, tfx.orchestration.kubeflow.container_entrypoint]
env:
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_POD_NAME
valueFrom:
fieldRef: {fieldPath: metadata.name}
- name: KFP_POD_UID
valueFrom:
fieldRef: {fieldPath: metadata.uid}
- name: KFP_NAMESPACE
valueFrom:
fieldRef: {fieldPath: metadata.namespace}
- name: WORKFLOW_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''workflows.argoproj.io/workflow'']'}
- name: KFP_RUN_ID
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipeline/runid'']'}
- name: ENABLE_CACHING
valueFrom:
fieldRef: {fieldPath: 'metadata.labels[''pipelines.kubeflow.org/enable_caching'']'}
envFrom:
- configMapRef: {name: metadata-grpc-configmap, optional: true}
#image: tensorflow/tfx:1.14.0
#image: dockerproxy.com/tensorflow/tfx:1.14.0
image: docker.nju.edu.cn/tensorflow/tfx:1.13.0
imagePullPolicy: Never
volumeMounts:
- mountPath: /tfx/pipelines/tfx_user_code_Transform-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
name: transform-module
readOnly: True
- mountPath: /tfx/pipelines/detect_anomalies_in_wafer_schema
name: schema-path
readOnly: True
- mountPath: /tfx/tfx_pv
name: tfx-pv
inputs:
parameters:
- {name: pipeline-root}
outputs:
artifacts:
- {name: mlpipeline-ui-metadata, path: /mlpipeline-ui-metadata.json}
metadata:
labels:
add-pod-env: "true"
pipelines.kubeflow.org/pipeline-sdk-type: tfx
pipelines.kubeflow.org/kfp_sdk_version: 1.8.0
pipelines.kubeflow.org/enable_caching: "true"
arguments:
parameters:
- {name: pipeline-root, value: /tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema}
#- {name: pipeline-root, value: hdfs:///pipelines/detect_anomolies_on_wafer_tfdv_schema}
serviceAccountName: pipeline-runner
Attention:
- Container has standalone file system, and when a container finishes, its file system also finishes, namely all files in it nor exit any more. pipeline_root of tfx pipeline needs to be a persistent directory which can be read and written by all components of the tfx pipeline, so in this example, use persistent volume for it. And persistent volume is better to be a standalone disk, not share disk with OS, since one persistent volume can not limit not to access the whole disk, if sharing one disk with OS, may affect OS.
- Each component of a tfx pipeline runs in a container in one pod, namely one component one pod. And usually, pods run on different nodes of kubernetes cluster, this is why using kubernetes cluster -- distribute pods to multiple nodes, so in this example, use
nfsfor pipeline_root's volume. And, nfs is better to be a standalone disk, since nfs can not limit not to access the whole disk, if sharing one disk with OS, may affect OS, not secure.
Note:
- mirror websites of hub.docker.com
汇总国内可用镜像
DaoCloud 镜像站
加速地址：https://docker.m.daocloud.io
支持：Docker Hub、GCR、K8S、GHCR、Quay、NVCR 等
对外免费：是
网易云
加速地址：https://hub-mirror.c.163.com
支持：Docker Hub
对外免费：是
Docker 镜像代理
加速地址：https://dockerproxy.com
支持：Docker Hub、GCR、K8S、GHCR
对外免费：是
百度云
加速地址：https://mirror.baidubce.com
支持：Docker Hub
对外免费：是
南京大学镜像站
加速地址：https://docker.nju.edu.cn
支持：Docker Hub、GCR、GHCR、Quay、NVCR 等
对外免费：是
上海交大镜像站
加速地址：https://docker.mirrors.sjtug.sjtu.edu.cn/
支持：Docker Hub、GCR 等
限制：无
阿里云
加速地址：https://<your_code>.mirror.aliyuncs.com
支持：Docker Hub
限制：需要登录账号获取CODE [2]
- check trace stack of failed linux process which runs in backgroud
strace -e trace=none -p <PID>
Refernece:
