[ERROR: Directory '*py3-none-any.whl' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.]
kubectl lgos trainer-pod-name -n kubeflow
-->
, pipeline_info=id: "detect_anomolies_on_wafer_tfdv_schema"
, pipeline_run_id='detect-anomolies-on-wafer-tfdv-schema-lnz6m')
INFO:absl:Train on the 'train' split when train_args.splits is not set.
INFO:absl:Evaluate on the 'eval' split when eval_args.splits is not set.
INFO:absl:udf_utils.get_fn {'custom_config': '{"epochs": 50}', 'eval_args': '{}', 'module_path': 'detect_anomalies_in_wafer_trainer@/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl', 'train_args': '{\n "num_steps": 21\n}'} 'run_fn'
INFO:absl:Installing '/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl' to a temporary directory.
INFO:absl:Executing: ['/usr/bin/python', '-m', 'pip', 'install', '--target', '/tmp/tmp3dt6mzvy', '/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl']
ERROR: Directory '/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
[notice] A new release of pip is available: 23.1.2 -> 23.3.2
[notice] To update, run: pip install --upgrade pip
INFO:absl:MetadataStore with gRPC connection initialized
ERROR:absl:Execution 27 failed.
INFO:absl:Cleaning up stateless execution info.
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 510, in
main(sys.argv[1:])
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 502, in main
execution_info = component_launcher.launch()
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/launcher.py", line 574, in launch
executor_output = self._run_executor(execution_info)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/launcher.py", line 449, in _run_executor
executor_output = self._executor_operator.run_executor(execution_info)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/python_executor_operator.py", line 135, in run_executor
return run_with_executor(execution_info, executor)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/python_executor_operator.py", line 58, in run_with_executor
result = executor.Do(execution_info.input_dict, output_dict,
File "/usr/local/lib/python3.8/dist-packages/tfx/components/trainer/executor.py", line 174, in Do
run_fn = udf_utils.get_fn(exec_properties, 'run_fn')
File "/usr/local/lib/python3.8/dist-packages/tfx/components/util/udf_utils.py", line 56, in get_fn
return import_utils.import_func_from_module(module_path, fn_name)
File "/usr/local/lib/python3.8/dist-packages/tfx/utils/import_utils.py", line 59, in import_func_from_module
wheel_context_manager.enter()
File "/usr/local/lib/python3.8/dist-packages/tfx/components/util/udf_utils.py", line 307, in enter
install_to_temp_directory(dependency, temp_dir=self.temp_directory)
File "/usr/local/lib/python3.8/dist-packages/tfx/components/util/udf_utils.py", line 348, in install_to_temp_directory
subprocess.check_call(install_command)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-m', 'pip', 'install', '--target', '/tmp/tmp3dt6mzvy', '/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl']' returned non-zero exit status 1.
[ANALYSIS]
1. Run the failed command: /usr/bin/python, -m, pip, install, --target, /tmp/tmp3dt6mzvy, tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl-path-on-host , on host, to see if ok.
$ /home/maye/anaconda3/bin/python -m pip install --target /tmp/tmptse6gaxa pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
Processing ./pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
Installing collected packages: tfx-user-code-Trainer
Successfully installed tfx-user-code-Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db
$
And, *.whl is zip file, not need setup.py in it.
2. run the failed command in the container, to see if ok.
kubectl exec -it trainer-pod-name -n kubeflow -- bash
--> entered pod trainer:
root@detect-anomolies-on-wafer-tfdv-schema-lnz6m-331885160:/tfx/src# /usr/bin/python -m pip install --target /tmp/tmp3dt6mzvy /tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
ERROR: Directory '/tfx/pipelines/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
3. see if the needed packages wheel and setuptools for intalling *.whl, exist in the container.
kubectl exec -it trainer-pod-name -n kubeflow -- bash
--> entered pod trainer:
root@detect-anomolies-on-wafer-tfdv-schema-lnz6m-331885160:/usr/bin# /usr/bin/python
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import wheel
>>> import setuptools
>>>
4. cd the Directory '*py3-none-any.whl' which is not installable. Neither 'setup.py' nor 'pyproject.toml' found. to see what in it.
kubectl exec -it trainer-pod-name -n kubeflow -- bash
--> entered pod trainer:
root@detect-anomolies-on-wafer-tfdv-schema-lnz6m-331885160:/tfx/src# cd path-of-*py3-none-any.whl
root@detect-anomolies-on-wafer-tfdv-schema-lnz6m-331885160: path-of *py3-none-any.whl # ls
--> nothing shown
This means Directory *py3-none-any.whl is an empty directory.
5. check the hostPath which mounts to paht-of-*py3-none-any.whl, to see if ok.
There is directory: /home/maye/pipelnes/detect_anomolies_on_wafer_tfdv_schema/_wheels/tfx_user_code_Trainer-0.0+35148d2579a5a421da4bda3bd371de44bf8888bb4ea4f5cc424f859c6e4db9db-py3-none-any.whl
on host, which is an newly created empty directory.
The volumes.hostPath.path is spelled wrong: "pipelnes", the right one is: "pipelines", in the file pipeline.yaml, and if the volumes.hostPath.path not exists on host, it will be created.
And the true *.whl is File, not Directory.
Attention:
volumeMounts.mountPath can not contain parent directory which does not exists, if so, no error will be raised, bu the mount Path just not work.
[SOLUTION]
The volumes.hostPath.path is spelled wrong: "pipelnes", the right one is: "pipelines", in the file pipeline.yaml.
[ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/tfx/pipelines/tfx_user_code_Transform-0.0+45557d2573d165741daa21563f9ee3af93d0a19280c704d6d8657b3b24997e67-py3-none-any.whl']
upstream_nodes: "ImportExampleGen"
upstream_nodes: "schema_importer"
downstream_nodes: "Trainer"
execution_options {
caching_options {
}
}
, pipeline_info=id: "detect_anomolies_on_wafer_tfdv_schema"
, pipeline_run_id='detect-anomolies-on-wafer-tfdv-schema-nv76s')
INFO:absl:Analyze the 'train' split and transform all splits when splits_config is not set.
INFO:absl:udf_utils.get_fn {'module_file': None, 'module_path': 'detect_anomalies_in_wafer_trainer@/tfx/pipelines/tfx_user_code_Transform-0.0+45557d2573d165741daa21563f9ee3af93d0a19280c704d6d8657b3b24997e67-py3-none-any.whl', 'preprocessing_fn': None} 'preprocessing_fn'
INFO:absl:Installing '/tfx/pipelines/tfx_user_code_Transform-0.0+45557d2573d165741daa21563f9ee3af93d0a19280c704d6d8657b3b24997e67-py3-none-any.whl' to a temporary directory.
INFO:absl:Executing: ['/usr/bin/python', '-m', 'pip', 'install', '--target', '/tmp/tmpzkuihpwz', '/tfx/pipelines/tfx_user_code_Transform-0.0+45557d2573d165741daa21563f9ee3af93d0a19280c704d6d8657b3b24997e67-py3-none-any.whl']
WARNING: Requirement '/tfx/pipelines/tfx_user_code_Transform-0.0+45557d2573d165741daa21563f9ee3af93d0a19280c704d6d8657b3b24997e67-py3-none-any.whl' looks like a filename, but the file does not exist
Processing /tfx/pipelines/tfx_user_code_Transform-0.0+45557d2573d165741daa21563f9ee3af93d0a19280c704d6d8657b3b24997e67-py3-none-any.whl
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/tfx/pipelines/tfx_user_code_Transform-0.0+45557d2573d165741daa21563f9ee3af93d0a19280c704d6d8657b3b24997e67-py3-none-any.whl'
[notice] A new release of pip is available: 23.1.2 -> 24.0
[notice] To update, run: pip install --upgrade pip
INFO:absl:MetadataStore with gRPC connection initialized
ERROR:absl:Execution 25 failed.
INFO:absl:Cleaning up stateless execution info.
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 510, in <module>
main(sys.argv[1:])
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 502, in main
execution_info = component_launcher.launch()
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/launcher.py", line 574, in launch
executor_output = self._run_executor(execution_info)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/launcher.py", line 449, in _run_executor
executor_output = self._executor_operator.run_executor(execution_info)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/beam_executor_operator.py", line 112, in run_executor
return python_executor_operator.run_with_executor(execution_info, executor)
File "/usr/local/lib/python3.8/dist-packages/tfx/orchestration/portable/python_executor_operator.py", line 58, in run_with_executor
result = executor.Do(execution_info.input_dict, output_dict,
File "/usr/local/lib/python3.8/dist-packages/tfx/components/transform/executor.py", line 574, in Do
preprocessing_fn = self._GetPreprocessingFn(inputs_for_fn_resolution,
File "/usr/local/lib/python3.8/dist-packages/tfx/components/transform/executor.py", line 372, in _GetPreprocessingFn
fn = udf_utils.get_fn(
File "/usr/local/lib/python3.8/dist-packages/tfx/components/util/udf_utils.py", line 56, in get_fn
return import_utils.import_func_from_module(module_path, fn_name)
File "/usr/local/lib/python3.8/dist-packages/tfx/utils/import_utils.py", line 59, in import_func_from_module
wheel_context_manager.__enter__()
File "/usr/local/lib/python3.8/dist-packages/tfx/components/util/udf_utils.py", line 307, in __enter__
install_to_temp_directory(dependency, temp_dir=self.temp_directory)
File "/usr/local/lib/python3.8/dist-packages/tfx/components/util/udf_utils.py", line 348, in install_to_temp_directory
subprocess.check_call(install_command)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-m', 'pip', 'install', '--target', '/tmp/tmpzkuihpwz', '/tfx/pipelines/tfx_user_code_Transform-0.0+45557d2573d165741daa21563f9ee3af93d0a19280c704d6d8657b3b24997e67-py3-none-any.whl']' returned non-zero exit status 1.
time="2024-02-13T07:35:46.008Z" level=info msg="sub-process exited" argo=true error="<nil>"
time="2024-02-13T07:35:46.008Z" level=error msg="cannot save artifact /mlpipeline-ui-metadata.json" argo=true error="stat /mlpipeline-ui-metadata.json: no such file or directory"
Error: exit status 1
(base) maye@maye-Inspiron-5547:~$
[SOLUTION]
This error is due to that file '/tfx/pipelines/tfx_user_code_Transform-0.0+45557d2573d165741daa21563f9ee3af93d0a19280c704d6d8657b3b24997e67-py3-none-any.whl' does not exist, owing to filename wrong.
