[ERROR： mysql_real_connect failed: errno: , error]
example 1
kubectl logs tfx-component-Transform-pod-name -n kubeflow
-->
INFO:absl:MetadataStore with gRPC connection initialized
WARNING:absl:mlmd client InternalError: mysql_real_connect failed: errno: , error:
ERROR:absl:[Transform] Input resolution error: Error while resolving inputs for Transform
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/ml_metadata/metadata_store/metadata_store.py", line 219, in _call_method
response.CopyFrom(grpc_method(request, timeout=self._grpc_timeout_sec))
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 1030, in call
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "mysql_real_connect failed: errno: , error: "
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-01-30T09:52:37.395417001+00:00", grpc_status:13, grpc_message:"mysql_real_connect failed: errno: , error: "}"
kubectl logs metadata-grpc-deployment-pod-name -n kubeflow
-->
W0130 09:49:08.262073 4255 metadata_store_service_impl.cc:747] PutContexts failed: Given node already exists: type_id: 12
name: "detect-anomolies-on-wafer-tfdv-schema-lnz6m"
INTERNAL: Cannot create node for type_id: 12 name: "detect-anomolies-on-wafer-tfdv-schema-lnz6m"mysql_query failed: errno: Duplicate entry '12-detect-anomolies-on-wafer-tfdv-schema-lnz6m' for key 'Context.type_id', error: Duplicate entry '12-detect-anomolies-on-wafer-tfdv-schema-lnz6m' for key 'Context.type_id'
W0130 09:49:26.348727 4258 metadata_store_service_impl.cc:869] PutParentContexts failed: Given parent_context already exists: child_id: 27
parent_id: 1
INTERNAL: mysql_query failed: errno: Duplicate entry '27-1' for key 'ParentContext.PRIMARY', error: Duplicate entry '27-1' for key 'ParentContext.PRIMARY' [mysql-error-info='\x08\xa6\x08']
E0130 09:50:47.328933 4263 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 09:50:47.329303 4263 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 09:52:27.551858 4294 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 09:52:27.552182 4294 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 09:52:37.185184 4281 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 09:52:37.185544 4281 metadata_store_service_impl.cc:944] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 09:54:15.667371 4323 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 09:54:15.667706 4323 metadata_store_service_impl.cc:944] Failed to connect to the database: mysql_real_connect failed: errno: , error:
(base) maye@maye-Inspiron-5547:~$
example 2
kubectl logs ImportExampleGen-pod-name -n kubeflow
-->
downstream_nodes: "StatisticsGen"
downstream_nodes: "Trainer"
downstream_nodes: "Transform"
execution_options {
caching_options {
}
}
INFO:absl:MetadataStore with gRPC connection initialized
WARNING:absl:mlmd client InternalError: mysql_real_connect failed: errno: , error:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/ml_metadata/metadata_store/metadata_store.py", line 219, in _call_method
response.CopyFrom(grpc_method(request, timeout=self._grpc_timeout_sec))
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 1030, in call
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "mysql_real_connect failed: errno: , error: "
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"mysql_real_connect failed: errno: , error: ", grpc_status:13, created_time:"2024-01-30T16:22:08.164717626+00:00"}"
kubectl logs metadata-grpc-deployment-pod-name -n kubeflow
-->
W0130 09:52:27.552182 4294 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 09:52:37.185184 4281 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 09:52:37.185544 4281 metadata_store_service_impl.cc:944] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 09:54:15.667371 4323 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 09:54:15.667706 4323 metadata_store_service_impl.cc:944] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 13:39:44.226889 4366 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 13:39:44.227226 4366 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 15:01:49.176071 4370 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 15:01:49.176560 4370 metadata_store_service_impl.cc:944] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 15:01:56.749688 4371 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/to fix this issue.
W0130 15:01:56.750033 4371 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0130 15:02:18.890461 4377 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https
Retry the run, the same component runs ok:
-->
INFO:absl:Value type <class 'list'> of key _beam_pipeline_args in exec_properties is not supported, going to drop it
INFO:absl:Cleaning up stateless execution info.
INFO:absl:Execution 29 succeeded.
INFO:absl:Cleaning up stateful execution info.
INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: "/tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema/ImportExampleGen/examples/29"
custom_properties {
key: "input_fingerprint"
value {
string_value: "split:train,num_files:1,total_bytes:48638225,xor_checksum:1703857824,sum_checksum:1703857824\nsplit:eval,num_files:1,total_bytes:12654996,xor_checksum:1703857858,sum_checksum:1703857858"
}
}
custom_properties {
key: "span"
value {
int_value: 0
}
}
, artifact_type: name: "Examples"
properties {
key: "span"
value: INT
}
properties {
key: "split_names"
value: STRING
}
properties {
key: "version"
value: INT
}
base_type: DATASET
)]}) for execution 29
INFO:absl:MetadataStore with gRPC connection initialized
INFO:root:Component ImportExampleGen is finished.
time="2024-01-30T16:31:52.235Z" level=info msg="sub-process exited" argo=true error="
time="2024-01-30T16:31:52.235Z" level=info msg="/mlpipeline-ui-metadata.json -> /var/run/argo/outputs/artifacts/mlpipeline-ui-metadata.json.tgz" argo=true
time="2024-01-30T16:31:52.235Z" level=info msg="Taring /mlpipeline-ui-metadata.json"
(base) maye@maye-Inspiron-5547:~$
example 3:
- On ml-pipeline-ui run details web page, component SchemaImporte raise error:
This step is in Error state with this message: Error (exit code 1): failed to put file: Get "http://minio-service.kubeflow:9000/mlpipeline/?location=": dial tcp: lookup minio-service.kubeflow on 10.96.0.10:53: no such host
10.96.0.10:53 is nameserver:port, minio-service.kubeflow is domain name of minio sevice. This error means that domain name minio-service.kubeflow can not be found on nameserver (namely coredns), If connection timeout, it also raises this error.
kubectl logs ImportExampleGen-pod-name -n kubeflow:
INFO:absl:MetadataStore with gRPC connection initialized
INFO:absl:[ImportExampleGen] Resolved inputs: ({},)
WARNING:absl:mlmd client InternalError: mysql_real_connect failed: errno: , error:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/ml_metadata/metadata_store/metadata_store.py", line 219, in _call_method
response.CopyFrom(grpc_method(request, timeout=self._grpc_timeout_sec))
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 1030, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "mysql_real_connect failed: errno: , error: "
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"mysql_real_connect failed: errno: , error: ", grpc_status:13, created_time:"2024-02-11T15:36:48.865413939+00:00"}"
kubectl logs coredns-f75499545-442p5 -n kube-system:
(base) maye@maye-Inspiron-5547:~$ kubectl logs coredns-f75499545-442p5 -n kube-system
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
.:53
[INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2fa5f046dd37f1cf888b953814040d180f52594972691cd6ff41be96639138a43908
CoreDNS-1.9.3
linux/amd64, go1.18.2, 45b0a11
[WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[WARNING] plugin/health: Local health request to "http://:8080/health" failed: Get "http://:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[WARNING] plugin/health: Local health request to "http://:8080/health" took more than 1s: 1.222982373s
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:52979->192.168.0.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:55635->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:44341->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:52290->192.168.0.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:47994->192.168.0.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:60898->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:34241->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:41686->192.168.0.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:47675->192.168.0.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:35167->192.168.0.1:53: i/o timeout
[ERROR] plugin/errors: 2 mysql. A: read udp 10.244.1.184:39310->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:33139->192.168.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:55212->192.168.0.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:35375->192.168.0.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:32853->192.168.0.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:33556->192.168.0.1:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.184:60026->192.168.0.1:53: i/o timeout
After "Forwarding IPv4 and letting iptables see bridged traffic" on each node of the kubernetes cluster, a little better,
still sometimes "mysql_real_connect failed: errno: , error" :
root@maye-laptop:/home/maye# cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
> overlay
> br_netfilter
> EOF
overlay
br_netfilter
root@maye-laptop:/home/maye# sudo modprobe overlay
root@maye-laptop:/home/maye# sudo modprobe br_netfilter
root@maye-laptop:/home/maye# # sysctl params required by setup, params persist across reboots
root@maye-laptop:/home/maye# cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
> net.bridge.bridge-nf-call-iptables = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> net.ipv4.ip_forward = 1
> EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
root@maye-laptop:/home/maye# sudo sysctl --system
* Applying /etc/sysctl.d/10-console-messages.conf ...
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
* Applying /etc/sysctl.d/10-kernel-hardening.conf ...
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-link-restrictions.conf ...
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/10-magic-sysrq.conf ...
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
* Applying /etc/sysctl.d/10-ptrace.conf ...
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf ...
vm.mmap_min_addr = 65536
* Applying /usr/lib/sysctl.d/30-tracker.conf ...
fs.inotify.max_user_watches = 65536
* Applying /usr/lib/sysctl.d/50-default.conf ...
net.ipv4.conf.default.promote_secondaries = 1
sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
fs.protected_regular = 1
fs.protected_fifos = 1
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
ping the ips which has error in log of coredns, both ok:
(base) maye@maye-Inspiron-5547:~$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=63 time=2.10 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=63 time=2.10 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=63 time=2.37 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=63 time=1.78 ms
^C
--- 192.168.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 1.776/2.086/2.367/0.209 ms
(base) maye@maye-Inspiron-5547:~$ ping 10.244.1.184
PING 10.244.1.184 (10.244.1.184) 56(84) bytes of data.
64 bytes from 10.244.1.184: icmp_seq=1 ttl=63 time=2.04 ms
64 bytes from 10.244.1.184: icmp_seq=2 ttl=63 time=2.08 ms
64 bytes from 10.244.1.184: icmp_seq=3 ttl=63 time=3.59 ms
64 bytes from 10.244.1.184: icmp_seq=4 ttl=63 time=3.14 ms
^C
--- 10.244.1.184 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 2.035/2.711/3.592/0.674 ms
(base) maye@maye-Inspiron-5547:~$
Attention:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X reset iptables, after it, "Forwarding IPv4 and letting iptables see bridged traffic" is cleaned up, need to do it again.
[1]
[ANALYSIS]
1. What's the meaning of "mysql_real_connect failed: errno: , error" ?
"mysql_real_connect failed: errno: , error":
mysql_real_connect failed, but errno and error are empty string, errno and error are from error message of database server, so this means that the database server has not received the request of connecting at all, not something wrong during connecting, if so, the database server will give error message.
2. so why the database server does not receive the request of connecting ?
From the log of pod metadata-grpc-deployment, It can be seen that in example 1 "mysql_real_connect failed: errno: , error" occured after "mysql_query failed: errno: Duplicate entry...", so is there any correlation between them ? but before transform, there are other components (such as StatisticsGen), and all components have the same context pipeline-run, and all components need to put context pipeline-run, and meet the error: "Duplicated entry", but other components before transform just ignore the error, and have not re-connected to mysql and failed. So there should be no correlation between "Error: Dulicated entry" and "mysql_real_connect failed: errno: , error".
In example 2, before "E0130 15:01:56.749688 4371 mysql_metadata_source.cc:174] MySQL database was not initialized.", the last log message is "W0130 13:39:44.227226 4366 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error: ", and the datetime is about 1.5h ago, This means that at datetime 0130 13:39:44.227226, "metadata_store_service_impl.cc:821] Failed to connect to the database" occured, and then metadata-grpc-deployment retried connecting to mysql and succeed since no "metadata_store_service_impl.cc:821] Failed to connect to the database" appeared in log after 0130 13:39:44.227226 until about 1.5h later. then at 0130 15:01:56.749688, metadata-grpc-deployment retried connecting to mysql and failed. And before it, no any warning or error said stopping connection to mysql due to something going wrong, so the stopping connection to mysql is normal operation, not due to something going wrong, it should be that the connection to mysql will be stopped if the connection is not interactive for some timeout. And after retrying the run, the same component failed due to "mysql_real_connection fail" runs ok.
In summary, based on upper analysis, it can be inferred that "mysql_real_connection fail" is not due to something going wrong in metadata-grpc-deployment or mysql, it is due to not good network state temporally.
# ml-metadata/ml_metadata/metadata_store/metadata_store_service_impl.cc
::grpc::Status MetadataStoreServiceImpl::PutParentContexts(
::grpc::ServerContext* context, const PutParentContextsRequest* request,
PutParentContextsResponse* response) {
std::unique_ptr<MetadataStore> metadata_store;
const ::grpc::Status connection_status =
ConnectMetadataStore(connection_config_, &metadata_store);
if (!connection_status.ok()) {
LOG(WARNING) << "Failed to connect to the database: "
<< connection_status.error_message();
return connection_status;
}
const ::grpc::Status transaction_status =
ToGRPCStatus(metadata_store->PutParentContexts(*request, response));
if (!transaction_status.ok()) {
LOG(WARNING) << "PutParentContexts failed: "
<< transaction_status.error_message();
}
return transaction_status;
}
# ml-metadata/ml_metadata/metadata_store/mysql_metadata_source.cc
Status MySqlMetadataSource::ConnectImpl() {
// Initialize the MYSQL object.
db_ = mysql_init(nullptr);
if (!db_) {
LOG(ERROR) << "MySQL error: " << mysql_errno(db_) << ": "
<< mysql_error(db_);
return BuildErrorStatus(absl::StatusCode::kInternal, "mysql_init failed",
mysql_errno(db_), mysql_error(db_));
}
// Explicitly setup the thread-local initializer.
MLMD_RETURN_WITH_CONTEXT_IF_ERROR(ThreadInitAccess(),
"MySql thread init failed at ConnectImpl");
// Set connection options
if (config_.has_ssl_options()) {
const MySQLDatabaseConfig::SSLOptions& ssl = config_.ssl_options();
// The method set mysql_options, and always return 0. The connection options
// are used in the `mysql_real_connect`.
mysql_ssl_set(db_, ssl.key().empty() ? nullptr : ssl.key().c_str(),
ssl.cert().empty() ? nullptr : ssl.cert().c_str(),
ssl.ca().empty() ? nullptr : ssl.ca().c_str(),
ssl.capath().empty() ? nullptr : ssl.capath().c_str(),
ssl.cipher().empty() ? nullptr : ssl.cipher().c_str());
my_bool verify_server_cert = ssl.verify_server_cert() ? 1 : 0;
mysql_options(db_, MYSQL_OPT_SSL_VERIFY_SERVER_CERT, &verify_server_cert);
}
mysql_options(db_, MYSQL_DEFAULT_AUTH, "mysql_native_password");
// Connect to the MYSQL server.
db_ = mysql_real_connect(
db_, config_.host().empty() ? nullptr : config_.host().c_str(),
config_.user().empty() ? nullptr : config_.user().c_str(),
config_.password().empty() ? nullptr : config_.password().c_str(),
/*db=*/nullptr, config_.port(),
config_.socket().empty() ? nullptr : config_.socket().c_str(),
/*clientflag=*/0UL);
if (!db_) {
LOG(ERROR)
<< "MySQL database was not initialized. Please ensure your "
"MySQL server is running. Also, this error might be caused by "
"starting from MySQL 8.0, mysql_native_password used by MLMD is not "
"supported as a default for authentication plugin. Please follow "
"<https://dev.mysql.com/blog-archive/"
"upgrading-to-mysql-8-0-default-authentication-plugin-"
"considerations/>"
"to fix this issue.";
return BuildErrorStatus(absl::StatusCode::kInternal,
"mysql_real_connect failed", mysql_errno(db_),
mysql_error(db_));
}
// Return an error if the default storage engine doesn't support transactions.
MLMD_RETURN_WITH_CONTEXT_IF_ERROR(
CheckTransactionSupport(),
"checking transaction support of default storage engine");
// Create the database if not already present and skip_db_creation is false.
if (!config_.skip_db_creation()) {
const std::string create_database_cmd =
absl::StrCat("CREATE DATABASE IF NOT EXISTS ", config_.database());
MLMD_RETURN_WITH_CONTEXT_IF_ERROR(RunQuery(create_database_cmd),
"Creating database ", config_.database(),
" in ConnectImpl");
}
// Switch to the database.
if (database_name_.empty()) {
database_name_ = config_.database();
}
const std::string use_database_cmd = absl::StrCat("USE ", database_name_);
MLMD_RETURN_WITH_CONTEXT_IF_ERROR(RunQuery(use_database_cmd),
"Changing to database ", database_name_,
" in ConnectImpl");
return absl::OkStatus();
}
3. "read udp xxx->xxx: i/o timeout" in log of coredns means domain name resolving timeout.
在使用 service name 即域名的场景下，Client Pod3 首先拿着域名去 coredns 解析成 ClusterIP，接着去请求 ClusterIP，最后通过 Kube-Proxy 把请求转发到目标后端 Pod。
由 coredns 后台关键日志 A: read udp xxx->xxx: i/o timeout可知 IPV4 解析超时，AAAA: read udp xxx->xxx: i/o timeout 可知 IPV6解析也超时。
3.1 check /etc/resolv.conf of pod metadata-grpc-deployment, domain name resolve configuration file
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# cat /etc/resolv.conf
search kubeflow.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
root@metadata-grpc-deployment-659594dfcb-g9jfl:/#
Note:
resolv.conf 文件的第二行是 nameserver，内容是 coredns 的 ClusterIP.
resolv.conf 文件的第一行是 search。当域名解析的时候，将域名依次添加后缀，例如：
flink-jobmanager.default.svc.cluster.local.default.svc.cluster.local
flink-jobmanager.default.svc.cluster.local.svc.cluster.local
flink-jobmanager.default.svc.cluster.local.cluster.local
flink-jobmanager.default.svc.cluster.local.openstacklocal
resolv.conf 文件的第三行是 options 其它项，常见配置是 ndots。ndots: 5 表示如果域名包含的 "." 少于5个，则先添加 search 后缀，再使用绝对域名；如果域名包含的 "." 大于等于5个，则先使用绝对域名，再添加 search 后缀。
#样例1 域名a.b.c.d.e，则域名的顺序如下
a.b.c.d.e.default.svc.cluster.local
a.b.c.d.e.svc.cluster.local
a.b.c.d.e.cluster.local
a.b.c.d.e.openstacklocal
a.b.c.d.e
#样例2 域名a.b.c.d.e.f，则域名的顺序如下
a.b.c.d.e.f
a.b.c.d.e.f.default.svc.cluster.local
a.b.c.d.e.f.svc.cluster.local
a.b.c.d.e.f.cluster.local
a.b.c.d.e.f.openstacklocal
3.2 抓取 Pod 网络数据包
mysql service ip and endpoint ip (namely mysql pod ip),
/etc/hosts of pod metadata-grpc-deployment:
(base) maye@maye-Inspiron-5547:~$ kubectl describe svc mysql -n kubeflow
Name: mysql
Namespace: kubeflow
Labels: application-crd-id=kubeflow-pipelines
Annotations: <none>
Selector: app=mysql,application-crd-id=kubeflow-pipelines
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.105.203.173
IPs: 10.105.203.173
Port: <unset> 3306/TCP
TargetPort: 3306/TCP
Endpoints: 10.244.0.61:3306
Session Affinity: None
Events: <none>
(base) maye@maye-Inspiron-5547:~$
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.244.0.53 metadata-grpc-deployment-659594dfcb-g9jfl
10.105.203.173 mysql
抓取 Pod 网络数据包:
# get container_id of pod mysql
(base) maye@maye-Inspiron-5547:~$ sudo crictl ps | grep mysql
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0000] validate service connection: validate CRI v1 runtime API for endpoint "unix:///var/run/dockershim.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/dockershim.sock: connect: no such file or directory"
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/dockershim.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/dockershim.sock: connect: no such file or directory"
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
2232b94224a7c 9da615fced53b 2 days ago Running mysql 0 26879802e4721 mysql-6b95d686-zsp98
(base) maye@maye-Inspiron-5547:~$
# get process id of container mysql
(base) maye@maye-Inspiron-5547:~$ sudo crictl inspect 2232b94224a7c | grep pid
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0000] validate service connection: validate CRI v1 runtime API for endpoint "unix:///var/run/dockershim.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/dockershim.sock: connect: no such file or directory"
"pid": 758744,
"pid": 1
"type": "pid"
(base) maye@maye-Inspiron-5547:~$
# enter net command space of container mysql from host,
# -t: target pid -n: 进入net命令空间
(base) maye@maye-Inspiron-5547:~$ nsenter -t 758744 -n
# capture network data packets of container mysql
# 10.105.203.173 is mysql pod ip, not mysql service ip,
# if using mysql service ip, no network data packet
# will be captured.
root@maye-Inspiron-5547:/home/maye# tcpdump 'src host 10.105.203.173 and src port 3306'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
# open a new terminal, enter pod metadata-grpc-deployment, and
# write curl-format file, for easy reading output of curl
# if "command not found", apt-get install vim
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# vim curl-format.txt
time_namelookup: %{time_namelookup}\n
time_connect: %{time_connect}\n
time_appconnect: %{time_appconnect}\n
time_redirect: %{time_redirect}\n
time_pretransfer: %{time_pretransfer}\n
time_starttransfer: %{time_starttransfer}\n
----------\n
time_total: %{time_total}\n
# "curl mysql" in pod metadata-grpc-deployment
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# curl -w "@curl-format.txt" -o /dev/null -l "http://mysql:3306"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (1) Received HTTP/0.9 when not allowed
time_namelookup: 0.000618
time_connect: 0.000865
time_appconnect: 0.000000
time_redirect: 0.000000
time_pretransfer: 0.000961
time_starttransfer: 0.001168
----------
time_total: 0.001173
# check output of tcpdump
root@maye-Inspiron-5547:/home/maye# tcpdump 'src host 10.244.0.61 and src port 3306'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:34:49.148968 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [.], ack 2695261422, win 501, options [nop,nop,TS val 829506470 ecr 3688927843], length 0
10:35:04.253051 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [.], ack 1, win 501, options [nop,nop,TS val 829521574 ecr 3688927843], length 0
10:35:18.893955 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [.], ack 2639, win 497, options [nop,nop,TS val 829536215 ecr 3688987834], length 0
10:35:18.894487 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 0:2279, ack 2639, win 501, options [nop,nop,TS val 829536215 ecr 3688987834], length 2279
10:35:18.895261 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 2279:9269, ack 2694, win 501, options [nop,nop,TS val 829536216 ecr 3688987834], length 6990
10:35:18.895322 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 9269:16259, ack 2694, win 501, options [nop,nop,TS val 829536216 ecr 3688987834], length 6990
10:35:18.895349 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 16259:18663, ack 2694, win 501, options [nop,nop,TS val 829536216 ecr 3688987835], length 2404
10:35:18.895402 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 18663:32643, ack 2694, win 501, options [nop,nop,TS val 829536216 ecr 3688987835], length 13980
10:35:18.895432 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 32643:46623, ack 2694, win 501, options [nop,nop,TS val 829536216 ecr 3688987835], length 13980
10:35:18.895469 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 46623:74583, ack 2694, win 501, options [nop,nop,TS val 829536216 ecr 3688987835], length 27960
10:35:18.895506 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 74583:98349, ack 2694, win 501, options [nop,nop,TS val 829536216 ecr 3688987835], length 23766
10:35:18.895521 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 98349:102543, ack 2694, win 501, options [nop,nop,TS val 829536216 ecr 3688987835], length 4194
10:35:18.895538 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 102543:105948, ack 2694, win 501, options [nop,nop,TS val 829536216 ecr 3688987835], length 3405
10:35:18.895666 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 105948:105957, ack 2694, win 501, options [nop,nop,TS val 829536216 ecr 3688987835], length 9
10:35:18.908310 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [.], ack 2819, win 501, options [nop,nop,TS val 829536229 ecr 3688987835], length 0
10:35:18.908471 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 105957:106474, ack 2819, win 501, options [nop,nop,TS val 829536229 ecr 3688987835], length 517
10:35:18.908704 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [P.], seq 106474:107095, ack 2874, win 501, options [nop,nop,TS val 829536229 ecr 3688987848], length 621
10:35:18.918327 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [S.], seq 3272080193, ack 728094467, win 64308, options [mss 1410,sackOK,TS val 2446773609 ecr 2119889463,nop,wscale 7], length 0
10:35:18.918611 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 1:79, ack 1, win 503, options [nop,nop,TS val 2446773609 ecr 2119889463], length 78
10:35:18.918679 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [.], ack 165, win 502, options [nop,nop,TS val 2446773609 ecr 2119889463], length 0
10:35:18.918769 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 79:90, ack 165, win 502, options [nop,nop,TS val 2446773609 ecr 2119889463], length 11
10:35:18.919111 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 90:280, ack 276, win 502, options [nop,nop,TS val 2446773610 ecr 2119889463], length 190
10:35:18.919385 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 280:291, ack 317, win 502, options [nop,nop,TS val 2446773610 ecr 2119889464], length 11
10:35:18.919529 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 291:313, ack 332, win 502, options [nop,nop,TS val 2446773610 ecr 2119889464], length 22
10:35:18.919857 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 313:337, ack 354, win 502, options [nop,nop,TS val 2446773610 ecr 2119889464], length 24
10:35:18.920079 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 337:441, ack 400, win 502, options [nop,nop,TS val 2446773611 ecr 2119889465], length 104
10:35:18.920312 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 441:877, ack 524, win 502, options [nop,nop,TS val 2446773611 ecr 2119889465], length 436
10:35:18.920524 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 877:1055, ack 593, win 502, options [nop,nop,TS val 2446773611 ecr 2119889465], length 178
10:35:18.920703 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 1055:1300, ack 667, win 502, options [nop,nop,TS val 2446773611 ecr 2119889465], length 245
10:35:18.920865 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 1300:1892, ack 813, win 501, options [nop,nop,TS val 2446773611 ecr 2119889465], length 592
10:35:18.921090 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 1892:2712, ack 1008, win 501, options [nop,nop,TS val 2446773612 ecr 2119889466], length 820
10:35:18.921306 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 2712:3271, ack 1159, win 501, options [nop,nop,TS val 2446773612 ecr 2119889466], length 559
10:35:18.921582 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 3271:4159, ack 1356, win 501, options [nop,nop,TS val 2446773612 ecr 2119889466], length 888
10:35:18.921818 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 4159:4524, ack 1472, win 501, options [nop,nop,TS val 2446773612 ecr 2119889466], length 365
10:35:18.922013 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 4524:4841, ack 1566, win 501, options [nop,nop,TS val 2446773613 ecr 2119889466], length 317
10:35:18.922197 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 4841:4945, ack 1612, win 501, options [nop,nop,TS val 2446773613 ecr 2119889467], length 104
10:35:18.922390 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 4945:5401, ack 1741, win 501, options [nop,nop,TS val 2446773613 ecr 2119889467], length 456
10:35:18.922587 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 5401:5602, ack 1819, win 501, options [nop,nop,TS val 2446773613 ecr 2119889467], length 201
10:35:18.922807 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 5602:6375, ack 2012, win 501, options [nop,nop,TS val 2446773613 ecr 2119889467], length 773
10:35:18.923018 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 6375:6618, ack 2089, win 501, options [nop,nop,TS val 2446773614 ecr 2119889467], length 243
10:35:18.923228 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 6618:6859, ack 2165, win 501, options [nop,nop,TS val 2446773614 ecr 2119889468], length 241
10:35:18.923374 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 6859:6883, ack 2176, win 501, options [nop,nop,TS val 2446773614 ecr 2119889468], length 24
10:35:18.923471 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 6883:6907, ack 2198, win 501, options [nop,nop,TS val 2446773614 ecr 2119889468], length 24
10:35:18.923960 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 6907:7353, ack 2439, win 501, options [nop,nop,TS val 2446773615 ecr 2119889468], length 446
10:35:18.924271 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 7353:7579, ack 2557, win 501, options [nop,nop,TS val 2446773615 ecr 2119889469], length 226
10:35:18.924710 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 7579:8047, ack 2813, win 501, options [nop,nop,TS val 2446773615 ecr 2119889469], length 468
10:35:18.925132 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 8047:8273, ack 2934, win 501, options [nop,nop,TS val 2446773616 ecr 2119889469], length 226
10:35:18.925311 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 8273:8297, ack 2945, win 501, options [nop,nop,TS val 2446773616 ecr 2119889470], length 24
10:35:18.925409 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 8297:8321, ack 2967, win 501, options [nop,nop,TS val 2446773616 ecr 2119889470], length 24
10:35:18.925710 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 8321:8698, ack 3156, win 501, options [nop,nop,TS val 2446773616 ecr 2119889470], length 377
10:35:18.925959 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 8698:8771, ack 3270, win 501, options [nop,nop,TS val 2446773617 ecr 2119889470], length 73
10:35:18.926112 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [P.], seq 8771:8795, ack 3281, win 501, options [nop,nop,TS val 2446773617 ecr 2119889471], length 24
10:35:18.926333 IP 10.244.0.61.mysql > 10.244.0.53.50552: Flags [F.], seq 8795, ack 3287, win 501, options [nop,nop,TS val 2446773617 ecr 2119889471], length 0
10:35:18.948936 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [.], ack 2883, win 501, options [nop,nop,TS val 829536270 ecr 3688987848], length 0
10:35:18.991204 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [S.], seq 3217401417, ack 3032625254, win 64308, options [mss 1410,sackOK,TS val 2446773682 ecr 2119889536,nop,wscale 7], length 0
10:35:18.991478 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 1:79, ack 1, win 503, options [nop,nop,TS val 2446773682 ecr 2119889536], length 78
10:35:18.991584 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [.], ack 165, win 502, options [nop,nop,TS val 2446773682 ecr 2119889536], length 0
10:35:18.991668 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 79:90, ack 165, win 502, options [nop,nop,TS val 2446773682 ecr 2119889536], length 11
10:35:18.992038 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 90:280, ack 276, win 502, options [nop,nop,TS val 2446773683 ecr 2119889536], length 190
10:35:18.992266 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 280:291, ack 317, win 502, options [nop,nop,TS val 2446773683 ecr 2119889537], length 11
10:35:18.992397 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 291:313, ack 332, win 502, options [nop,nop,TS val 2446773683 ecr 2119889537], length 22
10:35:18.992787 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 313:337, ack 354, win 502, options [nop,nop,TS val 2446773683 ecr 2119889537], length 24
10:35:18.993075 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 337:441, ack 400, win 502, options [nop,nop,TS val 2446773684 ecr 2119889537], length 104
10:35:18.993405 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 441:877, ack 524, win 502, options [nop,nop,TS val 2446773684 ecr 2119889538], length 436
10:35:18.993635 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 877:1055, ack 593, win 502, options [nop,nop,TS val 2446773684 ecr 2119889538], length 178
10:35:18.993871 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 1055:1300, ack 667, win 502, options [nop,nop,TS val 2446773684 ecr 2119889538], length 245
10:35:18.994327 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 1300:1892, ack 813, win 501, options [nop,nop,TS val 2446773685 ecr 2119889539], length 592
10:35:18.994654 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 1892:2712, ack 1008, win 501, options [nop,nop,TS val 2446773685 ecr 2119889539], length 820
10:35:18.994909 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 2712:3271, ack 1159, win 501, options [nop,nop,TS val 2446773686 ecr 2119889539], length 559
10:35:18.995152 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 3271:4159, ack 1356, win 501, options [nop,nop,TS val 2446773686 ecr 2119889540], length 888
10:35:18.995373 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 4159:4524, ack 1472, win 501, options [nop,nop,TS val 2446773686 ecr 2119889540], length 365
10:35:18.995612 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 4524:4841, ack 1566, win 501, options [nop,nop,TS val 2446773686 ecr 2119889540], length 317
10:35:18.996031 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 4841:4945, ack 1612, win 501, options [nop,nop,TS val 2446773687 ecr 2119889540], length 104
10:35:18.996297 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 4945:5401, ack 1741, win 501, options [nop,nop,TS val 2446773687 ecr 2119889541], length 456
10:35:18.996461 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 5401:5602, ack 1819, win 501, options [nop,nop,TS val 2446773687 ecr 2119889541], length 201
10:35:18.996628 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 5602:6375, ack 2012, win 501, options [nop,nop,TS val 2446773687 ecr 2119889541], length 773
10:35:18.996778 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 6375:6618, ack 2089, win 501, options [nop,nop,TS val 2446773687 ecr 2119889541], length 243
10:35:18.996966 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 6618:6859, ack 2165, win 501, options [nop,nop,TS val 2446773688 ecr 2119889541], length 241
10:35:18.997081 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 6859:6883, ack 2176, win 501, options [nop,nop,TS val 2446773688 ecr 2119889542], length 24
10:35:18.997156 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 6883:6907, ack 2198, win 501, options [nop,nop,TS val 2446773688 ecr 2119889542], length 24
10:35:18.997546 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 6907:7353, ack 2439, win 501, options [nop,nop,TS val 2446773688 ecr 2119889542], length 446
10:35:18.997779 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 7353:7579, ack 2557, win 501, options [nop,nop,TS val 2446773688 ecr 2119889542], length 226
10:35:18.998108 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 7579:8047, ack 2813, win 501, options [nop,nop,TS val 2446773689 ecr 2119889542], length 468
10:35:18.998344 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 8047:8273, ack 2934, win 501, options [nop,nop,TS val 2446773689 ecr 2119889543], length 226
10:35:18.998477 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 8273:8297, ack 2945, win 501, options [nop,nop,TS val 2446773689 ecr 2119889543], length 24
10:35:18.998557 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 8297:8321, ack 2967, win 501, options [nop,nop,TS val 2446773689 ecr 2119889543], length 24
10:35:18.998765 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 8321:8674, ack 3150, win 501, options [nop,nop,TS val 2446773689 ecr 2119889543], length 353
10:35:18.998868 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [P.], seq 8674:8698, ack 3161, win 501, options [nop,nop,TS val 2446773689 ecr 2119889543], length 24
10:35:18.999055 IP 10.244.0.61.mysql > 10.244.0.53.50560: Flags [F.], seq 8698, ack 3167, win 501, options [nop,nop,TS val 2446773690 ecr 2119889544], length 0
10:35:19.899884 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [.], ack 2888, win 501, options [nop,nop,TS val 829537220 ecr 3688988839], length 0
10:35:19.900019 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [F.], seq 107095, ack 2888, win 501, options [nop,nop,TS val 829537221 ecr 3688988839], length 0
10:35:19.900118 IP 10.244.0.61.mysql > 10.244.0.55.33038: Flags [.], ack 2889, win 501, options [nop,nop,TS val 829537221 ecr 3688988840], length 0
10:36:23.897945 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [S.], seq 3589510349, ack 1396676999, win 64308, options [mss 1410,sackOK,TS val 829601219 ecr 3689052837,nop,wscale 7], length 0
10:36:23.898484 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 1:79, ack 1, win 503, options [nop,nop,TS val 829601219 ecr 3689052838], length 78
10:36:23.898658 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [.], ack 76, win 503, options [nop,nop,TS val 829601219 ecr 3689052838], length 0
10:36:23.898946 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 79:90, ack 76, win 503, options [nop,nop,TS val 829601220 ecr 3689052838], length 11
10:36:23.899310 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 90:172, ack 108, win 503, options [nop,nop,TS val 829601220 ecr 3689052839], length 82
10:36:23.899651 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 172:183, ack 127, win 503, options [nop,nop,TS val 829601220 ecr 3689052839], length 11
10:36:23.899834 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 183:194, ack 164, win 503, options [nop,nop,TS val 829601220 ecr 3689052839], length 11
10:36:23.899984 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [.], ack 2802, win 497, options [nop,nop,TS val 829601221 ecr 3689052840], length 0
10:36:23.900693 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 194:2473, ack 2802, win 501, options [nop,nop,TS val 829601221 ecr 3689052840], length 2279
10:36:23.902201 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 2473:9463, ack 2857, win 501, options [nop,nop,TS val 829601223 ecr 3689052840], length 6990
10:36:23.902250 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 9463:16453, ack 2857, win 501, options [nop,nop,TS val 829601223 ecr 3689052840], length 6990
10:36:23.902332 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 16453:30433, ack 2857, win 501, options [nop,nop,TS val 829601223 ecr 3689052842], length 13980
10:36:23.902348 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 30433:44413, ack 2857, win 501, options [nop,nop,TS val 829601223 ecr 3689052842], length 13980
10:36:23.902503 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 44413:48607, ack 2857, win 501, options [nop,nop,TS val 829601223 ecr 3689052842], length 4194
10:36:23.902516 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 48607:76567, ack 2857, win 501, options [nop,nop,TS val 829601223 ecr 3689052842], length 27960
10:36:23.902522 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 76567:80761, ack 2857, win 501, options [nop,nop,TS val 829601223 ecr 3689052842], length 4194
10:36:23.902529 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 80761:100333, ack 2857, win 501, options [nop,nop,TS val 829601223 ecr 3689052842], length 19572
10:36:23.902607 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 100333:106151, ack 2857, win 501, options [nop,nop,TS val 829601223 ecr 3689052842], length 5818
10:36:23.911505 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [.], ack 2982, win 501, options [nop,nop,TS val 829601232 ecr 3689052842], length 0
10:36:23.911655 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 106151:106668, ack 2982, win 501, options [nop,nop,TS val 829601232 ecr 3689052842], length 517
10:36:23.911881 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 106668:107289, ack 3037, win 501, options [nop,nop,TS val 829601232 ecr 3689052851], length 621
10:36:23.919709 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [S.], seq 3711455167, ack 3637201687, win 64308, options [mss 1410,sackOK,TS val 2446838610 ecr 2119954464,nop,wscale 7], length 0
10:36:23.919953 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 1:79, ack 1, win 503, options [nop,nop,TS val 2446838611 ecr 2119954464], length 78
10:36:23.920149 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [.], ack 165, win 502, options [nop,nop,TS val 2446838611 ecr 2119954465], length 0
10:36:23.920269 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 79:90, ack 165, win 502, options [nop,nop,TS val 2446838611 ecr 2119954465], length 11
10:36:23.920587 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 90:280, ack 276, win 502, options [nop,nop,TS val 2446838611 ecr 2119954465], length 190
10:36:23.920776 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 280:291, ack 317, win 502, options [nop,nop,TS val 2446838611 ecr 2119954465], length 11
10:36:23.920879 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 291:313, ack 332, win 502, options [nop,nop,TS val 2446838611 ecr 2119954465], length 22
10:36:23.921244 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 313:337, ack 354, win 502, options [nop,nop,TS val 2446838612 ecr 2119954466], length 24
10:36:23.921433 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 337:441, ack 400, win 502, options [nop,nop,TS val 2446838612 ecr 2119954466], length 104
10:36:23.921672 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 441:877, ack 524, win 502, options [nop,nop,TS val 2446838612 ecr 2119954466], length 436
10:36:23.921837 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 877:1055, ack 593, win 502, options [nop,nop,TS val 2446838612 ecr 2119954466], length 178
10:36:23.922057 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 1055:1300, ack 667, win 502, options [nop,nop,TS val 2446838613 ecr 2119954467], length 245
10:36:23.922301 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 1300:1892, ack 813, win 501, options [nop,nop,TS val 2446838613 ecr 2119954467], length 592
10:36:23.922543 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 1892:2712, ack 1008, win 501, options [nop,nop,TS val 2446838613 ecr 2119954467], length 820
10:36:23.922765 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 2712:3271, ack 1159, win 501, options [nop,nop,TS val 2446838613 ecr 2119954467], length 559
10:36:23.922981 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 3271:4159, ack 1356, win 501, options [nop,nop,TS val 2446838614 ecr 2119954467], length 888
10:36:23.923188 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 4159:4524, ack 1472, win 501, options [nop,nop,TS val 2446838614 ecr 2119954468], length 365
10:36:23.923375 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 4524:4841, ack 1566, win 501, options [nop,nop,TS val 2446838614 ecr 2119954468], length 317
10:36:23.923563 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 4841:4945, ack 1612, win 501, options [nop,nop,TS val 2446838614 ecr 2119954468], length 104
10:36:23.923776 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 4945:5401, ack 1741, win 501, options [nop,nop,TS val 2446838614 ecr 2119954468], length 456
10:36:23.925204 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 5401:5602, ack 1819, win 501, options [nop,nop,TS val 2446838616 ecr 2119954468], length 201
10:36:23.925684 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 5602:6375, ack 2012, win 501, options [nop,nop,TS val 2446838616 ecr 2119954470], length 773
10:36:23.926054 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 6375:6618, ack 2089, win 501, options [nop,nop,TS val 2446838617 ecr 2119954470], length 243
10:36:23.926599 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 6618:6859, ack 2165, win 501, options [nop,nop,TS val 2446838617 ecr 2119954471], length 241
10:36:23.926811 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 6859:6883, ack 2176, win 501, options [nop,nop,TS val 2446838617 ecr 2119954471], length 24
10:36:23.926925 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 6883:6907, ack 2198, win 501, options [nop,nop,TS val 2446838618 ecr 2119954471], length 24
10:36:23.927391 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 6907:7353, ack 2439, win 501, options [nop,nop,TS val 2446838618 ecr 2119954472], length 446
10:36:23.927687 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 7353:7579, ack 2557, win 501, options [nop,nop,TS val 2446838618 ecr 2119954472], length 226
10:36:23.928048 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 7579:8047, ack 2813, win 501, options [nop,nop,TS val 2446838619 ecr 2119954472], length 468
10:36:23.928294 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 8047:8273, ack 2934, win 501, options [nop,nop,TS val 2446838619 ecr 2119954473], length 226
10:36:23.928506 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 8273:8297, ack 2945, win 501, options [nop,nop,TS val 2446838619 ecr 2119954473], length 24
10:36:23.928620 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 8297:8321, ack 2967, win 501, options [nop,nop,TS val 2446838619 ecr 2119954473], length 24
10:36:23.929050 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 8321:8698, ack 3156, win 501, options [nop,nop,TS val 2446838620 ecr 2119954473], length 377
10:36:23.929342 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 8698:8771, ack 3270, win 501, options [nop,nop,TS val 2446838620 ecr 2119954474], length 73
10:36:23.929496 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [P.], seq 8771:8795, ack 3281, win 501, options [nop,nop,TS val 2446838620 ecr 2119954474], length 24
10:36:23.929773 IP 10.244.0.61.mysql > 10.244.0.53.45742: Flags [F.], seq 8795, ack 3287, win 501, options [nop,nop,TS val 2446838620 ecr 2119954474], length 0
10:36:23.952933 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [.], ack 3046, win 501, options [nop,nop,TS val 829601274 ecr 3689052852], length 0
10:36:23.998908 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [S.], seq 3930087812, ack 3601153512, win 64308, options [mss 1410,sackOK,TS val 2446838690 ecr 2119954543,nop,wscale 7], length 0
10:36:23.999130 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 1:79, ack 1, win 503, options [nop,nop,TS val 2446838690 ecr 2119954544], length 78
10:36:23.999200 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [.], ack 165, win 502, options [nop,nop,TS val 2446838690 ecr 2119954544], length 0
10:36:23.999261 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 79:90, ack 165, win 502, options [nop,nop,TS val 2446838690 ecr 2119954544], length 11
10:36:23.999583 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 90:280, ack 276, win 502, options [nop,nop,TS val 2446838690 ecr 2119954544], length 190
10:36:23.999760 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 280:291, ack 317, win 502, options [nop,nop,TS val 2446838690 ecr 2119954544], length 11
10:36:23.999862 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 291:313, ack 332, win 502, options [nop,nop,TS val 2446838690 ecr 2119954544], length 22
10:36:24.000199 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 313:337, ack 354, win 502, options [nop,nop,TS val 2446838691 ecr 2119954545], length 24
10:36:24.000387 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 337:441, ack 400, win 502, options [nop,nop,TS val 2446838691 ecr 2119954545], length 104
10:36:24.000603 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 441:877, ack 524, win 502, options [nop,nop,TS val 2446838691 ecr 2119954545], length 436
10:36:24.000772 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 877:1055, ack 593, win 502, options [nop,nop,TS val 2446838691 ecr 2119954545], length 178
10:36:24.000979 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 1055:1300, ack 667, win 502, options [nop,nop,TS val 2446838692 ecr 2119954545], length 245
10:36:24.001163 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 1300:1892, ack 813, win 501, options [nop,nop,TS val 2446838692 ecr 2119954546], length 592
10:36:24.001348 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 1892:2712, ack 1008, win 501, options [nop,nop,TS val 2446838692 ecr 2119954546], length 820
10:36:24.001534 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 2712:3271, ack 1159, win 501, options [nop,nop,TS val 2446838692 ecr 2119954546], length 559
10:36:24.001718 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 3271:4159, ack 1356, win 501, options [nop,nop,TS val 2446838692 ecr 2119954546], length 888
10:36:24.001886 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 4159:4524, ack 1472, win 501, options [nop,nop,TS val 2446838692 ecr 2119954546], length 365
10:36:24.002045 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 4524:4841, ack 1566, win 501, options [nop,nop,TS val 2446838693 ecr 2119954547], length 317
10:36:24.002190 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 4841:4945, ack 1612, win 501, options [nop,nop,TS val 2446838693 ecr 2119954547], length 104
10:36:24.002339 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 4945:5401, ack 1741, win 501, options [nop,nop,TS val 2446838693 ecr 2119954547], length 456
10:36:24.002480 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 5401:5602, ack 1819, win 501, options [nop,nop,TS val 2446838693 ecr 2119954547], length 201
10:36:24.002627 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 5602:6375, ack 2012, win 501, options [nop,nop,TS val 2446838693 ecr 2119954547], length 773
10:36:24.002773 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 6375:6618, ack 2089, win 501, options [nop,nop,TS val 2446838693 ecr 2119954547], length 243
10:36:24.002918 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 6618:6859, ack 2165, win 501, options [nop,nop,TS val 2446838694 ecr 2119954547], length 241
10:36:24.003037 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 6859:6883, ack 2176, win 501, options [nop,nop,TS val 2446838694 ecr 2119954548], length 24
10:36:24.003142 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 6883:6907, ack 2198, win 501, options [nop,nop,TS val 2446838694 ecr 2119954548], length 24
10:36:24.003509 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 6907:7353, ack 2439, win 501, options [nop,nop,TS val 2446838694 ecr 2119954548], length 446
10:36:24.003734 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 7353:7579, ack 2557, win 501, options [nop,nop,TS val 2446838694 ecr 2119954548], length 226
10:36:24.004044 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 7579:8047, ack 2813, win 501, options [nop,nop,TS val 2446838695 ecr 2119954548], length 468
10:36:24.004252 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 8047:8273, ack 2934, win 501, options [nop,nop,TS val 2446838695 ecr 2119954549], length 226
10:36:24.004392 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 8273:8297, ack 2945, win 501, options [nop,nop,TS val 2446838695 ecr 2119954549], length 24
10:36:24.004493 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 8297:8321, ack 2967, win 501, options [nop,nop,TS val 2446838695 ecr 2119954549], length 24
10:36:24.004703 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 8321:8674, ack 3150, win 501, options [nop,nop,TS val 2446838695 ecr 2119954549], length 353
10:36:24.004816 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [P.], seq 8674:8698, ack 3161, win 501, options [nop,nop,TS val 2446838695 ecr 2119954549], length 24
10:36:24.005011 IP 10.244.0.61.mysql > 10.244.0.53.45746: Flags [F.], seq 8698, ack 3167, win 501, options [nop,nop,TS val 2446838696 ecr 2119954550], length 0
10:36:38.973239 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [.], ack 3046, win 501, options [nop,nop,TS val 829616294 ecr 3689052852], length 0
10:36:54.077077 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [.], ack 3046, win 501, options [nop,nop,TS val 829631398 ecr 3689052852], length 0
10:37:03.509561 IP 10.244.0.61.mysql > 10.244.0.53.55452: Flags [S.], seq 63881241, ack 1776191087, win 64308, options [mss 1410,sackOK,TS val 2446878200 ecr 2119994054,nop,wscale 7], length 0
10:37:03.509941 IP 10.244.0.61.mysql > 10.244.0.53.55452: Flags [P.], seq 1:79, ack 1, win 503, options [nop,nop,TS val 2446878201 ecr 2119994054], length 78
10:37:03.510113 IP 10.244.0.61.mysql > 10.244.0.53.55452: Flags [.], ack 75, win 503, options [nop,nop,TS val 2446878201 ecr 2119994055], length 0
10:37:03.510233 IP 10.244.0.61.mysql > 10.244.0.53.55452: Flags [P.], seq 79:116, ack 75, win 503, options [nop,nop,TS val 2446878201 ecr 2119994055], length 37
10:37:03.510266 IP 10.244.0.61.mysql > 10.244.0.53.55452: Flags [F.], seq 116, ack 75, win 503, options [nop,nop,TS val 2446878201 ecr 2119994055], length 0
10:37:03.510288 IP 10.244.0.61.mysql > 10.244.0.53.55452: Flags [R.], seq 117, ack 75, win 503, options [nop,nop,TS val 2446878201 ecr 2119994055], length 0
10:37:09.181089 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [.], ack 3046, win 501, options [nop,nop,TS val 829646502 ecr 3689052852], length 0
10:37:18.894103 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [.], ack 5684, win 497, options [nop,nop,TS val 829656215 ecr 3689107834], length 0
10:37:18.894643 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 107289:109568, ack 5684, win 501, options [nop,nop,TS val 829656215 ecr 3689107834], length 2279
10:37:18.895437 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 109568:116558, ack 5739, win 501, options [nop,nop,TS val 829656216 ecr 3689107834], length 6990
10:37:18.895494 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 116558:123548, ack 5739, win 501, options [nop,nop,TS val 829656216 ecr 3689107834], length 6990
10:37:18.895522 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 123548:125952, ack 5739, win 501, options [nop,nop,TS val 829656216 ecr 3689107835], length 2404
10:37:18.895571 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 125952:139932, ack 5739, win 501, options [nop,nop,TS val 829656216 ecr 3689107835], length 13980
10:37:18.895603 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 139932:153912, ack 5739, win 501, options [nop,nop,TS val 829656216 ecr 3689107835], length 13980
10:37:18.895636 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 153912:181872, ack 5739, win 501, options [nop,nop,TS val 829656216 ecr 3689107835], length 27960
10:37:18.895662 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 181872:205638, ack 5739, win 501, options [nop,nop,TS val 829656216 ecr 3689107835], length 23766
10:37:18.895680 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 205638:209832, ack 5739, win 501, options [nop,nop,TS val 829656216 ecr 3689107835], length 4194
10:37:18.895703 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 209832:213237, ack 5739, win 501, options [nop,nop,TS val 829656216 ecr 3689107835], length 3405
10:37:18.895842 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 213237:213246, ack 5739, win 501, options [nop,nop,TS val 829656216 ecr 3689107835], length 9
10:37:18.904454 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [.], ack 5864, win 501, options [nop,nop,TS val 829656225 ecr 3689107835], length 0
10:37:18.904612 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 213246:213763, ack 5864, win 501, options [nop,nop,TS val 829656225 ecr 3689107835], length 517
10:37:18.904895 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [P.], seq 213763:214384, ack 5919, win 501, options [nop,nop,TS val 829656225 ecr 3689107844], length 621
10:37:18.914731 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [S.], seq 3701260284, ack 2418110270, win 64308, options [mss 1410,sackOK,TS val 2446893605 ecr 2120009459,nop,wscale 7], length 0
10:37:18.914989 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 1:79, ack 1, win 503, options [nop,nop,TS val 2446893606 ecr 2120009459], length 78
10:37:18.915077 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [.], ack 165, win 502, options [nop,nop,TS val 2446893606 ecr 2120009460], length 0
10:37:18.915147 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 79:90, ack 165, win 502, options [nop,nop,TS val 2446893606 ecr 2120009460], length 11
10:37:18.915498 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 90:280, ack 276, win 502, options [nop,nop,TS val 2446893606 ecr 2120009460], length 190
10:37:18.915834 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 280:291, ack 317, win 502, options [nop,nop,TS val 2446893606 ecr 2120009460], length 11
10:37:18.915996 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 291:313, ack 332, win 502, options [nop,nop,TS val 2446893607 ecr 2120009460], length 22
10:37:18.916569 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 313:337, ack 354, win 502, options [nop,nop,TS val 2446893607 ecr 2120009461], length 24
10:37:18.916888 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 337:441, ack 400, win 502, options [nop,nop,TS val 2446893607 ecr 2120009461], length 104
10:37:18.917370 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 441:877, ack 524, win 502, options [nop,nop,TS val 2446893608 ecr 2120009462], length 436
10:37:18.917637 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 877:1055, ack 593, win 502, options [nop,nop,TS val 2446893608 ecr 2120009462], length 178
10:37:18.917829 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 1055:1300, ack 667, win 502, options [nop,nop,TS val 2446893608 ecr 2120009462], length 245
10:37:18.918062 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 1300:1892, ack 813, win 501, options [nop,nop,TS val 2446893609 ecr 2120009462], length 592
10:37:18.918306 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 1892:2712, ack 1008, win 501, options [nop,nop,TS val 2446893609 ecr 2120009463], length 820
10:37:18.918532 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 2712:3271, ack 1159, win 501, options [nop,nop,TS val 2446893609 ecr 2120009463], length 559
10:37:18.918846 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 3271:4159, ack 1356, win 501, options [nop,nop,TS val 2446893609 ecr 2120009463], length 888
10:37:18.919069 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 4159:4524, ack 1472, win 501, options [nop,nop,TS val 2446893610 ecr 2120009464], length 365
10:37:18.919257 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 4524:4841, ack 1566, win 501, options [nop,nop,TS val 2446893610 ecr 2120009464], length 317
10:37:18.919426 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 4841:4945, ack 1612, win 501, options [nop,nop,TS val 2446893610 ecr 2120009464], length 104
10:37:18.919650 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 4945:5401, ack 1741, win 501, options [nop,nop,TS val 2446893610 ecr 2120009464], length 456
10:37:18.921145 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 5401:5602, ack 1819, win 501, options [nop,nop,TS val 2446893612 ecr 2120009466], length 201
10:37:18.921347 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 5602:6375, ack 2012, win 501, options [nop,nop,TS val 2446893612 ecr 2120009466], length 773
10:37:18.921541 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 6375:6618, ack 2089, win 501, options [nop,nop,TS val 2446893612 ecr 2120009466], length 243
10:37:18.921732 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 6618:6859, ack 2165, win 501, options [nop,nop,TS val 2446893612 ecr 2120009466], length 241
10:37:18.921867 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 6859:6883, ack 2176, win 501, options [nop,nop,TS val 2446893612 ecr 2120009466], length 24
10:37:18.921959 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 6883:6907, ack 2198, win 501, options [nop,nop,TS val 2446893613 ecr 2120009467], length 24
10:37:18.922431 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 6907:7353, ack 2439, win 501, options [nop,nop,TS val 2446893613 ecr 2120009467], length 446
10:37:18.922713 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 7353:7579, ack 2557, win 501, options [nop,nop,TS val 2446893613 ecr 2120009467], length 226
10:37:18.923127 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 7579:8047, ack 2813, win 501, options [nop,nop,TS val 2446893614 ecr 2120009467], length 468
10:37:18.923420 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 8047:8273, ack 2934, win 501, options [nop,nop,TS val 2446893614 ecr 2120009468], length 226
10:37:18.923616 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 8273:8297, ack 2945, win 501, options [nop,nop,TS val 2446893614 ecr 2120009468], length 24
10:37:18.923711 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 8297:8321, ack 2967, win 501, options [nop,nop,TS val 2446893614 ecr 2120009468], length 24
10:37:18.923988 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 8321:8698, ack 3156, win 501, options [nop,nop,TS val 2446893615 ecr 2120009468], length 377
10:37:18.924212 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 8698:8771, ack 3270, win 501, options [nop,nop,TS val 2446893615 ecr 2120009469], length 73
10:37:18.924330 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [P.], seq 8771:8795, ack 3281, win 501, options [nop,nop,TS val 2446893615 ecr 2120009469], length 24
10:37:18.924548 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [F.], seq 8795, ack 3286, win 501, options [nop,nop,TS val 2446893615 ecr 2120009469], length 0
10:37:18.924754 IP 10.244.0.61.mysql > 10.244.0.53.56200: Flags [.], ack 3287, win 501, options [nop,nop,TS val 2446893615 ecr 2120009469], length 0
10:37:18.948966 IP 10.244.0.61.mysql > 10.244.0.55.40602: Flags [.], ack 5928, win 501, options [nop,nop,TS val 829656270 ecr 3689107845], length 0
10:37:18.999911 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [S.], seq 845172897, ack 158008937, win 64308, options [mss 1410,sackOK,TS val 2446893691 ecr 2120009544,nop,wscale 7], length 0
10:37:19.000202 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 1:79, ack 1, win 503, options [nop,nop,TS val 2446893691 ecr 2120009545], length 78
10:37:19.000276 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [.], ack 165, win 502, options [nop,nop,TS val 2446893691 ecr 2120009545], length 0
10:37:19.000354 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 79:90, ack 165, win 502, options [nop,nop,TS val 2446893691 ecr 2120009545], length 11
10:37:19.000771 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 90:280, ack 276, win 502, options [nop,nop,TS val 2446893691 ecr 2120009545], length 190
10:37:19.001051 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 280:291, ack 317, win 502, options [nop,nop,TS val 2446893692 ecr 2120009545], length 11
10:37:19.001164 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 291:313, ack 332, win 502, options [nop,nop,TS val 2446893692 ecr 2120009546], length 22
10:37:19.001462 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 313:337, ack 354, win 502, options [nop,nop,TS val 2446893692 ecr 2120009546], length 24
10:37:19.001634 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 337:441, ack 400, win 502, options [nop,nop,TS val 2446893692 ecr 2120009546], length 104
10:37:19.001869 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 441:877, ack 524, win 502, options [nop,nop,TS val 2446893692 ecr 2120009546], length 436
10:37:19.002043 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 877:1055, ack 593, win 502, options [nop,nop,TS val 2446893693 ecr 2120009547], length 178
10:37:19.002229 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 1055:1300, ack 667, win 502, options [nop,nop,TS val 2446893693 ecr 2120009547], length 245
10:37:19.002429 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 1300:1892, ack 813, win 501, options [nop,nop,TS val 2446893693 ecr 2120009547], length 592
10:37:19.002632 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 1892:2712, ack 1008, win 501, options [nop,nop,TS val 2446893693 ecr 2120009547], length 820
10:37:19.002848 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 2712:3271, ack 1159, win 501, options [nop,nop,TS val 2446893693 ecr 2120009547], length 559
10:37:19.003060 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 3271:4159, ack 1356, win 501, options [nop,nop,TS val 2446893694 ecr 2120009548], length 888
10:37:19.003266 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 4159:4524, ack 1472, win 501, options [nop,nop,TS val 2446893694 ecr 2120009548], length 365
10:37:19.003435 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 4524:4841, ack 1566, win 501, options [nop,nop,TS val 2446893694 ecr 2120009548], length 317
10:37:19.003594 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 4841:4945, ack 1612, win 501, options [nop,nop,TS val 2446893694 ecr 2120009548], length 104
10:37:19.003769 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 4945:5401, ack 1741, win 501, options [nop,nop,TS val 2446893694 ecr 2120009548], length 456
10:37:19.003939 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 5401:5602, ack 1819, win 501, options [nop,nop,TS val 2446893695 ecr 2120009548], length 201
10:37:19.004114 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 5602:6375, ack 2012, win 501, options [nop,nop,TS val 2446893695 ecr 2120009549], length 773
10:37:19.004287 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 6375:6618, ack 2089, win 501, options [nop,nop,TS val 2446893695 ecr 2120009549], length 243
10:37:19.004458 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 6618:6859, ack 2165, win 501, options [nop,nop,TS val 2446893695 ecr 2120009549], length 241
10:37:19.004586 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 6859:6883, ack 2176, win 501, options [nop,nop,TS val 2446893695 ecr 2120009549], length 24
10:37:19.004677 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 6883:6907, ack 2198, win 501, options [nop,nop,TS val 2446893695 ecr 2120009549], length 24
10:37:19.005196 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 6907:7353, ack 2439, win 501, options [nop,nop,TS val 2446893696 ecr 2120009549], length 446
10:37:19.005510 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 7353:7579, ack 2557, win 501, options [nop,nop,TS val 2446893696 ecr 2120009550], length 226
10:37:19.005909 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 7579:8047, ack 2813, win 501, options [nop,nop,TS val 2446893697 ecr 2120009550], length 468
10:37:19.006209 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 8047:8273, ack 2934, win 501, options [nop,nop,TS val 2446893697 ecr 2120009551], length 226
10:37:19.006362 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 8273:8297, ack 2945, win 501, options [nop,nop,TS val 2446893697 ecr 2120009551], length 24
10:37:19.006451 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 8297:8321, ack 2967, win 501, options [nop,nop,TS val 2446893697 ecr 2120009551], length 24
10:37:19.006684 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 8321:8674, ack 3150, win 501, options [nop,nop,TS val 2446893697 ecr 2120009551], length 353
10:37:19.006796 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [P.], seq 8674:8698, ack 3161, win 501, options [nop,nop,TS val 2446893697 ecr 2120009551], length 24
10:37:19.006983 IP 10.244.0.61.mysql > 10.244.0.53.56204: Flags [F.], seq 8698, ack 3167, win 501, options [nop,nop,TS val 2446893698 ecr 2120009552], length 0
^C
282 packets captured
282 packets received by filter
0 packets dropped by kernel
3.3 curl 获取 http 各阶段的响应时间:
(base) maye@maye-Inspiron-5547:~/github_repository/tensorflow_ecosystem/distribution_strategy$ kubectl get svc --all-namespaces | grep mysql
kubeflow mysql ClusterIP 10.105.203.173 <none> 3306/TCP 2d3h
(base) maye@maye-Inspiron-5547:~/github_repository/tensorflow_ecosystem/distribution_strategy$
# enter pod metadata-grpc-deployment
(base) maye@maye-Inspiron-5547:~$ kubectl exec -it metadata-grpc-deployment-659594dfcb-g9jfl -n kubeflow -- bash
# if "command not found", apt-get install vim
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# vim curl-format.txt
time_namelookup: %{time_namelookup}\n
time_connect: %{time_connect}\n
time_appconnect: %{time_appconnect}\n
time_redirect: %{time_redirect}\n
time_pretransfer: %{time_pretransfer}\n
time_starttransfer: %{time_starttransfer}\n
----------\n
time_total: %{time_total}\n
# if "command not found", apt-get install curl
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# curl -w "@curl-format.txt" -o /dev/null -l "http://10.105.203.173:3306"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (1) Received HTTP/0.9 when not allowed
time_namelookup: 0.000031
time_connect: 0.000666
time_appconnect: 0.000000
time_redirect: 0.000000
time_pretransfer: 0.000736
time_starttransfer: 0.000742
----------
time_total: 0.000747
root@metadata-grpc-deployment-659594dfcb-g9jfl:/#
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# curl -w "@curl-format.txt" -o /dev/null -l "http://mysql:3306"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (1) Received HTTP/0.9 when not allowed
time_namelookup: 0.004395
time_connect: 0.004617
time_appconnect: 0.000000
time_redirect: 0.000000
time_pretransfer: 0.004689
time_starttransfer: 0.004826
----------
time_total: 0.004832
root@metadata-grpc-deployment-659594dfcb-g9jfl:/#
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# curl -w "@curl-format.txt" -o /dev/null -l "http://mysql.kubeflow.svc.cluster.local:3306"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0
curl: (1) Received HTTP/0.9 when not allowed
time_namelookup: 5.016010
time_connect: 5.016159
time_appconnect: 0.000000
time_redirect: 0.000000
time_pretransfer: 5.016236
time_starttransfer: 5.016441
----------
time_total: 5.016448
root@metadata-grpc-deployment-659594dfcb-g9jfl:/#
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# curl -w "@curl-format.txt" -o /dev/null -l "http://mysql.svc.cluster.local:3306"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0curl: (6) Could not resolve host: mysql.svc.cluster.local
time_namelookup: 0.000000
time_connect: 0.000000
time_appconnect: 0.000000
time_redirect: 0.000000
time_pretransfer: 0.000000
time_starttransfer: 0.000000
----------
time_total: 4.773346
root@metadata-grpc-deployment-659594dfcb-g9jfl:/#
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# curl -w "@curl-format.txt" -o /dev/null -l "http://mysql.cluster.local:3306"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0curl: (6) Could not resolve host: mysql.cluster.local
time_namelookup: 0.000000
time_connect: 0.000000
time_appconnect: 0.000000
time_redirect: 0.000000
time_pretransfer: 0.000000
time_starttransfer: 0.000000
----------
time_total: 4.927736
root@metadata-grpc-deployment-659594dfcb-g9jfl:/#
After comment out "search kubeflow.svc.cluster.local svc.cluster.local cluster.local" in file /etc/resolv.conf, tfx components raise error 100%:
"WARNING:absl:mlmd client InternalError: mysql_real_connect failed: errno: , error:",
meanwhile:
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# curl -w "@curl-format.txt" -o /dev/null -l "http://mysql:3306"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: mysql
time_namelookup: 0.000000
time_connect: 0.000000
time_appconnect: 0.000000
time_redirect: 0.000000
time_pretransfer: 0.000000
time_starttransfer: 0.000000
----------
time_total: 0.002766
root@metadata-grpc-deployment-659594dfcb-g9jfl:/#
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# mysql -h mysql --port 3306 -u root -p ""
Enter password:
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (-2)
root@metadata-grpc-deployment-659594dfcb-g9jfl:/#
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# mysql -h mysql.kubeflow.svc.cluster.local --port 3306 -u root -p ""
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 12815
Server version: 8.0.26 MySQL Community Server - GPL
Copyright (c) 2000, 2024, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
It can be seen from upper outputs that 'mysql' is unknown mysql server host, after commenting out "search kubeflow.svc.cluster.local svc.cluster.local cluster.local".
After not comment out "search kubeflow.svc.cluster.local svc.cluster.local cluster.local":
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# mysql -h mysql --port 3306 -u root -p ""
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 12824
Server version: 8.0.26 MySQL Community Server - GPL
Copyright (c) 2000, 2024, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
# 1/10 mysql_real_connect() default timeout is 10s, when exceed 10s:
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# mysql -h mysql --port 3306 -u root -p ""
Enter password:
ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (-2)
root@metadata-grpc-deployment-659594dfcb-g9jfl:/#
mysql_real_connect() default timeout is 10s, when exceed 10s, mysql -h mysql --port 3306 -u root -p "" raises error: "ERROR 2005 (HY000): Unknown MySQL server host 'mysql'", and this occurs 1/10 (once in 10 tries). This is the root cause of error: "WARNING:absl:mlmd client InternalError: mysql_real_connect failed: errno: , error: ".
everytime a tfx component pod raises error:
'grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "mysql_real_connect failed: errno: , error: "
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"mysql_real_connect failed: errno: , error: ", grpc_status:13, created_time:"2024-02-12T13:01:11.496861816+00:00"}',
at the same time, mysql pod raises error:
"E0212 13:01:11.495942 6506 mysql_metadata_source.cc:174] MySQL database was not initialized.".
2024-02-12 13:00:30.226071: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'examples' with dtype string and shape [?]
[[{{node examples}}]]
WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.
2024-02-12 13:00:38.589615: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'serving_default_examples' with dtype string and shape [?]
[[{{node serving_default_examples}}]]
INFO:tensorflow:Assets written to: /tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema/Trainer/model/18/Format-Serving/assets
INFO:tensorflow:Assets written to: /tfx/tfx_pv/pipelines/detect_anomolies_on_wafer_tfdv_schema/Trainer/model/18/Format-Serving/assets
WARNING:absl:mlmd client InternalError: mysql_real_connect failed: errno: , error:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/ml_metadata/metadata_store/metadata_store.py", line 219, in _call_method
response.CopyFrom(grpc_method(request, timeout=self._grpc_timeout_sec))
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 1030, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "mysql_real_connect failed: errno: , error: "
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"mysql_real_connect failed: errno: , error: ", grpc_status:13, created_time:"2024-02-12T13:01:11.496861816+00:00"}"
W0212 12:55:42.167089 6494 metadata_store_service_impl.cc:821] Failed to connect to the database: mysql_real_connect failed: errno: , error:
E0212 13:01:11.495942 6506 mysql_metadata_source.cc:174] MySQL database was not initialized. Please ensure your MySQL server is running. Also, this error might be caused by starting from MySQL 8.0, mysql_native_password used by MLMD is not supported as a default for authentication plugin. Please follow <https://dev.mysql.com/blog-archive/upgrading-to-mysql-8-0-default-authentication-plugin-considerations/>to fix this issue.
W0212 13:01:11.496479 6506 metadata_store_service_impl.cc:52] Failed to connect to the database: mysql_real_connect failed: errno: , error:
(base) maye@maye-Inspiron-5547:~$
How to set mysql_real_connect() timeout:
- add a sentence to set mysql_real_connect() timeout in the function where mysql_init() and mysql_real_connect() are called.
"""
in file ml-metadata/ml_metadata/metadata_store/mysql_metadata_source.cc
"""
Status MySqlMetadataSource::ConnectImpl() {
// Initialize the MYSQL object.
db_ = mysql_init(nullptr);
if (!db_) {
LOG(ERROR) << "MySQL error: " << mysql_errno(db_) << ": "
<< mysql_error(db_);
return BuildErrorStatus(absl::StatusCode::kInternal, "mysql_init failed",
mysql_errno(db_), mysql_error(db_));
}
// Explicitly setup the thread-local initializer.
MLMD_RETURN_WITH_CONTEXT_IF_ERROR(ThreadInitAccess(),
"MySql thread init failed at ConnectImpl");
// Set connection options
if (config_.has_ssl_options()) {
const MySQLDatabaseConfig::SSLOptions& ssl = config_.ssl_options();
// The method set mysql_options, and always return 0. The connection options
// are used in the `mysql_real_connect`.
mysql_ssl_set(db_, ssl.key().empty() ? nullptr : ssl.key().c_str(),
ssl.cert().empty() ? nullptr : ssl.cert().c_str(),
ssl.ca().empty() ? nullptr : ssl.ca().c_str(),
ssl.capath().empty() ? nullptr : ssl.capath().c_str(),
ssl.cipher().empty() ? nullptr : ssl.cipher().c_str());
my_bool verify_server_cert = ssl.verify_server_cert() ? 1 : 0;
mysql_options(db_, MYSQL_OPT_SSL_VERIFY_SERVER_CERT, &verify_server_cert);
}
mysql_options(db_, MYSQL_DEFAULT_AUTH, "mysql_native_password");
// Connect to the MYSQL server.
db_ = mysql_real_connect(
db_, config_.host().empty() ? nullptr : config_.host().c_str(),
config_.user().empty() ? nullptr : config_.user().c_str(),
config_.password().empty() ? nullptr : config_.password().c_str(),
/*db=*/nullptr, config_.port(),
config_.socket().empty() ? nullptr : config_.socket().c_str(),
/*clientflag=*/0UL);
if (!db_) {
LOG(ERROR)
<< "MySQL database was not initialized. Please ensure your "
"MySQL server is running. Also, this error might be caused by "
"starting from MySQL 8.0, mysql_native_password used by MLMD is not "
"supported as a default for authentication plugin. Please follow "
"<https://dev.mysql.com/blog-archive/"
"upgrading-to-mysql-8-0-default-authentication-plugin-"
"considerations/>"
"to fix this issue.";
return BuildErrorStatus(absl::StatusCode::kInternal,
"mysql_real_connect failed", mysql_errno(db_),
mysql_error(db_));
}
// Return an error if the default storage engine doesn't support transactions.
MLMD_RETURN_WITH_CONTEXT_IF_ERROR(
CheckTransactionSupport(),
"checking transaction support of default storage engine");
// Create the database if not already present and skip_db_creation is false.
if (!config_.skip_db_creation()) {
const std::string create_database_cmd =
absl::StrCat("CREATE DATABASE IF NOT EXISTS ", config_.database());
MLMD_RETURN_WITH_CONTEXT_IF_ERROR(RunQuery(create_database_cmd),
"Creating database ", config_.database(),
" in ConnectImpl");
}
// Switch to the database.
if (database_name_.empty()) {
database_name_ = config_.database();
}
# add the following sentence in the upper function
# to set mysql_real_connect() timeout. e.g. 30s.
mysql_options(db_, MYSQL_OPT_CONNECT_TIMEOUT, 30);
- compile the c++ file.
- build container image of metadata-grpc-deployment, which contains the newly built c++ file.
- deploy metadata-grpc-deployment on kubernetes cluster, using the newly built container image.
4. the root cause of the error is that mysql service domain name resolving is slow so that mysql_real_connect() exceeds 10s, timeout. So if recording mysql service ip in file /etc/hosts of pod metadata-grpc-deployment, resolving mysql service domain name will be fast, just lookup in file /etc/hosts, no need coredns to resolve.
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.244.0.53 metadata-grpc-deployment-659594dfcb-g9jfl
root@metadata-grpc-deployment-659594dfcb-g9jfl:/#
(base) maye@maye-Inspiron-5547:~$ kubectl get svc --all-namespaces | grep mysql
kubeflow mysql ClusterIP 10.105.203.173 <none> 3306/TCP 2d
(base) maye@maye-Inspiron-5547:~$
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# vim /etc/hosts
# add the following line,
# 10.105.203.173 is CluseterIp of mysql service,
# if mysql service recreated, need to update
# it to the actual ClusterIp of mysql service.
10.105.203.173 mysql
After it, mysql -h mysql --port 3306 -u root -p "" is ok In pod metadata-grpc-deployment every time for 20 tries:
20/20
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# mysql -h mysql --port 3306 -u root -p ""
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 15737
Server version: 8.0.26 MySQL Community Server - GPL
Copyright (c) 2000, 2024, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> exit
Bye
root@metadata-grpc-deployment-659594dfcb-g9jfl:/#
[SOLUTION]
Option 1: Retry the run.
Option 2: Record CluterIp of mysql service in file /etc/hosts of pod metadata-grpc-deployment:
(base) maye@maye-Inspiron-5547:~$ kubectl get svc --all-namespaces | grep mysql
kubeflow mysql ClusterIP 10.105.203.173 <none> 3306/TCP 2d
(base) maye@maye-Inspiron-5547:~$
root@metadata-grpc-deployment-659594dfcb-g9jfl:/# vim /etc/hosts
# add the following line,
# 10.105.203.173 is CluseterIp of mysql service,
# if mysql service recreated, need to update
# it to the actual ClusterIp of mysql service.
10.105.203.173 mysql
Attention:
- If pod metadata-grpc-deployment, or pod mysql is recreated, need to Record CluterIp of mysql service in file /etc/hosts of pod metadata-grpc-deployment again, since new created pod has new ClusterIp, new file system.
Note:
- Retry the run, just rerun the failed component pod, namely, the component pod is the same one, not recreate a new one.
- On ml-pipeline-ui webpage, "Failed to retrieve ..." is the same Error: connecting to mysql failed.
- nodes in kubernetes cluster connect to mysql in container:
# see cluster ip of pod mysql
$ kubectl describe pod mysql-6b95d686-m4qh7 -n kubeflow
-->
Name: mysql-6b95d686-m4qh7
Namespace: kubeflow
Priority: 0
Service Account: mysql
Node: maye-inspiron-5547/192.168.0.104
Start Time: Tue, 16 Jan 2024 22:25:00 +0800
Labels: app=mysql
application-crd-id=kubeflow-pipelines
pod-template-hash=6b95d686
Annotations:
Status: Running
IP: 10.244.0.112
IPs:
IP: 10.244.0.112
Controlled By: ReplicaSet/mysql-6b95d686
Containers:
mysql:
Container ID: containerd://11423fcf65bfd081b901593a7ae284b79aac9123b96be8836a8f0d9371d5469a
Image: gcr.lank8s.cn/ml-pipeline/mysql:8.0.26
Image ID: gcr.lank8s.cn/ml-pipeline/mysql@sha256:882e55f40d61034a2bb8a1abab1353571ad2a33866f382350788eb34740528b5
Port: 3306/TCP
Host Port: 0/TCP
Args:
...
$ mysql --host 10.244.0.112 --port 3306 --user root --password # --host: cluster ip of mysql pod.
Enter password: # here, password is empty string, just press Enter key.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5616
Server version: 8.0.26 MySQL Community Server - GPL
Copyright (c) 2000, 2023, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+
1 row in set (0.08 sec)
mysql> show status like 'max%connections';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| Max_used_connections | 4 |
+----------------------+-------+
1 row in set (0.08 sec)
mysql>
Attention:
- "WARNING:absl:mlmd client InternalError: mysql_real_connect failed: errno: , error: " is raised by tfx component, not by kubeflow pipeline component, tfx component is sub-process of kubeflow pipeline component. If kubeflow pipeline component does not need object produced by tfx component, kubeflow pipeline component will compete successfully, namely:
"kubeflow detect-anomolies-on-wafer-tfdv-schema-tc4r4-1411469381 0/2 Completed ",
while tfx component raised error. So need to check log of tfx component pod to see if it is ok.
Note:
Any node or application in a container can connect to any pod in the same kubernetes cluster via its cluster ip.
References:
