Weird behavior of TcpServerConnection #311
Comments
|
@tessob thanks for submitting the issue. Could you please tell us more about your k8s environment? Are there any proxy/service mesh setup? |
|
@alparslanavci no, we have no proxy/service mesh. Our Service manifest is similar to this one. There are just two pods in separate namespace now. However cluster itself is bigger. Our JoinConfig on application start looks like this: and Discovery properties like this: and there are few additional messages that I don't realy know hot to interprete: I hope it would be helpful. |
|
Short update, I have checked out morning logs with 20K attempts to find other nodes and then taken a look at source codes to comprehend where the client has got the list of irrelevant endpoints. I reproduced this API call manually with the following kubectl proxy --port=0 &curl http://localhost:36891/api/v1/namespaces/XXXXXXXXX/endpoints/workflow-serviceand got this output: {
"kind": "Endpoints",
"apiVersion": "v1",
"metadata": {
"name": "workflow-service",
"namespace": "XXXXXXXXX",
"selfLink": "/api/v1/namespaces/XXXXXXXXX/endpoints/workflow-service",
"uid": "64ffb585-e68a-41bc-97bf-fb54e8f3fdc1",
"resourceVersion": "113176905",
"creationTimestamp": "2021-04-08T11:01:06Z",
"labels": {
"fluxcd.io/sync-gc-mark": "sha256.VJun3ud720pxL2mM3fjAbo3OoiEfk8bQDeZA8wZQx7c"
}
},
"subsets": [
{
"addresses": [
{
"ip": "10.154.24.57",
"nodeName": "XXXXXXXXX.compute.internal",
"targetRef": {
"kind": "Pod",
"namespace": "XXXXXXXXX",
"name": "workflow-tracker-service-5769dc77cf-9rnhp",
"uid": "7c84a0c5-7d9b-4cb9-b1d8-d6bb0d08979e",
"resourceVersion": "113176843"
}
},
{
"ip": "10.154.92.185",
"nodeName": "XXXXXXXXX.compute.internal",
"targetRef": {
"kind": "Pod",
"namespace": "XXXXXXXXX",
"name": "workflow-tracker-service-5769dc77cf-zf9pd",
"uid": "29737169-ff5e-49ba-902f-1756dc4e33db",
"resourceVersion": "113176895"
}
}
],
"ports": [
{
"name": "workflow-s",
"port": 8080,
"protocol": "TCP"
},
{
"name": "hazelcast",
"port": 5701,
"protocol": "TCP"
}
]
}
]
}So, as I told, I have just 2 pods relevant to the Hazelcast cluster. Hence I don't really understand what is the source IPs that Best regards, |
|
It's hard to tell where Hazelcast tries to connect to. Some more questions:
|
|
Hi @leszko,
In my dependencies I have only these two that can be related to Hazelcast: <dependency>
<artifactId>spring-boot-starter-cache</artifactId>
<groupId>org.springframework.boot</groupId>
</dependency>
<!-- https://mvnrepository.com/artifact/com.hazelcast/hazelcast-all -->
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-all</artifactId>
<version>4.2</version>
</dependency>My config is in very first message. Thanks, |
|
Short update, I have tested my application in the minikube locally and everything is OK. So the root-cause of such behavior highly likely in K8s cluster and/or deployment manifests. However, for me still unclear why Hazelcast is trying to establish connection with nodes that are irrelevant to service-name. |
|
It may be the phone home call. Could you try disabling phone home and check if it helps? |
|
After I disabled it I still have the same behavior in logs, so there should be another cause. |
|
Hi @tessob , were you able to find out the cause? I'm currently experiencing the same issue. I'm also in AWS EKS |
|
I'm also experiencing the same problem and it is also in AWS EKS. |
Sorry for A late reply )) |
|
Hey folks, I am facing same issue . java.lang.IllegalStateException: Unknown protocol: OPT @tessob Did you find the root cause of this? My issue is similar to what you have posted . |
|
@AshishWat No, just forget about Hazelcast after code observation and moved to Infinispan. So far so good. |
Hello guys!
I rather need a clarification, but not a help. Currently I'm trying to run application in K8s and everything is good so far, but I have a little bit weird logs:
It seems like current node constantly trying to establish connection with other pods "using all variety of irrelevant ports". Is it normal behavior or not? Is there a way to restrict it somehow?
My configs:
Best regards & Thank you in advance,
The text was updated successfully, but these errors were encountered: