avg-LoadCollection-milvus-proxy request latency meet a Sudden increase

[AnyangAngus]

Environment

• Milvus version: 2.4.1
• Deployment mode(standalone or cluster): cluster
• MQ type(rocksmq, pulsar or kafka): pulsar
• SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.2.12
• OS(Ubuntu or CentOS): Ubuntu
• CPU/Memory:
• GPU: null
• Others:
  Current Behavior
  avg-LoadCollection-milvus-proxy request latency meet a Sudden increase from 147 to 3.45K.
  even I load an empty collection need wait a lot of time.
  

[yhmo]

Observe the "Query Node" panel: Segment Load latency, Collection Load Num, Partition Load Num, Segment Load Num

[AnyangAngus]

Upgrade to v2.4.20+ Increase query node resource(memory or node number) since more collections are loaded, more memory is required. No need to unload collections if you have sufficient memory. Release unused collections if you have insufficient memory.

@yhmo As for increasing the query node resource, I check the querynode memory info which only consum a quarter of the pod memory. Is it necessary to increase the number of replicas in this situation？
On the other hand, Are there any IO issue which may effect the load latency? maybe some etcd or minio metrics and clean some catch may help?

"From the monitor ui, the "Load Segment Latency" shows at this time point the latency jumps from 200ms to 500ms, might be a storage lag, since this latency mainly depends on IO read." --------------------------- The log of all query nodes between this time range might be helpful. Export log by this script: https://github.com/milvus-io/milvus/tree/master/deployments/export-log

@yhmo Sorry, I can not download and upload logs from production env.
Dose The Load Segment Latency increase to 500ms a big problem? In contrast, a set of business operations include create a collection, build index, insert about 30 items, load and flush jump from 1min to 5min when collctions num accumulating up to 2000.
Should I remove flush after each insert to avoid too much flush that affect the system efficency

[yhmo]

Normally, no need to manually call flush(), just let milvus automatically trigger internal flush actions.
A flush action does the following things:

1. force all the growing segments of a collection to be flushed to storage as sealed segments
2. wait all the data nodes to consume timetick messages to the same point
   The flush action might wait a long time if some data node lags behind the current timetick point too much. In monitoring UI, we can observe the "TimeTick Lag Behind Now"
   

Some fixes for flush in v2.4.x:
v2.4.9 Optimized the use of locks and avoided double flush clustering buffer writer (#35490).
v2.4.13 Fixed a flush performance issue (#36741)
v2.4.18 Accelerated flushing speed by optimizing lock usage (#37897)

[AnyangAngus]

The latest version of v2.4 is v2.4.20, which fixed many bugs and enhancements, much better than v2.4.1. There were some enhancements for large number of collections, but "thoudsands of collections" is still a challenge for milvus. A task for "10k collections" enhancement is still in progress, hopefully can be done in the next minor version v2.4.21 or 2.4.22. If this problem doesn't block your business, I think you can keep using v2.4.1 until the enhancement is ready in v2.4.21 or 2.4.22.

@yhmo Dose it means that no matter how much resources I allocate to the system, how many Entities in each collections (maybe only 10 entites in each collecitons), the maximum number of collections milvus can maintain is around 2k to 3k?

[xiaofan-luan]

The latest version of v2.4 is v2.4.20, which fixed many bugs and enhancements, much better than v2.4.1. There were some enhancements for large number of collections, but "thoudsands of collections" is still a challenge for milvus. A task for "10k collections" enhancement is still in progress, hopefully can be done in the next minor version v2.4.21 or 2.4.22. If this problem doesn't block your business, I think you can keep using v2.4.1 until the enhancement is ready in v2.4.21 or 2.4.22.

@yhmo Dose it means that no matter how much resources I allocate to the system, how many Entities in each collections (maybe only 10 entites in each collecitons), the maximum number of collections milvus can maintain is around 2k to 3k?

Right not the answer is yes.

For multi tenant use case, we do recommend to think about using partition key instead of creating multiple collection for different users.

For the new release of 2.5.4, the collection number might be increased to 10K, but there is still a limit here

[AnyangAngus]

The latest version of v2.4 is v2.4.20, which fixed many bugs and enhancements, much better than v2.4.1. There were some enhancements for large number of collections, but "thoudsands of collections" is still a challenge for milvus. A task for "10k collections" enhancement is still in progress, hopefully can be done in the next minor version v2.4.21 or 2.4.22. If this problem doesn't block your business, I think you can keep using v2.4.1 until the enhancement is ready in v2.4.21 or 2.4.22.

@yhmo Dose it means that no matter how much resources I allocate to the system, how many Entities in each collections (maybe only 10 entites in each collecitons), the maximum number of collections milvus can maintain is around 2k to 3k?

Right not the answer is yes.

For multi tenant use case, we do recommend to think about using partition key instead of creating multiple collection for different users.

For the new release of 2.5.4, the collection number might be increased to 10K, but there is still a limit here

@xiaofan-luan As for system bottleneck, If 2k collection numbers can trigger system bottleneck which can not improve by increasing the cluster resources， How many entites can trigger system bottleneck? or we can increase resources to add Entities bottlenecks?