[Request] 希望可以自定义分块长度和加入reranker模型

[FAVENO]

🥰 Feature Description

不同embedding 模型max token不一样，用默认的分块长度有的模型会截断。另外希望可以自定义reranker模型。现在的好像即使加了环境变量也没有调用reranker 模型?

🧐 Proposed Solution

可以自定义分块长度和提供自定义reranker 模型的接口

📝 Additional Information

No response

[lobehubbot]

👀 @FAVENO

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.
非常感谢您提交 issue。我们会尽快调查此事，并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

[arvinxx]

不同embedding 模型max token不一样，用默认的分块长度有的模型会截断。

可以，后续也会作为 Embedding 模型的配置之一

现在的好像即使加了环境变量也没有调用reranker 模型?

Reranker 目前还没实现，等知识库 2.0 中会添加

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Different embedding models have different max tokens, and some models will be truncated using the default block length.

Yes, it will also be used as one of the configurations of the Embedding model in the future.

Now it seems that even if environment variables are added, the reranker model is not called?

Reranker is not implemented yet and will be added in Knowledge Base 2.0.