소스 검색

Remove unnecessary issubclass check (#23455)

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
tags/1.7.2
Yongtao Huang 2 달 전
부모
커밋
6b8b31ff64
No account linked to committer's email address
2개의 변경된 파일1개의 추가작업 그리고 10개의 파일을 삭제
  1. 0
    0
      api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenizer.py
  2. 1
    10
      api/core/rag/splitter/fixed_text_splitter.py

api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py → api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenizer.py 파일 보기


+ 1
- 10
api/core/rag/splitter/fixed_text_splitter.py 파일 보기

from typing import Any, Optional from typing import Any, Optional


from core.model_manager import ModelInstance from core.model_manager import ModelInstance
from core.model_runtime.model_providers.__base.tokenizers.gpt2_tokenzier import GPT2Tokenizer
from core.model_runtime.model_providers.__base.tokenizers.gpt2_tokenizer import GPT2Tokenizer
from core.rag.splitter.text_splitter import ( from core.rag.splitter.text_splitter import (
TS, TS,
Collection, Collection,
Literal, Literal,
RecursiveCharacterTextSplitter, RecursiveCharacterTextSplitter,
Set, Set,
TokenTextSplitter,
Union, Union,
) )




return [len(text) for text in texts] return [len(text) for text in texts]


if issubclass(cls, TokenTextSplitter):
extra_kwargs = {
"model_name": embedding_model_instance.model if embedding_model_instance else "gpt2",
"allowed_special": allowed_special,
"disallowed_special": disallowed_special,
}
kwargs = {**kwargs, **extra_kwargs}

return cls(length_function=_character_encoder, **kwargs) return cls(length_function=_character_encoder, **kwargs)





Loading…
취소
저장