소스 검색

Remove unnecessary issubclass check (#23455)

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
tags/1.7.2
Yongtao Huang 2 달 전
부모
커밋
6b8b31ff64
No account linked to committer's email address
2개의 변경된 파일1개의 추가작업 그리고 10개의 파일을 삭제
  1. 0
    0
      api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenizer.py
  2. 1
    10
      api/core/rag/splitter/fixed_text_splitter.py

api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py → api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenizer.py 파일 보기


+ 1
- 10
api/core/rag/splitter/fixed_text_splitter.py 파일 보기

@@ -5,14 +5,13 @@ from __future__ import annotations
from typing import Any, Optional

from core.model_manager import ModelInstance
from core.model_runtime.model_providers.__base.tokenizers.gpt2_tokenzier import GPT2Tokenizer
from core.model_runtime.model_providers.__base.tokenizers.gpt2_tokenizer import GPT2Tokenizer
from core.rag.splitter.text_splitter import (
TS,
Collection,
Literal,
RecursiveCharacterTextSplitter,
Set,
TokenTextSplitter,
Union,
)

@@ -45,14 +44,6 @@ class EnhanceRecursiveCharacterTextSplitter(RecursiveCharacterTextSplitter):

return [len(text) for text in texts]

if issubclass(cls, TokenTextSplitter):
extra_kwargs = {
"model_name": embedding_model_instance.model if embedding_model_instance else "gpt2",
"allowed_special": allowed_special,
"disallowed_special": disallowed_special,
}
kwargs = {**kwargs, **extra_kwargs}

return cls(length_function=_character_encoder, **kwargs)



Loading…
취소
저장