Просмотр исходного кода

chunk-overlap None check (#2781)

Co-authored-by: jyong <jyong@dify.ai>
tags/0.5.9
Jyong 1 год назад
Родитель
Сommit
6454e1d644
Аккаунт пользователя с таким Email не найден
1 измененных файлов: 6 добавлений и 1 удалений
  1. 6
    1
      api/core/indexing_runner.py

+ 6
- 1
api/core/indexing_runner.py Просмотреть файл

@@ -416,9 +416,14 @@ class IndexingRunner:
if separator:
separator = separator.replace('\\n', '\n')

if 'chunk_overlap' in segmentation and segmentation['chunk_overlap']:
chunk_overlap = segmentation['chunk_overlap']
else:
chunk_overlap = 0

character_splitter = FixedRecursiveCharacterTextSplitter.from_encoder(
chunk_size=segmentation["max_tokens"],
chunk_overlap=segmentation.get('chunk_overlap', 0),
chunk_overlap=chunk_overlap,
fixed_separator=separator,
separators=["\n\n", "。", ".", " ", ""],
embedding_model_instance=embedding_model_instance

Загрузка…
Отмена
Сохранить