Pārlūkot izejas kodu

be better chunks before graphrag (#1811)

### What problem does this PR solve?

#1594

### Type of change

- [x] Refactoring
tags/v0.9.0
Kevin Hu pirms 1 gada
vecāks
revīzija
fe797bcc66
Revīzijas autora e-pasta adrese nav piesaistīta nevienam kontam
1 mainītis faili ar 2 papildinājumiem un 3 dzēšanām
  1. 2
    3
      rag/app/naive.py

+ 2
- 3
rag/app/naive.py Parādīt failu

@@ -273,14 +273,13 @@ def chunk(filename, binary=None, from_page=0, to_page=100000,
raise NotImplementedError(
"file type not supported yet(pdf, xlsx, doc, docx, txt supported)")
if kwargs.get("section_only", False):
return [t for t, _ in sections]
st = timer()
chunks = naive_merge(
sections, int(parser_config.get(
"chunk_token_num", 128)), parser_config.get(
"delimiter", "\n!?。;!?"))
if kwargs.get("section_only", False):
return chunks
res.extend(tokenize_chunks(chunks, doc, eng, pdf_parser))
cron_logger.info("naive_merge({}): {}".format(filename, timer() - st))

Notiek ielāde…
Atcelt
Saglabāt