瀏覽代碼

refine code to prevent exception (#1231)

### What problem does this PR solve?


### Type of change

- [x] Refactoring
tags/v0.8.0
KevinHuSh 1 年之前
父節點
當前提交
15bf9f8c25
沒有連結到貢獻者的電子郵件帳戶。
共有 1 個檔案被更改,包括 4 行新增4 行删除
  1. 4
    4
      rag/app/naive.py

+ 4
- 4
rag/app/naive.py 查看文件

txt += l txt += l
sections = [] sections = []
for sec in txt.split("\n"): for sec in txt.split("\n"):
if num_tokens_from_string(sec) > 10 * parser_config.get("chunk_token_num", 128):
if num_tokens_from_string(sec) > 10 * int(parser_config.get("chunk_token_num", 128)):
sections.append((sec[:int(len(sec)/2)], "")) sections.append((sec[:int(len(sec)/2)], ""))
sections.append((sec[int(len(sec)/2):], "")) sections.append((sec[int(len(sec)/2):], ""))
else: else:
elif re.search(r"\.json$", filename, re.IGNORECASE): elif re.search(r"\.json$", filename, re.IGNORECASE):
callback(0.1, "Start to parse.") callback(0.1, "Start to parse.")
sections = JsonParser(parser_config.get("chunk_token_num", 128))(binary)
sections = JsonParser(int(parser_config.get("chunk_token_num", 128)))(binary)
sections = [(l, "") for l in sections if l] sections = [(l, "") for l in sections if l]
callback(0.8, "Finish parsing.") callback(0.8, "Finish parsing.")
st = timer() st = timer()
chunks = naive_merge( chunks = naive_merge(
sections, parser_config.get(
"chunk_token_num", 128), parser_config.get(
sections, int(parser_config.get(
"chunk_token_num", 128)), parser_config.get(
"delimiter", "\n!?。;!?")) "delimiter", "\n!?。;!?"))
res.extend(tokenize_chunks(chunks, doc, eng, pdf_parser)) res.extend(tokenize_chunks(chunks, doc, eng, pdf_parser))

Loading…
取消
儲存