瀏覽代碼

feat: disable token counting in GPT2Tokenizer by returning 0.

Signed-off-by: -LAN- <laipz8200@outlook.com>
tags/0.15.6-alpha.1
-LAN- 7 月之前
父節點
當前提交
193aaaeab6
No account linked to committer's email address
共有 1 個文件被更改,包括 2 次插入1 次删除
  1. 2
    1
      api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py

+ 2
- 1
api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py 查看文件

@@ -25,7 +25,8 @@ class GPT2Tokenizer:
# future = _executor.submit(GPT2Tokenizer._get_num_tokens_by_gpt2, text)
# result = future.result()
# return cast(int, result)
return GPT2Tokenizer._get_num_tokens_by_gpt2(text)
# return GPT2Tokenizer._get_num_tokens_by_gpt2(text)
return 0

@staticmethod
def get_encoder() -> Any:

Loading…
取消
儲存