浏览代码

smooth term weight (#3510)

### What problem does this PR solve?

#3499

### Type of change

- [x] Performance Improvement
tags/v0.14.0
Kevin Hu 11 个月前
父节点
当前提交
289034f36e
共有 1 个文件被更改,包括 2 次插入2 次删除
  1. 2
    2
      rag/nlp/term_weight.py

+ 2
- 2
rag/nlp/term_weight.py 查看文件

idf2 = np.array([idf(df(t), 1000000000) for t in tks]) idf2 = np.array([idf(df(t), 1000000000) for t in tks])
wts = (0.3 * idf1 + 0.7 * idf2) * \ wts = (0.3 * idf1 + 0.7 * idf2) * \
np.array([ner(t) * postag(t) for t in tks]) np.array([ner(t) * postag(t) for t in tks])
wts = [math.exp(s) for s in wts]
wts = [math.pow(s, 2) for s in wts]
tw = list(zip(tks, wts)) tw = list(zip(tks, wts))
else: else:
for tk in tks: for tk in tks:
idf2 = np.array([idf(df(t), 1000000000) for t in tt]) idf2 = np.array([idf(df(t), 1000000000) for t in tt])
wts = (0.3 * idf1 + 0.7 * idf2) * \ wts = (0.3 * idf1 + 0.7 * idf2) * \
np.array([ner(t) * postag(t) for t in tt]) np.array([ner(t) * postag(t) for t in tt])
wts = [math.exp(s) for s in wts]
wts = [math.pow(s, 2) for s in wts]
tw.extend(zip(tt, wts)) tw.extend(zip(tt, wts))


S = np.sum([s for _, s in tw]) S = np.sum([s for _, s in tw])

正在加载...
取消
保存