首页 PaddleHub 帖子详情
paddlehub文本分类使用macrof1问题
收藏
快速回复
PaddleHub 问答预训练模型 1360 4
paddlehub文本分类使用macrof1问题
收藏
快速回复
PaddleHub 问答预训练模型 1360 4

paddlehub的TextClassifierTask提供了三个评价指标:accuracy、matthews和f1,众所周知,要评估类别之间不平衡的分类问题,在考虑准确率之外必须考虑查全率,故而有f1指标和Matthews指标的诞生。但是据我实测,TextClassifierTask的f1和Matthews都只对二分类问题有用。这并不奇怪,这二者的定义本身也是在二分类问题上的,只是我们可以推广到多分类问题上,成为micro/macro-f1指标。paddlehub究竟有没有准备多分类的更加可靠的评价工具呢?

 

在多分类问题中使用paddlehub提供的Matthews:

[2020-04-01 19:20:59,542] [ TRAIN] - step 810 / 2539: loss=0.12797 matthews=0.00000 acc=0.95937 [step/sec: 0.80]
[2020-04-01 19:21:11,955] [ TRAIN] - step 820 / 2539: loss=0.13541 matthews=0.00000 acc=0.96484 [step/sec: 0.81]
[2020-04-01 19:21:24,331] [ TRAIN] - step 830 / 2539: loss=0.10855 matthews=0.00000 acc=0.96875 [step/sec: 0.81]
[2020-04-01 19:21:36,635] [ TRAIN] - step 840 / 2539: loss=0.11830 matthews=1.00000 acc=0.96406 [step/sec: 0.81]
[2020-04-01 19:21:48,879] [ TRAIN] - step 850 / 2539: loss=0.10088 matthews=0.00000 acc=0.97031 [step/sec: 0.82]
[2020-04-01 19:21:48,880] [ INFO] - Evaluation on dev dataset start
[2020-04-01 19:21:50,759] [ EVAL] - [dev dataset evaluation result] loss=0.11813 matthews=0.00000 acc=0.95460 [step/sec: 2.87]

在多分类问题中使用paddlehub提供的f1:

[2020-04-02 09:54:45,881] [ TRAIN] - step 110 / 2539: loss=2.00415 f1=0.00000 acc=0.43555 [step/sec: 0.87]
[2020-04-02 09:54:57,282] [ TRAIN] - step 120 / 2539: loss=1.91379 f1=0.00000 acc=0.45156 [step/sec: 0.88]
[2020-04-02 09:55:08,684] [ TRAIN] - step 130 / 2539: loss=1.79662 f1=0.00000 acc=0.49844 [step/sec: 0.88]
[2020-04-02 09:55:20,117] [ TRAIN] - step 140 / 2539: loss=1.68962 f1=0.00000 acc=0.54531 [step/sec: 0.87]
[2020-04-02 09:55:31,565] [ TRAIN] - step 150 / 2539: loss=1.57737 f1=0.00000 acc=0.55937 [step/sec: 0.87]
[2020-04-02 09:55:31,566] [ INFO] - Evaluation on dev dataset start
[2020-04-02 09:55:33,442] [ EVAL] - [dev dataset evaluation result] loss=2.03867 f1=0.00000 acc=0.47656 [step/sec: 3.50]
[2020-04-02 09:55:45,055] [ TRAIN] - step 160 / 2539: loss=1.41776 f1=0.00000 acc=0.62500 [step/sec: 0.86]
[2020-04-02 09:55:56,542] [ TRAIN] - step 170 / 2539: loss=1.46184 f1=0.00000 acc=0.62031 [step/sec: 0.87]
[2020-04-02 09:56:08,013] [ TRAIN] - step 180 / 2539: loss=1.29388 f1=0.00000 acc=0.64531 [step/sec: 0.87]
[2020-04-02 09:56:19,521] [ TRAIN] - step 190 / 2539: loss=1.28302 f1=0.00000 acc=0.63906 [step/sec: 0.87]
[2020-04-02 09:56:31,012] [ TRAIN] - step 200 / 2539: loss=1.34620 f1=0.00000 acc=0.60469 [step/sec: 0.87]
[2020-04-02 09:56:31,014] [ INFO] - Evaluation on dev dataset start
[2020-04-02 09:56:32,885] [ EVAL] - [dev dataset evaluation result] loss=1.83094 f1=0.00000 acc=0.54167 [step/sec: 3.51]

1
收藏
回复
全部评论(4)
时间顺序
路人白魔法师
#2 回复于2020-04

看了下源代码,在paddlehub/finetune/evaluate,的确都是针对二分类问题的。

f1:

def calculate_f1_np(preds, labels):
preds = np.array(preds)
labels = np.array(labels)

tp = np.sum((labels == 1) & (preds == 1))
tn = np.sum((labels == 0) & (preds == 0))
fp = np.sum((labels == 0) & (preds == 1))
fn = np.sum((labels == 1) & (preds == 0))
p = tp / (tp + fp) if (tp + fp) else 0
r = tp / (tp + fn) if (tp + fn) else 0
f1 = (2 * p * r) / (p + r) if p + r else 0
return f1

可以自己改成macro-f1,不过我得斟酌一下怎么改……

0
回复
路人白魔法师
#3 回复于2020-04

问题来了,怎么在在线平台改paddlehub的源代码……本地很好弄,但是我得用在线算力啊……

0
回复
红皮小猪
#4 回复于2020-04

aistudio上的环境可以修改paddlehub的源码,找到安装位置,替换一下

0
回复
夏克洛帕尔
#5 回复于2020-09
看了下源代码,在paddlehub/finetune/evaluate,的确都是针对二分类问题的。 f1: def calculate_f1_np(preds, labels): preds = np.array(preds) labels = np.array(labels) tp = np.sum((labels == 1) & (preds == 1)) tn = np.sum((labels == 0) & (preds == 0)) fp = np.sum((labels == 0) & (preds == 1)) fn = np.sum((labels == 1) & (preds == 0)) p = tp / (tp + fp) if (tp + fp) else 0 r = tp / (tp + fn) if (tp + fn) else 0 f1 = (2 * p * r) / (p + r) if p + r else 0 return f1 可以自己改成macro-f1,不过我得斟酌一下怎么改……
展开

你好 请问楼主知道怎么改了吗 能否分享下

0
回复
需求/bug反馈?一键提issue告诉我们
发现bug?如果您知道修复办法,欢迎提pr直接参与建设飞桨~
在@后输入用户全名并按空格结束,可艾特全站任一用户