tests analysers stops display values of fps and accuracy

AIStudio796930 发布于2019-12

The most probably this commit:
122b37c
causes test analysers:
test_analyzer_image_classification, test_analyzer_int8_image_classification, etc.
stops to display pieces of information about test results: accuracy and fps.

now is:

PROFILE: interrupts/evictions/bytes = 491/24/27656
[       OK ] Analyzer_int8_image_classification.quantization (10843 ms)
[----------] 1 test from Analyzer_int8_image_classification (10843 ms total)
[----------] Global test environment tear-down

[==========] 1 test from 1 test case ran. (10843 ms total)
[  PASSED  ] 1 test.

but should be:

PROFILE: interrupts/evictions/bytes = 2017/141/75104
I1203 20:07:59.107672 49943 helper.h:322] ====== threads: 1, thread id: 0 ======
I1203 20:07:59.107692 49943 helper.h:324] ====== batch size: 1, iterations: 1000, repetitions: 1 ======
I1203 20:07:59.107695 49943 helper.h:326] ====== batch latency: 20.1677ms, number of samples: 1000, sample latency: 20.1677ms, fps: 49.5842, data type: int8_t ======
I1203 20:07:59.113900 49943 tester_helper.h:520] INT8: avg fps: 49.5842, avg latency: 20.1677 ms
I1203 20:07:59.113971 49943 tester_helper.h:506] --- Accuracy summary --- 
I1203 20:07:59.113976 49943 tester_helper.h:507] Accepted top1_accuracy drop threshold: 0.01. (condition: (FP32_top1_accuracy  - INT8_top1_accuracy ) <= threshold)
I1203 20:07:59.113979 49943 tester_helper.h:511] FP32: avg top1_accuracy 0.0000
I1203 20:07:59.113982 49943 tester_helper.h:513] INT8: avg top1_accuracy 0.7630
[       OK ] Analyzer_int8_image_classification.quantization (35896 ms)
[----------] 1 test from Analyzer_int8_image_classification (35896 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (35896 ms total)
[  PASSED  ] 1 test.

全部评论(5)

AIStudio796931

#2 回复于2019-12

Does this happen in local env or Paddle CI? I ran these analyzer tests locally(e.g. by ./test_analyzer_image_classification --infer_model=/Paddle/build/third_party/inference_demo/resnet50/model/), and the accuracy and fps info is still printed on screen.

The commit only provides a config option to mute all logs while inferencing, but it's set to false by default.

AIStudio796930

#3 回复于2019-12

This lack of results was revealed to us last night during automated nightly tests. Our investigation showed that the perpetrator may be just this commit.

AIStudio796930

#4 回复于2019-12

sample run command:

OMP_NUM_THREADS=1 paddle_paddle/build/paddle/fluid/inference/tests/api/test_analyzer_int8_image_classification 
--enable_int8=true 
--enable_fp32=false 
--infer_data=/data/PaddlePaddle/1G/imagenet/val.bin 
--warmup  
--batch_size=1   
--iterations=1000  
--paddle_num_threads=1 
--infer_model=/repos/paddle_paddle/build/third_party/inference_demo/int8v2/vgg19/model

AIStudio796931

#5 回复于2019-12

It is likely to be fixed in PR#21573.
You can build with the latest develop branch and let us know if it still happens.

AIStudio796930

#6 回复于2019-12

Thank you (:
Now all works