Doris 从 0.13 升级到 0.14 后 fe 经常无响应，进程还在，但无法连接

fe.log

2021-05-25 09:52:09,521 INFO (leaderCheckpointer|77) [DatabaseTransactionMgr.replayUpsertTransactionState():1422] replay a committed transaction TransactionState. transaction id: 5784133, label: insert_2abcb4f71da040bc-86ccce070b24dcd2, db id: 2050773, table id list: 4619317, callback id: -1, coordinator: FE: 172.16.66.121, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1621852917795, commit time: 1621852918050, finish time: -1, reason:
2021-05-25 09:52:09,521 INFO (leaderCheckpointer|77) [DatabaseTransactionMgr.replayUpsertTransactionState():1425] replay a visible transaction TransactionState. transaction id: 5784133, label: insert_2abcb4f71da040bc-86ccce070b24dcd2, db id: 2050773, table id list: 4619317, callback id: -1, coordinator: FE: 172.16.66.121, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1621852917795, commit time: 1621852918050, finish time: 1621852918062, reason:
2021-05-25 09:52:09,521 INFO (leaderCheckpointer|77) [TxnStateCallbackFactory.addCallback():38] add callback of txn state : 5511520. current callback size: 401776
2021-05-25 09:52:09,522 INFO (leaderCheckpointer|77) [LoadManager.replayCreateLoadJob():248] LOAD_JOB=5511520, msg={replay create load job}
2021-05-25 09:52:09,522 INFO (leaderCheckpointer|77) [DatabaseTransactionMgr.replayUpsertTransactionState():1422] replay a committed transaction TransactionState. transaction id: 5784132, label: insert_f20224b4f5bb4afe-8037a54bdcec775a, db id: 2050773, table id list: 4420506, callback id: -1, coordinator: FE: 172.16.66.121, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1621852917795, commit time: 1621852918068, finish time: -1, reason:
2021-05-25 09:52:09,522 INFO (leaderCheckpointer|77) [DatabaseTransactionMgr.replayUpsertTransactionState():1425] replay a visible transaction TransactionState. transaction id: 5784132, label: insert_f20224b4f5bb4afe-8037a54bdcec775a, db id: 2050773, table id list: 4420506, callback id: -1, coordinator: FE: 172.16.66.121, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1621852917795, commit time: 1621852918068, finish time: 1621852918084, reason:
2021-05-25 09:52:09,522 INFO (leaderCheckpointer|77) [TxnStateCallbackFactory.addCallback():38] add callback of txn state : 5511521. current callback size: 401777
2021-05-25 09:52:09,522 INFO (leaderCheckpointer|77) [LoadManager.replayCreateLoadJob():248] LOAD_JOB=5511521, msg={replay create load job}
2021-05-25 09:52:09,522 INFO (leaderCheckpointer|77) [DatabaseTransactionMgr.replayUpsertTransactionState():1422] replay a committed transaction TransactionState. transaction id: 5784143, label: insert_6e86ce2bc9424fb6-8071661350c6f117, db id: 2050773, table id list: 2300264, callback id: -1, coordinator: FE: 172.16.66.121, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1621852918225, commit time: 1621852918256, finish time: -1, reason:
2021-05-25 09:52:09,522 INFO (leaderCheckpointer|77) [DatabaseTransactionMgr.replayUpsertTransactionState():1422] replay a committed transaction TransactionState. transaction id: 5784144, label: insert_307d569580744db9-adf1fb712466d54f, db id: 2050773, table id list: 4417637, callback id: -1, coordinator: FE: 172.16.66.121, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1621852918225, commit time: 1621852918262, finish time: -1, reason:
2021-05-25 09:52:09,523 INFO (leaderCheckpointer|77) [DatabaseTransactionMgr.replayUpsertTransactionState():1425] replay a visible transaction TransactionState. transaction id: 5784143, label: insert_6e86ce2bc9424fb6-8071661350c6f117, db id: 2050773, table id list: 2300264, callback id: -1, coordinator: FE: 172.16.66.121, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1621852918225, commit time: 1621852918256, finish time: 1621852918268, reason:
2021-05-25 09:52:09,523 INFO (leaderCheckpointer|77) [TxnStateCallbackFactory.addCallback():38] add callback of txn state : 5511522. current callback size: 401778
2021-05-25 09:52:09,523 INFO (leaderCheckpointer|77) [LoadManager.replayCreateLoadJob():248] LOAD_JOB=5511522, msg={replay create load job}
2021-05-25 09:52:09,523 INFO (leaderCheckpointer|77) [DatabaseTransactionMgr.replayUpsertTransactionState():1422] replay a committed transaction TransactionState. transaction id: 5784145, label: insert_1366191d0a434d0c-ac43cb0121839c2e, db id: 2050773, table id list: 4619317, callback id: -1, coordinator: FE: 172.16.66.121, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1621852918225, commit time: 1621852918273, finish time: -1, reason:

fe.out

2021-05-25 09:52:45 WARN TIOStreamTransport:112 - Error closing output stream.
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
at org.apache.thrift.transport.TSocket.close(TSocket.java:235)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:303)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-05-25 09:52:45 WARN TIOStreamTransport:112 - Error closing output stream.
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
at org.apache.thrift.transport.TSocket.close(TSocket.java:235)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:303)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-05-25 09:53:03 WARN TIOStreamTransport:112 - Error closing output stream.
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
at org.apache.thrift.transport.TSocket.close(TSocket.java:235)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:303)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-05-25 09:53:03 WARN TIOStreamTransport:112 - Error closing output stream.
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
at org.apache.thrift.transport.TSocket.close(TSocket.java:235)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:303)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

be.WARNING

W0525 09:52:05.203552 21949 thrift_rpc_helper.cpp:66] retrying call frontend service after 1000 ms, address=TNetworkAddress(hostname=172.16.66.121, port=9020), reason=THRIFT_EAGAIN (timed out)
W0525 09:52:05.210522 21948 thrift_rpc_helper.cpp:66] retrying call frontend service after 1000 ms, address=TNetworkAddress(hostname=172.16.66.121, port=9020), reason=THRIFT_EAGAIN (timed out)
W0525 09:52:19.827529 21950 routine_load_task_executor.cpp:310] consuming failed
W0525 09:52:19.827626 24795 broker_scan_node.cpp:373] Scanner[0] process failed. status=cancelled
W0525 09:52:19.828949 21940 fragment_mgr.cpp:230] Got error while opening fragment 8e6941be1cec4589-83bf55bc4258d069: Internal error: cancelled
W0525 09:52:19.829425 21940 stream_load_executor.cpp:90] fragment execute failed, query_id=8e6941be1cec4589-83bf55bc4258d068, err_msg=cancelled, id=8e6941be1cec4589-83bf55bc4258d068, job_id=2680334, txn_id=6094086, label=bill_cost_load-2680334-8e6941be1cec4589-83bf55bc4258d068-6094086
W0525 09:52:19.916626 21951 routine_load_task_executor.cpp:310] consuming failed
W0525 09:52:19.916709 24812 broker_scan_node.cpp:373] Scanner[0] process failed. status=cancelled
W0525 09:52:19.917073 21928 fragment_mgr.cpp:230] Got error while opening fragment 50e1225e94654500-84640984cc653987: Internal error: cancelled
W0525 09:52:19.917433 21928 stream_load_executor.cpp:90] fragment execute failed, query_id=50e1225e94654500-84640984cc653986, err_msg=cancelled, id=50e1225e94654500-84640984cc653986, job_id=2680329, txn_id=6094087, label=bill_income_load-2680329-50e1225e94654500-84640984cc653986-6094087
W0525 09:52:24.828547 21950 thrift_rpc_helper.cpp:66] retrying call frontend service after 1000 ms, address=TNetworkAddress(hostname=172.16.66.121, port=9020), reason=THRIFT_EAGAIN (timed out)
W0525 09:52:24.916708 21951 thrift_rpc_helper.cpp:66] retrying call frontend service after 1000 ms, address=TNetworkAddress(hostname=172.16.66.121, port=9020), reason=THRIFT_EAGAIN (timed out)
W0525 09:52:26.345885 21996 utils.cpp:75] fail to finish_task. host=172.16.66.121, port=9020, error=finishTask failed: unknown result
W0525 09:52:26.352545 21996 task_worker_pool.cpp:279] finish task failed. status_code=0
W0525 09:52:26.403069 21993 utils.cpp:75] fail to finish_task. host=172.16.66.121, port=9020, error=finishTask failed: unknown result
W0525 09:52:26.413558 21993 task_worker_pool.cpp:279] finish task failed. status_code=0
W0525 09:52:32.358542 21996 utils.cpp:62] master client, retry finishTask: THRIFT_EAGAIN (timed out)
W0525 09:52:32.413555 21993 utils.cpp:62] master client, retry finishTask: THRIFT_EAGAIN (timed out)