Skip to content

Heartbeat not received when a transfer is in progress #131

@TAlonglong

Description

@TAlonglong

Describe the bug
When transfer large files or the connection is slow, heartbeats is not received during transfer.

To Reproduce

# Your code here

Expected behavior
I expect the heartbeat to be received during transfer and not the connection to be restarted due missed heartbeats when transfer in progress.

Actual results
Move-it-client logs:

[2022-10-03 05:47:57,500 DEBUG    trollmoves.client] Receiving (SUB) pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA file meos@<move-it-server> 2022-10-03T05:47:57.497649 v1.01 application/json {"
antenna": "XL-BAND", "sensor": ["mersi", "hiras", "vass"], "platform_name": "Fengyun-3D", "start_time": "2022-10-03T05:38:56", "end_time": "2022-10-03T05:47:57", "process_time": "20
22-10-03T05:38:56", "uid": "rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "type": "1", "orbit_number": "25305", "uri": "/data/rawd
ata1/rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "request_address": "<move-it-server>:9109"}
[2022-10-03 05:47:57,500 DEBUG    trollmoves.heartbeat_monitor] Resetting heartbeat alarm timer to 60.0 sec
[2022-10-03 05:47:57,501 INFO     trollmoves.client] Requesting: pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA push ubuntu@fengyun-a 2022-10-03T05:47:57.501200 v1.01 application/json {"
antenna": "XL-BAND", "sensor": ["mersi", "hiras", "vass"], "platform_name": "Fengyun-3D", "start_time": "2022-10-03T05:38:56", "end_time": "2022-10-03T05:47:57", "process_time": "20
22-10-03T05:38:56", "uid": "rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "type": "1", "orbit_number": "25305", "uri": "/data/rawd
ata1/rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "request_address": "<move-it-server>:9109", "destination": "scp://fengyun-a/data/p
ytroll/fengyun3/rawdata"}
[2022-10-03 05:47:57,502 DEBUG    trollmoves.client] Send and recv timeout is 240.00 seconds
[2022-10-03 05:48:57,603 DEBUG    trollmoves.heartbeat_monitor] Missing heartbeat alarm!
[2022-10-03 05:49:29,711 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:29,712 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:29,712 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:29,809 DEBUG    trollmoves.client] Receiving (SUB) pytroll://heartbeat/move_it_server beat eumetcast@sater6 2022-10-03T05:49:29.808737 v1.01 application/json {"min_interval": 30}
[2022-10-03 05:49:29,810 DEBUG    trollmoves.heartbeat_monitor] Resetting heartbeat alarm timer to 900.0 sec
[2022-10-03 05:49:57,078 DEBUG    trollmoves.client] Receiving (SUB) pytroll://heartbeat/move_it_server beat ubuntu@fengyun-a 2022-10-03T05:49:57.076502 v1.01 application/json {"min_interval": 30}
[2022-10-03 05:49:57,078 DEBUG    trollmoves.heartbeat_monitor] Resetting heartbeat alarm timer to 900.0 sec
[2022-10-03 05:49:57,654 DEBUG    trollmoves.heartbeat_monitor] Missing heartbeat alarm!
[2022-10-03 05:49:57,820 DEBUG    trollmoves.client] Receiving (SUB) pytroll://heartbeat/move_it_server beat eumetcast@sater5 2022-10-03T05:49:57.818825 v1.01 application/json {"min_interval": 30}
[2022-10-03 05:49:57,821 DEBUG    trollmoves.heartbeat_monitor] Resetting heartbeat alarm timer to 900.0 sec
[2022-10-03 05:49:59,743 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:59,744 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:59,745 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:59,839 DEBUG    trollmoves.client] Receiving (SUB) pytroll://heartbeat/move_it_server beat eumetcast@sater6 2022-10-03T05:49:59.839160 v1.01 application/json {"min_interval": 30}
[2022-10-03 05:49:59,840 DEBUG    trollmoves.heartbeat_monitor] Resetting heartbeat alarm timer to 900.0 sec
[2022-10-03 05:50:18,015 DEBUG    trollmoves.client] Receiving (REQ) pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA file meos@<move-it-server> 2022-10-03T05:50:17.936793 v1.01 application/json {"antenna": "XL-BAND", "sensor": ["mersi", "hiras", "vass"], "platform_name": "Fengyun-3D", "start_time": "2022-10-03T05:38:56", "end_time": "2022-10-03T05:47:57", "process_time": "2022-10-03T05:38:56", "uid": "rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "type": "1", "orbit_number": "25305", "uri": "/data/rawdata1/rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "request_address": "<move-it-server>:9109", "destination": "scp://fengyun-a/data/pytroll/fengyun3/rawdata"}
[2022-10-03 05:50:18,015 DEBUG    trollmoves.client] Server done sending file
[2022-10-03 05:50:18,015 DEBUG    trollmoves.client] Add rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data to file cache
[2022-10-03 05:50:18,016 DEBUG    trollmoves.client] Sending a public 'ack' of completed transfer: pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA ack ubuntu@fengyun-a 2022-10-03T05:50:18.016318 v1.01 application/json {"antenna": "XL-BAND", "sensor": ["mersi", "hiras", "vass"], "platform_name": "Fengyun-3D", "start_time": "2022-10-03T05:38:56", "end_time": "2022-10-03T05:47:57", "process_time": "2022-10-03T05:38:56", "uid": "rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "type": "1", "orbit_number": "25305", "uri": "/data/rawdata1/rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "request_address": "<move-it-server>:9109"}
[2022-10-03 05:50:18,023 DEBUG    trollmoves.client] publishing pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA file ubuntu@fengyun-a 2022-10-03T05:50:18.017102 v1.01 application/json {"antenna": "XL-BAND", "sensor": ["mersi", "hiras", "vass"], "platform_name": "fengyun 3d", "start_time": "2022-10-03T05:38:56", "end_time": "2022-10-03T05:47:57", "process_time": "2022-10-03T05:38:56", "uid": "rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "type": "1", "orbit_number": "25305", "uri": "ssh://fengyun-a/data/pytroll/fengyun3/rawdata/rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "origin": "<move-it-server>:9109"}
[2022-10-03 05:50:18,023 DEBUG    trollmoves.client] Remove uid 20f5d7994f0033bd9db078f826867045: []
[2022-10-03 05:50:18,024 WARNING  trollmoves.client] Missing a heartbeat, restarting the subscriber to dict_values(['tcp://<move-it-server>:9019']).
[2022-10-03 05:50:19,026 DEBUG    trollmoves.client] Exiting listener tcp://<move-it-server>:9019
[2022-10-03 05:50:19,026 DEBUG    trollmoves.client] Starting listener tcp://<move-it-server>:9019
[2022-10-03 05:50:19,026 INFO     trollmoves.client] Subscribing to tcp://<move-it-server>:9019 with topics ['/XLBANDANTENNA/FENGYUN3D/RAWDATA', '/heartbeat/move_it_server', '/heartbeat/move_it_RAWDATA-FENGYUN3D']
[2022-10-03 05:50:19,027 INFO     posttroll.subscriber] Subscriber adding address tcp://<move-it-server>:9019 with topics ['pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA', 'pytroll://heartbeat/move_it_server', 'pytroll://heartbeat/move_it_RAWDATA-FENGYUN3D']

Environment Info:

  • OS: Linux
  • Trollmoves Version: 0.12.0

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions