[Fix] otaclient_stub: fix always export self ECU id in available_ecu_ids

[Bodong-Yang]

Description

Note
This PR superseded #249 .

This PR fixes unexpected behavior that otaclient_stub.ECUStatusStorage always add self ECU id into available_ecu_ids field when exporting. The bug is introduced when otaclient_stub is fully refactored in #212 .

Concepts related to ECUs for loop status querying

otaclient_stub keeps track of all reachable ECUs' status by loop polling the status API of all directly connected sub ECUs and query self ECU's internal status API. It categorizes ECUs into three groups as follow:

1. available_ecu_ids: a field that defined in ecu_info.yaml and merging from sub ECUs' status's available_ecu_ids field, and exported in status API response. This serves information purpose only and should be passed to upper agent as it. For web.auto agent and being used to generate update request,
2. tracked_ecus(active_ota_ecus): ECUs that is listed in the update request and actively doing OTA update,
3. all_reachable_ecus: ECUs that are reachable and status response can be seen, including all ECUs defined in ecu_info.yaml(self ECU and all directly connected sub ECUs), and all further child ECUs.

Cause of the bug

Previously otaclient_stub.ECUStatusStorage's implementation internally mixes the concepts of available_ecu_ids, tracked_ecus(active_ota_ecus) and all_reachable_ecus into single attribute, effectively making ECUStatusStorage only uses all_reachable_ecus internally and mistakenly export this value in the status API response as available_ecu_ids.

Bug fix

Current behavior

1. self ECU id is always being included in the available_ecu_ids, even it is not in ecu_info.yaml.

Behaivor after fix

1. available_ecu_ids exported in status API response should be the same as ecu_info.yaml and merging from all sub ECUs' available_ecu_ids,
2. for ECUStatusStorage, available_ecu_ids, active_ota_ecus and all_reachable_ecus are clearly distinguished internally:
   a. available_ecu_ids will only be updated by ecu_info.yaml and merging from sub ECUs' status report,
   b. overall_ecu_status_report is only generated against active_ota_ecus(besides lost_ecus, this property is still generated against all_reachable_ecus),
   c. export API will always export all status stored(status of all_reachable_ecus), except for ECUs that become unreachable or disconnected for a while.

Related links & ticket

1. [Feature&refinement] implement new multiple ECU OTA update spec, refactoring otaclient_stub module #212
2. doc: update document especially available_ecu_ids #248
3. OTA service API handler design document: document is updated to include new information related to available_ecu_ids.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
otaclient/app
__init__.py	0	0	100%
__main__.py	3	3	0%	16–19
common.py	268	15	94%	55, 103, 185, 200, 275–277, 287, 296–298, 404, 416, 535–539
configs.py	71	0	100%
copy_tree.py	81	3	96%	45, 96, 125
downloader.py	270	44	84%	69, 82–83, 300, 305, 309, 327–328, 378–382, 401–411, 436–454, 463, 538–540, 556, 576–594
ecu_info.py	59	7	88%	78–79, 92, 97, 120–121, 130
errors.py	179	5	97%	55, 58, 116, 119, 139
interface.py	17	4	76%	32, 43, 47, 51
log_setting.py	26	7	73%	27–28, 55–65
main.py	31	2	94%	43–44
ota_client.py	318	87	73%	69–70, 92, 198–222, 259–261, 271–276, 286–288, 331–332, 337–344, 357–371, 377–380, 405–429, 443–448, 457, 460–468, 547, 552–570
ota_client_call.py	38	6	84%	42–44, 80–82
ota_client_service.py	29	3	90%	58–60
ota_client_stub.py	345	75	78%	81–90, 99–101, 106, 138, 162, 165, 168, 272, 298, 300, 326, 376, 438, 508–509, 548, 568–579, 583–597, 601–604, 656, 743–752, 784–834, 837
ota_metadata.py	314	31	90%	145, 150, 186–187, 197–201, 213, 271, 304–306, 323–326, 406, 409, 417–419, 432, 441–446, 719–730
ota_status.py	15	2	87%	36, 44
proxy_info.py	51	7	86%	82–87, 122–129, 137–138
update_stats.py	105	2	98%	162, 172
otaclient/app/boot_control
__init__.py	4	0	100%
_cboot.py	286	139	51%	91–99, 108–121, 127–130, 134, 141, 145, 149, 153–157, 161–165, 170–189, 192–235, 241, 244, 247, 250, 253, 256, 259, 262–263, 266–267, 270–274, 277–278, 293, 296, 329, 356–359, 369–372, 380, 419, 436–439, 448–451, 456–458, 467–473, 476, 483, 513–515, 552–554, 557–567, 570–575
_common.py	363	155	57%	65–74, 83–84, 88–89, 97–98, 109–110, 115–120, 125–130, 139, 148, 152, 156, 160, 174–177, 187–191, 196, 200, 210–214, 222–234, 245–249, 253–257, 269–271, 283–285, 296–310, 320–341, 360–376, 393–404, 412–418, 467, 489–496, 518–539, 577–578, 632, 636–637, 640–648, 652–655, 705–706, 709, 723–731, 746–747, 816–819, 840–843, 856–857, 866
_errors.py	47	19	60%	43–69, 89–90
_grub.py	402	90	78%	202, 224, 241–244, 250–254, 280–281, 288–308, 318–320, 328–330, 334, 337, 340, 343, 365–366, 384–385, 433, 468, 488–493, 506, 533–542, 679–681, 708–711, 720, 759–763, 807, 814, 833–836, 843–846, 849, 856, 883–885, 906–908, 911–916, 919–924
_rpi_boot.py	260	121	53%	89–91, 97–152, 176–178, 184–186, 199–201, 207–209, 222–243, 248, 252, 256, 260, 294, 324, 327–329, 337, 346–348, 358–361, 365–372, 451–455, 474–477, 482, 485, 506–511, 514–522, 525–533, 545–548, 552–554, 557
configs.py	57	2	96%	46–47
firmware.py	32	5	84%	63, 65, 76–78
protocol.py	28	5	82%	40, 44, 48, 52, 60
selecter.py	39	27	31%	45–69, 77–94
otaclient/app/create_standby
__init__.py	12	5	58%	30–36
common.py	216	45	79%	74, 77–78, 82–95, 141, 189–197, 200–203, 207, 218, 292–300, 312, 356–361, 377–378, 392–396, 421–422
interface.py	19	3	84%	55, 59, 63
rebuild_mode.py	99	9	91%	83–101, 128
otaclient/app/proto
__init__.py	31	3	90%	37, 44–45
_common.py	419	49	88%	85, 163, 170, 182–184, 203, 208, 219, 256, 262, 267, 298, 302, 306, 363, 380, 403, 464, 471, 474, 494, 501, 503, 529, 535, 538, 540, 565, 571, 574, 576, 605, 609, 611, 625, 642, 670, 673, 677, 680, 708, 714, 761–766, 797
_ota_metafiles_wrapper.py	84	13	85%	37, 40–42, 112–116, 122–125
_otaclient_v2_pb2_wrapper.py	276	36	87%	87, 90–93, 175, 183, 197, 207, 210–213, 258, 261, 264–265, 285, 305, 385, 452, 505, 513–515, 519–522, 525–530, 551, 559, 573, 581, 595
streamer.py	43	8	81%	32, 47, 65–66, 71, 80–81, 99
wrapper.py	4	0	100%
otaclient/ota_proxy
__init__.py	29	10	66%	53–58, 102–123
__main__.py	21	21	0%	16–71
_consts.py	15	0	100%
cache_control.py	84	8	90%	50, 75, 95, 117, 125, 141, 147–148
config.py	18	0	100%
db.py	146	15	90%	75, 81, 103, 113–116, 145–147, 166, 199, 208–209, 229, 258, 300
errors.py	10	0	100%
orm.py	115	11	90%	37, 91, 96, 101, 107, 113, 140–141, 154, 231, 235
ota_cache.py	389	79	80%	91–92, 94–95, 214, 225, 252–254, 274, 290–293, 316–329, 358–362, 378, 439–440, 482–483, 553, 566–569, 633–634, 666–667, 678, 712–750, 797, 805–807, 878–881, 885–889, 924, 930, 971–973
server_app.py	138	39	72%	75, 78, 84, 100, 102, 161–170, 212–243, 256–266, 292–298, 312–314, 320–322
utils.py	23	1	96%	31
TOTAL	5929	1221	79%

[obi-t4]

@Bodong-Yang

available_ecu_ids: a field that defined in ecu_info.yaml and merging from sub ECUs' status's available_ecu_ids field, and exported in status API response.

sub ecu doesn't have this available_ecu_ids.
See https://github.com/tier4/ota-client/blob/main/docs/README.md?plain=1#L148

[obi-t4]

available_ecu_ids: a field that defined in ecu_info.yaml and merging from sub ECUs' status's available_ecu_ids field, and exported in status API response.

Ah, you're saying status API response, and sub ECU's statu's available_ecu_ids always empty, right?

[Bodong-Yang]

@Bodong-Yang

available_ecu_ids: a field that defined in ecu_info.yaml and merging from sub ECUs' status's available_ecu_ids field, and exported in status API response.

sub ecu doesn't have this available_ecu_ids. See https://github.com/tier4/ota-client/blob/main/docs/README.md?plain=1#L148

That is not a big problem, if sub ECU doesn't have available_ecu_ids, then the merging just merges empty list, which doesn't impact anything.
In contrast, this behavior of otaclient_stub can be treated as forward-compatibility to multi-layer structure. If in the future we support multi-layer structure, and sub ECUs also have their own child ECUs, we can support it without any changes to the code.

[obi-t4]

It is working as expected!

[obi-t4]

👍