Open AD Kit WG Meeting 2022/03/10

[ghost]

Administrative

• Previous meeting minutes

Attendees

Chaired by David

• Andreas Färber (SUSE)
• Armağan Arslan (LeoDrive.ai)
• Bonolo M (AWF)
• Christian John (AWF)
• Christophe Le Dorze (SUSE)
• David W (A3VG)
• Daisuke Hashimoto (T4)
• Fumihito Ito (T4)
• Hiroshi Igata (T4)
• Kasper Ornstein Mecklenburg (Arm)
• Kei Kato (T4)
• Lalith Vipulananthan (T4)
• M. Fatih Cirit (LeoDrive.ai)
• Mark Abrams (SUSE)
• Naoyori Tanzawa (T4)
• Ryohsuke Mitsudome (T4)
• Samet Kütük (LeoDrive.ai)
• Ted Lien (MIH)
• Tsigigenet Dessalgn
• Yoshiya Fujii (T4)

Agenda

• Introduction of new attendees if present
• Announcements (David) - 5 mins
• Review of backlog actions (Lalith) - 5 mins
• Discussion topics
  • Autoware WG best practices and how to get started with Core/Universe (Bonolo) - 15 mins
  • Update of ConOps/OpsCon for Bus ODD (Hashimoto) - 10 mins
  • Follow-up questions for SUSE's proposal (Lalith) - 15 mins

Announcements

• The Strategic Planning Committee and Open AD Kit Go to Market WG have asked for a more concrete roadmap for Open AD Kit 2.0
• ITRI is in discussion with ADLINK to get Arm-based HW to set up a fully SOAFEE-enabled environment that can be used with the Open AD Kit (ITRI's implementation is currently Intel based)

Backlog

• Discussion around the merge of the OAD and HW WGs
  • Armagan, Lalith, David collecting AWF member input on possible merge of OAD and HW WGs. Initial meeting scheduled for this week.
• Decide on nominee for co-chair
  • Only Lalith was nominated and so he will be put forward to the Technical Steering Commitee for ratification. However, also considering the possibility of a third chair for the proposed merge of the OAD and HW
• Call for input about scope of Open AD Kit 2.0
  • Lalith suggested using Slack to encourage discussion around 2.0 scope, since WG meetings are only every 2 weeks and not all members are able to join every meeting -> Slack enables more frequent async discussion.
  • Bonolo recommended using GitHub discussions instead, since the AWF doesn't have a premium subscription for Slack so conversation history is not preserved, which leads nicely on to...

Discussion topics

Autoware WG best practises and how to get started with Core/Universe

• Best practices document
• Quick guide slides

Autoware WG best practices

• Biggest change is around meeting minutes - these should be in Workgroup Meetings category with the appropriate label added eg: "Meeting: openadkit_wg"

David: How can Discussions be used to have lower level discussions, such as around 2.0 scope?
Bonolo: Use the Design category for those kind of discussions. As the discussion becomes more refined, it can split out into Epics and Issues.

Lalith: Should we label those Design discussions with anything in particular?
Bonolo: You can create labels, but don't go overboard! The Maintainer role may be required, so ask Bonolo to create the label if needed.

David: Is there a date for migration from GitLab to GitHub?
Bonolo: The GitLab wiki has been copied over already, so just start using GitHub from now on.

Lalith: Can GitHub projects be used to track backlog tasks?
Bonolo: Projects were previously used to track tasks that were separated out to different companies (eg: writing documentation, writing tests, confirming tests on specific hardware), and should have a description, purpose, desired behaviour, definition of done

• Ask Bonolo if any help is needed to translate goals into issues etc!

How to get started with Core/Universe (quick!)

• More detail and discussion can be found in this week's AS WG meeting minutes

Update of ConOps/OpsCon for Bus ODD

Proposal for ConOps and OpsCons of the AWF Bus ODD demonstration system 2nd draft.pdf

• Updates to OpsCons for Daytime operation, Nighttime operation, picking up/dropping off passengers, set service route via HMI, operation mode (automatic/manual)
  • daytime operation is not possible due to license restrictions
  • nighttime operation (10pm-1am)
  • Letting on/dropping off passengers - door must be opened/closed manually by local government request
  • routes must be set before operation and cannot be changed without authorization from local government
  • fail-safe system added by ITRI will set mode to manual if any malfunction/errors detected

Fatih: This is just discussing Autoware capabilities right?
Hashimoto: If we want to demonstrate Open AD Kit capabilities to Bus ODD, we need to add the concept of Open AD Kit to the Bus ODD ConOps document, for example OTA.

Armağan: Is this all approved by the ODD WG?
Hashimoto: The OpsCons were all based on information provided by ITRI, however this document is not authorized by the ODD WG.
Tanzawa: The document is being developed in parallel with use case development.

Samet: ITRI Bus ODD is based on Intel, but Open AD Kit is based on ARM. How will this work?
Tanzawa: We cannot change the hardware, so will adapt to ITRI's hardware configuration. On the other hand, we can also develop a Reference Design which is not restricted by ITRI hardware for Open AD Kit 2.0.

Follow-up questions for SUSE's proposal

Tier4_Questions_and_Responses_from_SUSE.xlsx

• After SUSE's presentation in the previous WG meeting, T4 asked some follow-up questions and SUSE provided initial answers in the file attached above
• Below is a summary (or closer to a transcription in places) of the discussion at the end of the meeting that focused mostly on performance and application to RTOS

Performance

Q1) Does containerization have any effect on performance?
Q4) Does orchestration have any effect on performance?

• In general, runtime binaries in containers have no performance issues but services running in containers that heavily utilise networking may have additional latency. Container orchestration (at least the initial orchestration method proposed by SUSE) involves an overlay interface which may cause additional overhead
• If you compare a monolithic, old-style application to micro-services, there may be a communication overhead due to the difference in application design. The difference in overhead would be smaller if you compared a non-containerised multi-process communication compared to a multi-container app

Fatih: Is it possible to create a real-time app with multiple containers? Most real-time OS use specific scheduling for applications. Would they work on containers which virtualises many things away?
Mark: We don't really know what container orchestration with real-time operating systems will look like.

RTOS

Q2) Is SUSE's containerization being used for any real-time mission-critical embedded systems in the market?
Q7) When using k3s, is real-time operating system in the scope? If yes, is there any performance and workload impact on RTOS? e.g. increasing boot up time, periodic monitoring with apps on RTOS.

• Currently, SUSE's solutions for containerization are designed and built around the community supported Kubernetes, and K3s is a distribution of Kubernetes.
• Kubernetes does not have a current solution for real-time OS for mixed criticality. A special instruction set may be required for real-time OS (eg: telling a binary it has a certain amount of time in which to run an operation, scheduling of those operations, shifting operations out and moving in something more critical) and none of this is really taken into consideration for the current orchestration systems built into Kubernetes. In addition, there is a need for time-sensitive networking.
• Because of the heavy networking component in Kubernetes, some exploration is being done with the Cilium CNI (Container Network Interface) to see if it could do time sensitive networking. One approach would be to schedule real-time operations on a different processor to the non-critical operations - to the orchestration system, it's just another processor you're scheduling to.
• Kubernetes is targeted at declarative scheduling of operations - you declare the state you want the system to be in, but you don't have any timing to make sure those things happen in a reasonable amount of time. It's just about making sure there is sufficient resource (CPU/memory etc) is available to the application/micro-service
• Still being explored, not a lot of answers at this time

Andreas: A container is just a user space process running on the OS. A container at runtime can make use of scheduling modes, but how to get an orchestrator to declaratively set up things the right way is still being discussed. Inter-container networking runs via another container, but prone to time-sensitivity and needs enhancements changes to deliver packages in that way (also need to consider routing logic)

Fatih: K3s is a light-weight Kubernetes distribution maintained by Rancher Labs, owned by SUSE

Mark: Kubernetes (also called "K8s") came out of Google. Rancher were working on their own orchestrator, but decided to move to Kubernetes from v2 onwards. k3s is smaller because of removal of some overhead (some cloud providers had put their own requirements into the public source tree of Kubernetes eg: Amazon, Google, Alibaba, plus storage providers code) -> k3s removes all of that extra code, and that extra stuff can be added back in via add-ons.

Fatih: K3s is used to run multiple Docker containers in a system. Are we going to use K3s for Docker container orchestration for Autoware?

Mark: Docker is a container runtime that implements the CRI (Container Runtime Interface). Docker was the standard container runtime for Kubernetes for a long time, but has been deprecated because it has a lot of its own extra stuff that is already implemented in Kubernetes. Challenge with Docker is that it must be managed - Docker containers do not restart themselves on segfault or application error, and must be told where to run.

• So Docker gives the ability to containerize an application, but Kubernetes gives the ability to manage the lifecycle of containers and find places them for run and "schedule" (schedule meaning CPU, RAM and kernel devices, rather than time/date).
• Is Kubernetes an exact fit for automotive? Possibly not due to previously mentioned real-time issues, but it is a much closer fit for where we need to be if we start to containerize our workloads.

Fatih: Currently using Docker images for CI/CD testing of Autoware, as well as providing native and Docker installation methods. Should we provide a Kubernetes installation as well?

Mark: Should be no need to change the container, so can continue using Docker - just convert the docker run command into a Kubernetes configuration file. The Kubernetes configuration file gives the declarative ability to say how much resources needs to be allocated, using this container with these arguments etc (similar to Docker Swarm). Recommendation is to provide an example Kubernetes configuration file.

Fatih: Docker is just used to abstract away required libraries. Do you recommend using Kubernetes for development or just for deployment?

Mark: Can be used for development as well. Rancher Desktop allows you to have a local Kubernetes installation and built it into your development cycle. Would recommend using Kubernetes in the development cycle.

• Note that there are different types of configuration
  • deployment: simply runs your pods and makes sure they have an appropriate place to run and that they are scheduled
  • service: allows you to communicate between pods. If you had two micro-services, you can put them in the same pod or in separate pods. If in separate pods, then you need a service in configuration to allow them to communicate

Andreas: There is still a lot of investigation ongoing to see where to improve via the SOAFEE community which T4 is part of, so T4 can join some of those discussions.

Action Items

• @LalithVipulananthan to set up a new Zoom meeting for the WG that allows participants to join from a browser instead of having to install the Zoom client
• @LalithVipulananthan to create a Discussion for the Open AD Kit WG to assess the suitability of ITRI's Bus ODD current demo hardware for use with the Open AD Kit
  • created .gitignore #88
• @LalithVipulananthan to create a Discussion for Open AD Kit 2.0 scope definition
  • https://github.com/autowarefoundation/autoware/discussions/89