[OSPP 2024] DolphinScheduler idea discussion thread

[lenboo]

DolphinScheduler has participated in the OSPP 2024 event [1] and can publish 4 projects.

If you have good ideas, you can post your ideas here for discussion. The idea includes 3 contents

Project Description
Project Output Standard
Technical Requirements
Deadline April 22, 2024.

Last year project is in here: #13768
Mentor guide: https://summer-ospp.ac.cn/help/mentor/

Our community have 4 max topic for this year.

[wangxj3]

Describe:
Present situation: For enterprises that use Apache dolphinscheduler, Apache dolphinscheduler is more often used as the core scheduling system for data processing. However, there are often other business systems in the enterprise, and most of these business systems have requirements such as scheduled task job management. At the same time, the role of the business system may also be applied to a certain aspect of data processing.
Planned renovation:

1. Add DS-plugin Java plug-in
   (1) The business system is referenced through Maven. The business system implements fixed interfaces and uses fixed annotations to complete the creation of scheduled task executors. The plug-in includes registration of Apache dolphinscheduler api-server, workflow, node creation, scheduled task creation, etc. Core functions.
   (2) Business system application processing can be referenced through the parent-child process, making it a part of data flow.
   2.Apache dolphinscheduler api-server and master-server add corresponding processing logic
   (1) api-server adds registration, task creation processor and logic.
   (2) master-server adds task triggering and result processing logic
   (3) Add external system nodes

Output requirements:
Complete the requirements and submit the code to the dev branch.

skills requirement:
Familiar with Java language, preferably familiar with front-end code.

描述：
当前状况：对于应用Apache dolphinscheduler的企业，更多的是将Apache dolphinscheduler作为数据加工的核心调度系统，但企业中往往存在其他业务系统，而这些业务系统大多数存在定时任务作业管理等需求。同时，业务系统的作用也可能会应用到数据加工的某一个环节。
计划改造：
1.增加DS-plugin Java插件
（1）业务系统通过Maven进行引用，业务系统通过实现固定接口，使用固定注解完成定时任务执行器的创建，插件中包含对Apache dolphinscheduler api-server的注册，工作流、节点创建，定时任务创建等核心功能。
（2）对于业务系统应用加工可以通过父子流程的方式进行引用，使其成为数据流转的一环。
2.Apache dolphinscheduler api-server、master-server增加对应处理逻辑
（1）api-server增加注册、任务创建processor和逻辑。
（2）master-server增加任务触发，结果处理逻辑
（3）增加外部系统节点

产出要求：
完成需求，并将代码提交到dev分支。

技术要求：
熟悉Java语言，最好熟悉前端代码。

[fuchanghai]

Background:
At present, k8s tasks are assembled in the form of low code. When the user's needs are more complex, users are supported to use customized yml. We first define the overall parsing excuse and support pod-type tasks first. As for other types of tasks, we can slowly implement them later.

Plan :
1.Reconstruct the k8s task structure to support both custom mode and low-code mode
2.Supports the ability to re-monitor existing k8s tasks after the master or worker is restarted, instead of killing them first and then starting them.
3.Abstract the interface that supports the yml function and derive an implementation class that supports the pod type

Output requirements:
Complete the requirements and submit the code to the dev branch.

skills requirement:
Familiar with Java language, k8s ,preferably familiar with front-end code.

背景:
目前k8s的任务都是以低代码的形式组装的。 当用户需求比较复杂时，支持用户使用自定义的yml。 我们首先定义整体解析借口并首先支持 Pod 类型的任务。 至于其他类型的任务，我们可以稍后慢慢实施。

计划 ：
1.重构k8s任务结构，同时支持自定义模式和低代码模式
2.支持master或worker重启后重新监控已有的k8s任务，而不是先kill掉再启动。
3.抽象支持yml功能的接口，并派生出支持pod类型的实现类

输出要求：
完成需求并将代码提交到dev分支。

技能要求：
熟悉Java语言、k8s，熟悉前端代码优先。

[pinkfloyds]

Difficulty: Simple

Background:
Right now, we support two kind of usage of JAVA tasks,
The first type is JAVA, user can write java code, once the worker receive this kind of task, it will compile the java code to class, and then execute the class file.
The second type is JAR, user can submit a fat jar, once the worker receive this kind of task, it will directly execute the fat jar.
This issue is want to remove the first type of JAVA task, since this is more like a demo, no one will submit java code, and in most of the time, the java application is not written within one file.
So we only need to support submit Jar.

Plan:

1. Remove JAVA type.
2. Add NORMAL JAR, we would better support user submit jars rather than a fat jar. The command may look like below, this should be more usefully.

java $JAVA_OPTS \
  -Dxx.yy.args=1 \
  -cp "xx.jar":"xx.conf" \
  xx.Application args

So we need to change the current task mode into: Fat JAR, NORMAL JAR.
3. Supplement relevant documents.

Output requirements:
Complete the requirements and submit the code to the dev branch.

skills requirement:
Familiar with Java language,preferably familiar with front-end code.

issus: #15819

难度：简单

背景:
目前，我们支持两种JAVA任务的使用，
第二种类型是JAR，用户可以提交一个 Fat JAR包，一旦工作程序接收到这种任务，它将直接执行这个 Fat JAR包。
这个问题是想要移除第一种类型的JAVA任务，因为这更像是一个演示，没有人会提交Java代码，并且在大多数情况下，JAVA应用程序不是写在一个文件里面的。
所以我们只需要支持提交Jar。

计划:

1. 去掉 JAVA类型。
2. 添加 NORMAL JAR ，将更好地支持用户提交jar，而不是 Fat jar。命令可能如下所示，这应该很有用。

java $JAVA_OPTS \
  -Dxx.yy.args=1 \
  -cp "xx.jar":"xx.conf" \
  xx.Application args

因此，我们需要将当前的任务模式更改为：Fat JAR、NORMAL JAR。
3. 补充相关文档。

输出要求：
完成需求并将代码提交给开发分支。

技能要求：
熟悉Java语言，最好熟悉前端代码。

问题：#15819

[zhuxt2015]

Adding Raft Registration Plugin for Dolphinscheduler

• Difficulty: Advanced

• Project Description:
  Currently, Dolphinscheduler already has three registration plugins: Zookeeper, Mysql, and Etcd. However, all these three registration plugins rely on external components, increasing the operational complexity of Dolphinscheduler.
  Adding a Raft registration plugin
  The Raft plugin has two advantages:

  1. Dolphinscheduler no longer relies on external components. It utilizes the Raft communication protocol to achieve master high availability and communication between master and worker nodes.
  2. The deployment of high availability architecture becomes more straightforward.

• Plan:

  1. Introduce an existing Raft project such as Apache Ratis or Sofastack Sofa-JRaft.
  2. Use Raft to implement the storage of master/worker metadata, capable of receiving node start/stop notifications.
  3. Use Raft to implement worker load balancing.
  4. Use Raft to implement acquiring global locks during execution failover.

• Deliverable Requirements:
  Complete the requirements, submit a PR, and merge it into the main branch.

• Technical Requirements:

  1. Understanding of the Raft protocol
  2. Familiarity with Dolphinscheduler's plugin mechanism
  3. Proficiency in the Java language

Dolphinscheduler 增加Raft注册插件

• 难度: 进阶

• 项目描述
  当前Dolphinscheduler已经有Zookeeper、Mysql、Etcd三种注册插件了，但是这三种注册插件都需要依赖外部组件，额外增加Dolphinscheduler的运维难度。
  新增一种Raft注册插件
  Raft插件两个优点：
  1、Dolphinscheduler不再依赖外部组件，利用Raft通讯协议实现master高可用以及master和worker之间交流
  2、高可用架构的部署方式更加简便

• 计划
  1、引入一个已有的Raft项目比如Apache Ratis或者sofastacksofa-jraft
  2、使用Raft实现存储master/worker的元数据，能够获取节点的起停通知
  3、使用Raft实现worker的负载均衡
  4、使用Raft实现执行故障转移时获取全局锁。

• 产出要求
  完成需求，提交PR并合入主分支

• 技术要求

1. 理解 Raft协议
2. 熟悉 Dolphinscheduler 的插件机制
3. 熟悉 Java 语言

• issus: 10874