diff --git a/README-ZH.md b/README-ZH.md index ac5708b..f65cd1f 100644 --- a/README-ZH.md +++ b/README-ZH.md @@ -75,12 +75,11 @@ DataSphereStudio - [DSS 的 Exchangis AppConn 插件安装指南](https://github.com/WeDataSphere/Exchangis/blob/master/docs/zh_CN/ch1/exchangis_appconn_deploy_cn.md) -- [DSS 的 Qualitis AppConn 插件安装指南](https://github.com/WeBankFinTech/Qualitis/blob/master/docs/zh_CN/ch1/%E6%8E%A5%E5%85%A5%E5%B7%A5%E4%BD%9C%E6%B5%81%E6%8C%87%E5%8D%97.md) - - [DSS 的 Streamis AppConn 插件安装指南](https://github.com/WeBankFinTech/Streamis/blob/main/docs/zh_CN/0.2.0/development/StreamisAppConn%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) - [DSS 的 Prophecis AppConn 插件安装指南](https://github.com/WeBankFinTech/Prophecis/blob/master/docs/zh_CN/Deployment_Documents/Prophecis%20Appconn%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) +- [DSS 的 Dolphinscheduler AppConn 插件安装指南](zh_CN/安装部署/DolphinScheduler插件安装文档.md) ## 谁在使用 DataSphere Studio diff --git a/README.md b/README.md index e558237..324e58d 100644 --- a/README.md +++ b/README.md @@ -1,104 +1,99 @@ -DataSphere Studio +DataSphereStudio ==== [![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html) English | [中文](README-ZH.md) -## Introduction -        DataSphere Studio (DSS for short) is WeDataSphere, a big data platform of WeBank, a self-developed one-stop data application development management portal. +## Introduce -        DataSphere Studio is positioned as a data application development framework, and the closed loop covers the entire process of data application development. With a unified UI, the workflow-like graphical drag-and-drop development experience meets the entire lifecycle of data application development from data import, desensitization cleaning, data analysis, data mining, quality inspection, visualization, scheduling to data output applications, etc. +       DataSphere Studio (DSS for short) is a one-stop data application development and management framework developed by WeBank. -        With a pluggable framework architecture, DSS is designed to allow users to quickly integrate new data application tools, or replace various tools that DSS has integrated. +       Under a unified UI, DataSphere Studio provides a workflow-based graphical drag-and-drop development experience, which will satisfy the requirements from data exchange, desensitization cleaning, analysis and mining, quality inspection, visual display, timing scheduling to Data output applications, etc., data application development full-process scenario requirements. -## Integrated data application components +       **DSS is designed with a pluggable integration framework, allowing users to easily and quickly replace various functional components that DSS has integrated, or add new functional components according to their needs. ** +## Integrated application tools -        DSS has integrated a variety of upper-layer data application systems by implementing multiple AppConns, which can basically meet the data development needs of users. +       DSS has integrated a variety of upper-layer data application systems by implementing multiple AppConns, which can basically meet the data development needs of users.        **If desired, new data application systems can also be easily integrated to replace or enrich DSS's data application development process.** [Click me to learn how to quickly integrate new application systems](en_US/Development_Documentation/Third-party_System_Access_Development_Guide.md) -| Component | Description | DSS0.X Compatibility(recommend DSS0.9.1) | DSS1.0 Compatibility(recommend DSS1.0.1) | +|Utility | Description | DSS0.X compatible version (DSS0.9.1 recommended) | DSS1.0 compatible version (DSS1.1.0 recommended) | | --------------- | -------------------------------------------------------------------- | --------- | ---------- | -| [**Linkis**](https://github.com/apache/incubator-linkis) | Apache Linkis, builds a layer of computation middleware, by using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can easily access the underlying engines such as MySQL/Spark/Hive/Presto/Flink, etc. | recommend Linkis0.11.0 (**Released**) | recommend Linkis1.0.3 (**Released**)| -| [**DataApiService**](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/en_US/Using_Document/DataApiService_Usage_Documentation.md) | (Third-party applications built into DSS) Data API service. The SQL script can be quickly published as a Restful interface, providing Rest access capability to the outside world. | Not supported | recommend DSS1.0.1 (**Released**) | -| [**Scriptis**](https://github.com/WeBankFinTech/DataSphereStudio) | (Third-party applications built into DSS) Support online script writing such as SQL, Pyspark, HiveQL, etc., submit to [Linkis](https://github.com/WeBankFinTech/Linkis ) to perform data analysis web tools. | recommend DSS0.9.1 (**Released**) | recommend DSS1.0.1 (**Released**) | -| [**Schedulis**](https://github.com/WeBankFinTech/Schedulis) | Workflow task scheduling system based on Azkaban secondary development, with financial-grade features such as high performance, high availability and multi-tenant resource isolation. | recommend Schedulis0.6.1 (**Released**) | >= Schedulis0.6.2 (**Released**) | -| **EventCheck** | (Third-party applications built into DSS) Provides cross-business, cross-engineering, and cross-workflow signaling capabilities. | recommend DSS0.9.1 (**Released**) | recommend DSS1.0.1 (**Released**) | -| **SendEmail** | (Third-party applications built into DSS) Provides the ability to send data, all the result sets of other workflow nodes can be sent by email | recommend 0.9.1 (**Released**) | recommend 1.0.1 (**Released**) | -| [**Qualitis**](https://github.com/WeBankFinTech/Qualitis) | Data quality verification tool, providing data verification capabilities such as data integrity and correctness | recommend Qualitis0.8.0 (**Released**) | >= Qualitis0.9.1 (**Released**) | -| [**Streamis**](https://github.com/WeBankFinTech/Streamis) | Streaming application development management tool. It supports the release of Flink Jar and Flink SQL, and provides the development, debugging and production management capabilities of streaming applications, such as: start-stop, status monitoring, checkpoint, etc. | Not supported | >= Streamis0.1.0 (**Released**) | -| [**Prophecis**](https://github.com/WeBankFinTech/Prophecis) | A one-stop machine learning platform that integrates multiple open source machine learning frameworks. Prophecis' MLFlow can be connected to DSS workflow through AppConn. | Not supported | >= Prophecis 0.3.0 (**Released**) | -| [**Exchangis**](https://github.com/WeBankFinTech/Exchangis) | A data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources, the upcoming Exchangis1. 0, will be connected with DSS workflow | not supported | >= Exchangis1.0.0 (**Developing**) | -| [**Visualis**](https://github.com/WeBankFinTech/Visualis) | A data visualization BI tool based on the second development of Davinci, an open source project of CreditEase, provides users with financial-level data visualization capabilities in terms of data security. | recommend Visualis0.5.0 (**Released**) | >= Visualis1.0.0 (**Developing**) | -| **UserManager** | (Third-party applications built into DSS) Automatically initialize all user environments necessary for a new DSS user, including: creating Linux users, various user paths, directory authorization, etc. | recommend DSS0.9.1 (**Released**) | Planned in DSS1.0.2 (**Developing**) | -| [**DolphinScheduler**](https://github.com/apache/dolphinscheduler) | Apache DolphinScheduler, a distributed and scalable visual workflow task scheduling platform, supports one-click publishing of DSS workflows to DolphinScheduler. | Not supported | >= DolphinScheduler1.3.6, planned in DSS1.1.0 (**Developing**) | -| **UserGuide** | (Third-party applications to be built into DSS) It mainly provides help documentation, beginner's guide, Dark mode skinning, etc. | Not supported | Planning in DSS1.1.0 (**Developing**) | -| **DataModelCenter** | (Third-party applications to be built into DSS) It mainly provides the capabilities of data warehouse planning, data model development and data asset management. Data warehouse planning includes subject domains, data warehouse layers, modifiers, etc.; data model development includes indicators, dimensions, metrics, wizard-based table building, etc.; data assets are connected to Apache Atlas to provide data lineage capabilities. | Not supported | Planning in DSS1.2.0 (**Developing**) | -| [**Airflow**](https://github.com/apache/airflow) | Supports publishing DSS workflows to Apache Airflow for scheduling. | recommend DSS0.9.1, not yet merged | Not supported | +| [**Linkis**](https://github.com/apache/incubator-linkis) | Computing middleware Apache Linkis, by providing standard interfaces such as REST/WebSocket/JDBC/SDK, upper-layer applications can easily connect and access underlying engines such as MySQL/Spark/Hive/Presto/Flink. | Linkis0.11.0 is recommended (**Released* *) | >= Linkis1.1.1 (**released**) | +| [**DataApiService**](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/en_US/Using_Document/DataApiService_Usage_Documentation.md) | (DSS has built-in third-party application tools) data API service. The SQL script can be quickly published as a Restful interface, providing Rest access capability to the outside world. | Not supported | DSS1.1.0 recommended (**released**)| +| [**Scriptis**](https://github.com/WeBankFinTech/DataSphereStudio) | (DSS has built-in third-party application tools) support online writing of SQL, Pyspark, HiveQL and other scripts, and submit to [Linkis](https ://github.com/WeBankFinTech/Linkis) data analysis web tool. | Recommended DSS0.9.1 (**Released**) | Recommended DSS1.1.0 (**Released**) | +| [**Schedulis**](https://github.com/WeBankFinTech/Schedulis) | Workflow task scheduling system based on Azkaban secondary development, with financial-grade features such as high performance, high availability and multi-tenant resource isolation. | Recommended Schedulis0.6.1 (**released**) | >= Schedulis0.7.0 (**coming soon**) | +| **EventCheck** | (a third-party application tool built into DSS) provides signal communication capabilities across business, engineering, and workflow. | Recommended DSS0.9.1 (**Released**) | Recommended DSS1.1.0 (**Released**) | +| **SendEmail** | (DSS has built-in third-party application tools) provides the ability to send data, all the result sets of other workflow nodes can be sent by email | DSS0.9.1 is recommended (**released**) | Recommended DSS1.1.0 (**Released**) | +| [**Qualitis**](https://github.com/WeBankFinTech/Qualitis) | Data quality verification tool, providing data verification capabilities such as data integrity and correctness | Qualitis0.8.0 is recommended (**Released **) | >= Qualitis0.9.2 (**coming soon**) | +| [**Streamis**](https://github.com/WeBankFinTech/Streamis) | Streaming application development management tool. It supports the release of Flink Jar and Flink SQL, and provides the development, debugging and production management capabilities of streaming applications, such as: start-stop, status monitoring, checkpoint, etc. | Not supported | >= Streamis0.2.0 (**coming soon**) | +| [**Prophecis**](https://github.com/WeBankFinTech/Prophecis) | A one-stop machine learning platform that integrates multiple open source machine learning frameworks. Prophecis' MLFlow can be connected to DSS workflow through AppConn. | Not supported | >= Prophecis 0.3.2 (**coming soon**) | +| [**Exchangis**](https://github.com/WeBankFinTech/Exchangis) | A data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources, the upcoming Exchangis1. 0, will work with DSS workflow | not supported | = Exchangis1.0.0 (**coming soon**) | +| [**Visualis**](https://github.com/WeBankFinTech/Visualis) | A data visualization BI tool based on the secondary development of Davinci, an open source project of CreditEase, provides users with financial-level data visualization capabilities in terms of data security. | Recommended Visualis0.5.0 |= Visualis1.0.0 (**coming soon**) | +| [**DolphinScheduler**](https://github.com/apache/dolphinscheduler) | Apache DolphinScheduler, a distributed and easily scalable visual workflow task scheduling platform, supports one-click publishing of DSS workflows to DolphinScheduler. | Not supported | DolphinScheduler1.3.X, planned in DSS1.1.0 (**released**) | +| **UserGuide** | (DSS will be built-in third-party application tools) contains help documents, beginner's guide, Dark mode skinning, etc. | Not supported | >= DSS1.1.0 (**released**) | +| **DataModelCenter** | (the third-party application tool that DSS will build) mainly provides data warehouse planning, data model development and data asset management capabilities. Data warehouse planning includes subject domains, data warehouse hierarchies, modifiers, etc.; data model development includes indicators, dimensions, metrics, wizard-based table building, etc.; data assets are connected to Apache Atlas to provide data lineage capabilities . | Not supported | Planned in DSS1.2.0 (**under development**) | +| **UserManager** | (DSS has built-in third-party application tools) automatically initialize all user environments necessary for a new DSS user, including: creating Linux users, various user paths, directory authorization, etc. | Recommended DSS0.9.1 (**Released**) | Planning | +| [**Airflow**](https://github.com/apache/airflow) | Supports publishing DSS workflows to Apache Airflow for scheduled scheduling. | PR not yet merged | Not supported | -## Download +       Due to the high risk of script execution supported by DataSphere Studio, the isolation of the WeDataSphere Demo environment has not been completed. Considering that everyone is consulting the Demo environment, it is decided to first issue invitation codes to the community and accept trial applications from enterprises and organizations. -        Please go to the [DSS Releases Page](https://github.com/WeBankFinTech/DataSphereStudio/releases) to download a compiled version or a source code package of DSS. +       If you want to try the Demo environment, please join the DataSphere Studio community user group [Join the group to jump](#1), and contact the team members to get the invitation code. -## Compile and deploy +       DataSphereStudio Demo environment user registration page: [Click me to enter](https://dss-open.wedatasphere.com/#/register) -        Please follow [Compile Guide](en_US/Development_Documentation/Compilation_Documentation.md) to compile DSS from source code. +       DataSphereStudio Demo environment login page: [Click me to enter](https://dss-open.wedatasphere.com/#/login) -        Please refer to [Deployment Documents](en_US/Installation_and_Deployment/DSS_Single-Server_Deployment_Documentation.md) to do the deployment. -## Demo Trial environment +## Documentation directory -        The function of DataSphere Studio supporting script execution has high security risks, and the isolation of the WeDataSphere Demo environment has not been completed. Considering that many users are inquiring about the Demo environment, we decided to first issue invitation codes to the community and accept trial applications from enterprises and organizations. +       Please visit [English Documentation](en_US) for a complete list of DSS documents. -        If you want to try out the Demo environment, please join the DataSphere Studio community user group (**Please refer to the end of the document**), and contact **WeDataSphere Group Robot** to get an invitation code. +       The following is the installation guide for DSS-related AppConn plugins: -        DataSphereStudio Demo environment user registration page: [click me to enter](https://dss-open.wedatasphere.com/#/register) +- [Visualis AppConn Plugin Installation Guide for DSS](https://github.com/WeBankFinTech/Visualis/blob/master/visualis_docs/en_US/Visualis_appconn_install_en.md) -        DataSphereStudio Demo environment login page: [click me to enter](https://dss-open.wedatasphere.com/#/login) +- [Schedulis AppConn Plugin Installation Guide for DSS](en_US/Installation_and_Deployment/SchedulisAppConn_Plugin_Installation_Documentation.md) -## Documents +- [Qualitis AppConn Plugin Installation Guide for DSS](https://github.com/WeBankFinTech/Qualitis/blob/master/docs/zh_CN/ch1/%E6%8E%A5%E5%85%A5%E5%B7%A5%E4%BD%9C%E6%B5%81%E6%8C%87%E5%8D%97.md) -        For a complete list of documents for DSS1.0, see [DSS-Doc](en_US) +- [Exchangis AppConn Plugin Installation Guide for DSS](https://github.com/WeDataSphere/Exchangis/blob/master/docs/en_US/ch1/exchangis_appconn_deploy_en.md) -        The following is the installation guide for DSS-related AppConn plugins: +- [Streamis AppConn Plugin Installation Guide for DSS](https://github.com/WeBankFinTech/Streamis/blob/main/docs/en_US/0.2.0/development/StreamisAppConnInstallationDocument.md) -- [Visualis AppConn Plugin Installation Guide](en_US/Installation_and_Deployment/VisualisAppConn_Plugin_Installation_Documentation.md) +- [Prophecis AppConn Plugin Installation Guide for DSS](https://github.com/WeBankFinTech/Prophecis/blob/master/docs/zh_CN/Deployment_Documents/Prophecis%20Appconn%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) -- [Schedulis AppConn Plugin Installation Guide](en_US/Installation_and_Deployment/SchedulisAppConn_Plugin_Installation_Documentation.md) -- [Qualitis AppConn Plugin Installation Guide](en_US/Installation_and_Deployment/QualitisAppConn_Plugin_Installation_Documentation.md) +## Who is using DataSphere Studio -- [Exchangis AppConn Plugin Installation Guide](en_US/Installation_and_Deployment/ExchangisAppConn_Plugin_Installation_Documentation.md) +       We have created a [Who is using DSS](https://github.com/WeBankFinTech/DataSphereStudio/issues/1) issue for user feedback and documentation of who is using DSS, you are welcome to register. -## Who is using DSS +       DSS Since the open source release in 2019, there have been more than 700 test companies and 1,000+ sandbox test users, covering industries such as finance, telecommunications, manufacturing, and the Internet. -        We opened an issue for users to feedback and record who is using DSS. -        Since the first release of DSS in 2019, it has accumulated more than 700 trial companies and 1000+ sandbox trial users, which involving diverse industries, from finance, banking, tele-communication, to manufactory, internet companies and so on. +## Documentation +       DataSphere Studio uses GitBook for management, and the entire project will be organized into a GitBook e-book for everyone to download and use. -## Document statement +       WeDataSphere A unified document reading entry will be provided in the future. For the usage of GitBook, please refer to: [GitBook Documentation](http://caibaojian.com/gitbook/) -        DataSphere Studio uses GitBook for management, and the entire project will be organized into a GitBook e-book for everyone to download and use. -        WeDataSphere will provide a unified document reading entry in the future. For the usage of GitBook, please refer to: [GitBook Documentation](http://caibaojian.com/gitbook/)。 +## contribute -## Contributing +       We welcome and look forward to more contributors to build DSS, whether it is code, documentation, or other forms of contribution that can help the community. -        Contributions are always welcomed, we need more contributors to build DSS together. either code, or doc, or other supports that could help the community. -        For code and documentation contributions, please follow the contribution guide. +## Contact Us -## Communication +       Any questions and suggestions about DSS, please submit an issue for tracking processing and experience sharing. -        For any questions or suggestions, please kindly submit an issue. +       You can also scan the QR code below to join our WeChat/QQ group for faster response. -        You can scan the QR code below to join our WeChat and QQ group to get more immediate response. +![comminicate](https://github.com/WeBankFinTech/DataSphereStudio/blob/master/images/zh_CN/readme/communication.png) - ![communication](https://github.com/WeBankFinTech/DataSphereStudio/blob/master/images/zh_CN/readme/communication.png) ## License -        DSS is under the Apache 2.0 license. See the [License](https://github.com/WeBankFinTech/DataSphereStudio/blob/master/LICENSE) file for details. +       DSS is under the Apache 2.0 license. See the [License](https://github.com/WeBankFinTech/DataSphereStudio/LICENSE) file for details. diff --git a/en_US/Design_Documentation/FlowExecution/README.md b/en_US/Design_Documentation/FlowExecution/README.md new file mode 100644 index 0000000..48148ec --- /dev/null +++ b/en_US/Design_Documentation/FlowExecution/README.md @@ -0,0 +1,92 @@ +FlowExecution +------------------------- +FlowExecution：The workflow real-time execution module provides interface services for workflow execution, uses the Entrance module of linkis, and inherits some classes of linkis-entrance for adaptive transformation. +Such as inheriting the PersistenceEngine to implement the persistence operation of the dss workflow task, and overriding the EntranceExecutionJob class to implement the workflow node execution, state transition, kill and other operations. Finally, the parsed and processed tasks are delivered to the linkis service through linkis-computation-client. + + +### 1 business architecture + +User use function points: + +| component name | First-level module | Second-level module | Function point | +|---------------------|------------------|-----------------|-----------------| +| DataSphereStudio | Workflow | Workflow Execution | Execute | +| | | | Check to execute | +| | | | fail and rerun | +| | | | Execution history view | + +![](images/workflow_execution_uml.png) + +### 2. Introduction to FlowExecution interface/class function: + +| Core Interface/Class | Core Function | +|---------------------------|------------------------------| +| FlowEntranceRestfulApi | Provides a restful interface for workflow execution, such as task execution, status acquisition | +| WorkflowPersistenceEngine | Overrides the persist method of the PersistenceEngine of linkis, converts the jobRequest to a workflowTask and persists it to the workflow_task table of dss | +| WorkflowQueryService | Provides interface services such as workflow task creation and status update | +| WorkflowExecutionInfoService | Provides services such as creation and query of workflow execution information | +| FlowEntranceEngine | Executor inherited from linkis, implements the execute method, will call flowParser to parse the flow and use it as the entry of the runJob method | +| FlowExecutionExecutorManagerImpl | Inherit the ExecutorManager of the child linkis, rewrite the createExecutor method, so that the created executor is the FlowEntranceEngine of dss | +| FlowExecutionParser | CommonEntranceParser inherited from linkis, rewrites the parseToJob method to return FlowEntranceJob of dss | +| DefaultFlowExecution | Provides the runJob() method to convert all scheduled nodes of FlowEntranceJob to runnable state, and add the running nodes to the task queue. The timing thread pool polls the linkis task status corresponding to each node. If the status task is completed is removed from the queue. | +| FlowEntranceJobParser | Define the parse() method, the subclass contains various parsing implementation classes, such as parsing workflow from jobRequest, parsing the params attribute of workflow nodes | +| FlowEntranceJob | EntranceExecutionJob inherited from linkis, rewrites run(), kill(), onStatusChanged() and other methods to provide job execution entry and status callback processing. | +| DefaultNodeRunner | The node task running thread, converts the node task to LinkisJob and delivers it to linkis, and provides methods for initiating task status query and task cancellation to linkis | +| NodeSkipStrategy | Strategy interface for judging whether a node is skipped, including three implementation strategies: execution, rerun on failure, and selected execution | +| FlowContext | Contains workflow context status information, and provides methods such as getRunningNodes and getSucceedNodes to obtain information such as running nodes and successful nodes of the workflow | + + + +### 3. Workflow execution process link: +![](images/flowexecution.drawio.png) + +### 4. Data Structure/Storage Design +Workflow execution information table: +```roomsql +CREATE TABLE `dss_workflow_execute_info` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `task_id` bigint(20) NOT NULL COMMENT 'task id', + `status` int(1) DEFAULT NULL COMMENT 'status，0：faild 1：success，', + `flow_id` bigint(20) NOT NULL COMMENT 'flowId', + `version` varchar(200) DEFAULT NULL COMMENT 'Workflow bml version number', + `failed_jobs` text COMMENT 'execution failed node', + `Pending_jobs` text COMMENT 'Node not executed', + `skipped_jobs` text COMMENT 'execute skip node', + `succeed_jobs` text COMMENT 'Execute successful node', + `createtime` datetime NOT NULL COMMENT 'create time', + `running_jobs` text COMMENT 'running jobs', + `updatetime` datetime DEFAULT NULL COMMENT 'update time', + PRIMARY KEY (`id`) + ) ENGINE=InnoDB DEFAULT CHARSET=utf8; +``` +Workflow task information table: +```roomsql +CREATE TABLE `dss_workflow_task` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `instance` varchar(50) DEFAULT NULL COMMENT 'An instance of Entrance, consists of IP address of the entrance server and port', + `exec_id` varchar(50) DEFAULT NULL COMMENT 'execution ID, consists of jobID(generated by scheduler), executeApplicationName , creator and instance', + `um_user` varchar(50) DEFAULT NULL COMMENT 'User name', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'submitUser name', + `execution_code` text COMMENT 'Run script. When exceeding 6000 lines, script would be stored in HDFS and its file path would be stored in database', + `progress` float DEFAULT NULL COMMENT 'Script execution progress, between zero and one', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the log files', + `result_location` varchar(200) DEFAULT NULL COMMENT 'File path of the result', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `created_time` datetime DEFAULT NULL COMMENT 'Creation time', + `updated_time` datetime DEFAULT NULL COMMENT 'Update time', + `run_type` varchar(50) DEFAULT NULL COMMENT 'Further refinement of execution_application_time, e.g, specifying whether to run pySpark or SparkR', + `err_code` int(11) DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `err_desc` text COMMENT 'Execution description. Generated when the execution of script fails', + `execute_application_name` varchar(200) DEFAULT NULL COMMENT 'The service a user selects, e.g, Spark, Python, R, etc', + `request_application_name` varchar(200) DEFAULT NULL COMMENT 'Parameter name for creator', + `script_path` varchar(200) DEFAULT NULL COMMENT 'Path of the script in workspace', + `params` text COMMENT 'Configuration item of the parameters', + `engine_instance` varchar(50) DEFAULT NULL COMMENT 'An instance of engine, consists of IP address of the engine server and port', + `task_resource` varchar(1024) DEFAULT NULL, + `engine_start_time` time DEFAULT NULL, + `label_json` varchar(200) DEFAULT NULL COMMENT 'label json', + PRIMARY KEY (`id`), + KEY `created_time` (`created_time`), + KEY `um_user` (`um_user`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; +``` \ No newline at end of file diff --git a/en_US/Design_Documentation/FlowExecution/images/flowexecution.drawio.png b/en_US/Design_Documentation/FlowExecution/images/flowexecution.drawio.png new file mode 100644 index 0000000..728a435 Binary files /dev/null and b/en_US/Design_Documentation/FlowExecution/images/flowexecution.drawio.png differ diff --git a/en_US/Design_Documentation/FlowExecution/images/workflow_execution_uml.png b/en_US/Design_Documentation/FlowExecution/images/workflow_execution_uml.png new file mode 100644 index 0000000..31302ba Binary files /dev/null and b/en_US/Design_Documentation/FlowExecution/images/workflow_execution_uml.png differ diff --git a/en_US/Design_Documentation/Orchestrator/README.md b/en_US/Design_Documentation/Orchestrator/README.md new file mode 100644 index 0000000..c1622e2 --- /dev/null +++ b/en_US/Design_Documentation/Orchestrator/README.md @@ -0,0 +1,156 @@ +Orchestrator Architecture Design +------------------------- +Orchestrator：The orchestration module provides interface services such as adding, deleting, modifying, querying, importing and exporting orchestrated under the project, and serves as a unified input for each orchestration implementation (such as workflow). Connect to the project service upward, and connect to the specific orchestration implementation (such as workflow service) downward. + +### 1 business architecture +User use function point： + + +|Component name | First-level module | Second-level module | Function point | +|---------------------|------------------|-----------------|-----------------| +| DataSphereStudio | Orchestration Mode | New Orchestration Mode | Create a New Orchestration | +| | | Edit Arrangement Mode | Edit Arranged Field Information | +| | | Delete Arrangement Mode | Delete Arrangement | +| | | Open Arrangement Mode | Open Arrangement to perform drag-and-drop development of choreography nodes | +| | | View the list of orchestration versions | View the historical versions of the orchestration mode, you can open and view a version, or roll back a version | +| | | Orchestration mode rollback | Roll back to a historical version of the orchestration (a version will be added after the orchestration is released) | + +![](images/orchestrator_uml.png) + +### 一、Orchestrator Architecture： +![](images/orchestrator_arch.png) + +### 二、Orchestrator module design： +Second-level module core class introduction： + +**dss-orchestrator-core** + +The core module of Orchestrator defines top-level interfaces such as DSSOrchestrator, DSSOrchestratorContext, and DSSOrchestratorPlugin. + +| Core top-level interface/class | Core functionality | +|---------------------------|------------------------------| +| DSSOrchestrator | Defines the method of obtaining the properties of the orchestration, such as obtaining the editorial ranking, associated appconn, context information, etc. | +| DSSOrchestratorContext | Defines the context information of the orchestrator, and provides methods such as obtaining the orchestration plug-in class | +| DSSOrchestratorPlugin | The top-level interface for orchestrating plugins, defines the init method, and the subclass contains the imported and exported plugin implementation classes | +| DSSOrchestratorRelation |Defines the method of obtaining the associated properties of the orchestration, such as obtaining the orchestration mode, and obtaining the appconn associated with the orchestration | + +**dss-orchestrator-db** + +Defines the unified entry of the dao layer method of orchestration. + +**dss-orchestrator-conversion-standard** + +Defines the interface specification for orchestration and conversion to third-party systems, including top-level interfaces such as ConversionOperation, ConversionRequestRef, and ConversionService. +The core module of Orchestrator defines top-level interfaces such as DSSOrchestrator, DSSOrchestratorContext, and DSSOrchestratorPlugin. + +| Core Interface/Class | Core Function | +|--------------------- |------------------------------------------| +| ConversionOperation | Defines the conversion core convert method, the input parameter is ConversionRequestRef, and returns ResponseRef | +| DSSToRelConversionRequestRef | Defines the basic parameters of the conversion request, such as userName, workspace, dssProject and other information | +| ConversionIntegrationStandard | The following core methods are defined: getDSSToRelConversionService (used to support the orchestration of DSS and convert it to the workflow of the scheduling system) | +| ConversionService | Defines methods for getting labels and getting ConversionIntegrationStandard | + + +**dss-orchestrator-loader** + +Used to load orchestration-related appconn, such as workflow-appconn, and load subclasses of DSSOrchestratorPlugin, such as ExportDSSOrchestratorPlugin. + +| Core Interface/Class | Core Function | +|--------------------- |---------------------------------------------| +| OrchestratorManager | The getOrCreateOrchestrator method is defined to load the appconn associated with the orchestrator, which will be cached after the first load to avoid repeated loading | +| LinkedAppConnResolver | Defines the interface for obtaining appconn according to the user | +| SpringDSSOrchestratorContext | The initialization method of the class will load all subclasses of DSSOrchestratorPlugin and cache them in memory | + +**dss-framework-orchestrator-server** + +The Orchestrator framework service provides interfaces such as adding, deleting, modifying, checking, and rolling back the orchestration front-end, as well as rpc services such as orchestration import and export. + +**dss-framework-orchestraotr-publish** + +Provides publishing-related plug-ins, such as arranging import and export implementation classes, arranging compressed package generation, and parsing implementation classes. + +| Core Interface/Class | Core Function | +|--------------------- |---------------------------------------------| +| ExportDSSOrchestratorPlugin | Defines the orchestration export interface | +| ImportDSSOrchestratorPlugin | The orchestration import interface is defined | +| MetaWriter | Provides the function of outputting the arranged table field information to the metadata file in a specific format| +| MetaReader | Provides the function of parsing the arranged metadata file to generate table field content| + +#### Create an orchestration sequence diagram (delete and edit operations are similar): + +![](images/Create_an_orchestration_sequence_diagram.png) + +#### Import and arrange sequence diagrams (export operations are similar)： + +![](images/Import_Orchestration_Sequence_Diagram.png) + +### Data Structure/Storage Design +orchestrator information sheet: +```roomsql +CREATE TABLE `dss_orchestrator_info` ( +`id` bigint(20) NOT NULL AUTO_INCREMENT, +`name` varchar(255) NOT NULL COMMENT 'orchestrator name', +`type` varchar(255) NOT NULL COMMENT 'orchestrator type，E.g：workflow', +`desc` varchar(1024) DEFAULT NULL COMMENT 'description', +`creator` varchar(100) NOT NULL COMMENT 'creator', +`create_time` datetime DEFAULT NULL COMMENT 'create time', +`project_id` bigint(20) DEFAULT NULL COMMENT 'project id', +`uses` varchar(500) DEFAULT NULL COMMNET 'uses', +`appconn_name` varchar(1024) NOT NULL COMMENT 'Orchestrate the associated appconn，E.g:workflow', +`uuid` varchar(180) NOT NULL COMMENT 'uuid', +`secondary_type` varchar(500) DEFAULT NULL COMMENT 'Orchestrate of the second type，E.g:workflow-DAG', +`is_published` tinyint(1) NOT NULL DEFAULT '0' COMMENT 'Is it published', +`workspace_id` int(11) DEFAULT NULL COMMENT 'workspace id', +`orchestrator_mode` varchar(100) DEFAULT NULL COMMENT 'orchestrator mode，The value obtained is dic_key(parent_key=p_arrangement_mode) in dss_dictionary', +`orchestrator_way` varchar(256) DEFAULT NULL COMMENT 'orchestrator way', +`orchestrator_level` varchar(32) DEFAULT NULL COMMENT 'orchestrator level', +`update_user` varchar(100) DEFAULT NULL COMMENT 'update user', +`update_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', +PRIMARY KEY (`id`) USING BTREE, +UNIQUE KEY `unique_idx_uuid` (`uuid`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; +``` + +orchestrator version information table: +```roomsql +CREATE TABLE `dss_orchestrator_version_info` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `orchestrator_id` bigint(20) NOT NULL COMMENT 'associated orchestration id', + `app_id` bigint(20) DEFAULT NULL COMMENT 'The id of the orchestration implementation, such as flowId', + `source` varchar(255) DEFAULT NULL COMMENT 'source', + `version` varchar(255) DEFAULT NULL COMMENT 'verison', + `comment` varchar(255) DEFAULT NULL COMMENT 'description', + `update_time` datetime DEFAULT NULL COMMENT 'update time', + `updater` varchar(32) DEFAULT NULL COMMENT 'updater', + `project_id` bigint(20) DEFAULT NULL COMMENT 'project id', + `content` varchar(255) DEFAULT NULL COMMENT '', + `context_id` varchar(200) DEFAULT NULL COMMENT 'context id', + `valid_flag` INT(1) DEFAULT '1' COMMENT 'Version valid flag, 0: invalid; 1: valid', + PRIMARY KEY (`id`) USING BTREE +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; +``` + +And the scheduling system orchestration association table: +```roomsql +CREATE TABLE `dss_orchestrator_ref_orchestration_relation` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'primary key ID', + `orchestrator_id` bigint(20) NOT NULL COMMENT 'The orchestration mode id of dss', + `ref_project_id` bigint(20) DEFAULT NULL COMMENT 'The project ID associated with the scheduling system', + `ref_orchestration_id` int(11) DEFAULT NULL COMMENT 'The id of the scheduling system workflow (the orchestrationId returned by calling the OrchestrationOperation service of SchedulerAppConn)', + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; +``` + +### 5.interface design + + +### 6.non-functional design +#### 6.1 Safety +Using the special ID in the cookie, GateWay needs to use a special decryption algorithm to identify it. +#### 6.2 Performance +can meet performance requirements. +#### 6.3 Capacity +not involving +#### 6.4 High Availability +Deployable Multi-Active + \ No newline at end of file diff --git a/en_US/Design_Documentation/Orchestrator/images/Create_an_orchestration_sequence_diagram.png b/en_US/Design_Documentation/Orchestrator/images/Create_an_orchestration_sequence_diagram.png new file mode 100644 index 0000000..2b2bcae Binary files /dev/null and b/en_US/Design_Documentation/Orchestrator/images/Create_an_orchestration_sequence_diagram.png differ diff --git a/en_US/Design_Documentation/Orchestrator/images/Import_Orchestration_Sequence_Diagram.png b/en_US/Design_Documentation/Orchestrator/images/Import_Orchestration_Sequence_Diagram.png new file mode 100644 index 0000000..c375c3d Binary files /dev/null and b/en_US/Design_Documentation/Orchestrator/images/Import_Orchestration_Sequence_Diagram.png differ diff --git a/en_US/Design_Documentation/Orchestrator/images/orchestrator_arch.png b/en_US/Design_Documentation/Orchestrator/images/orchestrator_arch.png new file mode 100644 index 0000000..21f08c9 Binary files /dev/null and b/en_US/Design_Documentation/Orchestrator/images/orchestrator_arch.png differ diff --git a/en_US/Design_Documentation/Orchestrator/images/orchestrator_uml.png b/en_US/Design_Documentation/Orchestrator/images/orchestrator_uml.png new file mode 100644 index 0000000..51de76e Binary files /dev/null and b/en_US/Design_Documentation/Orchestrator/images/orchestrator_uml.png differ diff --git a/en_US/Design_Documentation/UserGuide/DSS-UserGuide_Module_Design.md b/en_US/Design_Documentation/UserGuide/DSS-UserGuide_Module_Design.md new file mode 100644 index 0000000..f93dfa7 --- /dev/null +++ b/en_US/Design_Documentation/UserGuide/DSS-UserGuide_Module_Design.md @@ -0,0 +1,51 @@ +# DSS-UserGuide module design + +### Introduce + +The DSS user manual module is a newly added function module of DSS1.0, which is used to provide user guidance for DSS users. It contains many problems and solutions encountered during the use of DSS, and also includes some function points. Users can self-service search for solutions to problems encountered. It can also be used to associate error codes in the later stage, which supports directly locating the solutions that have been entered in the knowledge base after the pop-up error codes. The guide module stores the files in the fields of the table in the form of html. It needs to parse the md file and convert it into html. Since some files have links and need to be jumped, it is necessary to build a gitbook to display and manage these documents. In order to be efficient To synchronize the dss-guide-user module, package the files on gitLab, upload and decompress them to the specified directory of the server where gitbook is located, and scan the specified directory regularly through guide-user to achieve the purpose of synchronization. + +## Introduction to the main modules of dss_guide + +The DSS_Guide module mainly contains the definitions of Restful, Service, Dao, and Entity. + +### GuideGroupService + +It is used to solve GuideGroup's ability to add, query, modify, save, delete, etc. It also has the ability to synchronize Summary.md. The guide module can parse this file, and then locate the file that needs to be read and write it to the database regularly according to the configuration paths of the parsed directories at all levels in the file, so as to complete the synchronization. When the service is running on other servers, In order to avoid repeated installation of gitbook, the guide module needs to pass the ip of the server where the configuration file is located, and then automatically synchronize the file to the server where the guide module is located for display. + +### GuideContentService + +It is used to handle the save, query, update and delete operations of GuideContent. + +### GuideChapterService + +It is used to deal with the specific content of manual chapters, including chapter search, ID-based query, deletion, saving, etc. + +### GuideCatalogService + +It is used to synchronize the knowledge base, support batch insertion of directory content, and implement operations such as saving, deleting, and querying the directory structure classification. + + +### Core flow chart + +![](./images/16559707626688.png) + + +### data structure + +![](./images/1653309930194.png) + +### dss_guide_group + +Divide the grouping of dss_guide, including group_id, path (access path), title, etc. + +### dss_guide_chapter + +Used to store the detailed content of the dss_guide chapter, including catalog_id, title, content, content_html. Associate with the content of dss_guide_catalog. + +### dss_guide_content + +It is used to store the description content after grouping, and will be planned under the corresponding group. Contains title, type, content, content_html, etc. + +### dss_guide_catalog + +It is used to classify the content of dss_guide, which is equivalent to the directory structure of the knowledge base and has a hierarchical directory relationship. diff --git a/en_US/Design_Documentation/UserGuide/images/1653309535303.png b/en_US/Design_Documentation/UserGuide/images/1653309535303.png new file mode 100644 index 0000000..c731a5d Binary files /dev/null and b/en_US/Design_Documentation/UserGuide/images/1653309535303.png differ diff --git a/en_US/Design_Documentation/UserGuide/images/1653309930194.png b/en_US/Design_Documentation/UserGuide/images/1653309930194.png new file mode 100644 index 0000000..f7006ab Binary files /dev/null and b/en_US/Design_Documentation/UserGuide/images/1653309930194.png differ diff --git a/en_US/Design_Documentation/UserGuide/images/16559707626688.png b/en_US/Design_Documentation/UserGuide/images/16559707626688.png new file mode 100644 index 0000000..c7e2f8f Binary files /dev/null and b/en_US/Design_Documentation/UserGuide/images/16559707626688.png differ diff --git a/en_US/Design_Documentation/Workspace/README.md b/en_US/Design_Documentation/Workspace/README.md new file mode 100644 index 0000000..5a64006 --- /dev/null +++ b/en_US/Design_Documentation/Workspace/README.md @@ -0,0 +1,96 @@ +Introduce +------------------------- +Workspace：The workspace module belongs to the project service and provides the interface implementation of all the functions of the workspace. It mainly includes these interfaces: the interface for obtaining dictionary table information, the workspace menu, the crud operation interface for component permissions, +Workspace role, user's crud operation interface. The favorite interface of the menu bar component in the upper left corner of the workspace. + +User use function points: + +| Primary Module | Secondary Module | User Function | +|-------------|----------- |---------------- | +| Workspace | Workspace Management (Workspace Admins only) | Query, Add, Edit, Delete Workspace Users | +| | | Permission settings for the menu component of the workspace role | +| | | Home Settings for Workspace Roles | +| | Menu bar in the upper left corner of the home page | Favorite components to the left list | +| | | Subscribe to Favorite Components, which will be displayed in the upper navigation bar | +| | | Click a component to jump to the corresponding component page | + +### Restful/Service class function introduction: + +| Core Interface/Class | Core Function | +|---------------------------|------------------------------| +| DSSDictionaryRestful、DSSDictionaryServiceImpl | Provide dictionary information acquisition interface, query corresponding records from dictionary table through parameter key or parentKey | +| DSSWorkspacePrivRestful、DSSWorkspacePrivServiceImpl | Provides viewing and editing functions for the permission information of the menu component of the workspace role | +| DSSWorkspaceRestful、DSSWorkspaceServiceImpl | Provides basic functional interfaces of workspace, such as creating workspace, obtaining workspace list, obtaining permission information of menu components, etc. | +| DSSWorkspaceRoleRestful、DSSWorkspaceRoleServiceImpl | Provides query and creation interfaces for workspace roles | +| DSSWorkspaceUserRestful、DSSWorkspaceUserServiceImpl | Provides an interface for adding, deleting, modifying, and querying workspace users | + +### User function UML class diagram: +![](images/workspace_uml.drawio.png) + +## database table design + +Workspace basic information table: +```roomsql +CREATE TABLE `dss_workspace` ( +`id` bigint(20) NOT NULL AUTO_INCREMENT, +`name` varchar(255) DEFAULT NULL COMMENT 'workspace name', +`label` varchar(255) DEFAULT NULL COMMENT 'label', +`description` varchar(255) DEFAULT NULL COMMENT 'description', +`create_by` varchar(255) DEFAULT NULL COMMENT 'creater', +`create_time` datetime DEFAULT NULL COMMENT 'create time', +`department` varchar(255) DEFAULT NULL COMMENT 'department', +`product` varchar(255) DEFAULT NULL COMMENT 'product，reserved field', +`source` varchar(255) DEFAULT NULL COMMENT 'reserved field', +`last_update_time` datetime DEFAULT NULL, +`last_update_user` varchar(30) DEFAULT NULL COMMENT 'last update user', +`workspace_type` varchar(20) DEFAULT NULL comment 'workspace type', +PRIMARY KEY (`id`), +UNIQUE KEY `name` (`name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +``` +Workspace dictionary table： +```roomsql +CREATE TABLE `dss_workspace_dictionary` ( + `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'primary key ID', + `workspace_id` int(11) DEFAULT '0' COMMENT 'Space ID, default is 0, all spaces have', + `parent_key` varchar(200) DEFAULT '0' COMMENT 'parent key', + `dic_name` varchar(200) NOT NULL COMMENT 'dictionary name', + `dic_name_en` varchar(300) DEFAULT NULL COMMENT 'dictionary english name', + `dic_key` varchar(200) NOT NULL COMMENT 'key is equivalent to encoding, the space starts with w_, and the project is p_', + `dic_value` varchar(500) DEFAULT NULL COMMENT 'the value corresponding to the key', + `dic_value_en` varchar(1000) DEFAULT NULL COMMENT 'The value corresponding to the key (English)', + `title` varchar(200) DEFAULT NULL COMMENT 'title', + `title_en` varchar(400) DEFAULT NULL COMMENT 'english title', + `url` varchar(200) DEFAULT NULL COMMENT 'url', + `url_type` int(1) DEFAULT '0' COMMENT 'url type: 0-internal system, 1-external system; default is internal', + `icon` varchar(200) DEFAULT NULL COMMENT 'icon', + `order_num` int(2) DEFAULT '1' COMMENT 'order', + `remark` varchar(1000) DEFAULT NULL COMMENT 'remnark', + `create_user` varchar(100) DEFAULT NULL COMMENT 'creater', + `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `update_user` varchar(100) DEFAULT NULL COMMENT 'update user', + `update_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', + PRIMARY KEY (`id`), + UNIQUE KEY `idx_unique_workspace_id` (`workspace_id`,`dic_key`), + KEY `idx_parent_key` (`parent_key`), + KEY `idx_dic_key` (`dic_key`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='data dictionary table'; +``` +Component Menu Information Sheet +```roomsql +CREATE TABLE `dss_workspace_menu` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `name` varchar(64) DEFAULT NULL COMMENT 'memu name', + `title_en` varchar(64) DEFAULT NULL COMMENT 'English title of menu', + `title_cn` varchar(64) DEFAULT NULL COMMENT 'Chinese title of menu', + `description` varchar(255) DEFAULT NULL COMMENT 'description', + `is_active` tinyint(1) DEFAULT '1' COMMENT 'is active', + `icon` varchar(255) DEFAULT NULL COMMENT 'icon', + `order` int(2) DEFAULT NULL COMMENT 'order', + `create_by` varchar(255) DEFAULT NULL COMMENT 'creater', + `create_time` datetime DEFAULT NULL COMMENT 'create time', + `last_update_time` datetime DEFAULT NULL, + `last_update_user` varchar(30) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +``` \ No newline at end of file diff --git a/en_US/Design_Documentation/Workspace/images/workspace_uml.drawio.png b/en_US/Design_Documentation/Workspace/images/workspace_uml.drawio.png new file mode 100644 index 0000000..7618f56 Binary files /dev/null and b/en_US/Design_Documentation/Workspace/images/workspace_uml.drawio.png differ diff --git a/en_US/Design_Documentation/appconn/DSS_Access_Scheduling_System.md b/en_US/Design_Documentation/appconn/DSS_Access_Scheduling_System.md new file mode 100644 index 0000000..41a578d --- /dev/null +++ b/en_US/Design_Documentation/appconn/DSS_Access_Scheduling_System.md @@ -0,0 +1,81 @@ +# DSS access scheduling system + +## background + +There are many batch timing scheduling systems currently used in the field of big data, such as Azkaban, Dophinscheduler, Airflow, etc. DataSphere Studio (DSS) supports publishing workflows designed by users to different scheduling systems for scheduling. Currently, the default support for publishing to Azkaban. The user has completed the DAG design of the workflow in DSS, including workflow resource file loading, workflow parameter setting, data import, data quality check, node code writing, visual report design, email output, node resource file upload and running parameter setting After the operation, it can be debugged and executed in DSS. After verifying that the execution of all nodes is correct, it will be released to the scheduling system, and the scheduling system will schedule and execute regularly according to the configuration of the scheduled task. + +### release method + +The DSS integration new scheduling system needs to be accessed by AppConn. Users need to define the corresponding XXXSchedulerAppConn according to different scheduling systems, and the transformation integration specification and the structured integration specification are defined in the SchedulerAppConn. The transformation integration specification contains the transformation of DSS project-level content and DSS workflow-level content to third-party scheduling systems. The access of DSS to the dispatching system can be divided into the following two types: + +1. Engineering level release + +It refers to converting all the workflows in the project, and uniformly packaging and uploading the converted content to the scheduling system. There are mainly the ProjectPreConversionRel interface, which defines the workflow that needs to be converted in the project. + +2. Workflow level release + +It refers to the conversion according to the granularity of the workflow, and only the content of the workflow is packaged and uploaded to the scheduling system. The current workflow definition of DSS is stored in the BML file in the form of Json, and the metadata information of the workflow is stored in the database. + +## The main steps + +### Parser + +JsonToFlowParser is used to convert the Json of the workflow into Workflow. Workflow is the standard format for operating workflow in DSS. It includes the node information of the workflow, the edge information of the workflow, the parent workflow, the child workflow, the workflow resource file, Workflow property file, workflow creation time, update time, workflow user, workflow proxy user, workflow metadata information such as name, ID, description, type, whether it is a root workflow, etc. These are parsed according to the content of Json and converted into DSS operable Workflow objects, such as AzkabanWorkflow and DolphinSchedulerWorkflow. + +### Converter + +Turn the Workflow of DSS into a workflow that can be recognized by the access scheduling system. Each scheduling system has its own definition for the workflow. If you convert the nodes of the DSS workflow into Azkaban's job format files, or into DolphinScheduler's tasks, you can also convert them in reverse, and convert the workflow of the scheduling system into a workflow that can be loaded and displayed by DSS, and the dependencies of the workflow. , the connection of the node is converted into the dependency of the corresponding scheduling system. You can also check whether the workflow nodes under the project have nodes with the same name in Converter. For example, the use of nodes with the same name is not allowed in Azkaban's scheduling system. + +WorkflowConVerter defines the workflow conversion output directory structure, including workflow storage directory, workflow resource file storage, and establishment of sub-workflow storage directory. For example, Azkaban also includes creating a project conversion directory in the project-level conversion operation, and establishing a workflow conversion directory according to the workflow situation in the project. Convert Workflow to dolphinSchedulerWorkflow or ScheduleWorkFlow in convertToRel + +NodeConverter defines the output content of node conversion: such as Azkaban's ConvertNode, it will convert the node content of the workflow into the corresponding Job file content. Including the name, type, dependency of the conversion node, the execution command of the node (depending on linkis-jobtype parsing), the configuration parameters of the node, the label of the node, etc. Finally, it is stored in the format defined by the Job file. The Converter of DolphinScheduler converts the nodes in DSS into tasks in DolphinScheduler, and builds the execution script of Shell type Task, and converts the node content of DSS into the parameters required for the execution of the custom dss-dolphinscheduler-client.sh script. + +```--java + addLine.accept("LINKIS_TYPE", dssNode.getNodeType()); //Workflow Node Type + addLine.accept("PROXY_USER", dssNode.getUserProxy()); //proxy user + addObjectLine.accept("JOB_COMMAND", dssNode.getJobContent()); //Excuting an order + addObjectLine.accept("JOB_PARAMS", dssNode.getParams()); //Node execution parameters + addObjectLine.accept("JOB_RESOURCES", dssNode.getResources()); //Node execution resource file + addObjectLine.accept("JOB_SOURCE", sourceMap); //Node's source information + addLine.accept("CONTEXT_ID", workflow.getContextID()); //context ID + addLine.accept("LINKIS_GATEWAY_URL", Configuration.getGateWayURL()); //linkis gateway address + addLine.accept("RUN_DATE", "${system.biz.date}"); //run date variable +``` + +### Tunning + +It is used to complete the overall adjustment operation before the project is released. In the implementation of Azkaban, the path setting of the project and the storage path setting of the workflow are mainly completed.Because at this time it is possible to operate the project=》workflow=》sub-workflow，It is convenient for setting operation from outside to inside. For example, the storage of workflow depends on the storage location of the project, and the storage of child workflow depends on the location of the parent workflow. The child node calculation is completed in FlowTuning, and the end node is automatically added. + +## Scheduling the AppConn implementation + +### AbstractSchedulerAppConn + +The abstract class of scheduling AppConn, the new scheduling system AppConn access can directly inherit this abstract class, it implements the SchedulerAppConn interface, and inherits AbstractOnlySSOAppConn, to get through the SSO login between DSS and the scheduling system. For example, the already integrated DolphinSchedulerAppConn and SchedulisAppConn both inherit this abstract class. + +This abstract class contains two types of Standard + +The first is ConversionIntegrationStandard to support workflows that transform DSS orchestration into a scheduling system + +The second is SchedulerStructureIntegrationStandard, a structured integration specification for DSS and scheduling systems + +### ConversionIntegrationStandard + +Conversion integration specification for scheduling systems, including DSSToRelConversionService for converting DSS orchestration into scheduling system workflows. An interface is also reserved to support converting the workflow of the scheduling system into the orchestration of DSS + +### AbstractSchedulerStructureIntegrationStandard + +The scheduling system organizational structure integration specification is specially used for the organizational structure management of the scheduling system, mainly including engineering services and orchestration services. + +### ProjectService + +* The unified creation, update, deletion and duplicate checking operations of the project are realized. +* It is used to open up the engineering system of DSS engineering and access third-party application tools, and realize the collaborative management of engineering. +* If the scheduling system needs to open up the engineering system with DSS, it needs to implement all the interfaces of engineering services in the structured integration specification. + +### OrchestrationService + +The orchestration service is used for the unified orchestration specification of the scheduling system, and has the following functions: + +* Unified orchestration specification, specially used to open up the orchestration system of DSS and SchedulerAppConn (scheduling system). +* For example: connect DSS workflow and Schedules workflow. +* Please note that if the docking SchedulerAppConn system does not support management workflow itself, you do not need to implement this interface. diff --git a/en_US/Design_Documentation/appconn/appconn.md b/en_US/Design_Documentation/appconn/appconn.md new file mode 100644 index 0000000..b435d94 --- /dev/null +++ b/en_US/Design_Documentation/appconn/appconn.md @@ -0,0 +1,29 @@ +DSS-AppConn Design Documentation +------ +## Introduce +    The principle of the original AppJoint is to define a top-level interface AppJoint. The third party implements this interface and stores its own connection information in the DSS table, and implements a "proxy service" in DSS that communicates with the third-party system. At the initial stage of initialization , creating an instance of the service through the reflection mechanism, and using the connection information in the table, DSS can use the "proxy service" to establish HTTP communication with the third-party system, thereby invoking the third-party system. However, there are shortcomings in the design of AppJoint. Each connected application instance needs to generate an AppJoint instance. Different instances of the same application are not logically associated. The application instance AppConn of each system is DSS1.0. The top-level interface, in DSS1.0, its own orchestration mode, workflow, single-task node, etc., are all instances of AppConn. In addition, third-party systems that access DSS need to implement the AppConn interface to implement DSS. Integrate with third-party systems to call third-party applications. Logically, AppConn has a higher abstraction logic than AppJoint. AppConn is similar to a class instance, while AppJoint is similar to an instance. + +### Introduction to related modules +|Level 1 Module | Level 2 Module | Function Introduction| +|-------------|-----------|----------------| +|dss-appconn|appconns|Access DSS to implement AppConn related specification implementation code| +| |dss-appconn-core|Appconn interface and basic class definition| +| |dss-appconn-loader|Instantiation, loading and assembly of the AppConn compiled package of the connected application| +| |dss-appconn-manager|Interact with framework modules and manage related AppConn instance information| +| |dss-scheduler-appconn|Abstract AppConn Definition for Scheduling System Implementation| +| |linkis-appconn-engineplugin|Implement the relevant specifications of linkis appconn and open up the interaction between DSS AppConn and Linkis| + + + +| Core interface/class | Core functions | +|---------------------------|------------------------------| +| DSSDictionaryRestful、DSSDictionaryServiceImpl | Provide dictionary information acquisition interface, query corresponding records from dictionary table through parameter key or parentKey | +| DSSWorkspacePrivRestful、DSSWorkspacePrivServiceImpl | Provides viewing and editing functions for the permission information of the menu component of the workspace role | +| DSSWorkspaceRestful、DSSWorkspaceServiceImpl | Provides basic functional interfaces of workspace, such as creating workspace, obtaining workspace list, obtaining permission information of menu components, etc. | +| DSSWorkspaceRoleRestful、DSSWorkspaceRoleServiceImpl | Provides query and creation interfaces for workspace roles | +| DSSWorkspaceUserRestful、DSSWorkspaceUserServiceImpl | Provides an interface for adding, deleting, modifying, and querying workspace users | + +### AppConn Architecture Diagram +![](./images/appconn_class_uml.png) +![](./images/appconn_structure.png) +![](./images/appconn_load_process.png) diff --git a/en_US/Design_Documentation/appconn/images/appconn_class_uml.png b/en_US/Design_Documentation/appconn/images/appconn_class_uml.png new file mode 100644 index 0000000..3c6765d Binary files /dev/null and b/en_US/Design_Documentation/appconn/images/appconn_class_uml.png differ diff --git a/en_US/Design_Documentation/appconn/images/appconn_load_process.png b/en_US/Design_Documentation/appconn/images/appconn_load_process.png new file mode 100644 index 0000000..733958d Binary files /dev/null and b/en_US/Design_Documentation/appconn/images/appconn_load_process.png differ diff --git a/en_US/Design_Documentation/appconn/images/appconn_structure.png b/en_US/Design_Documentation/appconn/images/appconn_structure.png new file mode 100644 index 0000000..2287a65 Binary files /dev/null and b/en_US/Design_Documentation/appconn/images/appconn_structure.png differ diff --git a/en_US/Design_Documentation/labelRoute/DSS_label-based_routing_forwarding.md b/en_US/Design_Documentation/labelRoute/DSS_label-based_routing_forwarding.md new file mode 100644 index 0000000..ccc414f --- /dev/null +++ b/en_US/Design_Documentation/labelRoute/DSS_label-based_routing_forwarding.md @@ -0,0 +1,92 @@ +# DSS label-based routing forwarding + +## introduction + +Labels are a newly introduced concept in DSS1.0. Labels are used in many places in the DSS framework as the basis for forwarding routes. For example, in the multi-environment support of DSS, each request is labeled with an environment, such as DEV, PROD, etc. These can be used to determine which service the current request is being sent to, or which AppConn Instance should currently be used. + +## Format of the label + +route type label + +``` +labels={"route":"dev"} +``` + +Get type request + +``` +dssurl?labels=dev +``` + +The above two are the more common formats used for Label in DSS + +## Label-based routing forwarding + +1. Service forwarding + +Service forwarding refers to forwarding HTTP requests based on tag matching to the corresponding service instance. Currently, it is mainly implemented by DSS-GateWay. By defining the DSS-GateWay plug-in and deploying it to the lib directory of Linkis-GateWay, DSS can be implemented. The request is forwarded to the DSS service instance. + +* Get the tag directly from the request. The current DSS request contains an environment tag like DEV or PROD. The GET type request is directly in the URl parameter, and the POST type request is placed in the body of the request, with json format to submit. After DSS-GateWay intercepts the request sent to DSS, it will parse the label and service name in the request. Then, according to the service name and label, go to the registration center such as Euraka to find the corresponding service instance. According to the matching rules, when the service name is completely matched, it will be sent to the service instance first. When there is no complete match, it will be sent to the service instance with the highest matching similarity, and if there is no match at all, it will be sent to the default DSS service instance project. + +``` +object DSSGatewayParser { + //The URL extraction regular matching rules of DSS can refer to the front-end request URL of DSS for analysis + val DSS_HEADER = normalPath(API_URL_PREFIX) + "rest_[a-zA-Z][a-zA-Z_0-9]*/(v\\d+)/dss/" + val DSS_URL_REGEX = (DSS_HEADER + "([^/]+)/" + "([^/]+)/.+").r + val DSS_URL_DEFAULT_REGEX = (DSS_HEADER + "([^/]+).+").r + + //This method is mainly used in AppConn applications, such as Visualis url extraction + val APPCONN_HEADER = normalPath(API_URL_PREFIX) + "rest_[a-zA-Z][a-zA-Z_0-9]*/(v\\d+)/([^/]+)/" + val APPCONN_URL_DEFAULT_REGEX = (APPCONN_HEADER + "([^/]+).+").r + +} +``` + +* RouteLabelParser introduces the Label-related dependencies of Linkis into the application, and then defines the label name of the service in the yaml of the application. After the service is started, the metadata information of the label will be registered with the registration center. Registries are all tagged, and there may be multiple tags. Then implement RouteLabelParser in Linkis's GateWay, and after GateWay intercepts the request, parse out the relevant route type label list in parse, and then forward the request by Linkis-gateway's label-based forwarding rules. This is another tag-based forwarding that requires the use of the tag module in Linkis. + +``` + + org.apache.linkis + linkis-instance-label-client + ${linkis.version} + +``` + +2、AppConn Instance Forward + +In DSS1.0, the concept of AppConn is used, and the original AppJoint is replaced, and the third-party system is integrated in the way of AppConn. And the biggest feature of AppConn is that it can support multiple application instances, and each application instance has its own label. Add a new table dss_appconn_instance to the database: + +``` +CREATE TABLE `dss_appconn_instance` ( + `id` int(20) NOT NULL AUTO_INCREMENT COMMENT 'primary key', + `appconn_id` int(20) NOT NULL COMMENT 'appconn primary key', + `label` varchar(128) NOT NULL COMMENT 'instance label', + `url` varchar(128) DEFAULT NULL COMMENT 'access third-party urls', + `enhance_json` varchar(1024) DEFAULT NULL COMMENT 'json format configuration', + `homepage_uri` varchar(255) DEFAULT NULL COMMENT 'homepage uri', + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=66 DEFAULT CHARSET=utf8mb4 COMMENT='Instance table of dss instance'; +``` + +When each AppConn is deployed, its Instance information of AppConn will be registered in DSS. And use the above database table to record the label, access address, homepage address, configuration information, etc. corresponding to the instance. + +After each appConn instance has a label, in the DSS framework, each time it interacts with AppConn, it will search for the corresponding operation instance object according to the label information of the current request, and then send a specific request to the object. . + +``` + //Get label content from execution parameters + val labels = engineExecutorContext.getProperties.get("labels").toString + //Obtain the corresponding appconn instance based on the tag content and the loaded AppConn + getAppInstanceByLabels(labels, appConn) match { + case Some(appInstance) => + val developmentIntegrationStandard = appConn.asInstanceOf[OnlyDevelopmentAppConn].getOrCreateDevelopmentStandard + //Obtain the corresponding execution service from the development specification according to the instance information + val refExecutionService = developmentIntegrationStandard.getRefExecutionService(appInstance) +``` + +Please use EnvDSSLabel for the label in DSS, including the key and value of the label. Inherited from the label system GenericLabel in linkis. + +``` +//Build a DSS label object, the default key is: DSSEnv +DSSLabel envDSSLabel = new EnvDSSLabel(rollbackOrchestratorRequest.getLabels().getRoute()); +``` + diff --git a/en_US/Design_Documentation/project/DSS_Engineering_Module_Design_Documentation.md b/en_US/Design_Documentation/project/DSS_Engineering_Module_Design_Documentation.md new file mode 100644 index 0000000..798cb39 --- /dev/null +++ b/en_US/Design_Documentation/project/DSS_Engineering_Module_Design_Documentation.md @@ -0,0 +1,96 @@ +# DSS Engineering + + + +In actual development and production, projects are often used to manage and develop a type of data application. A project can be an actual type of data application, including workflow, single task, etc. The projects under each workspace are isolated from each other. + + + +## 1. Architecture Design + +- DSS itself can create and manage projects. Including create, view, modify, delete and other functions. At the same time, DSS provides engineering integration specifications to complete the docking operation with external components. +- DSS projects are created and bound to each other synchronously with projects of external systems (or entities at the same level) through project integration specifications. +- The external system obtains the project corresponding to the user in the DSS through the project integration specification, and completes the unified management of the underlying entities. +- The external system obtains the user's project authority in DSS through the project integration specification, and further restricts the operation authority of the native project. + +![project](images\project.png) + + + +### 2.1.1, project creation + +Brief process description: Create a DSS project 》Create a third-party application project 》Save the relationship between the project and user permissions + +flow chart: + +![Create project flow diagrams](images/project-create.png) + +### 2.1.2, project editing + +Brief process description: Edit user permission relationship 》Edit third-party project information 》Edit DSS project basic information + + + +flow chart: + +![Create project flow diagrams](images/project-edit.png) + +### 2.1.3, Project deletion + +Brief process description: determine whether there is delete permission "delete third-party application project "delete DSS project + +### + +## 3、Database table structure design + +``` +--dss Engineering Basic Information Sheet +dss_project: + id + name + source + description + user_id + username + workspace_id + create_time + create_by + update_time + update_by + org_id + visibility + is_transfer + initial_org_id + isArchive + pic + star_num + product + application_area + business + is_personal + create_by_str + update_by_str + dev_process + orchestrator_mod + visible + + --dss project and user authority relationship table + dss_project_user: + id + project_id + username + workspace_id + priv + last_update_time + + --Relationship between third-party application engineering and dss engineering + dss_appconn_project_relation: + id + project_id + appconn_instance_id + appconn_instance_project_id + +``` + + + diff --git a/en_US/Design_Documentation/project/images/project-create.png b/en_US/Design_Documentation/project/images/project-create.png new file mode 100644 index 0000000..1e42fe7 Binary files /dev/null and b/en_US/Design_Documentation/project/images/project-create.png differ diff --git a/en_US/Design_Documentation/project/images/project-edit.png b/en_US/Design_Documentation/project/images/project-edit.png new file mode 100644 index 0000000..24698e3 Binary files /dev/null and b/en_US/Design_Documentation/project/images/project-edit.png differ diff --git a/en_US/Design_Documentation/project/images/project.png b/en_US/Design_Documentation/project/images/project.png new file mode 100644 index 0000000..317e6b0 Binary files /dev/null and b/en_US/Design_Documentation/project/images/project.png differ diff --git a/en_US/Design_Documentation/publish/Workflow_Release_Design.md b/en_US/Design_Documentation/publish/Workflow_Release_Design.md new file mode 100644 index 0000000..0a7e5bb --- /dev/null +++ b/en_US/Design_Documentation/publish/Workflow_Release_Design.md @@ -0,0 +1,19 @@ +## Workflow release design + +## I. Overview + +In actual production applications, the development center is responsible for debugging the business-related workflows. After the debugging is completed, the workflows will be released to the scheduling system for scheduled batch scheduling jobs to realize business automation. + +### 2. Workflow release architecture design + +Workflow publishing is a relatively complex process, which involves functions such as importing, exporting, and publishing workflows. + +##### 1. Publish the process call link diagram: + +![img](images/workflow-publish.png) + +##### 2. Description of important steps + +- Export: Export from Dev Center (dev), including workflow and 3rd party node export. Compress the generated workflow json file into a zip package through the bml service and upload it to the bml file service center. +- Import: Import to production center (prod). Download the zip file saved in the bml file service center, parse the json file, obtain the workflow arrangement information, and save it to the database. +- Publish: Convert the dss workflow arrangement information obtained by importing into the workflow arrangement information available to the scheduling system, compress it into a zip package, and publish it to the wtss scheduling system. diff --git a/en_US/Design_Documentation/publish/images/workflow-publish.png b/en_US/Design_Documentation/publish/images/workflow-publish.png new file mode 100644 index 0000000..bdb57aa Binary files /dev/null and b/en_US/Design_Documentation/publish/images/workflow-publish.png differ diff --git a/en_US/Design_Documentation/workflow/DSS_Workflow_Architecture_Design.md b/en_US/Design_Documentation/workflow/DSS_Workflow_Architecture_Design.md new file mode 100644 index 0000000..31dfcbc --- /dev/null +++ b/en_US/Design_Documentation/workflow/DSS_Workflow_Architecture_Design.md @@ -0,0 +1,60 @@ +## DSS workflow architecture design + +Workflow : refers to "the automation of part or the whole of the business process in the computer application environment". It is an abstract and general description of the business rules between the workflow and its operation steps. Data development work can be greatly facilitated by using workflow. + +### 1. Workflow Architecture + +![workflow](images/workflow.png) + + + +### 2. the introduction of the second module + +##### dss-workflow-server + +The workflow core module includes the workflow CRUD function, the workflow publishing function, the CRUD function of the workflow appconn node, the workflow cs service function, and the PPC external service function. + +| Core service | Core function | +| --------- | -------- | +| DSSFlowService | Contains workflow and sub-workflow CRUD, as well as workflow version management methods | +| WorkflowNodeService | Contains workflow node CRUD, copy, import, export and other methods | +| PublishService | Provides functions related to workflow publishing | +| DSSWorkflowReceiver | PRC Task Call Receiver | +| DSSWorkflowChooser | PRC Task Invocation Selector | + +##### dss-flow-execution-server + +The workflow execution module includes functions related to workflow execution, including real-time workflow execution, selected execution, rerun on failure, and workflow kill. + +##### dss-workflow-sdk + +The workflow toolkit module provides external workflow resource file parsing functions. + +| Core Class | Core Function | +| ---------------- | -------------------------------------------- | +| FlowToJsonParser | Used to parse a flow into a DSSJsonFlow that DSS can use | +| json2flow | Used to parse the workflow json into the required workflow object | + +##### dss-workflow-common + +Workflow basic public module, which abstracts the public entity class and saves it in this module. + +##### dss-linkis-node-execution + +dss calls linkis to execute the node module, which implements the task execution related interfaces provided by linkis. + +| Core Class | Core Function | +| ----------------------- | ------------------------------------------------------------ | +| LinkisNodeExecution | Contains core methods such as task running, task status, task results, task cancellation, and task logs | +| LinkisExecutionListener | Monitor task execution | +| DefaultRetryHandler | Provide a retry mechanism | + +##### dss-workflow-conversion-standard + +The workflow transformation specification module defines the specification for transforming the DSS workflow into a workflow that can be used by other external applications. + +| Core Class | Core Function | +| ------------------------------------- | -------------- | +| ProjectConversionIntegrationStandard | Engineering Conversion Specification | +| WorkflowConversionIntegrationStandard | Workflow Transformation Specification | + diff --git a/en_US/Design_Documentation/workflow/images/workflow.png b/en_US/Design_Documentation/workflow/images/workflow.png new file mode 100644 index 0000000..6f2b580 Binary files /dev/null and b/en_US/Design_Documentation/workflow/images/workflow.png differ diff --git a/en_US/Development_Documentation/AppConn_Development_Guide.md b/en_US/Development_Documentation/AppConn_Development_Guide.md new file mode 100644 index 0000000..7c64617 --- /dev/null +++ b/en_US/Development_Documentation/AppConn_Development_Guide.md @@ -0,0 +1,533 @@ +# AppConn Development Guide + +## 1. Introduction + +The third-party system accesses DSS by implementing an ```AppConn```. DSS provides a variety of ```AppConn``` to facilitate the fast access of third-party systems in different application scenarios. + +The following is a detailed introduction to the various basic ``AppConn``, each of which represents a capability of the DSS framework abstraction: + +- ```OnlySSOAppConn``. If your third-party system wants to complete SSO login-free jump with DSS, you need to use the capabilities provided by this AppConn. +- ```OnlyStructureAppConn```. If your third-party system wants to have a unified organizational structure with DSS, such as unified management operations such as project creation, update, and deletion, and unified management of role rights management, you need to inherit this AppConn. +- ```OnlyDevelopmentAppConn``. This AppConn needs to be implemented if your third-party system wants to be integrated as a node of the DSS workflow. +- ````OptionalAppConn```. The AppConn is an optional non-mandatory implementation of the AppConn specification. It is used to assist the third-party AppConn to provide some special operation capabilities. These operations have nothing to do with the framework logic of DSS, and are not the capabilities that the DSS framework requires the third-party AppConn to have. + +On top of the above basic AppConn, DSS provides several combined AppConns to facilitate integration and docking, which are: + +- ```SchedulerAppConn```. This AppConn is a subclass of ```OnlySSOAppConn``` and ```OnlyStructureAppConn```, used to synchronize a DSS workflow to a third-party scheduling system, supplementing the scheduling capabilities of DSS workflow; if you want to connect with other schedulers system, you need to implement the AppConn. +- ```SecondlyAppConn```. It is a subclass of ```OnlySSOAppConn``` and ```OnlyStructureAppConn```, indicating that the third-party system wants to get through both SSO and organizational structure. +- ```ThirdlyAppConn```. It is a subclass of ```OnlySSOAppConn``, ```OnlyStructureAppConn`` and ```OnlyDevelopmentAppConn```, indicating that the third-party system not only wants to get through the SSO and organizational structure, but also wants to be integrated as a node of the DSS workflow Come in. + +You can choose the corresponding ``AppConn``` to implement according to your actual needs. + +## 2. Detailed introduction of basic AppConn + +The specific implementation of ```AppConn``` includes the following layers, the first layer is the ```AppConn``` layer, the second layer is the ```AppStandard``` specification layer, and the third layer is ``` The Service```` layer, the fourth layer is the ````Operation``` layer, the main business logic is implemented in the ```Operation``` layer, and the parameter transmission is unified using ```Ref`` ` Implementation class. + +Its organizational structure relationship is expressed in a tree structure as follows: + +```html +|-- [OnlySSOAppConn](#21-onlyssoappconn--open-sso-login-free jump): The third-party system and DSS complete the SSO login-free jump, the default implementation has been provided, and the user does not need any method of this AppConn. +| |-- SSOIntegrationStandard: SSO login-free jump specification for DSS and third-party systems, the default implementation has been provided, and users do not need any methods of this AppConn. +| | |-- SSORequestService: Provides a general service capability that can send front-end or back-end requests to third-party AppConn systems integrated with DSS. +| | | |-- SSORequestOperation: Send front-end or back-end requests to third-party AppConn systems integrated with DSS. +| | |-- SSOPluginService: Corresponding to `SSOBuilderService`, after the third-party system introduces the DSS SSO Jar, the Jar will use the Service to parse the background `HttpServletRequest` of the third-party system to obtain the required DSS entity information. +| | | |-- SSOMsgParseOperation: **_No need to pay attention, if you are interested, please refer to the architecture design document_**. +| | | |-- DssMsgCacheOperation::**_No need to pay attention, if you are interested, please refer to the architecture design document_**. +| | | |-- WorkspaceInfoOperation::**_No need to pay attention, if you are interested, please refer to the architecture design document_**. +| | |-- [SSOUserService](#214-dss-user-sync-with-third-party-appconn-): DSS user sync service with third-party AppConn users. **_Can be implemented on demand_**. +| | | |-- SSOUserCreationOperation: Used to request a third-party AppConn to create a user with the same name. +| | | |-- SSOUserUpdateOperation: Used to modify the basic information of third-party AppConn users. +| | | |-- SSOUserGetOperation: Used to request the third-party AppConn to obtain the user information of the same name. +| | | |-- SSOUserDeletionOperation: reserved interface for deleting third-party AppConn users. +| | |-- SSOBuilderService: Corresponding to `SSOPluginService`, when initiating an Http request to a third-party system, it is used to spell out the special URL of a third-party system that supports SSO password-free access capability. +| | | |-- SSOUrlBuilderOperation: **_No need to pay attention, if you are interested, please refer to the architecture design document_**. +| | | |-- DssMsgBuilderOperation: **_No need to pay attention, if you are interested, please refer to the architecture design document_**. +|-- [OnlyStructureAppConn](#22-onlystructureappconn--access-dss-organizational structure specification): Unify the organizational structure with DSS, such as the unified management operations of creating, updating, and deleting projects. +| |-- StructureIntegrationStandard: The second-level specification of DSS, which is the organizational structure specification, mainly provides engineering management service capabilities, role management service capabilities and third-party system state management capabilities. +| | |-- ProjectService: An engineering system used to connect DSS projects with connected third-party systems to realize collaborative management of projects. +| | | |-- ProjectCreationOperation: Requests the third-party system to create a third-party refProject that is associated one-to-one with the DSS project. +| | | |-- ProjectUpdateOperation: Requests a third-party system to update the associated third-party refProject. +| | | |-- ProjectDeletionOperation: Requests the third-party system to delete the associated third-party refProject. +| | | |-- ProjectSearchOperation: Request a third-party system to query the project that contains projectName. +| | |-- RoleService: Unified role specification, used to connect the role system between DSS and each integrated access system. **_Reserved specification, no need to implement _** for the time being. +| | | |-- RoleCreationOperation: Requests the third-party system to create a third-party refRole that is one-to-one associated with the DSS role. +| | | |-- RoleUpdateOperation: Requests a third-party system to update a third-party refRole that is already one-to-one associated with a DSS role. +| | | |-- RoleDeletionOperation: Requests the third-party system to delete the associated third-party refRole. +| | | |-- RoleUrlOperation: When the user sets the permission information of this role of the third-party system on the DSS front-end, it returns the jumpURL of the front-end jumping page of the role of the third-party system. +| | |-- AppStatusService: The third-party application status check specification. **_Reserved specification, no need to implement _** for the time being. +| | | |-- AppStatusOperation: Check the status of the third-party system. If the third-party system is found to be abnormal, it can be alerted in time or displayed to the front-end users. +|-- [OnlyDevelopmentAppConn](#23-onlydevelopmentappconn--access-dss-development process specification): If your third-party system wants to be integrated as a node of DSS workflow, you need to implement this AppConn. +| |-- DevelopmentIntegrationStandard: DSS development process specification. Can directly inherit `AbstractDevelopmentIntegrationStandard`, including five `DevelopmentService` that need to be implemented by the user. +| | |-- RefCRUDService: Job management specification, mainly used to manage jobs of third-party application tools (named refJob). +| | | |-- RefCreationOperation, the creation operation of the Job (named refJob) of the third-party application tool. +| | | |-- RefCopyOperation, the copy operation of the Job (named refJob) of the third-party application tool. +| | | |-- RefUpdateOperation, the update operation of the Job (named refJob) of the third-party application tool. +| | | |-- RefDeletionOperation, the delete operation of the Job (named refJob) of the third-party application tool. +| | |-- RefExecutionService: Job execution specification, mainly used to execute jobs of third-party application tools. +| | | |-- RefExecutionOperation: Execute the refJob of the third-party AppConn. +| | |-- RefExportService: Job export specification, mainly used to export jobs of third-party application tools. +| | | |-- RefExportOperation: Supports exporting third-party AppConn jobs as Linkis BML materials or `InputStream` byte streams. +| | |-- RefImportService: Job import specification, mainly used to import jobs of third-party application tools. +| | | |-- RefImportOperation: By passing in Linkis BML material or `InputStream` byte stream, the third-party AppConn needs to support converting it into a refJob. +| | |-- RefQueryService: Job query specification, mainly used to open the Job page of a third-party application tool. +| | | |-- RefQueryJumpUrlOperation: used to return a jumpURL that can be jumped when the user double-clicks the workflow node on the front end. +| | | |-- RefQueryOperation: Get some information about the refJob of the third-party AppConn under refProject. +|-- [OptionalAppConn](#24-optionalappconn--optional non-mandatory implementation-appconn-specification): Assist third-party AppConn to provide some special `Operation` capabilities. +| |-- OptionalIntegrationStandard: Optional specification. +| | |-- OptionalService: A generic `Service` service provided for third-party AppConn to build some `Operation` with special capabilities for use by DSS-embedded application tools. +| | | |-- OptionalOperation: An `Operation` for the third-party system with special capabilities. +``` + +### 2.1 OnlySSOAppConn - get through SSO without login jump + +If your third-party system wants to complete SSO login-free jump with DSS, you need to use the capabilities provided by this AppConn. + +`OnlySSOAppConn` provides the default abstract class `AbstractOnlySSOAppConn`, which already provides a default implementation of the first-level specification. + +**_Please note: In general, the relevant `AppConn` subclasses will actively inherit this abstract class, so you do not need to implement any methods of `OnlySSOAppConn`_**. + +If your third-party system only wants to implement SSO login-free jump with DSS, then you don't even need to re-write an `OnlySSOAppConn` implementation, you can just use `SSOAppConn` directly. + +How to use `SSOAppConn`? Just add a new record to the dss_appconn table in DSS, specifying the reference field as sso. + +For the introduction of the dss_appconn table, please refer to: [Introduction to the dss_appconn table] (Third-party system access DSS development guide.md#331-dss_appconn-table). + +The core of **_`OnlySSOAppConn` is to require a third-party AppConn to import the SSO Jar package of DSS as required to complete the code implementation and introduction of related interfaces, as follows_**: + +DSS provides the core SSO Jar package for SSO login-free jump. The third-party system needs to import the SSO Jar package and trust the DSS user in the Filter to complete the user's login-free jump. + +The design scheme of DSS SSO login-free jump is as follows: + +![SSO免登录跳转](../Images/Development_doc/Third-part_System_Access_Development_Guide/SSO_password-free_jump.png) + +At present, the third-party AppConn system can connect to the first-level SSO login-free jump specification of DSS in two ways: one is the connection method of Spring Web applications; the other is the connection method of non-Spring Web applications. + +#### 2.1.1 Spring Web application implements DSS first-level specification + +Define a ```Configuration```, as shown below: + +```java +@Configuration +public class DSSConfiguration { + + @Bean + public FilterRegistrationBean dssSSOFilter(@Autowired SSOPluginFilter ssoPluginFilter) { + FilterRegistrationBean filter = new FilterRegistrationBean<>(); + filter.setName("dssSSOFilter"); + filter.setFilter(ssoPluginFilter); + // Specify the priority, the order must be before the user login judgment filter of the third-party application + filter.setOrder(-1); + return filter; + } +} + +``` + +Please note: If ```@Autowired SSOPluginFilter ssoPluginFilter``` is introduced by means of dependency injection, you need to add ```com.webank.wedatasphere``` to ```@ComponentScan(basePackages=” of SpringBoot main class com.webank.wedatasphere”)````; otherwise, use the following directly: +```java +@Configuration +public class DSSConfiguration { + + @Bean + public FilterRegistrationBean dssSSOFilter() { + FilterRegistrationBean filter = new FilterRegistrationBean<>(); + filter.setName("dssSSOFilter"); + filter.setFilter(new SpringOriginSSOPluginFilter()); + // Specify the priority, the order must be before the user login judgment filter of the third-party application + filter.setOrder(-1); + return filter; + } +} +``` + +#### 2.1.2 Non-Spring web application + +You need to inherit ``OriginSSOPluginFilter`` by yourself, and add the filter to the web container. The order must be before the user login judgment filter of the third-party application. + +The specific code reference is shown in the following figure: +https://github.com/WeBankFinTech/Schedulis/blob/branch-0.6.1/azkaban-web-server/src/main/java/azkaban/webapp/servlet/DSSOriginSSOFilter.java + +![Schedulis实现DSS一级规范](../Images/Development_doc/Third-part_System_Access_Development_Guide/Schedulis_implements_DSS_Level1_specification.png) + +#### 2.1.3 Implements UserInterceptor + +Please note: Whether it is a Spring Web application or a non-Spring Web application, you need to implement a ```UserInterceptor ```. + +According to different scenarios, DSS provides the following three ``UserInterceptor``` sub-interfaces: + +- ``DSSInternalUserInterceptor``: If the third-party application is a DSS internal application, implement this interface; +- ``HttpRequestUserInterceptor````: If the user login information (such as: User object) of the third-party application is stored in cookies, you need to implement this interface. The main purpose of this interface is to hope that the third-party system will write the user Go to ``HttpServletRequest```` and return, so that the latter Filter can detect the user in the cookie and release it directly. You can refer to the implementation of Schedulis: https://github.com/WeBankFinTech/Schedulis/blob/branch -0.6.1/azkaban-web-server/src/main/java/azkaban/webapp/servlet/WTSSHttpRequestUserInterceptor.java +- ``HttpSessionUserInterceptor``: If the user login information of the third-party application (such as: User object) is stored in the HttpSession, you need to implement this interface. The main purpose of this interface is to hope that the third-party system will write the user to ``HttpSession``, so that the following Filter can detect the user in the cookie and let it go directly; + +Please note: **If it is a Spring Web application, you need to mark the implemented ```UserInterceptor``` with ```@Component```, so that ```SpringOriginSSOPluginFilter``` can be loaded into the ``` UserInterceptor```. ** + +#### 2.1.4 DSS user synchronization with third-party AppConn + +From DSS1.1.0, the DSS admin module provides the function of adding LDAP users. + +Since some upper-layer application tools themselves also provide user management functions (store user information in the database), in order to open up the user system between DSS users and third-party AppConn, `OnlySSOAppConn` also provides a `SSOUserService` service, which is used when DSS users When a user adds or modifies, it can support synchronously adding or modifying third-party AppConn users. + +Since `SSOUserService` is not a service that must be implemented, if your third-party application does not have user management functions, you can skip this section directly. + +If you want to get through the user system of third-party AppConn and DSS, please directly implement the following `Operation`: + +- `SSOUserCreationOperation`: Added user operation, when the DSS Admin module creates a new user, or when the user logs in to DSS for the first time, synchronously requests the third-party AppConn to create a user. +- `SSOUserUpdateOperation`: update user operation, when the DSS Admin module updates a user, synchronously request a third-party AppConn to update a user. +- `SSOUserDeletionOperation`: delete user operation, when the DSS Admin module deletes a user, synchronously requests the third-party AppConn to delete a user. +- `SSOUserGetOperation`: Request the third-party AppConn to obtain the user information of the third-party AppConn with the unique English username username. + +You only need to implement these four abstract classes directly in AppConn, no need to implement a `SSOUserService` manually, the DSS framework will automatically recognize and load these four operations. + +### 2.2 OnlyStructureAppConn - Access to DSS Organizational Structure Specification + +```OnlyStructureAppConn``` requires that the user must return a ```StructureIntegrationStandard``` object, which is: DSS organization structure specification. + +Users can directly inherit ```AbstractStructureIntegrationStandard```, which contains three ```StructureService``` that need to be implemented by users, namely: + +- ``ProjectService```, engineering integration specification. It is used to open up the engineering system of DSS engineering and access third-party application tools, and realize the collaborative management of engineering. +- ``RoleService``, the role integration specification. This specification is a reserved specification, and the DSS framework layer has not yet been connected with these two specifications, and the user can directly return null. +- ``AppStatusService``, the third-party application status check specification. This specification is a reserved specification, and the DSS framework layer has not yet been connected with these two specifications, and the user can directly return null. + +#### 2.2.1 Engineering Integration Specification + +``ProjectService``` realizes the unified creation, update, deletion and duplicate checking of projects. The project specification structure is shown in the following figure: + +![Organizational Structure Specification](../Images/Development_doc/Third-part_System_Access_Development_Guide/Organizational_Structure_Specification.png) + +##### 2.2.1.1 StructureOperation Introduce + +The engineering integration specification ``ProjectService``, requires that the ```StructureOperation``` must be implemented as follows: + +- ``ProjectCreationOperation``, the project creation protocol. +- ``ProjectUpdateOperation``, project update protocol. +- ``ProjectDeletionOperation``, the project deletion protocol. +- ``ProjectSearchOperation``, the project duplication checking protocol. + +The interface definition of ``ProjectCreationOperation``` is given below. Other ProjectOperations also provide rich interface annotations, which will not be posted here. + +```java + import com.webank.wedatasphere.dss.standard.app.structure.StructureOperation; + import com.webank.wedatasphere.dss.standard.app.structure.project.ref.DSSProjectContentRequestRef; + import com.webank.wedatasphere.dss.standard.app.structure.project.ref.ProjectResponseRef; + import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + + public interface ProjectCreationOperation> + extends StructureOperation { + + /** + * This method attempts to request the third-party utility to create a third-party refProject that is associated one-to-one with the DSS project. + * If the creation is successful, please return a ProjectResponseRef containing the project ID (named refProjectId) of the third-party application tool, + * So that DSS can then use refProjectId to manage this project (named refProject) of the third-party utility. + *
+ * The returned refProjectId is the basis for other ProjectOperation projects that can operate this third-party application tool. DSS is calling other + * During ProjectOperation, the refProjectId will be passed in as a method parameter, so that the user can normally find the corresponding refProject and perform the corresponding operation. + * @param projectRef contains DSS project information. + * @return returns a ProjectResponseRef containing refProjectId + * @throws ExternalOperationFailedException Thrown if an exception occurs during creation. + */ + ProjectResponseRef createProject(R projectRef) throws ExternalOperationFailedException; + + } +``` + +As can be seen from the above, ```ProjectCreationOperation``` is a generic interface with ```>```, which requires the user's implementation class to specify a specific generic type. + +Please note: each ``ProjectOperation``` interface is a generic interface, the user must specify the appropriate generic class as required when implementing, so that the DSS framework layer can normally find the ```ProjectRequestRef``` and complete it successfully transfer. + +It is recommended that users inherit ```AbstractStructureOperation``` when implementing the corresponding ```ProjectOperation```. The following is an example of ```VisualisAppConn```: + +```java + public class VisualisProjectCreationOperation extends AbstractStructureOperation + implements ProjectCreationOperation { + } +``` + +##### 2.2.1.2 Introduction to ProjectRequestRef System of StructureOperation + +Here we will focus on the ``ProjectOperation``` method entry parameter ```ProjectRequestRef``` and return value ```ProjectResponseRef``` system. + +According to the needs, ``ProjectRequestRef``` mainly has the following basic definitions: + +- ```DSSProjectContentRequestRef```, contains DSS project information, such as: ```DSSProject``` object, DSS project permission information (mainly accessUsers, editUsers, releaseUsers). It is mainly provided to ```ProjectCreationOperation``` and ```ProjectUpdateOperation``` to be used when creating and updating projects of third-party applications. +- ```RefProjectContentRequestRef```, contains the project ID information of the third-party application tool, such as: ```refProjectId```, which is convenient for users to find the refProject corresponding to the third-party application tool and perform corresponding operations, such as: `` `ProjectDeletionOperation```, ```ProjectUpdateOperation``` and ```ProjectSearchOperation``` are used when deleting, updating and searching. +- ```ProjectUpdateRequestRef``` is a subclass of ```DSSProjectContentRequestRef``` and ```RefProjectContentRequestRef```. When a third-party application tool updates a project, it needs not only the relevant information of the DSS project, but also the information of the refProject. ID information. + +Each of the above ```ProjectRequestRef``` has provided a specific implementation class in the corresponding interface, you can directly import and use when implementing the corresponding ```ProjectOperation```, such as ```DSSProjectContentRequestRef.DSSProjectContentRequestRefImpl ````. + +##### 2.2.1.3 Introduction to ProjectResponseRef System of StructureOperation + +The ```ProjectResponseRef``` is relatively simple, only requires the user to return ```refProjectId```, the DSS framework will store the ```refProjectId``` in the database, which is convenient for calling each ```ProjectOperation`` `, you can pass in ```refProjectId``` as needed. + +Among them, only ```ProjectCreationOperation``` and ```ProjectSearchOperation``` require a ```ProjectResponseRef``` containing a ```refProjectId```, while ```ProjectUpdateOperation``` and ` ``ProjectDeletionOperation``` is only required to return a ```ResponseRef```, which simply tells the DSS framework whether the delete and update operations were successful. + +You can also refer to the implementation of VisualisAppConn: [Please click me to view](https://github.com/WeDataSphere/DataSphereStudio/tree/dev-1.1.0/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/project) + +### 2.3 OnlyDevelopmentAppConn - access to DSS development process specification + +```OnlyDevelopmentAppConn``` requires that the user must return a ```DevelopmentIntegrationStandard``` object, which is: DSS development process specification. + +DSS 开发流程规范用于打通并关联 DSS 的 Job 与集成的第三方 AppConn 的一个 Job，并在 DSS 的编排器（如：DSS 工作流）中对第三方 AppConn 的 Job 进行统一管理，DSS 编排器会提供通用的从需求 -> 设计 -> 开发 -> 调试 -> 导出 -> 导入 -> 发布 的全流程数据应用开发管理能力。 + +Users can directly inherit ```AbstractDevelopmentIntegrationStandard```, which contains five ```DevelopmentService``` that need to be implemented by users, namely: + +- ```RefCRUDService```, Job management specification, mainly used to manage jobs of third-party application tools (named ```refJob```). +- ``RefExecutionService``, Job execution specification, mainly used to execute jobs of third-party application tools. +- ``RefExportService``, Job export specification, mainly used to export jobs of third-party application tools. +- ``RefImportService``, Job import specification, mainly used to import jobs of third-party application tools. +- ``RefQueryService``, Job query specification, mainly used to open the Job page of third-party application tools. + +At present, DSSOrchestrator mainly needs to use development process specifications, such as: DSS workflow, addition, deletion, modification, and query operations of workflow nodes of various third-party application tools, import and export operations, real-time node execution, and opening third-party node front-end pages and other operations. + +##### 2.3.1.1 DevelopmentService Introduce + +Since the architecture design of each ``DevelopmentService```` is exactly the same, here is a detailed introduction to the Job management specification ```RefCRUDService```. + +The ```DevelopmentOperation``` that must be implemented by the Job Management Specification ```RefCRUDService``` is as follows: + +- ```RefCreationOperation```, the creation protocol for the Job (named ```refJob```) of the third-party utility. +- ```RefCopyOperation```, the copy protocol for the Job (named ```refJob```) of a third-party utility. +- ```RefUpdateOperation```, the update protocol for the third-party utility's Job (named ```refJob```). +- ```RefDeletionOperation```, the deletion protocol for the third-party utility's Job (named ```refJob```). + +The interface definition of ```RefCreationOperation``` is given below, and other ```DevelopmentOperation``` also provide rich interface comments, which will not be posted here. + +```java + import com.webank.wedatasphere.dss.standard.app.development.ref.DSSJobContentRequestRef; + import com.webank.wedatasphere.dss.standard.app.development.ref.RefJobContentResponseRef; + import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + + public interface RefCreationOperation> + extends DevelopmentOperation { + + /** + * This method will attempt to request a third-party utility to create a third-party refJob under refProject that is one-to-one associated with the DSS Job. + * If the creation is successful, please return a RefJobContentResponseRef containing the job ID (named refJobId) of the third-party application tool, + * So that DSS can use refJobId to manage the refJob under the refProject corresponding to the third-party application tool. + *
+ * The returned refJobId is the basis for other DevelopmentOperation jobs that can operate on this third-party utility. DSS is calling other + * During DevelopmentOperation, the refJobId will be passed in as a method parameter, so that the user can normally find the corresponding refJob and perform the corresponding operation. + * + */ + RefJobContentResponseRef createRef(K requestRef) throws ExternalOperationFailedException; + + } + +``` + +Here we focus on the implementation reference of ```RefQueryJumpUrlOperation``` and ```RefExecutionOperation```. + +The following is an implementation reference for ```RefQueryJumpUrlOperation```: + +```java + public class TestRefQueryOperation + extends AbstractDevelopmentOperation + implements RefQueryJumpUrlOperation { + + public QueryJumpUrlResponseRef query(ThirdlyRequestRef.QueryJumpUrlRequestRefImpl ref) throws ExternalOperationFailedException { + String jumpUrl = ref.getSSOUrlBuilderOperation().setReqUrl(getBaseUrl()).redirectTo(getBaseUrl() + "/jobUi/1") + // nodeId is mainly provided for "open first, then create" type of workflow nodes. For details, please refer to How to Add Workflow Nodes in DSS Workflow #1.2 Special Connections for Third-Party Node Types + .addQueryParameter(QueryJumpUrlConstant.NODE_ID.getKey(), QueryJumpUrlConstant.NODE_ID.getValue()) + .addQueryParameter("projectId", ref.getProjectRefId().toString()) + // The id value of the refJob corresponding to the third-party system, the "id" here is just an example, the third-party system is not necessarily the name, it depends on RefCreationOperation + // The refJobContent value brought back. + .addQueryParameter("id", String.valueOf(ref.getRefJobContent().get("id"))) + .getBuiltUrl(); + return QueryJumpUrlResponseRef.newBuilder().setJumpUrl(jumpUrl).success(); + } + } +``` + +The following is an implementation reference for ```RefExecutionOperation```: + +```java + public class TestRefExecutionOperation + extends LongTermRefExecutionOperation + implements Killable { + + /** + * Asynchronously submitted to the third-party AppConn, requesting the execution of the refJob. + * Usually, the third-party AppConn will return an executionId, please put the executionId into {@code RefExecutionAction}, + * So that the following state(), result() and kill() methods can request the third-party AppConn to operate through the executionId in this RefExecutionAction. + * @return contains the RefExecutionAction of the executionId + * @throws ExternalOperationFailedException Thrown when the submission fails + */ + @Override + protected RefExecutionAction submit(RefExecutionRequestRef.RefExecutionProjectWithContextRequestRef requestRef) throws ExternalOperationFailedException { + // submitTo is pseudocode, which means submitting a request to a third-party system and obtaining the executionId + String executionId = submitTo(requestRef); + // Please define an implementation class of AbstractRefExecutionAction to store the extra information you want + AbstractRefExecutionAction action = new AbstractRefExecutionAction(){}; + action.setExecutionRequestRefContext(requestRef.getExecutionRequestRefContext()); + action.setId(id); + return action; + } + + /** + * Get the status of a refJob that has been submitted to a third-party AppConn. + * You can also get the status of refJob and the execution progress of refJob in this method, and then call + * The RefExecutionAction.getExecutionRequestRefContext().updateProgress() method updates the execution progress of the refJob, + * So that DSS can directly display the real-time progress information of the refJob on the front end. + * @param action contains RefExecutionAction of executionId + * @return returns the state of refJob + * @throws ExternalOperationFailedException Thrown when getting status fails + */ + @Override + public RefExecutionState state(RefExecutionAction action) throws ExternalOperationFailedException { + // getStateFrom is pseudo code representing the state of requesting executionId from a third-party system + String state = getStateFrom(action); + // toRefExecutionState is pseudo code that translates the state of the third-party system into RefExecutionState + RefExecutionState state = toRefExecutionState(state); + // (Optional) You can also get the execution progress of refJob and update it at the same time + // (Not required) getProgressFrom is pseudocode, which indicates the progress of requesting executionId from the third-party system + long progress = getProgressFrom(action); + // (optional) update progress + action.getExecutionRequestRefContext().updateProgress(progress); + return state; + } + + /** + * Get the result of a refJob that has been submitted to a third-party AppConn. + * Please note: This method will only be called when the state returned by the state() method is successful. + * @param action contains RefExecutionAction of executionId + * @return returns the result of refJob + * @throws ExternalOperationFailedException Thrown when the result fails to be obtained + */ + @Override + public ExecutionResponseRef result(RefExecutionAction action) throws ExternalOperationFailedException { + // Please confirm first: Does the task of executionId have a result set? If there is a result set, do you want to make it available to downstream workflow nodes? + // If you don't want either, just simply return the result, like this: + if(isSucceed) { + return ExecutionResponseRef.newBuilder().success(); + } else { + return ExecutionResponseRef.newBuilder().setException(t).build(); + } + // If you want both, you need to write the result set as follows: + // Please confirm the result set type first. For details, please refer to the result set type supported by ExecutionRequestRefContext + // Here we take the table type result set as an example + ResultSetWriter resultSetWriter = null; + try { + resultSetWriter = action.getRequestRef().getExecutionRequestRefContext().createTableResultSetWriter(); + // getColumnsFrom is pseudo code to get all field information + List columns = getColumnsFrom(action); + resultSetWriter.addMetaData(new TableMetaData(columns.toArray(new Column[0]))); + // getDataFrom is pseudo code to get all table data + List> resultList = getDataFrom(action); + if(resultList != null && !resultList.isEmpty()) { + resultList.forEach(list -> resultSetWriter.addRecord(new TableRecord(list.toArray()))); + } + resultSetWriter.flush(); + action.getRequestRef().getExecutionRequestRefContext().sendResultSet(resultSetWriter); + } catch (Exception e) { + LOGGER.error("fetch result failed!", e); + // To facilitate troubleshooting, you can also use the following methods to print more information + action.getRequestRef().getExecutionRequestRefContext().appendLog("Richer log information."); + throw new ExternalOperationFailedException(90176, "fetch result failed!", e); + } finally{ + IOUtils.closeQuietly(resultSetWriter); + } + return ExecutionResponseRef.newBuilder().success(); + } + + /** + * Request to kill the refJob of the third-party AppConn. + * Note: also return true if refJob has completed execution (success or failure). + * @param action contains RefExecutionAction of executionId + * @return returns the result of kill refJob, true for kill success, false for kill failure + */ + @Override + public boolean kill(RefExecutionAction action) { + //killFrom is pseudocode, which means to request kill executionId from a third-party system + return killFrom(action); + } + + } +``` + +It should be noted that all ``DevelopmentOperation``` are also generic interfaces, and the user must specify the appropriate generic class as required when implementing, so that the DSS framework layer can normally find the ```DevelopmentRequestRef``` The call completed successfully. + +It is recommended that users inherit ```AbstractDevelopmentOperation``` when implementing the corresponding ```DevelopmentOperation```. The following is an example of ```VisualisAppConn```: + + +##### 2.3.1.2 DevelopmentRequestRef Introduce + +``DevelopmentRequestRef``` The system is more complicated, so let's start with the basic definition: + +- ```DSSJobContentRequestRef```, contains the information of the DSS workflow node, ```dssJobContent``` stores the basic information of the workflow node, all keys have been included in ```DSSJobContentConstant```. +- ```RefJobContentRequestRef```, contains the refJob ID information of the third-party application tool, such as: ```refJobContent```, which is convenient for users to find the refJob corresponding to the third-party application tool and perform corresponding operations, such as: `` `RefUpdateOperation```, ```RefDeletionOperation```, etc. are used in delete, update and lookup. +- ``DSSContextRequestRef``, contains the contextId information of the DSS workflow. +- ``ProjectRefRequestRef``, contains the refProjectId of the third-party application tool. +- ````UpdateRequestRef```, a sub-interface of ````DSSJobContentRequestRef``` and ```RefJobContentRequestRef```. +- ```CopyRequestRef```, is a sub-interface of ```RefJobContentRequestRef```, and contains the copied workflow version information. +- ```ImportRequestRef```, a sub-interface of ```CopyRequestRef```, contains resourceMap , which represents the metadata file of the workflow node, through which the workflow node can be restored. +- ```QueryJumpUrlRequestRef```, including ```SSOUrlBuilderOperation```, can help to encapsulate the URL of the front-end page corresponding to the third-party system refJob. + +We also provide specific implementation classes for each of the above ```DevelopmentRequestRef```, there are currently two implementation systems: ```OnlyDevelopmentRequestRef``` and ```ThirdlyRequestRef```. + +When you read the source code, you can see that ```OnlyDevelopmentRequestRef``` and ```ThirdlyRequestRef``` define exactly the same subclass, +The only difference is that ```OnlyDevelopmentRequestRef``` is for third-party ```AppConn`` that does not have a project concept (ie its corresponding ```AppConn``` does not implement ```OnlyStructureAppConn```). ` used, +And ```ThirdlyRequestRef``` is for those third-party ```AppConn``` with project concept. + +It should be noted that each ```DevelopmentRequestRef``` implementation class of ```ThirdlyRequestRef``` and ```OnlyDevelopmentRequestRef``` also defines the implementation class of each scene as needed. You can, according to your actual needs, Select the corresponding implementation class to fill in the generic type required by each ``DevelopmentOperation```. + +Here is an example of ```RefCreationOperation```. + +Since Visualis has the concept of a project, and Visualis has opened up the context capability of DSS workflow (that is, the result set of the upstream node can be directly used as the view of the Visualis widget for visual chart analysis, so the definition of ```VisualisRefCreationOperation``` is as follows: + +```java +public class VisualisRefCreationOperation + extends VisualisDevelopmentOperation + implements RefCreationOperation { + // Omit its internal specific implementation method... + } +``` + +```VisualisRefCreationOperation``` selected ```ThirdlyRequestRef.DSSJobContentWithContextRequestRef``` as the generic class K required by ```RefCreationOperation>```. + +The following is the definition of ```ThirdlyRequestRef.DSSJobContentWithContextRequestRef```: + +```java +class DSSJobContentWithContextRequestRef extends DevelopmentRequestRefImpl + implements DSSJobContentRequestRef, DSSContextRequestRef, + ProjectRefRequestRef {} +``` + +It can be seen that `DSSJobContentWithContextRequestRef` also implements `DSSContextRequestRef` (representing context capabilities) and `ProjectRefRequestRef` (representing engineering capabilities). ability, +When creating `DSSJobContentWithContextRequestRef`, context information and project information `set` will be automatically entered, so that `VisualisRefCreationOperation` can use these two capabilities normally. + +##### 2.3.1.3 DevelopmentResponseRef Introduce + +``DevelopmentResponseRef``` The system is relatively simple, as follows: + +- ```RefJobContentResponseRef```, corresponding to ```RefJobContentRequestRef```, contains the refJob ID information of third-party application tools, such as: ```refJobContent```, which is convenient for users to find the corresponding third-party application tools refJob to operate accordingly. +- ```ExportResponseRef```, corresponding to ```ImportRequestRef```, where resourceMap refers to the metadata file exported by the refJob from the third-party application system, through which the workflow node can be restored. +- ```QueryJumpUrlResponseRef```, returns a jumpUrl, which is used to jump to the front-end page corresponding to the third-party system refJob through this jumpUrl when the front-end double-clicks the DSS workflow node. + + +### 2.4 OptionalAppConn - Optional non-mandatory implementation of the AppConn specification + +It is used to assist the third-party `AppConn` to provide some special `Operation` capabilities. These `Operation` have nothing to do with the framework logic of DSS, and are not the capabilities that the DSS framework requires the third-party `AppConn` to have. + +However, some applications embedded in DSS (in the `dss-apps` directory) may use some incoming external AppConns and expect these third-party external AppConns to have some special capabilities. + +For example: `DolphinSchedulerTokenGetOperation` provided by `DolphinSchedulerAppConn` to get the `token` of the DolphinScheduler user. + +If users want to use `OptionalAppConn`, they need to introduce `OptionalAppConn` on their corresponding AppConn class, for example: + +```java +public class DolphinSchedulerAppConn extends AbstractSchedulerAppConn implements OptionalAppConn { + // Omit its internal specific implementation method... +} +``` + +It only needs to be introduced simply, and then the user only needs to directly write the relevant `OptionalOperation` implementation class, the DSS framework will automatically scan the entire AppConn jar package, and automatically register these `OptionalOperation` into its AppConn. + +We strongly recommend that users inherit from the `AbstractOptionalOperation` class. + +Finally, a brief introduction to the difference between `OptionalOperation` and `RefQueryOperation`: + +- `OptionalOperation`, provided by the second-level specification `OnlyStructureAppConn`, is used to initiate some additional system requests to the third-party external system, or obtain some additional information of the third-party external system. +- `RefQueryOperation`, which is provided by the third-level specification `OnlyDevelopmentAppConn` to obtain additional information about the refJob of a third-party external system. + +Therefore, `RefQueryOperation` is dedicated to the third-level specification - the development process specification, and is generally only called by a specific DSSOrchestrator (eg: DSS workflow). + +The positioning of `OptionalOperation` is to provide flexible expansion capabilities to facilitate users to initiate some additional system requests to third-party external systems. It does not have any usage restrictions and can be used either by DSS-embedded applications (in the `dss-apps` directory) or by a specific DSSOrchestrator. \ No newline at end of file diff --git a/en_US/Development_Documentation/Front-end_compilation_documentation.md b/en_US/Development_Documentation/Front-end_compilation_documentation.md new file mode 100644 index 0000000..b85c2bc --- /dev/null +++ b/en_US/Development_Documentation/Front-end_compilation_documentation.md @@ -0,0 +1,64 @@ +# DSS front-end compilation documentation + +## start the process + +### 1. Install Node.js + +Download Node.js to your computer and install it. + +Download address: [http://nodejs.cn/download/](http://nodejs.cn/download/) (The latest stable version may have incompatibility problems, you can use the tested versions v10.16.2, v- 14.15.0) Back-end students suggest that it be done under Linux + +**This step is only required for the first use. ** + +### Second, the installation project + +Execute the following command in the terminal command line: + +```shell script +cd DataSphereStudio/web +lerna bootstrap +```` + +Instruction introduction: +1. Go to the root directory of the project package: `cd DataSphereStudio/web` +2. Dependencies required to install the project: `lerna bootstrap` +3. If the lerna command is not installed, you can install it through the `npm install lerna -g` command + +### Three, packaging project + +You can package the project and generate the compressed code by executing the following command on the terminal command line: + +```shell script +npm run build +```` + +After the command is successfully executed, a folder named "dist" will appear in the project root directory, which is the packaged code. You can put this folder directly into your static server. + +### Fourth, run the project + +If you want to run the project on a local browser and change the code to see the effect, you need to execute the following command in the terminal command line: + +```shell script +npm run serve +```` + +Access the app in a browser (Chrome is recommended) via the link: [http://localhost:8080/](http://localhost:8080/) . + +When you run the project in this way, the effect of your code changes will be dynamically reflected on the browser. + +**Note: Because the project adopts front-end and back-end separate development, when running on the local browser, you need to set the browser to cross-domain to access the back-end interface:** + +Here is how the chrome browser sets up cross-domain: + +- Cross-domain configuration under windows system: + 1. Close all chrome browsers. + 2. Create a new chrome shortcut, right-click "Properties", select "Target" in the "Shortcut" tab, and add ```--args --disable-web-security --user-data-dir=C:\ MyChromeDevUserData```` + 3. Open chrome browser via shortcut + +- Cross-domain configuration under mac system: + +Execute the following command in the terminal command line (you need to replace yourname in the path, if it does not work, please check the location of the MyChromeDevUserData folder on your machine and copy the path to ```--user-data-dir=`` in the following command ` behind) + +```shell script +open -n /Applications/Google\ Chrome.app/ --args --disable-web-security --user-data-dir=/Users/yourname/MyChromeDevUserData/ +```` \ No newline at end of file diff --git a/en_US/Development_Documentation/How_to_Add_Workflow_Node_in_DSS_Workflow.md b/en_US/Development_Documentation/How_to_Add_Workflow_Node_in_DSS_Workflow.md new file mode 100644 index 0000000..aabad0e --- /dev/null +++ b/en_US/Development_Documentation/How_to_Add_Workflow_Node_in_DSS_Workflow.md @@ -0,0 +1,210 @@ + +## 1. Front-end addition + +DSS workflow supports two new ways of adding workflow nodes: + +- Workflow nodes of the Scriptis script type; +- Workflow nodes for third-party AppConn. + +Please confirm the type of workflow node you want to add first. + +Please note: Workflow nodes of the Scriptis script type must be of a script type already supported by Scriptis. If it is not a script type already supported by Scriptis, please refer to：[Added Scriptis script type](How_to_add_script_types_in_Scriptis.md)。 + +### 1.1 Modify the nodeType.js file + +Modify the ```web/src/apps/workflows/service/nodeType.js``` file as follows: + +``` + // 1. Import the SVG icon of the workflow node + import shell from '../module/process/images/shell.svg'; + + // 2. Specify the nodeType of the workflow node + const NODETYPE = { + SHELL: 'linkis.shell.sh' + } + + // 3. Specify the workflow node and the suffix of its corresponding script type + const ext = { + [NODETYPE.SHELL]: 'shell' + } + + // 4. Bind the icon constants, corresponding to the imported SVG icons in the first step. + const NODEICON = { + [NODETYPE.SHELL]: { + icon: shell, + class: {'shell': true} + } + } +``` + +### 1.2 Special docking of third-party node types + +If you want to add a workflow node of a third-party AppConn, in general, the third-party system page will be embedded in an Iframe. + +If you don't want to add a workflow node by embedding a third-party system page through an Iframe, but just want to add some parameters to the property bar on the right side of the workflow node, you can skip this chapter directly. + +In general, there are two ways to create third-party nodes: + +- Create first, then open. You cannot directly drag and drop a workflow node to the UI canvas. You need to pop up a new node dialog box on the DSS workflow interface. The DSS background calls the ```RefCreationOperation``` corresponding to the third-party ```AppConn``` to create After successfully corresponding to the third-party job, the workflow node can be displayed on the workflow UI interface. Double-click to open and jump to the third-party system interface corresponding to the workflow node. +- Open first, then create. You can directly drag and drop a workflow node to the UI interface, double-click the node to open and jump to the corresponding third-party system interface, and click Save on the third-party system interface to trigger the workflow node save operation on the DSS side. + +The first is the general case, such as the Widget node of Visualis, which does not require any special handling. + +The second case involves the communication between the front-end Iframes, which requires some modifications to the third-party system pages. + +The saving process of the third-party system interface is usually as follows: + + The user clicks save on the third-party system interface -> the third-party system interface initiates an Ajax request, requesting a new task in the background of the third-party system -> the background of the third-party system is successfully saved, and returns the id of the task to the front-end of the third-party system . + +In order to connect with the DSS workflow, the saving process of the third-party system interface needs to be changed to the following process: + + The front-end of the third-party system obtains the nodeId from the URL -> the user clicks save on the third-party system interface -> the third-party system interface initiates an Ajax request to request a new task in the background of the third-party system -> the background of the third-party system is successfully saved , return the id of the task to the front-end of the third-party system -> the front-end of the third-party system transmits the task id to the DSS front-end through Iframe communication technology. + +As you can see, the difference is that two steps need to be added to the first and last, namely: + +- The third-party system front end gets the nodeId from the URL. nodeId is used to specify the received DSS workflow node when transferring data to the parent domain page. Please refer to the following data format description. +- The third-party system front-end transmits the task id to the DSS front-end through the Iframe communication technology. + +For how to include nodeId in the URL of the front-end of the third-party system, please refer to: [Third-party system access DSS development guide](AppConn_Development_Guide.md), which is used to encapsulate the Iframe URL of the third-party node page, and the front-end will use this URL to jump to third-party systems. +How does the third-party system interface in the Iframe pass data to the parent page? + +As follows: + +```html + // Pass data to parent domain page + window.parent.postMessage('The data to be transmitted must be a character string, please see the data format below！', '*'); +``` + +Note： +- Pass string data to avoid browser compatibility issues, json data can be processed using ```stringify```, ```parse```; +- It is recommended that the second parameter of ``postMessage```` is '*', because to display the specified domain, you need to know the corresponding domain name of the parent page; +- ````postMessage``` can only be called after the iFrame is loaded. + +The data format of the first parameter of ```postMessage``` needs to be a json that has been converted to a string. The format is as follows: + +```json +{ + "type": "Third-party AppConn types, such as: visualis", + "nodeId": "node key for DSS workflow", + "data": { + "action": "add or delete, which means the event type is add or delete", + "jobContent": { + // It is generally recommended to pass the unique id of the task node, and the DSS workflow will put this map into the jobContent of the DSS workflow node. + // Here is an example from Visualis + "widgetId": 122 + } + } +} +``` + +pay attention：**The data format of the first parameter of ```postMessage``` needs to be a json that has been converted to a string**。 + +## 2. New background + +Prerequisite: You need to implement an AppConn first. For how to implement an AppConn, please refer to：[Third-party system access DSS development guide](AppConn_Development_Guide.md)。 + +### 2.1 Add icons file + +The icon for the new workflow node needs to be added to the ```dss-appconn/appconns/dss--appconn/src/main/icons``` directory. + +When you add an icon, please directly copy the code of the icon and put it into the file. For example: create a new widget.icon file, and copy the relevant SVG code into this file. + +### 2.2 init.sql adds the dml sql of the workflow node + +The following shows how to add a workflow node as follows: + +```mysql-sql + +-- 1. Add a workflow node +delete from `dss_workflow_node` where `appconn_name` = 'Specific AppConnName, such as: Visualis'; +insert into `dss_workflow_node` (`name`, `appconn_name`, `node_type`, `jump_type`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon_path`) + values('Workflow node name, such as: display','Specific AppConnName, such as: visualis','Workflow Node Type，such as：linkis.appconn.visualis.display',1,'1','1','0','1','icons/display.icon'); +select @workflow_node_id:=id from `dss_workflow_node` where `name` = 'Workflow node name, such as: display'; + +-- 2. Bind a workflow node to a category (node type of the workflow UI interface), currently supported types are: +-- 1-Data exchange; 2-Data development; 3-Data quality; 4-Data visualization; 5-Data output; 6-Signal node; 7-Function node; 8-Machine learning +delete from `dss_workflow_node_to_group` where `node_id`=@workflow_node_id; +INSERT INTO `dss_workflow_node_to_group`(`node_id`,`group_id`) values (@workflow_node_id,4); + +-- 3. Add an input input box to the right property bar of the workflow node +delete from `dss_workflow_node_to_ui` where `workflow_node_id`=@workflow_node_id; +delete from `dss_workflow_node_ui` where `key` = 'input The key value of the input box, must be in English and globally unique'; +insert into `dss_workflow_node_ui`(`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) + values ('input The key value of the input box, must be in English and globally unique','Chinese prompt description','English description','the title of the input box','Input name','Input',1,NULL,NULL,0,NULL,0,1,1,1,'node'); +select @workflow_node_ui_id:=id from `dss_workflow_node_ui` where `key` = 'input The key value of the input box, must be in English and globally unique'; +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@workflow_node_id, @workflow_node_ui_id); + +-- 4. Add a front-end validation rule to the input input box +delete from `dss_workflow_node_ui_validate` where id=100; +insert into `dss_workflow_node_ui_validate` (id, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values(100, 'NumInterval','[1,15]','驱动器内存大小，默认值：2','Drive memory size, default value: 2','blur'); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (@workflow_node_ui_id,100); + +``` + +#### 2.2.1 Add a workflow node + +```dss_workflow_node``` Explanation of core fields： + +- `appconn_name`，The name of the associated AppConn +- `node_type`，Workflow node type, must start with linkis.[appconn/engineconn]. If it is a Linkis engine node, such as sparkSQL, it must be: linkis.spark.sql; if it is a third-party AppConn, such as a Visualis widget, it must be: linkis.appconn.visualis.widget. +- `support_jump`，Whether to support jump URL, that is, whether the workflow node supports Iframe embedding +- `jump_type`，Jump type: 1 means it is an external node, 2 means it is a Scriptis node. If supportJump is false, this field has no meaning. +- `submit_to_scheduler`，[Reserved field] Indicates whether it can be released to the scheduling system, 1 means it can be released +- `enable_copy`，Whether the node supports copying, 1 means copying +- `should_creation_before_node`，Whether the workflow node needs to be created with a pop-up window before dragging it to the workflow canvas, 1 means it needs to be created with a pop-up window first +- `icon_path`，Workflow node icon path, the icon of each workflow node is stored in the corresponding AppConn + +#### 2.2.2 Bind workflow nodes to a category + +```dss_workflow_node_to_group``` Tables are used to bind workflow nodes to a category (node type of the workflow UI interface). + +Currently, the types defined in the ``dss_workflow_node_group``` table are: + +- 1- Data exchange +- 2- Data Development +- 3- Data Quality +- 4-Data visualization +- 5-Data output +- 6-Signal Node +- 7-function node +- 8-Machine Learning + +If the existing ```node_group_id``` does not meet your needs, you can insert a new workflow node classification into the ```dss_workflow_node_group``` table by yourself. + +#### 2.2.3 add input box + +```dss_workflow_node_ui``` Explanation of core fields： + +- `key`, the key value of the form control, for example: spark.executor.memory, must be globally unique, the key will be stored in the node params property of flow json. +- `description`, description, default prompt for input type. +- `lable_name`, the title of the input control. +- `ui_type`, the input control type, currently supported: 'SELECT, TextArea, Radio, Checkbox, Mutil-SELECT, Binder, Uploader'. Among them, Binder is used to bind upstream nodes, and Uploader is a file upload control. +- `required`, whether the input control is required, 1 is required +- `value`, the selectable value of the input control (such as SELECT, etc.). +- `default_value`, the default value of the input control +- `is_hidden`, whether to hide the input control by default, 1 is hidden +- `condition`, if it is hidden by default, the control will only be displayed when the condition is satisfied, for example: spark.driver.memory=5G, it means that the input control key is spark.driver.memory and the value is 5G, triggering Show this control. +- `is_advanced`, whether it is an advanced parameter, the advanced parameter is hidden by default, and will be displayed only when the user clicks "Show Advanced Parameters" in the property bar on the right. 1 is an advanced parameter +- `order`, the display order, the smaller the order, the higher the order. +- `node_menu_type`, the type of input input box, 1 represents the input input box of the property bar on the right side of the workflow node, and 0 represents the input input box of the new dialog box of the workflow node +- `is_base_info`, whether it is the basic information of the property bar on the right side of the workflow node, 1 is the basic information +- `position`, the location where the key-value key-value pair is stored, including: node, startup, runtime. Where startup means stored in node -> params -> startup of flow json, runtime means stored in node -> params -> runtime of flow json, node means stored in node of flow json. + +#### 2.2.4 Add a front-end validation rule to the input input box + +```dss_workflow_node_ui_validate``` core field explanation: + +- `id`, because one ```dss_workflow_node_ui_validate``` is allowed to be used by multiple ```dss_workflow_node_ui```, so the user needs to decide the id value by himself. When the user selects an id, he must ensure that the id is unique in the whole table. +- `validate_type`, the validation type, there are: None, NumInterval, FloatInterval, Include, Regex, OPF' +- `validate_range`, validation range, for example: [1,15] +- `error_msg`, the Chinese error message when the verification fails +- `error_msg_en`, the English error message when the verification fails +- `trigger`, the trigger rule for verification, including: blur, change. Among them, blur refers to losing the mouse focus, and change refers to when the content changes. + +The NumInterval and FloatInterval requirements must meet the range of values limited by `validate_range`, for example: [1,15]. + +Include requires that the content of `validate_range` must be included. + +Regex requires a regular expression that must satisfy `validate_range`. + +The OPF requirement must be one of the `validate_range` array, for example: [\"apple\",\"banana\"], indicating that the value of the input control must be an apple or a banana. diff --git a/en_US/Development_Documentation/How_to_add_script_types_in_Scriptis.md b/en_US/Development_Documentation/How_to_add_script_types_in_Scriptis.md new file mode 100644 index 0000000..2aa95f8 --- /dev/null +++ b/en_US/Development_Documentation/How_to_add_script_types_in_Scriptis.md @@ -0,0 +1,26 @@ +## 1. New background + +In ```linkis-computation-governance\linkis-manager\label-common\src\main\java\com\webank\wedatasphere\linkis\manager\label\entity\engine\EngineType.scala```, +Added a new enumeration engine type. + +For how to implement a new Linkis engine, please refer to: [How to implement a new engine](https://linkis.apache.org/zh-CN/docs/latest/development/new_engine_conn). + +## 2. New front-end + +Find the ``web/src/common/config/scriptis.js``` file and add a new record to its array. + +The record is of type object and can be configured with the following properties: + +- rule: regular expression, used to match regular file name suffixes; +- executable: whether the script type can be executed; +- lang: Which language is used in the monaco editor, corresponding to its highlighting, associative words and other functions. You can refer to the languages ​​supported by the official website of monaco editor, or customize the language; +- application: corresponds to an engine type of Linkis and is used to pass the parameter executeApplicationName during websocket or http polling; +- runType: corresponds to the script type of a Linkis engine, used to pass the parameter runType during websocket or http polling; +- ext: the file suffix of the script type, such as: ````.hql```; +- scriptType: The script type supported by the system, which is used for judgment when creating a new script. This attribute is used to distinguish script types at the front end because both application and runType may be duplicated; +- abbr: the suffix name of the script type (this attribute is deprecated and will be deleted later); +- label: The name of the script type displayed on the UI interface of the user's new script, with the first letter capitalized; +- logo: The script's icon. The storage path is: ```web/src/apps/scriptis/assets/styles/home.scss```, please add a new LOGO at the end of the file. +- isCanBeNew: Whether it is allowed to be newly created, which is used to display in scenarios where new scripts can be created, such as workspaces; +- isCanBeOpen: Whether it is allowed to be opened, it can be double-clicked or right-clicked to open in modules such as workspace (this property is valid in the foreground, and the background also checks whether the script can be opened); +- flowType: The type used in the workflow. \ No newline at end of file diff --git a/en_US/Development_Documentation/Third-party_System_Access_Development_Guide.md b/en_US/Development_Documentation/Third-party_System_Access_Development_Guide.md index e69de29..125e524 100644 --- a/en_US/Development_Documentation/Third-party_System_Access_Development_Guide.md +++ b/en_US/Development_Documentation/Third-party_System_Access_Development_Guide.md @@ -0,0 +1,372 @@ +# Third-party system access DSS development guide + +There are four steps in total: + +1. Create a brand new AppConn directory in DSS; +2. Implement a new AppConn in the background; +3. Supplement relevant documents required by AppConn; +4. If you want to access the DSS workflow as a workflow node, you need to complete the addition of the workflow node. + +Details as follows: + +## 1. Create a new AppConn directory in DSS + +The directory format requirements of the new AppConn are as follows, please create a new AppConn module according to the required format: + +```html + +|-- DataSphereStudio +| |-- dss-appconn // The root directory of the AppConn framework +| | |-- appconns // All AppConn storage directories +| | | |-- dss--appconn // Added root directory of AppConn +| | | | |-- src +| | | | | |-- main +| | | | | | |-- assembly // The configuration directory for the DSS-loadable appconnName.zip package +| | | | | | | |-- distribution.xml // packaged config file +| | | | | | |-- icons // If you want to use it as a workflow node, here is the icon used to store each workflow node +| | | | | | |-- java // The directory where AppConn code is stored +| | | | | | |-- resources // The resource file storage directory of the AppConn +| | | | | | | |-- appconn.properties // The AppConn configuration file, must be named appconn.properties +| | | | | | | |-- init.sql // dml sql file to import AppConn information into DSS database +| | | | |-- pom.xml // Added AppConn pom file + +``` + +## 2. Implement a new AppConn in the background + +Please refer to: [AppConn Development Guide](AppConn_Development_Guide.md). + +## 3. Supplementary relevant documents required by AppConn + +Mainly divided into the following steps: + +- Supplement pom.xml file +- Supplementary icons file +- Added init.sql +- Supplement distribution.xml +- What appconn.properties does + +### 3.1 Supplemental pom.xml file + +Please refer to: + +```xml + + + + dss + com.webank.wedatasphere.dss + 1.1.0 + ../../../pom.xml + + 4.0.0 + + dss-${appconnName}-appconn + + + + + com.webank.wedatasphere.dss + dss-appconn-core + ${dss.version} + + + linkis-common + org.apache.linkis + + + json4s-jackson_2.11 + org.json4s + + + + + + + com.webank.wedatasphere.dss + dss-development-process-standard-execution + ${dss.version} + + + + + org.apache.linkis + linkis-cs-client + ${linkis.version} + compile + + + + org.apache.linkis + linkis-httpclient + ${linkis.version} + + + linkis-common + org.apache.linkis + + + json4s-jackson_2.11 + org.json4s + + + + + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + + org.apache.maven.plugins + maven-jar-plugin + + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + + src/main/java + + **/*.xml + + + + + src/main/resources + + **/application.yml + **/bootstrap.yml + **/log4j2.xml + + + + + +``` + +### 3.2 icons file + +If the AppConn wants to access the DSS workflow as a workflow node, the icon of the relevant workflow node needs to be stored in the DSS workflow ```dss-appconn/appconns/dss--appconn/src/main/icons``` directory. + +When you add an icon, please directly copy the code of the icon and put it into the file. For example: create a new widget.icon file, and copy the relevant SVG code into this file. + +### 3.3 Added init.sql + +Here is only an example of basic dml sql that can be used in the top menu bar to connect an external system to DSS. If you want to connect AppConn as a workflow node, you should also refer to [How to add a DSS workflow Workflow Node](How_to_Add_Workflow_Node_in_DSS_Workflow.md). + +As follows: + +```mysql-sql + +select @old_dss_appconn_id:=id from `dss_appconn` where `appconn_name` = 'Specific AppConnName，eg：Visualis'; + +delete from `dss_workspace_menu_appconn` WHERE `appconn_id` = @old_dss_appconn_id; +delete from `dss_appconn_instance` where `appconn_id` = @old_dss_appconn_id; +delete from `dss_appconn` where `appconn_name`='Specific AppConnName，eg：Visualis'; + +-- Except appconn_class_path and resource users do not need to pay attention, other fields need to pay attention +-- The following is a detailed introduction +INSERT INTO `dss_appconn` (`appconn_name`, `is_user_need_init`, `level`, `if_iframe`, `is_external`, `reference`, `class_name`, `appconn_class_path`, `resource`) VALUES ('具体的AppConnName，如：Visualis', 0, 1, NULL, 0, NULL, 'com.webank.wedatasphere.dss.appconn.visualis.VisualisAppConn', '', ''); + +select @dss_appconn_id:=id from `dss_appconn` where `appconn_name` = 'Specific AppConnName，eg：Visualis'; + +-- Fields that users need to pay attention to：menu_id、title_en、title_cn、desc_en、desc_cn、labels_en、labels_cn、is_active、access_button_en、access_button_cn、manual_button_url +-- The following is a detailed introduction +INSERT INTO `dss_workspace_menu_appconn`(`appconn_id`,`menu_id`,`title_en`,`title_cn`,`desc_en`,`desc_cn`,`labels_en`,`labels_cn`,`is_active`,`access_button_en`,`access_button_cn`,`manual_button_en`,`manual_button_cn`,`manual_button_url`,`icon`,`order`,`create_by`,`create_time`,`last_update_time`,`last_update_user`,`image`) + values(@dss_appconn_id,2,'appconn English title','This is the Chinese name of the engine','This is English description.','This is the Chinese description.','label1,label2','Label 1, Label 2','1','enter AppConnName1','into the engine1','user manual','User Manual','Please add user manual address',NULL,10,'system',NULL,'system',NULL,NULL); + +-- Fields that users need to pay attention to:enhance_json、homepage_uri +-- The following is a detailed introduction +INSERT INTO `dss_appconn_instance` (`appconn_id`, `label`, `url`, `enhance_json`, `homepage_uri`) VALUES (@dss_appconn_id, 'DEV', 'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/', '', ''); + +``` + +#### 3.3.1 dss_appconn table + +```dss_appconn``` The table is the core table of DSS. Except appconn_class_path and resource users do not need to pay attention, other fields need to be concerned. + +The specific concerns are as follows: + +- ```is_user_need_init```，Whether the user needs to initialize, this field is reserved and can be directly set to 0 +- ```if_iframe```，Whether DSS can open third-party AppConn pages through Iframe embedding, 1 means it can be opened through Iframe, 0 means it can be opened through front-end routing; +- ```is_external```，When opening a third-party AppConn page, whether to open it by opening a new browser tab, 1 means opening a new browser tab, 0 means opening in this page. +- ```reference```，Associated AppConn. Some AppConns want to reuse the implemented AppConn directly, you can directly use this field to specify the corresponding AppConn name. For example, if an AppConn only wants to get through with DSS for a password-free jump (ie, the first-level specification), this field can be directly filled with sso (ie, associated with SSOAppConn); +- ```class_name```，The main class for this AppConn. This field is empty if ```reference``` is not empty. + +#### 3.3.2 dss_workspace_menu_appconn table + +The ```dss_workspace_menu_appconn``` table needs the user's attention, as follows: + +- Other fields do not need to be changed, and related fields can be modified as needed:menu_id、title_en、title_cn、desc_en、desc_cn、labels_en、labels_cn、is_active、access_button_en、access_button_cn、manual_button_url +- ```menu_id```，That is, the classification id of the top menu bar, currently divided into: 1-data exchange; 2-data analysis; 3-production operation and maintenance; 4-data quality; 5-administrator function; 6-data application +- If the existing ```menu_id``` does not meet your needs, you can insert a new menu category into the ```dss_workspace_menu``` table by yourself. +- ```is_active```，That is, whether the AppConn can be accessed, if it is 1, it means available; if it is 0, it means stay tuned +- ```manual_button_url```，URL address of the user manual + +#### 3.3.2 dss_appconn_instance table + +The ```dss_appconn_instance``` table needs the user's attention, as follows: + +- ```enhance_json```，The additional parameters of AppConn are strings in map json format, which have the same function as appconn.properties. They are all configured with additional parameters for the AppConn. +- ```homepage_uri```，The URI of the home page. Note that it's a URI, not a URL. For example: if the home page URL is: http://ip:port/test/home, then the home page URI is: test/home + + +### 3.4 distribution.xml file + +The following reference example does not need to be modified except for appconnName which needs to be modified by the user according to the actual name, and can be directly copied and used. + +Please copy and use as needed. For details, please refer to: + +```xml + + + dss-appconnName-appconn + + dir + + true + + appconnName + + + + + lib + true + true + false + true + true + + + + + + + ${basedir}/conf + + * + + 0777 + conf + unix + + + + ${basedir}/src/main/resources + + init.sql + + 0777 + db + + + + ${basedir}/src/main/icons + + * + + 0777 + icons + + + +``` + +### 3.5 What appconn.properties does + +appconn.properties is used to configure additional parameters of the AppConn. If your AppConn does not define any parameters, you can also not create the appconn.properties file. + +appconn.properties has the same effect as the ```enhance_json``` field of the ````dss_appconn_instance``` table. + +The benefits of using appconn.properties are: + +- ```enhance_json``` is a string in map json format, there are certain thresholds for modification, and it needs to be converted before it can be used; +- Modifying ``enhance_json`` requires changing the database, which is far less simple than directly changing the AppConn configuration file. + +We recommend that you use appconn.properties to configure the parameters of AppConn, you can also choose to use one of them according to your actual needs. + +## 4. the new workflow node + +Please refer to: [How to Add Workflow Node in DSS Workflow](How_to_Add_Workflow_Node_in_DSS_Workflow.md). + +## 5. How DSS loads new AppConn + +Use ```install-appconn.sh``` to install the new AppConn as follows: + +```shell script + cd $DSS_HOME/bin + sh install-appconn.sh +``` + +If it is an existing AppConn, when the AppConn changes, please use ```appconn-refresh.sh``` to refresh the AppConn, as follows: + +```shell script + cd $DSS_HOME/bin + sh appconn-refresh.sh +``` + +Note: You need to put the packaged AppConn zip in the AppConn installation directory in advance. + +## 6. AppConn Contribution + +**If you want to contribute the AppConn plugin to the community, please contact the community and communicate with the community before implementing the AppConn**. And before release, you also need to write `AppConn plugin installation document`, and `AppConn plugin usage document`. +For plugin installation documentation, please refer to: [```VisualisAppConn``` Plugin installation documentation](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/en_US/Installation_and_Deployment/VisualisAppConn_Plugin_Installation_Documentation.md)。 + +For plugin installation documentation, please refer to: [```VisualisAppConn``` Plugin use documentation](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/en_US/Installation_and_Deployment/VisualisAppConn_Plugin_Installation_Documentation.md#4-use-of-visualisappconn)。 + +## 7. How DSS uses AppConn to interact with third-party applications + +The interaction between DSS and the third-party application system is completed through the corresponding AppConn instance. + +When the AppConn is deployed to the specified directory, the ``dss-framework-project-server``` service will load or **refresh** all AppConns when it starts, and store it in the material library. + +Other DSS microservices will pull all AppConn materials on demand from ``dss-framework-project-server``` and dynamically load them into the service process to be called as AppConn instances. + +![DSS frame design](../Images/Development_doc/Third-part_System_Access_Development_Guide/DSS_frame_design.png) \ No newline at end of file diff --git a/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/DSS_frame_design.png b/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/DSS_frame_design.png new file mode 100644 index 0000000..ada7581 Binary files /dev/null and b/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/DSS_frame_design.png differ diff --git a/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/Organizational_Structure_Specification.png b/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/Organizational_Structure_Specification.png new file mode 100644 index 0000000..2f2332c Binary files /dev/null and b/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/Organizational_Structure_Specification.png differ diff --git a/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/SSO_password-free_jump.png b/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/SSO_password-free_jump.png new file mode 100644 index 0000000..fa2aa17 Binary files /dev/null and b/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/SSO_password-free_jump.png differ diff --git a/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/Schedulis_implements_DSS_Level1_specification.png b/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/Schedulis_implements_DSS_Level1_specification.png new file mode 100644 index 0000000..914a291 Binary files /dev/null and b/en_US/Images/Development_doc/Third-part_System_Access_Development_Guide/Schedulis_implements_DSS_Level1_specification.png differ diff --git a/en_US/Images/Install_and_Deploy/DSS&Linkis_one-click_deployment_document_stand-alone_version/eureka.png b/en_US/Images/Install_and_Deploy/DSS&Linkis_one-click_deployment_document_stand-alone_version/eureka.png new file mode 100644 index 0000000..40e1a25 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DSS&Linkis_one-click_deployment_document_stand-alone_version/eureka.png differ diff --git a/en_US/Images/Install_and_Deploy/DSSUserGuide_Deploy/userguide_1.png b/en_US/Images/Install_and_Deploy/DSSUserGuide_Deploy/userguide_1.png new file mode 100644 index 0000000..953ebca Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DSSUserGuide_Deploy/userguide_1.png differ diff --git a/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img.png b/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img.png new file mode 100644 index 0000000..b48eb62 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img.png differ diff --git a/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_1.png b/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_1.png new file mode 100644 index 0000000..7c03d50 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_1.png differ diff --git a/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_2.png b/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_2.png new file mode 100644 index 0000000..ebf825e Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_2.png differ diff --git a/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_3.png b/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_3.png new file mode 100644 index 0000000..35f0b57 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_3.png differ diff --git a/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_4.png b/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_4.png new file mode 100644 index 0000000..160b7e3 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DSS_Debug_Documentation/img_4.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img.png new file mode 100644 index 0000000..0358b6e Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_1.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_1.png new file mode 100644 index 0000000..188e580 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_1.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_10.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_10.png new file mode 100644 index 0000000..8030e7f Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_10.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_11.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_11.png new file mode 100644 index 0000000..8030e7f Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_11.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_12.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_12.png new file mode 100644 index 0000000..d909331 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_12.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_13.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_13.png new file mode 100644 index 0000000..6c79232 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_13.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_14.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_14.png new file mode 100644 index 0000000..98298b7 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_14.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_2.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_2.png new file mode 100644 index 0000000..f453ede Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_2.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_3.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_3.png new file mode 100644 index 0000000..9ee7f19 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_3.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_4.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_4.png new file mode 100644 index 0000000..1df8914 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_4.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_5.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_5.png new file mode 100644 index 0000000..3df6f50 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_5.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_6.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_6.png new file mode 100644 index 0000000..f9b5013 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_6.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_7.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_7.png new file mode 100644 index 0000000..771df63 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_7.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_8.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_8.png new file mode 100644 index 0000000..76d3f74 Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_8.png differ diff --git a/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_9.png b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_9.png new file mode 100644 index 0000000..589c6af Binary files /dev/null and b/en_US/Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_9.png differ diff --git a/en_US/Images/Install_and_Deploy/ExchangisAppConn_Deployment/DSS-Exchangis.png b/en_US/Images/Install_and_Deploy/ExchangisAppConn_Deployment/DSS-Exchangis.png deleted file mode 100644 index c7ee122..0000000 Binary files a/en_US/Images/Install_and_Deploy/ExchangisAppConn_Deployment/DSS-Exchangis.png and /dev/null differ diff --git a/en_US/Images/Install_and_Deploy/QualitisAppConn_Deployment/dss-qualitis.png b/en_US/Images/Install_and_Deploy/QualitisAppConn_Deployment/dss-qualitis.png deleted file mode 100644 index 6dd68f1..0000000 Binary files a/en_US/Images/Install_and_Deploy/QualitisAppConn_Deployment/dss-qualitis.png and /dev/null differ diff --git a/en_US/Images/Install_and_Deploy/QualitisAppConn_Deployment/workflow-qualitis.png b/en_US/Images/Install_and_Deploy/QualitisAppConn_Deployment/workflow-qualitis.png deleted file mode 100644 index 7ad1933..0000000 Binary files a/en_US/Images/Install_and_Deploy/QualitisAppConn_Deployment/workflow-qualitis.png and /dev/null differ diff --git a/en_US/Images/Install_and_Deploy/Standalone_Deployment_Documentation/eureka.png b/en_US/Images/Install_and_Deploy/Standalone_Deployment_Documentation/eureka.png deleted file mode 100644 index 70c7903..0000000 Binary files a/en_US/Images/Install_and_Deploy/Standalone_Deployment_Documentation/eureka.png and /dev/null differ diff --git a/en_US/Images/Install_and_Deploy/VisualisAppConn_Deployment/DSS-Visualis.png b/en_US/Images/Install_and_Deploy/VisualisAppConn_Deployment/DSS-Visualis.png deleted file mode 100644 index 8b2e9f9..0000000 Binary files a/en_US/Images/Install_and_Deploy/VisualisAppConn_Deployment/DSS-Visualis.png and /dev/null differ diff --git a/en_US/Images/Install_and_Deploy/VisualisAppConn_Deployment/Workflow-Visualis.png b/en_US/Images/Install_and_Deploy/VisualisAppConn_Deployment/Workflow-Visualis.png deleted file mode 100644 index 2050232..0000000 Binary files a/en_US/Images/Install_and_Deploy/VisualisAppConn_Deployment/Workflow-Visualis.png and /dev/null differ diff --git a/en_US/Images/Using_Document/Scriptis/hive-6.png b/en_US/Images/Using_Document/Scriptis/hive-6.png new file mode 100644 index 0000000..f1cb9d6 Binary files /dev/null and b/en_US/Images/Using_Document/Scriptis/hive-6.png differ diff --git a/en_US/Images/Using_Document/Scriptis/hive1.png b/en_US/Images/Using_Document/Scriptis/hive1.png new file mode 100644 index 0000000..1932286 Binary files /dev/null and b/en_US/Images/Using_Document/Scriptis/hive1.png differ diff --git a/en_US/Images/Using_Document/Scriptis/hive2.png b/en_US/Images/Using_Document/Scriptis/hive2.png new file mode 100644 index 0000000..8232e3b Binary files /dev/null and b/en_US/Images/Using_Document/Scriptis/hive2.png differ diff --git a/en_US/Images/Using_Document/Scriptis/hive3.png b/en_US/Images/Using_Document/Scriptis/hive3.png new file mode 100644 index 0000000..523f1e1 Binary files /dev/null and b/en_US/Images/Using_Document/Scriptis/hive3.png differ diff --git a/en_US/Images/Using_Document/Scriptis/hive4.png b/en_US/Images/Using_Document/Scriptis/hive4.png new file mode 100644 index 0000000..77e2658 Binary files /dev/null and b/en_US/Images/Using_Document/Scriptis/hive4.png differ diff --git a/en_US/Images/Using_Document/Scriptis/hive5.png b/en_US/Images/Using_Document/Scriptis/hive5.png new file mode 100644 index 0000000..f2ecae3 Binary files /dev/null and b/en_US/Images/Using_Document/Scriptis/hive5.png differ diff --git a/en_US/Images/Using_Document/Scriptis/hive7.png b/en_US/Images/Using_Document/Scriptis/hive7.png new file mode 100644 index 0000000..efe7141 Binary files /dev/null and b/en_US/Images/Using_Document/Scriptis/hive7.png differ diff --git a/en_US/Images/Using_Document/Scriptis/home.png b/en_US/Images/Using_Document/Scriptis/home.png new file mode 100644 index 0000000..1564c51 Binary files /dev/null and b/en_US/Images/Using_Document/Scriptis/home.png differ diff --git a/en_US/Images/Using_Document/Scriptis/udf-3.png b/en_US/Images/Using_Document/Scriptis/udf-3.png new file mode 100644 index 0000000..64bad9b Binary files /dev/null and b/en_US/Images/Using_Document/Scriptis/udf-3.png differ diff --git a/en_US/Images/Using_Document/Scriptis/udf1.png b/en_US/Images/Using_Document/Scriptis/udf1.png new file mode 100644 index 0000000..0866f38 Binary files /dev/null and b/en_US/Images/Using_Document/Scriptis/udf1.png differ diff --git a/en_US/Images/Using_Document/Scriptis/udf2.png b/en_US/Images/Using_Document/Scriptis/udf2.png new file mode 100644 index 0000000..8bb046f Binary files /dev/null and b/en_US/Images/Using_Document/Scriptis/udf2.png differ diff --git a/en_US/Images/Using_Document/workspace/ws_img1.png b/en_US/Images/Using_Document/workspace/ws_img1.png new file mode 100644 index 0000000..bc01094 Binary files /dev/null and b/en_US/Images/Using_Document/workspace/ws_img1.png differ diff --git a/en_US/Images/Using_Document/workspace/ws_img2.png b/en_US/Images/Using_Document/workspace/ws_img2.png new file mode 100644 index 0000000..b68a923 Binary files /dev/null and b/en_US/Images/Using_Document/workspace/ws_img2.png differ diff --git a/en_US/Images/Using_Document/workspace/ws_img3.png b/en_US/Images/Using_Document/workspace/ws_img3.png new file mode 100644 index 0000000..4769068 Binary files /dev/null and b/en_US/Images/Using_Document/workspace/ws_img3.png differ diff --git a/en_US/Images/Using_Document/workspace/ws_img5.png b/en_US/Images/Using_Document/workspace/ws_img5.png new file mode 100644 index 0000000..de7fd5f Binary files /dev/null and b/en_US/Images/Using_Document/workspace/ws_img5.png differ diff --git a/en_US/Images/Using_Document/workspace/ws_img6.png b/en_US/Images/Using_Document/workspace/ws_img6.png new file mode 100644 index 0000000..00caf49 Binary files /dev/null and b/en_US/Images/Using_Document/workspace/ws_img6.png differ diff --git a/en_US/Installation_and_Deployment/DSS&Linkis_one-click_deployment_document_stand-alone_version.md b/en_US/Installation_and_Deployment/DSS&Linkis_one-click_deployment_document_stand-alone_version.md new file mode 100644 index 0000000..37af6b0 --- /dev/null +++ b/en_US/Installation_and_Deployment/DSS&Linkis_one-click_deployment_document_stand-alone_version.md @@ -0,0 +1,398 @@ +# DataSphere Studio & Linkis one-click deployment document stand-alone version + + +### 1. Basic software installation + +- Required command tools (before the official installation, the script will automatically detect whether these commands are available. If they do not exist, they will try to install them automatically. If the installation fails, the user needs to manually install the following basic shell command tools): + + *telnet;tar;sed;dos2unix;mysql;yum;java;unzip;zip;expect* + + +- Software to be installed: + + MySQL (5.5+); JDK (above 1.8.0_141); Python (both 2.x and 3.x are supported); Nginx + + +- The following services must be accessible from this machine: + + Hadoop (**2.7.2, other versions of Hadoop need to compile Linkis**), the installed machine must support the execution of the `hdfs dfs -ls /` command + + Hive (**2.3.3, other versions of Hive need to compile Linkis**), the installed machine must support the execution of the `hive -e "show databases"` command + + Spark (**supports all versions above 2.0**), the installed machine must support the execution of the `spark-sql -e "show databases"` command + + +- Tips: + + If the user is installing Hadoop for the first time, you can refer to: [Hadoop single-machine deployment](https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html) ；For distributed deployment of Hadoop, please refer to: [Hadoop Distributed Deployment](https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/ClusterSetup.html) + + If the user is installing Hive for the first time, please refer to: [Hive Quick Installation and Deployment](https://cwiki.apache.org/confluence/display/Hive/GettingStarted) + + If the user is installing Spark for the first time, the On Yarn mode can refer to: [Spark on Yarn deployment](https://spark.apache.org/docs/2.4.3/running-on-yarn.html) + +### 2. Create a user + +1. Assume **deployment user is hadoop account** (it may not be hadoop user, but it is recommended to use Hadoop super user for deployment, here is just an example) + + +2. Create a deployment user on all machines that need to be deployed for installation. The following command creates a deployment user hadoop + + ```shell + sudo useradd hadoop + ```` + +3. Because the Linkis service uses the `sudo -u ${linux-user}` method to switch the engine to execute the job, the deployment user needs to have sudo permissions, and it is password-free. Follow the steps below to modify the deployment user permissions + + Edit the /etc/sudoers file: + + ```shell + vi /etc/sudoers + ```` + + Add the following to the /etc/sudoers file: + + ```` + hadoop ALL=(ALL) NOPASSWD: NOPASSWD: ALL + ```` + +4. Make sure that the server where DSS and Linkis are deployed can execute commands such as hdfs , hive -e and spark-sql -e normally. In the one-click install script, the components are checked. + + +5. **If the user's Pyspark wants to have the drawing function, you need to install the drawing module on all installation nodes**. The command is as follows: + + ```shell + python -m pip install matplotlib + ``` + +### 3. Prepare the installation package + +- Users can compile their own or go to the release page to download the installation package：[DSS Release-1.1.0](https://github.com/WeBankFinTech/DataSphereStudio/releases/tag/1.1.0) + + **Please note: In order to use Visualis1.0.0 and Exchangis1.0.0 normally, please go to the releases page to download the latest one-click installation deployment package. ** + + **Please note: In order to use Visualis1.0.0 and Exchangis1.0.0 normally, please go to the releases page to download the latest one-click installation deployment package. ** + + + +- The hierarchical directory structure of DSS & Linkis one-click installation deployment package is as follows: + + ```text + ├── dss_linkis # One-click deployment home directory + ├── bin # for one-click installation, and one-click to start DSS + Linkis + ├── conf # Parameter configuration directory for one-click deployment + ├── wedatasphere-dss-x.x.x-dist.tar.gz # DSS backend installation package + ├── wedatasphere-dss-web-x.x.x-dist.zip # DSS front-end and Linkis front-end installation package + ├── wedatasphere-linkis-x.x.x-dist.tar.gz # Linkis backend installation package + ``` + +- If the user chooses to directly deploy by downloading the installation package, they can directly jump to [Modify Configuration](#1) + + +- If the user chooses to compile Linkis or DSS by themselves, please make sure to compile Linkis1.1.1 and the latest code that has pulled the DSS `master` branch. For the compilation method, please refer to:
+ [DSS backend compilation documentation](../Development_Documentation/Compilation_Documentation.md) + [DSS front-end compilation documentation](../Development_Documentation/Front-end_compilation_documentation.md) + [Linkis backend compilation documentation](https://linkis.apache.org/docs/latest/development/linkis_compile_and_package/) + [Linkis front-end compilation documentation](https://linkis.apache.org/docs/latest/development/web_build/) + + + 1. For the back-end installation package, you can directly replace the above Linkis back-end installation package or DSS back-end installation package with the relevant installation package after compilation. + + 2. For the front-end installation package, you need to pay special attention. The directory structure of the entire front-end installation package is as follows: + ```` + ├── wedatasphere-dss-web-x.x.x-dist # DSS and Linkis front-end installation package + ├── config.sh # parameter configuration script + ├── install.sh # Front-end deployment script + ├── dist # DSS front-end package + ├── dss # Front-end package used to store applications deployed with DSS with one click + ├── linkis # Linkis front-end package directory + ├── build.zip # Compressed Linkis front-end package + ```` + + 3. The DSS front-end package can be directly replaced with the relevant installation package compiled by the user. The Linkis front-end package needs to be compressed into an installation package named build.zip, and then replace xx/dss_linkis/wedatasphere-dss-web-x.x.x-dist with it build.zip under the /dss/linkis directory. + + 4. Users need to pay special attention when packaging wedatasphere-dss-web-x.x.x-dist.zip and build.zip, do not directly compress them in the parent directory, they should select all the files under the directory and then compress. + +### 4. modify the configuration + +- You need to modify 'config.sh' and 'db.sh' in 'xx/dss_linkis/conf'. + + +- Run 'config.sh' and modify the parameters as required. The parameters are described as follows: + +```properties +#################### One-click installation of basic deployment configurations #################### + +### deploy user（Deployment user. The default value is the current login user） +deployUser=hadoop + +### Linkis_VERSION（You are not advised to change the value if it is not mandatory） +LINKIS_VERSION=1.1.1 + +### DSS Web（Generally, no modification is required for the local installation. However, check whether the port is occupied. If it is occupied, modify an available port） +DSS_NGINX_IP=127.0.0.1 +DSS_WEB_PORT=8085 + +### DSS VERSION（You are not advised to change the value if it is not mandatory） +DSS_VERSION=1.1.0 + + +############## Linkis other default configuration information start ############## +### Specifies the user workspace, which is used to store the user's script files and log files. +### Generally local directory +##file:// required（You are not advised to change the value if it is not mandatory） +WORKSPACE_USER_ROOT_PATH=file:///tmp/linkis/ +### User's root hdfs path +##hdfs:// required（You are not advised to change the value if it is not mandatory） +HDFS_USER_ROOT_PATH=hdfs:///tmp/linkis +### Path to store job ResultSet:file or hdfs path +##hdfs:// required（You are not advised to change the value if it is not mandatory） +RESULT_SET_ROOT_PATH=hdfs:///tmp/linkis + +### Path to store started engines and engine logs, must be local（You are not advised to change the value if it is not mandatory） +ENGINECONN_ROOT_PATH=/appcom/tmp + + +###HADOOP CONF DIR #/appcom/config/hadoop-config（You are not advised to change the value if it is not mandatory） +HADOOP_CONF_DIR=/appcom/config/hadoop-config +###HIVE CONF DIR #/appcom/config/hive-config（You are not advised to change the value if it is not mandatory） +HIVE_CONF_DIR=/appcom/config/hive-config +###SPARK CONF DIR #/appcom/config/spark-config（You are not advised to change the value if it is not mandatory） +SPARK_CONF_DIR=/appcom/config/spark-config +###for install （You are not advised to change the value if it is not mandatory） +LINKIS_PUBLIC_MODULE=lib/linkis-commons/public-module + +##YARN REST URL spark engine required（Change the IP address and port number as required） +YARN_RESTFUL_URL=http://127.0.0.1:8088 + + +## Engine version +#SPARK_VERSION（Modify the version number according to the actual version） +SPARK_VERSION=2.4.3 +##HIVE_VERSION（Modify the version number according to the actual version） +HIVE_VERSION=2.3.3 +##PYTHON_VERSION（Modify the version number according to the actual version） +PYTHON_VERSION=python2 + +## LDAP is for enterprise authorization, if you just want to have a try, ignore it. +#LDAP_URL=ldap://localhost:1389/ +#LDAP_BASEDN=dc=webank,dc=com +#LDAP_USER_NAME_FORMAT=cn=%s@xxx.com,OU=xxx,DC=xxx,DC=com + +############## Other default configuration information for linkis end ############## + + +################### The install Configuration of all Linkis's Micro-Services ##################### +################### Users can modify the IP and port according to the actual situation ################### +# +# NOTICE: +# 1. If you just wanna try, the following micro-service configuration can be set without any settings. +# These services will be installed by default on this machine. +# 2. In order to get the most complete enterprise-level features, we strongly recommend that you install +# the following microservice parameters +# + +### EUREKA install information +### You can access it in your browser at the address below:http://${EUREKA_INSTALL_IP}:${EUREKA_PORT} +### Microservices Service Registration Discovery Center +LINKIS_EUREKA_INSTALL_IP=127.0.0.1 +LINKIS_EUREKA_PORT=9600 +#LINKIS_EUREKA_PREFER_IP=true + +### Gateway install information +#LINKIS_GATEWAY_INSTALL_IP=127.0.0.1 +LINKIS_GATEWAY_PORT=9001 + +### ApplicationManager +#LINKIS_MANAGER_INSTALL_IP=127.0.0.1 +LINKIS_MANAGER_PORT=9101 + +### EngineManager +#LINKIS_ENGINECONNMANAGER_INSTALL_IP=127.0.0.1 +LINKIS_ENGINECONNMANAGER_PORT=9102 + +### EnginePluginServer +#LINKIS_ENGINECONN_PLUGIN_SERVER_INSTALL_IP=127.0.0.1 +LINKIS_ENGINECONN_PLUGIN_SERVER_PORT=9103 + +### LinkisEntrance +#LINKIS_ENTRANCE_INSTALL_IP=127.0.0.1 +LINKIS_ENTRANCE_PORT=9104 + +### publicservice +#LINKIS_PUBLICSERVICE_INSTALL_IP=127.0.0.1 +LINKIS_PUBLICSERVICE_PORT=9105 + +### cs +#LINKIS_CS_INSTALL_IP=127.0.0.1 +LINKIS_CS_PORT=9108 + +########## Linkis微服务配置完毕 ########## + +################### The install Configuration of all DataSphereStudio's Micro-Services ##################### +#################### Non-commented parameters must be configured, and commented out parameters can be modified as needed #################### +# NOTICE: +# 1. If you just wanna try, the following micro-service configuration can be set without any settings. +# These services will be installed by default on this machine. +# 2. In order to get the most complete enterprise-level features, we strongly recommend that you install +# the following microservice parameters +# + +# Temporary ZIP package file for storing publications to Schedules +WDS_SCHEDULER_PATH=file:///appcom/tmp/wds/scheduler +### DSS_SERVER +### This service is used to provide dss-server capability. + +### project-server +#DSS_FRAMEWORK_PROJECT_SERVER_INSTALL_IP=127.0.0.1 +#DSS_FRAMEWORK_PROJECT_SERVER_PORT=9002 +### orchestrator-server +#DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP=127.0.0.1 +#DSS_FRAMEWORK_ORCHESTRATOR_SERVER_PORT=9003 +### apiservice-server +#DSS_APISERVICE_SERVER_INSTALL_IP=127.0.0.1 +#DSS_APISERVICE_SERVER_PORT=9004 +### dss-workflow-server +#DSS_WORKFLOW_SERVER_INSTALL_IP=127.0.0.1 +#DSS_WORKFLOW_SERVER_PORT=9005 +### dss-flow-execution-server +#DSS_FLOW_EXECUTION_SERVER_INSTALL_IP=127.0.0.1 +#DSS_FLOW_EXECUTION_SERVER_PORT=9006 +###dss-scriptis-server +#DSS_SCRIPTIS_SERVER_INSTALL_IP=127.0.0.1 +#DSS_SCRIPTIS_SERVER_PORT=9008 +########## DSS微服务配置完毕##### + + +############## other default configuration Other default configuration information ############## + +## java application default jvm memory（The stack size of the Java application. If the memory of the deployment machine is less than 8G, 128M is recommended； +## When it reaches 16G, at least 256M is recommended; if you want to have a very good user experience, it is recommended to deploy the machine with a memory of at least 32G） +export SERVER_HEAP_SIZE="128M" + +##sendemail configuration, only affects the sending email function in DSS workflow +EMAIL_HOST=smtp.163.com +EMAIL_PORT=25 +EMAIL_USERNAME=xxx@163.com +EMAIL_PASSWORD=xxxxx +EMAIL_PROTOCOL=smtp + +### Save the file path exported by the orchestrator service +ORCHESTRATOR_FILE_PATH=/appcom/tmp/dss +### Save DSS flow execution service log path +EXECUTION_LOG_PATH=/appcom/tmp/dss +############## other default configuration Other default configuration information ############## +``` + +- Please note: DSS recommends using LDAP for user login authentication. If you want to access the company's LDAP, you need to fill in the LDAP configuration parameters in the `config.sh` above. [How to install LDAP? ](https://web.mit.edu/rhel-doc/5/RHEL-5-manual/Deployment_Guide-en-US/s1-ldap-quickstart.html) + +- Modify database configuration. Please ensure that the configured database and the installation machine can be accessed normally, otherwise there will be an error of DDL and DML import failure, open `db.sh`, and modify the relevant configuration parameters as needed. The parameter descriptions are as follows: + +```properties +### Configure the DSS database +MYSQL_HOST=127.0.0.1 +MYSQL_PORT=3306 +MYSQL_DB=dss +MYSQL_USER=xxx +MYSQL_PASSWORD=xxx + +## The database configuration of the Hive metastore, which is used by Linkis to access the metadata information of Hive +HIVE_HOST=127.0.0.1 +HIVE_PORT=3306 +HIVE_DB=xxx +HIVE_USER=xxx +HIVE_PASSWORD=xxx +``` + +### Five, installation and use + +1. #### Stop all DSS and Linkis services on the machine + +- If you have never installed DSS and Linkis services, ignore this step + +2. #### Change the current directory to the bin directory + ```shell + cd xx/dss_linkis/bin + ```` +3. #### Execute the installation script + ```shell + sh install.sh + ```` +- The installation script will check various integrated environment commands. If not, please follow the prompts to install. The following commands are required: + + *yum; java; mysql; unzip; expect; telnet; tar; sed; dos2unix; nginx* + +- During installation, the script will ask you if you need to initialize the database and import metadata, both Linkis and DSS will ask, **The first installation must select Yes** + +- Check whether the installation is successful by checking the log information printed on the console. If there is an error message, you can check the specific error reason +- *This command should only be executed once unless the user wants to reinstall the entire app* + +4. #### start the service +- If the user's Linkis installation package is obtained by compiling and the user wants to enable the data source management function, then it is necessary to modify the configuration to enable this function, and no operation is required to use the downloaded installation package + ```shell + ## Switch to the Linkis configuration file directory + cd xx/dss_linkis/linkis/conf + + ## Open the configuration file linkis-env.sh + vi linkis-env.sh + + ## Change the following configuration to true + export ENABLE_METADATA_MANAGER=true + ```` +- If the user's Linkis installation package is obtained by compiling by yourself, try to change the password used later to be the same as the deployment user name before starting the service. No operation is required to use the downloaded installation package. + ```shell + ## Switch to the Linkis configuration file directory + cd xx/dss_linkis/linkis/conf/ + + ## Open the configuration file linkis-mg-gateway.properties + vi linkis-mg-gateway.properties + + ## change Password + wds.linkis.admin.password=hadoop + ```` +- Execute the startup service script in the xx/dss_linkis/bin directory + + ```shell + sh start-all.sh + ``` + +- If the startup generates an error message, you can view the specific error reason. After startup, each microservice will perform **communication detection**, and if there is an abnormality, it can help users locate the abnormal log and cause + +5. #### Install default Appconn + + ```shell + # Switch the directory to dss. Normally, the dss directory is in the xx/dss_linkis directory. + cd xx/dss_linkis/dss/bin + + # Execute the startup default Appconn script + sh install-default-appconn.sh + ```` + +- *This command can be executed once, unless the user wants to reinstall the entire application* + +6. #### Check if the verification is successful + +- Users can view the startup status of Linkis & DSS background microservices on the Eureka interface. By default, DSS has 7 microservices, and Linkis has 10 microservices (including 2 microservices after enabling the data source management function) **(Eureka address is configured in xx/dss_linkis/conf/config.sh)** + ![](../Images/Install_and_Deploy/DSS&Linkis_one-click_deployment_document_stand-alone_version/eureka.png) + +- Users can use **Google Chrome** to access the following front-end address: `http://DSS_NGINX_IP:DSS_WEB_PORT` ** The startup log will print this access address (this address is also configured in xx/dss_linkis/conf/config.sh )**. When logging in, the default administrator username and password are the deployment user hadoop (if the user wants to change the password, you can modify the wds.linkis.admin.password in the xx/dss_linkis/linkis/conf/linkis-mg-gateway.properties file parameter) + +7. #### stop service + ```shell + sh stop-all.sh + ```` +- If the user needs to stop all services, execute the command `sh stop-all.sh`, restart all services and execute `sh start-all.sh`, these two commands are executed in the xx/dss_linkis/bin directory + +### 6. Supplementary Instructions +- Considering the problem that the installation package is too large, Linkis only provides Hive, Python, Shell, Spark engine plug-ins by default. If users want to use other engines, please refer to the document: [Linkis Engine Installation](https://linkis.apache.org/docs/latest/deployment/engine_conn_plugin_installation/) +- DSS does not install the scheduling system by default. Users can choose to install Schedulelis or DolphinScheduler. The specific installation method is shown in the following table +- DSS only installs DateChecker, EventSender, EventReceiver AppConn by default. Users can refer to the documentation to install other AppConn, such as Visualis, Exchangis, Qualitis, Prophecis, Streamis. The scheduling system can use Schedulelis or DolphinScheduler + + | Component Name | Component Version Requirements | Component Deployment Link | AppConn Deployment Link | + |-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------| + | Schedulis | Schedulis0.7.0 | [Schedulis deploy](https://github.com/WeBankFinTech/Schedulis/blob/master/docs/schedulis_deploy_cn.md) | [Schedulis AppConn install](SchedulisAppConn_Plugin_Installation_Documentation.md) | + | Visualis | Visualis1.0.0 | [Visualis deploy](https://github.com/WeBankFinTech/Visualis/blob/master/visualis_docs/en_US/Visualis_deploy_doc_en.md) | [Visualis AppConn install](https://github.com/WeBankFinTech/Visualis/blob/master/visualis_docs/en_US/Visualis_appconn_install_en.md) | + | Exchangis | Exchangis1.0.0 | [Exchangis deploy](https://github.com/WeBankFinTech/Exchangis/blob/master/docs/en_US/ch1/exchangis_appconn_deploy_en.md) | [Exchangis AppConn install](https://github.com/WeBankFinTech/Exchangis/blob/master/docs/en_US/ch1/exchangis_appconn_deploy_en.md) | + | Qualitis |Qualitis0.9.2 | [Qualitis deploy](https://github.com/WeBankFinTech/Qualitis/blob/master/docs/en_US/ch1/QuickDeploy.md) | [Qualitis AppConn install](https://github.com/WeBankFinTech/Qualitis/blob/master/docs/zh_CN/ch1/%E6%8E%A5%E5%85%A5%E5%B7%A5%E4%BD%9C%E6%B5%81%E6%8C%87%E5%8D%97.md) | + | Prophecis | Prophecis0.3.2 | [Prophecis deploy](https://github.com/WeBankFinTech/Prophecis/blob/master/docs/zh_CN/QuickStartGuide.md) | [Prophecis AppConn install](https://github.com/WeBankFinTech/Prophecis/blob/master/docs/zh_CN/Deployment_Documents/Prophecis%20Appconn%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) | + | Streamis | Streamis0.2.0 | [Streamis deploy](https://github.com/WeBankFinTech/Streamis/blob/main/docs/zh_CN/0.2.0/Streamis%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) | [Streamis AppConn install](https://github.com/WeBankFinTech/Streamis/blob/main/docs/en_US/0.2.0/development/StreamisAppConnInstallationDocument.md) | + | DolphinScheduler | DolphinScheduler1.3.x | [DolphinScheduler deploy](https://dolphinscheduler.apache.org/zh-cn/docs/1.3.8/user_doc/standalone-deployment.html) | [DolphinScheduler AppConn install](DolphinScheduler_Plugin_Installation_Documentation.md) | diff --git a/en_US/Installation_and_Deployment/DSSUserGuide_Deploy_documentation.md b/en_US/Installation_and_Deployment/DSSUserGuide_Deploy_documentation.md new file mode 100644 index 0000000..b744ca6 --- /dev/null +++ b/en_US/Installation_and_Deployment/DSSUserGuide_Deploy_documentation.md @@ -0,0 +1,141 @@ +# DSSUserGuide Deploy documentation + +> Introduction: The help documentation module belongs to the dss-user-guide module and is used to provide dss project related materials. + +        When using the dss-user-guide module, you need to deploy the dss project service first. You can refer to the dss deployment related documents. The dss-user-guide document synchronization function uses timed tasks to synchronize every two hours, and the document needs to be updated to the system. Maintain a summary. The md file is used by the user-guide module to parse the location of the file and parse the content of the document. + + + +## 1、dss-guide-server.properties Knowledge Base Related Configuration Instructions + +### 1.1 Reference configuration + +````properties +#gitbook +#The ip address of the server where the document is stored +target.ip.address=127.0.0.1 +#The path to the server where the document is located +host.gitbook.path=/appcom/Install/ApacheInstall/gitbook_books +#Need to sync to server +target.gitbook.path=/appcom/Install/ApacheInstall +#Used to ignore parsing km down to the directory +summary.ignore.model=km +#Knowledge base synchronization method: gitbook database +guide.sync.model=gitbook +```` + +### 1.2 DSS1.1.0 config + +````properties +#gitbook +#The ip address of the server where the document is stored +target.ip.address=127.0.0.1 +#The path of the server where the document is located (the directory can be configured to the directory where the summary.md file is located, for example: /xxx/test1/SUMMARY.md) +host.gitbook.path=/xxx/test1 +#Dss1.1.0 does not support gitbook synchronization, so use database +guide.sync.model=gitbook +```` + + + +## 2、SUMMARY.mdDescription of file structure + +        SUMMARY.md this file is mainly used to maintain the location of the file and the file to the hierarchical structure after the file is parsed + +**E.g：** + +````SUMMARY.md +guide +- [学习引导]() +/workspaceManagement + - [工作空间管理]() + - [question]() + - [什么用户可以进入工作空间管理](/学习引导/工作空间管理/question/什么用户可以进入工作空间管理.md) + - [step]() + - [权限管理页面](/学习引导/工作空间管理/step/权限管理页面.md) + - [用户权限管理](/学习引导/工作空间管理/step/用户权限管理.md) +/workspaceHome + - [工作空间页面]() + - [question]() + - [为什么看不到应用商店了](/学习引导/工作空间页面/question/为什么看不到应用商店了.md) + - [step]() + - [工作空间页面介绍](/学习引导/工作空间页面/step/工作空间页面介绍.md) +/workflow + - [工作流]() + - [step]() + - [页面介绍](/学习引导/工作流/step/页面介绍.md) +knowledge +- [知识库]() + - [Demo案例]() + - [开发](/知识库/Demo案例/开发.md) + - [生产](/知识库/Demo案例/生产.md) + - [调试](/知识库/Demo案例/调试.md) + - [Dss常见问题]() + - [用户权限问题]() + - [新人权限申请](/知识库/DSS常见问题/用户权限问题/新人权限申请.md) + - [SparkHive等组件报权限问题](/知识库/DSS常见问题/用户权限问题/SparkHive等组件报权限问题.md) + - [没有队列权限不能提交到队列](/知识库/DSS常见问题/用户权限问题/没有队列权限不能提交到队列.md) +km +- [km]() + - [BDAP-IDE2.1.2功能介绍](/km/BDAP-IDE2.1.2功能介绍.md) + - [BDAP共享目录问题汇总](/km/BDAP共享目录问题汇总.md) +```` + +**note：** + +![企业微信截图_20220623173645](../Images/Install_and_Deploy/DSSUserGuide_Deploy/userguide_1.png) + +**Directory Structure Description：** + +- "-" + "space" + "[content]" + "()" means first level directory +- "space" + "space" + "-" + "space" + "[content]" + "()" for secondary directory +- "space" + "space" + "space" + "space" + "-" + "space" + "[content]" + "()" means three-level directory +- "space" + "space" + "space" + "space" + "space" + "space" + "-" + "space" + "[content]" + "()" means four-level directory + +**Note: () put the file to a relative path, and the file name cannot contain English characters ()** + + + +## 3.File picture description and configuration + +        Since some documents will have illustrations, the md file will be stored in the database after being parsed by the user-guide module. The image stored in the database is only the image to the path. Therefore, when inserting an image into the md document, the file needs to be stored in a relative path. , and then proxy to the folder where the server pictures are stored through nginx. + +The file directory structure is as follows: (Assume the file is stored in the sub-server /xxx/test1 directory) + +├── This is just an example.md
+│   ├── images
+│   │   └── 1.png
+├──  SUMMARY.md
+ +**例：** + +````md +This is just an example！！！ +Below is the picture +![我是图片](/images/1.png) +```` + +**nginx config：** + +````nginx +# The path of the server where the image is located +location /images { + root /xxx/test1/; + autoindex on; +} +```` + +**Note: This configuration needs to be added to the nginx configuration where the dss service is located, and it is necessary to ensure that the image to the ip:port has the same server.** + +        After the configuration is complete, restart the service to synchronize the files to the user-guide！ + + + + + + + + + + + diff --git a/en_US/Installation_and_Deployment/DSS_1.0.1_upgrade_to_1.1.0_using_documentation.md b/en_US/Installation_and_Deployment/DSS_1.0.1_upgrade_to_1.1.0_using_documentation.md new file mode 100644 index 0000000..48c0a15 --- /dev/null +++ b/en_US/Installation_and_Deployment/DSS_1.0.1_upgrade_to_1.1.0_using_documentation.md @@ -0,0 +1,107 @@ +# DataSphere Studio 1.0.1 upgrade to 1.1.0 using documentation + +### The upgrade steps are mainly divided into: +- service stopped +- Execute database upgrade scripts +- Replace the dss deployment directory with the new version package +- Configuration file addition, modification +- service start + +#### 1. Service stopped +Go to the deployment directory of dss, and execute the command in the directory to stop all services of dss: +```shell +cd ${DSS_DEPLOY_PATH} + +sh sbin/dss-stop-all.sh +```` +#### 2. Execute the database upgrade sql script + +How to get the upgrade sql script: + +- The decompressed directory of the installation package of dss1.1.0: db/version_update/from_v101_to_v110.sql +- Download from the github page, the address is: (to be uploaded and added) + +Then log in to the dss database and execute the source command: + +```roomsql +source ${your_path}/from_v101_to_v110.sql +``` +It can be executed successfully under normal circumstances. + +#### 3. Replace the dss deployment directory with the new version package + +**(Important: it is best to back up the database of the old version of dss first)** + +- Back up the following tables with structural changes through the mysqldump command: + +dss_appconn、dss_appconn_instance、dss_workflow_node、dss_onestop_user_favorites、dss_component_role、dss_onestop_menu_application + + + +These tables just modify the table name, and can also be backed up: + +dss_dictionary +dss_role +dss_admin_dept +dss_download_audit +dss_flow_relation +dss_flow_edit_lock +dss_onestop_menu +dss_menu_role + +- Back up the deployment directory of the old version of dss, take this directory as an example:/appcom/Install/DSSInstall +```shell +mv /appcom/Install/DSSInstall /appcom/Install/DSSInstall-bak +``` + +Put the installation package of dss1.1.0 in the temporary directory and decompress it: +```shell +mkdir /tmp/dss-pkg +mv wedatasphere-dss-1.1.0-dist.tar.gz /tmp/dss-pkg/ +cd /tmp/dss-pkg +tar zxvf wedatasphere-dss-1.1.0-dist.tar.gz +``` +The directory structure after decompression is as follows: +![img.png](../Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img.png) + +Then copy all the files in the dss-1.1.0 directory to the installation directory of dss1.1.0: +```shell +cd dss-1.1.0 +cp -r lib dss-appconns sbin /appcom/Install/DSSInstall/ +``` + +Copy the configuration file from the previous version: +```shell +cp -r /appcom/Install/DSSInstall-bak/conf /appcom/Install/DSSInstall/ +``` + +#### 4. Add and modify configuration + +New configuration added in the new version: dss-scriptis-server.properties, dss-guide-server.properties, +Copy directly from the dss1.1.0/conf directory: + +```shell +cp -r conf/dss-scriptis-server.properties /appcom/Install/DSSInstall/conf/ +cp -r conf/dss-guide-server.properties /appcom/Install/DSSInstall/conf/ +``` + +Configuration modification: + +1. Add to the dss.properties configuration file: +```properties +###If appconn does not implement all development specifications (node update, delete, copy, import, export operations), it needs to be added to the configuration to ignore the check +wds.dss.appconn.checker.development.ignore.list=workflow,sendemail +###If appconn does not implement all project specifications (add, delete, modify, check), it needs to be added to the configuration to ignore the check +wds.dss.appconn.checker.project.ignore.list= +``` + +#### 5. service start +OK, now you can start the new version of dss service, run the command in the **dss deployment directory** to start all services: + +```shell +sh sbin/dss-start-all.sh +``` + + + + diff --git a/en_US/Installation_and_Deployment/DSS_Debug_Documentation.md b/en_US/Installation_and_Deployment/DSS_Debug_Documentation.md new file mode 100644 index 0000000..f04a39f --- /dev/null +++ b/en_US/Installation_and_Deployment/DSS_Debug_Documentation.md @@ -0,0 +1,79 @@ + +# Debug related + +> No programmer can write code without any bugs in one go, so many programmers spend a considerable part of their time on debugging. Program debugging is a job that every programmer must face. The following will guide you how to perform dss. Remote debugging (based on DSS1.1.0 version). + +## step 1 Prepare DSS source code and compile + +```plain +git clone https://github.com/WeBankFinTech/DataSphereStudio.git +cd DataSphereStudio +#If necessary, you can switch to the corresponding branch +#git checkout dev-xxx +mvn -N install +mvn clean Install +``` + +## step 2 Deploy the DSS service on the server +If DSS has not been deployed, please refer to the deployment document: [DSS single-machine deployment document](DSS&Linkis_one-click_deployment_document_stand-alone_version.md) + +## step 3 Open debug port + +First, you need to identify the service where the package to be debugged is located, and determine the service to which it belongs according to the location of the code to be debugged. + +Then enter the ${DSS_HOME}/sbin/ext directory under the dss deployment directory, modify the startup script file of the corresponding service to be debugged, and open the remote call port: (take the workflow-server service as an example) + +```shell +cd ${DSS_HOME}/sbin/ext + +vim dss-workflow-server +``` +Find the DEBUG_PORT keyword in the script, and then enter the port number that needs to be opened **(to be able to connect from the local)** + +![img.png](../Images/Installation_and_Deployment/DSS_Debug_Documentation/img.png) + +Then you need to restart the corresponding service to make it take effect: + +``` +sh sbin/dss-daemon.sh restart workflow-server +``` +Note: If you are not sure about the service name, you can query it in the ${DSS_HOME}/sbin/common.sh script, +Just enter the keyword of the service to start the corresponding service: + +![img_1.png](../Images/Install_and_Deploy/DSS_Debug_Documentation/img_1.png) + + +## step 4 IDEA Compiler configuration remote debugging +Open the window as shown below and configure remote debugging ports, services, and modules: + +![img_2.png](../Images/Install_and_Deploy/DSS_Debug_Documentation/img_2.png) + +## step 5 start debugging + +Click the debug button in the upper right corner of the idea to start debugging: + +![img_3.png](../Images/Install_and_Deploy/DSS_Debug_Documentation/img_3.png) + +## step 6 Replace jar package + +After modifying the code locally, you can package the jar package of the corresponding module, and then replace the jar of the corresponding lib on the server. Just restart the service. + +```shell +cd ${DSS_HOME} + +## Upload the jar package to the server +rz -bye + +## Replace the uploaded jar package with the original jar package +cp ${your_jar_name} lib/${jar_path} +``` + +Note: If you don't know which service libs the jar exists in, you can search all the locations of the jar with the following command: +```shell +cd ${DSS_HOME} + +## Search all dss-orchestrator-common-*.jar packages in the lib directory +find lib/ -name "*dss-orchestrator-common*" +``` + +![img_4.png](../Images/Install_and_Deploy/DSS_Debug_Documentation/img_4.png) \ No newline at end of file diff --git a/en_US/Installation_and_Deployment/DSS_Single-Server_Deployment_Documentation.md b/en_US/Installation_and_Deployment/DSS_Single-Server_Deployment_Documentation.md deleted file mode 100644 index 18217e5..0000000 --- a/en_US/Installation_and_Deployment/DSS_Single-Server_Deployment_Documentation.md +++ /dev/null @@ -1,323 +0,0 @@ -# DataSphere Studio One-Click Installation Documentation - -## 1. Environment preparation before use - -### a. Basic software installation - -Command tools required by Linkis (before the official installation, the script will automatically detect whether these commands are available. If they do not exist, they will try to install them automatically. If the installation fails, the user needs to manually install the following basic shell command tools): - -- telnet -- tar -- sed -- dos2unix -- mysql -- yum -- java -- unzip -- expect - -Software to be installed: - -- MySQL (5.5+) - -- JDK (1.8.0_141 and above) - -- Python (both 2.x and 3.x are supported) - -- Nginx - - -The following services must be accessible from this machine: - -- Hadoop (**2.7.2, other versions of Hadoop need to compile Linkis**), the installed machine must support the execution of the ``` hdfs dfs -ls / ``` command - -- Hive (**2.3.3, other versions of Hive need to compile Linkis**), the installed machine must support the execution of the ``` hive -e "show databases" ``` command - -- Spark (**supports all versions above 2.0**), the installed machine must support the execution of the ```spark-sql -e "show databases" ``` command - -Tips: - -If you are installing Hadoop for the first time, you can refer to: [Hadoop single-machine deployment](https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html ); for distributed deployment of Hadoop, please refer to: [Hadoop Distributed Deployment](https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/ClusterSetup.html). - -If you are installing Hive for the first time, please refer to: [Hive Quick Installation and Deployment](https://cwiki.apache.org/confluence/display/Hive/GettingStarted). - -If you are installing Spark for the first time, you can refer to the On Yarn mode: [Spark on Yarn deployment](http://spark.apache.org/docs/2.4.3/running-on-yarn.html). - -### b. Create User - -        For example: **The deployment user is the hadoop account** (it may not be the hadoop user, but it is recommended to use the Hadoop super user for deployment, here is just an example) - -2. Create a deployment user on all machines that need to be deployed for installation - -```bash -sudo useradd hadoop -``` - -3. Because the Linkis service uses sudo -u ${linux-user} to switch engines to execute jobs, the deployment user needs to have sudo privileges and is password-free. - -```bash -vi /etc/sudoers -``` - -```properties - hadoop ALL=(ALL) NOPASSWD: NOPASSWD: ALL -``` - -4. Make sure that the server where DSS and Linkis are deployed can execute commands such as hdfs , hive -e and spark-sql -e normally. In the one-click install script, the components are checked. - -5. **If your Pyspark wants to have the drawing function, you also need to install the drawing module on all installation nodes**. The command is as follows: - - -```bash -python -m pip install matplotlib -``` - -### c.Installation preparation - -Compile by yourself or go to the component release page to download the installation package: - -2. Download the installation package - -- [wedatasphere-linkis-x.x.x-dist.tar.gz](https://github.com/WeBankFinTech/Linkis/releases) -- [wedatasphere-dss-x.x.x-dist.tar.gz](https://github.com/WeBankFinTech/DataSphereStudio/releases) -- [wedatasphere-dss-web-x.x.x-dist.zip](https://github.com/WeBankFinTech/DataSphereStudio/releases) - -3. Download the DSS & LINKIS one-click installation deployment package, and unzip it. The following is the hierarchical directory structure of the one-click installation deployment package: - -```text -├── dss_linkis # One-click deployment home directory - ├── bin # for one-click installation, and one-click to start DSS + Linkis - ├── conf # Parameter configuration directory for one-click deployment - ├── wedatasphere-dss-x.x.x-dist.tar.gz # DSS background installation package - ├── wedatasphere-dss-web-x.x.x-dist.zip # DSS front-end installation package - ├── wedatasphere-linkis-x.x.x-dist.tar.gz # Linkis installation package -``` - -### d. Modify the configuration - -        Open conf/config.sh and modify the relevant configuration parameters as needed: - -```bash -vi conf/config.sh -``` - -The parameter description is as follows: - -```properties -#################### Basic configuration for one-click installation and deployment #################### - -# Deployment user, the default is the current logged-in user, it is not recommended to modify it if it is not necessary -# deployUser=hadoop - -# Not required and not recommended to modify -# LINKIS_VERSION=1.0.2 - -### DSS Web, no modification required for native installation -#DSS_NGINX_IP=127.0.0.1 -#DSS_WEB_PORT=8088 - -# Not required and not recommended to modify -#DSS_VERSION=1.0.0 - -## The stack size of the Java application. If the memory of the deployment machine is less than 8G, 128M is recommended; when it reaches 16G, at least 256M is recommended; if you want to have a very good user experience, it is recommended that the memory of the deployment machine be at least 32G. -export SERVER_HEAP_SIZE="128M" - -################################################## ########## -##################### Linkis configuration starts ##################### -########## Non-commented parameters must be configured, and commented out parameters can be modified as needed ########## -################################################## ########## - -### DSS workspace directory -WORKSPACE_USER_ROOT_PATH=file:///tmp/linkis/ -### User HDFS root path -HDFS_USER_ROOT_PATH=hdfs:///tmp/linkis -### result set path: file or hdfs path -RESULT_SET_ROOT_PATH=hdfs:///tmp/linkis - -### Path to store started engines and engine logs, must be local -ENGINECONN_ROOT_PATH=/appcom/tmp - -#ENTRANCE_CONFIG_LOG_PATH=hdfs:///tmp/linkis/ - -### HADOOP configuration file path, must be configured -HADOOP_CONF_DIR=/appcom/config/hadoop-config -### HIVE CONF DIR -HIVE_CONF_DIR=/appcom/config/hive-config -### SPARK CONF DIR -SPARK_CONF_DIR=/appcom/config/spark-config - -# for install -#LINKIS_PUBLIC_MODULE=lib/linkis-commons/public-module - - -## YARN REST URL -YARN_RESTFUL_URL=http://127.0.0.1:8088 - -## Engine version configuration, if not configured, the default configuration is used -#SPARK_VERSION -#SPARK_VERSION=2.4.3 -##HIVE_VERSION -#HIVE_VERSION=1.2.1 -#PYTHON_VERSION=python2 - -## LDAP is for enterprise authorization, if you just want to have a try, ignore it. -#LDAP_URL=ldap://localhost:1389/ -#LDAP_BASEDN=dc=webank,dc=com -#LDAP_USER_NAME_FORMAT=cn=%s@xxx.com,OU=xxx,DC=xxx,DC=com - -# Microservices Service Registration Discovery Center -#LINKIS_EUREKA_INSTALL_IP=127.0.0.1 -#LINKIS_EUREKA_PORT=20303 -#LINKIS_EUREKA_PREFER_IP=true - -### Gateway install information -#LINKIS_GATEWAY_PORT =127.0.0.1 -#LINKIS_GATEWAY_PORT=9001 - -### ApplicationManager -#LINKIS_MANAGER_INSTALL_IP=127.0.0.1 -#LINKIS_MANAGER_PORT=9101 - -### EngineManager -#LINKIS_ENGINECONNMANAGER_INSTALL_IP=127.0.0.1 -#LINKIS_ENGINECONNMANAGER_PORT=9102 - -### EnginePluginServer -#LINKIS_ENGINECONN_PLUGIN_SERVER_INSTALL_IP=127.0.0.1 -#LINKIS_ENGINECONN_PLUGIN_SERVER_PORT=9103 - -### LinkisEntrance -#LINKIS_ENTRANCE_INSTALL_IP=127.0.0.1 -#LINKIS_ENTRANCE_PORT=9104 - -### publicservice -#LINKIS_PUBLICSERVICE_INSTALL_IP=127.0.0.1 -#LINKIS_PUBLICSERVICE_PORT=9105 - -### cs -#LINKIS_CS_INSTALL_IP=127.0.0.1 -#LINKIS_CS_PORT=9108 - -##################### Linkis configuration completed ##################### - -################################################## ########## -####################### DSS configuration starts ####################### -########## Non-commented parameters must be configured, and commented out parameters can be modified as needed ########## -################################################## ########## - -# Used to store temporary ZIP package files published to Schedules -WDS_SCHEDULER_PATH=file:///appcom/tmp/wds/scheduler - -### This service is used to provide dss-framework-project-server capability. -#DSS_FRAMEWORK_PROJECT_SERVER_INSTALL_IP=127.0.0.1 -#DSS_FRAMEWORK_PROJECT_SERVER_PORT=9002 - -### This service is used to provide dss-framework-orchestrator-server capability. -#DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP=127.0.0.1 -#DSS_FRAMEWORK_ORCHESTRATOR_SERVER_PORT=9003 - -### This service is used to provide dss-apiservice-server capability. -#DSS_APISERVICE_SERVER_INSTALL_IP=127.0.0.1 -#DSS_APISERVICE_SERVER_PORT=9004 - -### This service is used to provide dss-workflow-server capability. -#DSS_WORKFLOW_SERVER_INSTALL_IP=127.0.0.1 -#DSS_WORKFLOW_SERVER_PORT=9005 - -### dss-flow-Execution-Entrance -### This service is used to provide flow execution capability. -#DSS_FLOW_EXECUTION_SERVER_INSTALL_IP=127.0.0.1 -#DSS_FLOW_EXECUTION_SERVER_PORT=9006 - -### This service is used to provide dss-datapipe-server capability. -#DSS_DATAPIPE_SERVER_INSTALL_IP=127.0.0.1 -#DSS_DATAPIPE_SERVER_PORT=9008 - -##sendemail configuration, only affects the sending email function in DSS workflow -EMAIL_HOST=smtp.163.com -EMAIL_PORT=25 -EMAIL_USERNAME=xxx@163.com -EMAIL_PASSWORD=xxxxx -EMAIL_PROTOCOL=smtp -####################### DSS configuration ends ####################### -``` - -### f. Modify database configuration - -Please make sure that the configured database and the installation machine can be accessed normally, otherwise the DDL and DML import failure error will occur. - -```bash -vi conf/db.sh -``` - -```properties -### Configure DSS database -MYSQL_HOST=127.0.0.1 -MYSQL_PORT=3306 -MYSQL_DB=dss -MYSQL_USER=xxx -MYSQL_PASSWORD=xxx - -## Hive metastore database configuration, used for Linkis to access Hive metadata information -HIVE_HOST=127.0.0.1 -HIVE_PORT=3306 -HIVE_DB=xxx -HIVE_USER=xxx -HIVE_PASSWORD=xxx -``` - -## Second, installation and use - -### 1. Execute the installation script: - -```bash -sh bin/install.sh -```` - -### 2. Installation steps - -- The installation script will check various integrated environment commands. If not, please follow the prompts to install. The following commands are required: - - _yum java mysql unzip expect telnet tar sed dos2unix nginx_ - -- When installing, the script will ask if you need to initialize the database and import metadata, both Linkis and DSS will ask. - - -     **First time installation** must be Yes. - -### 3. Whether the installation is successful: - -     Check whether the installation is successful by viewing the log information printed on the console. - -     If there is an error message, you can check the specific error reason. - -### 4. Start the service - -#### (1) Start the service: - -     Execute the following command in the installation directory to start all services: - -```shell -sh bin/start-all.sh -```` - -     If the startup generates an error message, you can check the specific error reason. After startup, each microservice will perform **communication detection**, and if there is an abnormality, it can help users locate the abnormal log and cause. - -#### (2) Check if the startup is successful - -     You can view the startup status of Linkis & DSS background microservices on the Eureka interface. -![Eureka](../Images/Install_and_Deploy/Standalone_Deployment_Documentation/eureka.png) - -#### (3) Google Chrome access: - -Please use **Google Chrome** to visit the following frontend address: - -`http://DSS_NGINX_IP:DSS_WEB_PORT` **The startup log will print this access address**. When logging in, the administrator's username and password are both the deployment username. If the deployment user is hadoop, the administrator's username/password is: hadoop/hadoop. - -#### (4) Stop the service: - -     Execute the following command in the installation directory to stop all services: - -```bash -sh bin/stop-all.sh -``` diff --git a/en_US/Installation_and_Deployment/DolphinScheduler_Plugin_Installation_Documentation.md b/en_US/Installation_and_Deployment/DolphinScheduler_Plugin_Installation_Documentation.md new file mode 100644 index 0000000..bac482e --- /dev/null +++ b/en_US/Installation_and_Deployment/DolphinScheduler_Plugin_Installation_Documentation.md @@ -0,0 +1,200 @@ +# DolphinSchedulerAppConn installation documentation + +## 1.Preparation + +Before you deploy `DolphinSchedulerAppConn`, you must first start the `DolphinScheduler` deployment and ensure that the `DolphinScheduler` is available. + +**Please note: Currently `DolphinSchedulerAppConn` only supports DolphinScheduler 1.3.X. ** + +For the installation and deployment of `DolphinScheduler`, please refer to：[`DolphinScheduler` Install and deploy documentation](https://dolphinscheduler.apache.org/en-us/docs/1.3.8/user_doc/standalone-deployment.html) + +## 2. Download and compile + +The `DolphinSchedulerAppConn` plugin installation package can be downloaded from here: [Click me to download the `DolphinSchedulerAppConn` plugin installation package](). + +If you want to compile `DolphinSchedulerAppConn` yourself, the specific compilation steps are as follows: + +1. The code of clone DataSphereStudio + +2. Compile dss-dolphinscheduler-appconn separately + +```shell script + cd ${DSS_HOME}/dss-appconn/appconns/dss-dolphinscheduler-appconn + mvn clean install +``` +3. 在dss-dolphinscheduler-appconn/target/out目录下会生成dolphinscheduler文件夹，用户将其压缩成dolphinscheduler-appconn.zip文件即可 + +## 3. Configure and deploy + +### 3.1 appconn configuration and installation +- Put the `dolphinscheduler-appconn.zip` plugin installation package in the following directory and unzip it. + +```shell script + cd ${DSS_HOME}/dss/dss-appconns + unzip dolphinscheduler-appconn.zip +``` + +- Configuration parameters, please modify the configuration parameters of `appconn.properties` as needed. + +```shell script + cd ${DSS_HOME}/dss/dss-appconns/dolphinscheduler + vim appconn.properties +``` + +```properties +# [Required] Specify the administrator user of DolphinScheduler +wds.dss.appconn.ds.admin.user=admin +# [Required] Specify the token of the DolphinScheduler administrator user, which can be obtained from "Security Center -> Token Management" on the dolphinscheduler page +wds.dss.appconn.ds.admin.token= + +# [Please refer to] Currently only compatible with DolphinScheduler 1.3.X. +wds.dss.appconn.ds.version=1.3.9 + +# Used to configure the home path of dss-dolphinscheduler-client, it can be a specific path, please refer to the fourth step for details +wds.dss.appconn.ds.client.home=${DSS_DOLPHINSCHEDULER_CLIENT_HOME} + +# this property is used to add url prefix, if you add a proxy for dolphinscheduler url. +# for example: the normal dolphinscheduler url is http://ip:port/users/create, if you set +# this property, the real url will be http://ip:port/${wds.dss.appconn.ds.url.prefix}/users/create +#wds.dss.appconn.ds.url.prefix= +``` + +- Load the `DolphinScheduler` plugin + +```shell script +cd ${DSS_HOME}/bin +sh install-appconn.sh +# This script is an interactive installation program, you only need to follow the instructions, enter the string dolphinscheduler and the ip and port of the dolphinscheduler service to complete the installation +``` + +Please note: Do not enter `localhost` or `127.0.0.1` for the ip of dolphinscheduler, please enter the real IP. + +### 3.2 Modify the jar package +#### 3.2.1 Put dss-dolphinscheduler-token.jar under lib of dss-framework-project + +The role of this Jar package is to provide the `/api/rest_j/v1/dss/framework/project/ds/token` interface, which is used to request the interface of DolphinScheduler without password. + +How to obtain the Jar package: After DSS is compiled, it can be obtained from the `plugins/dolphinscheduler` directory: + +![img_9.png](../Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_9.png) + +Upload the Jar package to the DSS deployment directory: `${DSS_HOME}/lib/dss-framework/dss-framework-project-server/`，然后重启 `dss-framework-project-server` 服务： + +```shell +sh sbin/dss-daemon.sh restart project-server +``` + +#### 3.2.2 Put dolphinscheduler-prod-metrics.jar into DolphinScheduler's lib directory + +This step is to add the custom interface implementation Jar package of DolphinScheduler to the lib directory of the DolphinScheduler service, and restart the DolphinScheduler service to make it effective. + +Jar acquisition method: There are dolphinscheduler related plug-in packages in the plugins directory compiled from DSS, as shown in the figure: + +![img_6.png](../Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_6.png) + +Copy the Jar package to the lib directory deployed by DolphinScheduler: + +![img_7.png](../Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_7.png) + +Restart the DolphinScheduler service to make the Jar's custom interface take effect: + +```shell script +sh bin/stop-all.sh +sh bin/start-all.sh +``` + + +### 3.3 Modify the nginx configuration of DSS and add the request matching rule of the /dolphinscheduler path. + +This step is because the front end of the operation and maintenance center page will directly call the interface of the DolphinScheduler service to request data (`/dolphinscheduler` URI path prefix), + +So the request needs to be forwarded to the DolphinScheduler service. + +```shell script +vim /etc/nginx/conf.d/dss.conf +``` + +```shell script +location /dolphinscheduler { + proxy_pass http://127.0.0.1:12345;#The address of the backend dolphinscheduler service + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection upgrade; +} +``` + +After the modification, execute the command to reload the nginx configuration to make it take effect: + +```shell script +sudo nginx -s reload +``` + +### 3.4 Configure the url to the dispatch center + +update `${DSS_HOME}/conf/dss-workflow-server.properties` config： + +```properties +#This path corresponds to the page of the dolphinscheduler operation and maintenance center +wds.dss.workflow.schedulerCenter.url="/scheduler" +``` + +Then restart the workflow to make the configuration take effect: + +```shell script +sh sbin/dss-daemon.sh restart workflow-server +``` + +## 4. Deploy dss-dolphinscheduler-client + +In order for `DolphinScheduler` to normally schedule workflow node jobs in DataSphereStudio, you also need to install the dss-dolphinscheduler-client plugin, which is used to execute DSS workflow node jobs. + +### 4.1 Installation package preparation + +The `dss-dolphinscheduler-client` plugin installation package can be downloaded from here: [Click me to download the plugin installation package](https://osp-1257653870.cos.ap-guangzhou.myqcloud.com/WeDatasphere/DolphinScheduler/dss-dolphinscheduler-client.zip) + +If you want to compile `dss-dolphinscheduler-client` yourself, the specific compilation steps are as follows: + +1. The code of clone DataSphereStudio +2. Compile dss-dolphinscheduler-client separately + +```shell script +cd ${DSS_HOME}/plugins/dolphinscheduler/dss-dolphinscheduler-client +mvn clean install +``` + +### 4.2 Installation and deployment + +Please configure the environment variable `DSS_DOLPHINSCHEDULER_CLIENT_HOME` in `/home/${USER}/.bash_rc` first (if you specify an absolute path instead of this environment variable in `appconn.properties`, this environment variable can also be left unconfigured ). + +Configure `DSS_DOLPHINSCHEDULER_CLIENT_HOME` to the actual `dss-dolphinscheduler-client` root path. + +Unzip and install in the root path of `dss-dolphinscheduler-client`, as follows: + +```shell script +cd ${DSS_DOLPHINSCHEDULER_CLIENT_HOME} +unzip dss-dolphinscheduler-client.zip +``` + +Unzip it to complete the installation of `dss-dolphinscheduler-client`. + +Then you need to modify the ip and port of the linkis gateway in the configuration file conf/linkis.properties in dss-dolphinscheduler-client: + +![img_5.png](../Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_5.png) + + +## 5. Use of DolphinSchedulerAppConn + +### 5.1 Password-free jump + +Go to the DSS workspace home page, and then click Jump to DolphinScheduler on the top menu bar. + +![DolphinScheduler password-free jump](../Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_13.png) +![img_14](../Images/Install_and_Deploy/DolphinschedulerAppConn_deployment/img_14.png) + +### 5.2 Publish DSS workflows to DolphinScheduler + +Click the Publish button of the DSS workflow to publish the DSS workflow to DolphinScheduler with one click. + +### 5.3 dispatch center documentation + +For more information on the use of DSS dispatch center, please refer to：[Schedul Center Documentation](../User_Manual/Schedule_Center_Documentation.md) \ No newline at end of file diff --git a/en_US/Installation_and_Deployment/ExchangisAppConn_Plugin_Installation_Documentation.md b/en_US/Installation_and_Deployment/ExchangisAppConn_Plugin_Installation_Documentation.md deleted file mode 100644 index 0d8c9c6..0000000 --- a/en_US/Installation_and_Deployment/ExchangisAppConn_Plugin_Installation_Documentation.md +++ /dev/null @@ -1,86 +0,0 @@ -# ExchangisAppConn installation documentation - -This article mainly introduces the deployment, configuration and use of ExchangeisAppConn in DSS (DataSphere Studio) 1.0. - - - -## 1. Preparations for deploying ExchangisAppConn - -Before you deploy the ExchangisAppConn, you must start the Exchangis deployment and ensure that the basic functions are available. And Exchangis needs to introduce the following maven dependencies, which can achieve unified SSO capabilities -````xml - - com.webank.wedatasphere.dss - spring-origin-sso-integration-plugin - ${dss.version} - -```` -In the DSS1.0.0 version, Exchangis can achieve the SSO specification in the first-level specification of DSS, and can log in to Exchangeis without password on the home page. - - -## 2. Compilation of ExchangisAppConn materials - -DSS1.0.0 version, this step can be skipped, ExchangisAppConn uses the default SSO implementation - -## 3. Deployment and installation of ExchangisAppConn materials - -DSS1.0.0 version, ExchangisAppConn material installation can be skipped, but you need to insert the corresponding data in the database, the sql is as follows, you need to modify the ip and port of the Exhangis service in the first line -```roomsql -SET @EXCHANGIS_INSTALL_IP_PORT='127.0.0.1:9003'; -SET @URL = replace('http://EXCHANGIS_IP_PORT', 'EXCHANGIS_IP_PORT', @EXCHANGIS_INSTALL_IP_PORT); -SET @HOMEPAGE_URL = replace('http://EXCHANGIS_IP_PORT', 'EXCHANGIS_IP_PORT', @EXCHANGIS_INSTALL_IP_PORT); -SET @PROJECT_URL = replace('http://EXCHANGIS_IP_PORT', 'EXCHANGIS_IP_PORT', @EXCHANGIS_INSTALL_IP_PORT); -SET @REDIRECT_URL = replace('http://EXCHANGIS_IP_PORT/udes/auth', 'EXCHANGIS_IP_PORT', @EXCHANGIS_INSTALL_IP_PORT); - -delete from `dss_application` WHERE `name` = 'Exchangis'; -INSERT INTO `dss_application`(`name`,`url`,`is_user_need_init`,`level`,`user_init_url`,`exists_project_service`,`project_url`,`enhance_json`,`if_iframe`,`homepage_url`,`redirect_url`) VALUES ('Exchangis', @URL, 0, 1, NULL, 0, @PROJECT_URL, '', 1, @HOMEPAGE_URL, @REDIRECT_URL); - -select @dss_exchangis_applicationId:=id from `dss_application` WHERE `name` = 'Exchangis'; - -delete from `dss_onestop_menu` WHERE `name` = 'data exchange'; -INSERT INTO `dss_onestop_menu`(`name`, `title_en`, `title_cn`, `description`, `is_active`) values ('data exchange','data exchange','data exchange','data exchange',1); - -select @dss_onestop_menu_id:=id from `dss_onestop_menu` where `name` = 'data exchange'; - -delete from `dss_onestop_menu_application` WHERE title_en = 'Exchangis'; -INSERT INTO `dss_onestop_menu_application` (`application_id`, `onestop_menu_id`, `title_en`, `title_cn`, `desc_en`, `desc_cn`, `labels_en`, `labels_cn`, `is_active`, `access_button_en`, `access_button_cn`, `manual_button_en`, `manual_button_cn`, `manual_button_url`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`, `image`) -VALUES(@dss_exchangis_applicationId, @dss_onestop_menu_id, 'Exchangis','data exchange','Exchangis is a lightweight, high scalability, data exchange platform, support for structured and unstructured data transmission between heterogeneous data sources','Exchangis is a lightweight A highly scalable data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources, and has business features such as data permission control, high availability of node services, and multi-tenant resource isolation at the application layer. , and on the data layer, it has architectural features such as diversification of transmission architecture, module plug-in and low coupling of components.','Data Exchange','Data Exchange','1','Enter Exchangis','Enter Exchange', 'user manual','user manual','http://127.0.0.1:8088/wiki/scriptis/manual/workspace_cn.html','shujujiaohuan-logo',NULL,NULL,NULL,NULL,NULL,'shujujiaohuan -icon'); -``` - - - -## 4. Use of ExchangisAppConn -You can enter the front-end homepage of DSS, and then enter the homepage of the Exchangis application, as shown in the figure - -![Exchangis Embed DSS](../Images/Install_and_Deploy/ExchangisAppConn_Deployment/DSS-Exchangis.png) - - -## 5.How the ExchangisAppConn plugin works - -1. Microservices using ExchangeisAppConn - -The following microservices in DSS will interact with Exchangis through ExchangisAppConn to complete the specified functions. - -| Microservice name | Functions done with AppConn | Remark | -|-----------------|----------------|----------------------------------------| -| dss-framework-project-server | Unified authentication to achieve SSO | must | - - - - -2. Exchangis integration into dss needs to set the corresponding content in the following database tables - -| Table Name | Table Role | Remark | -|-----------------|----------------|----------------------------------------| -| dss_application | Application table, mainly to insert the basic information of the Exchangis application | must | -| dss_menu | Menu table, which stores the content displayed to the outside world, such as icons, names, etc. | must | -| dss_onestop_menu_application | The association table of menu and application for joint search | must | - - - -3. Follow-up development items - -3.1 ExchangisAppConn supports engineering specifications in primary specifications - -3.2 ExchangisAppConn supports the execution specification in the third-level specification, and supports the execution of tasks of the type Exchangis - -3.3 ExchangisAppConn supports import and export specifications in the third-level specification diff --git a/en_US/Installation_and_Deployment/QualitisAppConn_Plugin_Installation_Documentation.md b/en_US/Installation_and_Deployment/QualitisAppConn_Plugin_Installation_Documentation.md deleted file mode 100644 index a3e1075..0000000 --- a/en_US/Installation_and_Deployment/QualitisAppConn_Plugin_Installation_Documentation.md +++ /dev/null @@ -1,74 +0,0 @@ -# QualitisAppConn installation documentation - -This article mainly introduces the deployment, configuration and use of QualitisAppConn in DSS (DataSphere Studio) 1.0. - - - -## 1. Preparations for deploying QualitisAppConn - -Before you deploy QualitisAppConn, you must start the Qualitis deployment and ensure that the basic functions are available. - - - -## 2. Download and compile the QualitisAppConn plugin - -We provide the material package of QualitisAppConn, if you have already downloaded it, you can skip this step. If you want to compile QualitisAppConn yourself, the specific compilation steps are as follows: -1. clone the code of DataSphere Studio -2. Compile qualitis-appconn separately -```bash -cd {DSS_CODE_HOME}/dss-appconn/appconns/dss-qualitis-appconn -mvn clean install -``` - -## 3. Deployment and configuration of QualitisAppConn plugin - - -1. Obtain the qualitis-appconn.zip material package from the target directory in step 2 -2. Put it in the following directory and unzip it -```bash -cd {DSS_HOME}/dss/dss-appconns -unzip qualitis-appconn.zip -``` -3. Configure the relevant information of QualitisAppConn -``` bash -cd {DSS_INSTALL_HOME}/dss/bin -sh install-appconn.sh -The script is an interactive installation solution. You need to enter the string qualitis and the ip and port of the qualitis service to complete the installation. -``` -4. Restart the dss service to complete the update of the plugin - -## 4. Use of QualitisAppConn -You can enter the front-end homepage of DSS, and then enter the homepage of the Qualitis application, as shown in the figure. - -![Qualitis Embedded DSS](../Images/Install_and_Deploy/QualitisAppConn_Deployment/dss-qualitis.png) -
Figure
- -You can also use the visualization capabilities of Qualitis by using the visualization node of the workflow, as shown in Fig. -![Workflows use visualization nodes](../Images/Install_and_Deploy/QualitisAppConn_Deployment/workflow-qualitis.png) - - - -## 5. How the QualitisAppConn plugin works -This section is an extension of the installation and a brief explanation of how QualitisAppConn works. - -1. Microservices using QualitisAppConn - -The following microservices in DSS will interact with Qualitis through QualitisAppConn to complete specified functions. - -| Microservice name | Functions done with AppConn | Remark | -|-----------------|----------------|----------------------------------------| -| dss-framework-project-server | Complete engineering and organizational unification with qualitis-appconn | must | -| dss-workflow-server | Use the third-level specification to complete the creation, editing, import and export of nodes, etc.| must | -| appconn-engine | Use the third-level specification to complete the execution of the qualitis node | must | - -2. Qualitis integration into dss needs to set the corresponding content in the following database tables - -| Table Name | Table Role | Remark | -|-----------------|----------------|----------------------------------------| -| dss_application | Application table, mainly to insert basic information of qualitis application | must | -| dss_menu | Menu table, which stores the content displayed to the outside world, such as icons, names, etc.| must | -| dss_onestop_menu_application | The association table of menu and application for joint search | must | -| dss_appconn | Basic information of appconn, used to load appconn | must | -| dss_appconn_instance | Information about the qualitis instance, including the url information of qualitis itself | must | -| dss_workflow_node | Information that Qualitis needs to insert as a workflow node | If you want to use data quality checks, you must | - diff --git a/en_US/Installation_and_Deployment/SchedulisAppConn_Plugin_Installation_Documentation.md b/en_US/Installation_and_Deployment/SchedulisAppConn_Plugin_Installation_Documentation.md index 386e470..0607961 100644 --- a/en_US/Installation_and_Deployment/SchedulisAppConn_Plugin_Installation_Documentation.md +++ b/en_US/Installation_and_Deployment/SchedulisAppConn_Plugin_Installation_Documentation.md @@ -1,34 +1,55 @@ #Schedulis AppConn Installation Instructions + >Schedulis is an open-source workflow scheduling system of WeBank's big data platform room. By installing and configuring the AppConn plug-in in DSS, the workflow developed by DSS can be published to Schedulis for scheduled execution. # 1. Preparation before installation -    1. First, you need to install DSS, through one-click installation deployment script or separate installation, you can refer to the DSS installation documentation. -    2. After installing DSS, you can implement basic DSS interactive analysis and workflow execution tasks. If you need to publish DSS workflows to Scheduled for scheduling execution, you first need to install and run [Schedulis](https ://github.com/WeBankFinTech/Schedulis). + +- You need to install DSS first, please refer to: [DSS single-machine deployment document](DSS&Linkis_one-click_deployment_document_stand-alone_version.md)    2. After installing DSS, you can implement basic DSS interactive analysis and workflow execution tasks. If you need to publish DSS workflows to Scheduled for scheduling execution, you first need to install and run [Schedulis](https ://github.com/WeBankFinTech/Schedulis). + +- You need to install and run Schedulis first, you can refer to: [Schedulis deployment documentation](https://github.com/WeBankFinTech/Schedulis/blob/master/docs/schedulis_deploy_cn.md) + # 2. Compile the AppConn plugin package separately -    After executing DSS compilation, packaging and deployment, the related AppConn packages will be compiled at the same time. If you want to compile Schedules AppConn separately, you need to refer to the following steps: -```bash -cd {DSS_SOURCE_CODE_PROJECT}/dss-appconn/appconns/dss-schedulis-appconn -mvn clean install -``` -# 3. Install Scheduler AppConn -    In the DSS installation directory, the appconns directory already contains AppConn jar packages for third-party application tools connected to DSS, including the Schedulis AppConn jar package described in this document. -    When installing Schedulis, by executing the following script, you only need to simply enter the IP of the Schedulis deployment machine and the service Port. You can configure and complete the installation of the corresponding AppConn plug-in. When the script is executed, the init.sql script under the corresponding AppConn will be executed, and the corresponding database information will be inserted into the DSS table. -````sh -sh ${DSS_HOME}bin/appconn-install.sh - -# After executing the appcon-install installation script, enter the corresponding appconn name -# Follow the prompts to enter the IP corresponding to the schedulelis service, and the PORT +- The user first needs to check whether the schedulis directory exists under the xx/dss_linkis/dss/dss-appconns directory. If the user does not exist, the user needs to download the schedulis plugin +- The user installs Schedulis AppConn by executing the script appconn-install.sh, and only needs to enter the specific IP and Port of the machine where Schedulis WEB is deployed, and the AppConn plug-in installation can be completed. When the script is executed, the init.sql script under the corresponding AppConn will be executed, and the corresponding database information will be inserted into the DSS table +```shell +## Change directory to the installation directory of DSS +cd xx/dss + +## Execute the appconn-install installation script, enter the corresponding appconn name, and follow the prompts to enter the IP and PORT corresponding to the scheduleulis WEB service, +## Note that when the services are deployed on one machine, do not enter 127.0.0.1 or localhost for the IP, you must enter the real IP +sh bin/appconn-install.sh >> schedulis ->> 127.0.0.1 ->> 8089 -```` +>> xx.xx.xx.xx +>> 8085 +``` + +#### Configure the url for the "Go to dispatch center" button + +- Modify the `${DSS_HOME}/conf/dss-workflow-server.properties` configuration: + +```properties +#This path corresponds to the page of the dolphinscheduler operation and maintenance center +wds.dss.workflow.schedulerCenter.url="http://${schedulis_ip}:${schedulis_port}" +``` + +- Then restart the workflow to make the configuration take effect: + +```shell script +sh sbin/dss-daemon.sh restart workflow-server +``` + + +### 3. Scheduled JobType plugin installation + +- Users also need to install a JobType plugin for Schedulis: linkis-jobtype, please click [Linkis JobType installation document] (Schedulis_Linkis_JobType installation document.md). + +### 4. How to use Schedulelis -# 4. How to use Schedulis -    Schedulis service is deployed and Scheduled AppConn is installed, you can use Schedulis in DSS. In the app store, click Schedules to enter Schedules. During workflow development, you can publish DSS workflows to Schedules for scheduled execution with one click. +- Once the Schedulis service is deployed and the Schedulis AppConn is installed, Schedulis can be used in DSS. In the application component, users can click Schedulis to enter Schedulis, or during workflow development, one-click publishing of DSS workflows to Schedulis for scheduled execution. -# 5. Schedulis AppConn installation principle -    Schedulis related configuration information will be inserted into the following table, by configuring the following table, you can complete the use and configuration of Schedulis, when installing the Schedulis AppConn, the script will replace the init.sql under each AppConn, and Insert into the table. +### 5. Schedulis AppConn installation principle +- The relevant configuration information of Schedulis will be inserted into the following table. By configuring the following table, you can complete the use and configuration of Schedulis. When installing Schedulis AppConn, the script will replace the init.sql under each AppConn and insert it into the table. | Table Name | Table function | Remark | |-----------------|----------------|----------------------------------------| | dss_application | Application table, mainly to insert the basic information of schedule application | must | diff --git a/en_US/Installation_and_Deployment/Schedulis_Linkis_JobType_Installation_Documentation.md b/en_US/Installation_and_Deployment/Schedulis_Linkis_JobType_Installation_Documentation.md new file mode 100644 index 0000000..08e3318 --- /dev/null +++ b/en_US/Installation_and_Deployment/Schedulis_Linkis_JobType_Installation_Documentation.md @@ -0,0 +1,42 @@ +# Schedulis Linkis JobType installation documentation + +> This article mainly introduces the automatic deployment and installation steps of Schedulis' Linkis JobType. If you install it manually, please refer to Azkaban's JobType [installation steps](https://azkaban.github.io/azkaban/docs/latest/#job-types) + +1. Go to the Schedules directory + +```` +##Users first need to go to the installation directory of Schedulelis. The specific operation commands are as follows: +cd xx/schedulis_0.7.0_exec/plugins/jobtypes/linkis/bin +``` + +2. Modify config.sh configuration + +``` +## Linkis gateway url +LINKIS_GATEWAY_URL=http://127.0.0.1:9001 ## GateWay address for linkis + +## Linkis gateway token default WS-AUTH +LINKIS_GATEWAY_TOKEN=WS-AUTH ## The proxy token of Linkis, this parameter can use the default value + +## Azkaban executor host +AZKABAN_EXECUTOR_HOST=127.0.0.1 ## This IP is the machine IP if Schedulis is a stand-alone installation, or the Schedulis executor machine IP if it is a distributed installation + +## SSH Port +SSH_PORT=22 ## SSH port + +## Azkaban executor dir +AZKABAN_EXECUTOR_DIR=xx/schedulis_0.7.0_exec ## If Schedulis is a stand-alone installation, this directory is the installation directory of Schedulis. If it is a distributed installation, it is the installation directory of the executor. Note: You do not need to bring / at the end. + +## Azkaban executor plugin reload url +AZKABAN_EXECUTOR_URL=http://$AZKABAN_EXECUTOR_HOST:12321/executor?action=reloadJobTypePlugins ## You only need to modify the IP and port here. +``` + +3. Execute the installation script + +``` +sh install.sh +``` + + + + diff --git a/en_US/Installation_and_Deployment/VisualisAppConn_Plugin_Installation_Documentation.md b/en_US/Installation_and_Deployment/VisualisAppConn_Plugin_Installation_Documentation.md deleted file mode 100644 index 1837f48..0000000 --- a/en_US/Installation_and_Deployment/VisualisAppConn_Plugin_Installation_Documentation.md +++ /dev/null @@ -1,75 +0,0 @@ -# VisualisAppConn Installation Documentation - -This article mainly introduces the deployment, configuration and use of VisualisAppConn in DSS (DataSphere Studio) 1.0. - - - -## 1. Preparations for deploying VisualisAppConn - -Before you deploy VisualisAppConn, you must start Visualis deployment and ensure that the basic functions are available. - - - -## 2. Downloading and compiling the VisualisAppConn plugin - -We provide a material package for VisualisAppConn, if you have already downloaded it, you can skip this step. If you want to compile VisualisAppConn yourself, the specific compilation steps are as follows: -1. clone the code of DataSphere Studio -2. Compile visualis-appconn separately -```bash -cd {DSS_CODE_HOME}/dss-appconn/appconns/dss-visualis-appconn -mvn clean install -``` - -## 3.Deployment and configuration of the VisualisAppConn plugin - - -1. Obtain the visualis-appconn.zip material package from the target directory in step 2 -2. Put it in the following directory and unzip it -```bash -cd {DSS_HOME}/dss/dss-appconns -unzip visualis-appconn.zip -``` -3. Configure the relevant information of VisualisAppConn -``` bash -cd {DSS_INSTALL_HOME}/dss/bin -sh install-appconn.sh -The script is an interactive installation scheme. You need to enter the string visualis and the ip and port of the visualis service to complete the installation. -``` -4. Restart the dss service to complete the update of the plugin - -## 4. Use of VisualisAppConn -You can enter the front-end homepage of DSS, and then enter the homepage of the Visualis application, as shown in the figure - -![Visualis Embed DSS](../Images/Install_and_Deploy/VisualisAppConn_Deployment/DSS-Visualis.png) - -You can also use the visualization capabilities of Visualis by using the visualization node of the workflow, as shown in the figure - -![Workflow uses visualization nodes](../Images/Install_and_Deploy/VisualisAppConn_Deployment/Workflow-Visualis.png) - - - -## 5. How the VisualisAppConn plugin works -This section is an extension of the installation and a brief explanation of how VisualisAppConn works. - -1. Microservices using VisualisAppConn - -The following microservices in DSS will interact with Visualis through VisualisAppConn to complete the specified functions. - -|Microservice name | Function done using AppConn | Remarks | -|-----------------|----------------|----------------------------------------| -| dss-framework-project-server | Use visualis-appconn to complete the project and unify the organization | required | -| dss-workflow-server | Use the third-level specification to complete the creation, editing, import and export of nodes, etc. | required | -| appconn-engine | Use the third-level specification to complete the execution of the visualis node | required | - -2. Visualis integration into dss requires setting the corresponding content in the following database tables - -| Table name | Table role | Remarks | -|-----------------|----------------|----------------------------------------| -| dss_application | Application table, mainly to insert basic information of visualis application | required | -| dss_menu | Menu table, which stores the content displayed to the outside world, such as icons, names, etc. | Required | -| dss_onestop_menu_application | Association table of menu and application for joint search | required | -| dss_appconn | Basic information of appconn, used to load appconn | Required | -| dss_appconn_instance | information about the visualis instance, including the url information of visualis itself | required | -| dss_workflow_node | Information that Visualis needs to insert as a workflow node | If you want to use a visualization node, you must | - - diff --git a/en_US/User_Manual/DSS_How_To_Add_Users.md b/en_US/User_Manual/DSS_How_To_Add_Users.md new file mode 100644 index 0000000..04d4f1e --- /dev/null +++ b/en_US/User_Manual/DSS_How_To_Add_Users.md @@ -0,0 +1,61 @@ +# DataSphere Studio How to add users + +> DSS only provides an administrator account by default. User login authentication relies on Linkis' LDAP user login authentication system. This article will introduce in detail how to add a new DSS user. + +## 1. Basic introduction + +The user name of the DSS super administrator is the deployment user name. If the deployment user is hadoop, the administrator's user name and password are hadoop/hadoop. The specific user can be found in [DSS single-machine deployment document](../Installation/Deployment/DSS&Linkis One-click Deployment document stand-alone version.md) view. + +Adding a DSS user is mainly divided into the following steps: + +- Add LDAP user +- Improve environmental information for new users + +## 2. Add LDAP user + +DSS super administrators can create departments and users on the homepage, establish the company's hierarchy and personnel management system, and create departments and users. As shown below: + + ![Super administrator home page](images/Super_Administrator_Function.png) + +#### Create department and user: + +On the home page of the super administrator, click [Management Desk] to enter the management desk page. + +Super administrators can create, modify and delete departments (please note: a maximum of four levels can be established at the department level, including meta-company), meta-company cannot be deleted, only information can be modified. + +At the same time, the super administrator can create, modify and reset user passwords. + +As shown below: + + ![Create departments and users](images/Create_Departments_And_Users.png) + +After you create a user on the page, the DSS background will automatically request LDAP and create a user with the same name in LDAP for you. + +## 3. Improve environmental information for new users + +1. Due to the top-down multi-tenant isolation of DSS & Linkis, in order to allow logged-in users to use DSS normally, it is necessary to create a corresponding Linux user on the Linux server. The specific steps are as follows: + +- Create corresponding Linux users on all Linkis & DSS servers. +- If Hadoop is used, a corresponding Linux user needs to be created on the NameNode of Hadoop. +- Ensure that Linux users on the Linkis & DSS server can use commands such as `hdfs dfs -ls /` normally, and the user needs to be able to execute shell commands such as `spark-sql -e` and `hive -e` normally. +- Since each user's workspace is strictly isolated, you also need to create a workspace and HDFS directory for the user, as follows: + +```shell script +## Create user workspace directory and authorization +mkdir $WORKSPACE_USER_ROOT_PATH/${NEW_USER} +chmod 750 $WORKSPACE_USER_ROOT_PATH/${NEW_USER} + +## Create user HDFS directory and authorize +hdfs dfs -mkdir $HDFS_USER_ROOT_PATH/${NEW_USER} +hdfs dfs -chown ${NEW_USER}:${NEW_USER} $HDFS_USER_ROOT_PATH/${NEW_USER} +hdfs dfs -chmod 750 $HDFS_USER_ROOT_PATH/${NEW_USER} +``` + +`WORKSPACE_USER_ROOT_PATH` and `HDFS_USER_ROOT_PATH` are the workspace and HDFS root paths set when you install DSS with one click. + +If you don't set it, it defaults to: + +```shell script +WORKSPACE_USER_ROOT_PATH=file:///tmp/linkis +HDFS_USER_ROOT_PATH=hdfs:///tmp/linkis +``` diff --git a/en_US/User_Manual/Data_Service_User_Manual.md b/en_US/User_Manual/Data_Service_User_Manual.md new file mode 100644 index 0000000..1718e7f --- /dev/null +++ b/en_US/User_Manual/Data_Service_User_Manual.md @@ -0,0 +1,60 @@ +Data Service +---------- + +        DSS currently supports publishing SQL scripts as data service APIs and sharing them with other users. Business users can execute data service scripts and browse or download platform data directly by setting parameters without writing code and without big data platform users. +Data service is a kind of service provided to users who do not have database table permission for users with database table permission. This article will focus on publishing data service users (referred to as publishing users) and using data service users (referred to as using users) from two different perspectives. Be explained. +Due to business needs, the use of data services will have the following restrictions: +* Only supports Spark SQL queries +* Multiple result set queries are not supported +* Only supports query SELECT statement +* can only contain one SQL statement +* Semicolons are not allowed at the end of SQL statements + +        In addition to the above restrictions, the following notation is allowed. +```sql +USE DATABASE default; +SELECT * FROM default.book +``` + +**1. Create a data service** + +        Due to business needs, it is necessary to establish a data service and authorize others to use it. When the publishing user enters Scriptis, edits the new script file, writes Spark SQL statements, and embeds variables in the SQL statements, which is convenient for subsequent business personnel to set parameters by themselves. data can be obtained. +        After saving the Spark SQL script, you can click "Publish as Data API" at the top of the script editing bar (the "Publish as Data API" function, only some users have this permission, and the function button is not visible for users who do not have permission. ), fill in the new API information as shown in the figure below. +![](./images/createapiservice.png) + +Click Next for information on setting variables. +![](./images/createapiservice_param.png) + + +Publishing users can use the data service on the homepage of the workspace, through the application tool, enter the "Data Service" application to use the data service, and click "More" on the data service tab to enter the data service management interface + +![](./images/apiservicepage.png) + + + +**2. Use of data services** + +        After entering the data service page, you can see the data service list page that the user can use, where default means all data services by default, the user can click the corresponding The label filters out the data services you need to use, and you can filter by name, status, and submitter in the search box. + +        Users can click "Enter Use" to set the value of parameters in the filter conditions. The user is a subset of the published user data set. + +![](./images/useapiservice.png) + +**3. Modify data service** + +        A data service may be modified due to business needs. When the publishing user has modified the script of the data service, he can click "Update Data API". +To update the data service, you can select other data services to be bound. +![](./images/modifyapiservice.png) + +**4. Use postman to access data services** + +After the data service is released, it can be accessed using the api interface, which can be called directly to other systems. Submit a query as shown below: + +![](./images/postman1.png) + +The task execution ID is obtained, and then the task execution progress, log, result set, etc. can be obtained according to the ID. + +![](./images/postman2.png) + +Description: The token of the data service can be returned from the /queryById? interface (click to use it), and the field is userToken. Access to all interfaces must be authenticated by GateWay. The token of the data service can only be used for the management process of the data service. Authentication using postman requires the use of the cookie of the page and the key authentication method of linkis-gateway. In the head, add Token-Code: XXX here to specify the login key of linkis-gateway Token-User: XXX here to specify the login user of linkis-gateway. + diff --git a/en_US/User_Manual/Introduction_to_Data_Services.md b/en_US/User_Manual/Introduction_to_Data_Services.md new file mode 100644 index 0000000..84a79b0 --- /dev/null +++ b/en_US/User_Manual/Introduction_to_Data_Services.md @@ -0,0 +1,59 @@ +DSS Plugin - Data Api Services +---------- +## motivation +    In actual business, business personnel and data developers usually need to be connected, and business personnel are usually not familiar with data development, so when there is data demand, they usually need to communicate with data personnel. Data developers usually develop data according to the needs of the business side, and then complete the requirements by sharing scripts or data. +For data extraction, there are many scenarios in actual production, but most of the main structures are similar. Next, the scenarios of data provision will be described according to several aspects. +## traditional data extraction +    There are two ways when the business side provides the requirements, and the data developer develops the script for fetching data and hands over with the business side. +- The first type: directly share the script for fetching data to business personnel (usually requiring shared storage or other methods), and the business personnel run the script to obtain data by themselves. The advantages and disadvantages of this method are as follows: + Advantages: suitable for some temporary needs, the script is usually only used once or several times, which is more convenient. + Disadvantage: Because business personnel also need to execute scripts, business personnel need permissions for the libraries and tables involved in the script. There is a risk of business people tampering with the script. +- The second type: Data developers run the fetching scripts themselves, and then share the data with business personnel. The advantages and disadvantages of this method are as follows: + Advantages: suitable for some temporary needs, the script is usually only used once or several times, which is more convenient. + Disadvantages: There is a risk of security compliance, and data that the business does not have permission to view is at risk of being leaked. +## Data View +    Data views are similar to data services, but there are some differences in the process. When business users need data, they first put forward the demand, and then apply for the permission to use the data with the bill of lading. After the bill of lading is completed, the data developer can develop the demand and publish the developed script to the business user. +Due to the security requirements in the industry, access to data without permission requires Bill of Lading authorization. So the advantages and disadvantages of data views are similar to data services. +The difference is that the B/L application of the data view is initiated by the business user. +## Data Service (Api Service) +    Data service is a service provided to users with database table permissions, and it can also prevent tampering of published scripts. In terms of business, the way of data service, the business side first initiates the demand, and then the data developer develops the script and publishes it as a data service. +When publishing data services, the process of authorizing business users to use them requires authorization according to the company's security requirements. The bill of lading request is initiated by the data developer. After the authorization is passed, business users can use the data service. There are the following problems with the way the data is served. +Advantages: Scripts suitable for some fixed requirements, more convenient for scenarios where the script is used multiple times. At the same time, it can prevent tampering with the script. Business users can access data without granting database table-level permissions. It is more friendly, and business users can fetch numbers with simple operations. +## Data Services and Data Views +    The data view trial scenario is generally that the business side initiates a demand for data development from top to bottom. Before the demand is provided, the data development does not know the required library table permissions and the specific content of the development. +The data service is different. In some relatively fixed business scenarios, the data service is initiated from the bottom to the top, and the data developer specifies the set of data access. The business side uses this script, and the data that the business side can view can only be provided by data development. subset of the dataset. +## how to use +### compile +```shell +# Compiled and packaged by DSS +mvn -N install +mvn -DskipTests=true clean package +``` +    After the packaged tar package is installed and decompressed, the corresponding lib package can be found in dss-apps. If you want to extract the service, you can package it separately, and the related startup scripts and configurations need to be stripped and modified. Currently, it is recommended to compile and package with DSS. +### Startup and shutdown +```shell +# 1. Use batch start and stop scripts +sh sbin/start-all.sh +sh sbin/stop-all.sh + +# 2. Start and stop the service individually +sh sbin/dss-daemon.sh start dss-datapipe-server +sh sbin/dss-daemon.sh stop dss-datapipe-server +sh sbin/dss-daemon.sh restart dss-datapipe-server +``` +### Related Restrictions +    Currently, DSS data services can only be used within DSS Scritis. Before using, you need to pay attention to the following: +1. Only Spark SQL scripts are supported to be published as data services. +2. The script parameters of the data service need to be configured with the ${params} parameter in Scritis. +3. After publishing the data service, you can view the published data service in the data service module. +4. Only users with data service permissions in the workspace configuration can use the data service function. + +### Data Service Architecture +      List. Only the publisher or proxy user has permissions on the entire table. The architecture diagram of the data service is as follows: + + +### Data service usage +    Data Api Service is a part of DSS plug-ins and a part of DSS ecosystem. For the usage instructions of data services, please refer to the document [Introduction to Data Service Usage](). + + + diff --git a/en_US/User_Manual/Schedule_Center_Documentation.md b/en_US/User_Manual/Schedule_Center_Documentation.md new file mode 100644 index 0000000..76e6d27 --- /dev/null +++ b/en_US/User_Manual/Schedule_Center_Documentation.md @@ -0,0 +1,142 @@ +# Schedule Center Documentation + +The scheduling center is a visual workflow scheduling platform. The operation and maintenance large-screen function can visually monitor and count the running status of workflow tasks in real time. The scheduling function supports the running of workflows. + +Timing management, online and offline operations, etc., also support workflow management, such as rerun, resume, terminate, pause and other operations. + +The log function supports operations such as viewing and downloading workflow log information. + +**Note: If you want to use the dispatch center, please install it first DolphinSchedulerAppConn。** [How to install DolphinSchedulerAppConn](../Installation_and_Deployment/DolphinScheduler_Plugin_Installation_Documentation.md) + +## page overview + +The Dispatch Center page is shown below. + +![Overview](images/Schedule_Center/Page_Overview.png) + +The functions of each area on the homepage of the operation and maintenance center are as follows: + +- ①Project and workflow list: The list on the left side of the page displays all projects and workflows in the workspace, and you can switch projects here; +- ②Function module list: This area displays the operation and maintenance screen, workflow definition, workflow instance and task instance under the project in units of projects; +- ③ Operation area: perform operations on specific functions in this area; + +## Operation and maintenance large screen + +The large operation and maintenance screen is used to display the statistical chart monitoring information of all workflows in different time periods under the entire project, and display process status statistics, cycle instance completion status, workflow instance and success rate statistics, and process definition statistics. + +The operation steps are as follows: + +1. Log in to the platform, enter the operation and maintenance center, and select "Operation and Maintenance Large Screen" +2. View the workflow statistics under the currently selected item. The meanings and operations of different information are as follows: + +#### 2.1 Process Status Statistics + +Count the running status of all workflow instances under the selected project and display them in the form of a pie chart. + +![](images/Schedule_Center/Process_Status_Statistics.png) + +Users can click the **date icon** to select the statistical time period + +![](images/Schedule_Center/Process_Status_Statistics_By_Date.png) + +#### 2.2 Cycle instance completion + +Statistics on the task execution of periodic instances under the selected project, where the horizontal axis is time, and the vertical axis is the number of periodic instances. Users can click [Running Successfully] or [Running Failed] in the upper right corner to switch the cycle instance to be viewed. +![](images/Schedule_Center/Cycle_Instance_Completion.png) + +#### 2.3 Workflow instance and success rate statistics + +Count the number of execution workflow instances and success rate information in different time periods under the selected item, where the horizontal axis is time, and the vertical axis is the number of instances and success rate. Users can click [Today] and [Yesterday] in the upper right corner to switch the date to be viewed. +![](images/Schedule_Center/Workflow_Instance_And_Success_Rate_Statistics.png) + +#### 2.4 Process time ranking + +Displays the time-consuming descending ranking of task instances that have an end time in the corresponding time period. + +![](images/Schedule_Center/Process_Time_Ranking.png) + +## Workflow Definition + +Workflow definition displays the information of all workflows defined under the currently selected item, and can perform operations such as running, timing, online/offline, and timing management for a single workflow in the operation bar. +![](images/Schedule_Center/Workflow_Definition.png) + +- 1. How to run the currently selected workflow? + +Select the row of the workflow to be run, and click the **Run** button in the action bar. + +![](images/Schedule_Center/Run_The_Workflow.png) + +- 2. How to run the workflow regularly? + +Select the row of the workflow to be run, and click the **Timer** button in the action bar. + +In the pop-up box, set the parameters and configure the notification person and other information. After filling in, click the [Create] button to set the scheduled task. + +![](images/Schedule_Center/Run_A_Workflow_On_A_Schedule.png) + +- 3. Workflow online/offline switching + +Select a workflow that is not online, and click the **Go Online** and **Go Offline** buttons in its operation bar to go online and offline. + +- 4. Timing management + +Workflow timing management can perform operations such as online/offline switching, deletion or editing of the historical execution of the workflow and the running timing service. + +Select the workflow that needs timing management, and click the **Timing** button in its operation bar to enter the timing management interface of the workflow. As shown below: + +![](images/Schedule_Center/Timed_Page.png) + +Click the **Edit** button to edit the scheduled service of the workflow. + +![](images/Schedule_Center/Edit_Scheduled_Tasks.png) + + +## Workflow instance + +The workflow instance page displays the workflow status, running type, start time, running duration and other information in the form of a list, and can operate the running status of the workflow. + +![](images/Schedule_Center/Workflow_Instance.png) + +- 1. Rerun the workflow + +Select the workflow that needs to be rerun, and click the **Rerun** button in its operation bar. + +- 2. Resume workflow + +Select the workflow to be restored and click the **Restore** button in its action bar. + +- 3. Terminate the workflow + +Select the workflow to be terminated and click the ** Terminate ** button in its action bar. + +- 4. Pause the workflow + +Select the workflow that needs to be paused, and click the **Pause** button in its operation bar. + +- 5. Delete workflow + +Select the workflow that needs to be deleted and click the **Delete** button in its operation bar. + +- 6. View Gantt Chart + +Look at the Gantt chart of the workflow, which consists of the time and the runtime of each node under the workflow. + +Select the workflow that needs to view the Gantt chart, and click the **View Gantt Chart** button in its operation bar. + +## task instance + +The task instance page displays the basic information and running status of each node task in the workflow in the form of a list, and you can view the running log of the task instance. + +![](images/Schedule_Center/Task_Instance.png) + +- 1. View log + +View the running log of the task instance, and support operations such as downloading, refreshing, and full-screen viewing of the log. + +![](images/Schedule_Center/View_Logs.png) + +Click the **Download** button in the log window to download the log to the local. + +For a running task instance, click the **Refresh** button in the log window to refresh the log in real time. + +Click the **full screen** button in the log window to view the log in full screen. \ No newline at end of file diff --git a/en_US/User_Manual/Scriptis_Usage_Documentation.md b/en_US/User_Manual/Scriptis_Usage_Documentation.md new file mode 100644 index 0000000..8899b6d --- /dev/null +++ b/en_US/User_Manual/Scriptis_Usage_Documentation.md @@ -0,0 +1,22 @@ +Scriptis usage documentation +------ +## Introduce +    Use the computing engines such as Spark, Hive, and HBase of the big data platform to conduct interactive query and analysis, and support the daily use of data mining and analysts. Provides a graphical and multi-style interface, making it more convenient and simple for users to perform data analysis, script editing, testing, and querying. +![](./images/scriptis.png) + +## workspace +    The workspace is a file directory, and the user has all the permissions to perform file management operations and so on. Generally, it corresponds to a file directory deployed by the Linkis server, and each login user corresponds to a file directory, which stores files such as user scripts and result sets. +![](./images/scriptis_workspace.png) +    When you right-click the workspace folder, the right-click functions mainly include copy path, new directory, new script, and refresh. +![](./images/scriptis_workspace_dir.png) +    When the mouse right-clicks the file under the workspace folder, the script right-click function, the script right-click mainly includes punching to the side, copying the path, renaming, deleting, importing to hive (csv, txt, excel type files), importing to hdfs, etc. Function. +![](./images/scriptis_workspace_file.png) +## database +The database module obtains the hive library that the logged-in user has permission. The main functions of the right-click library include brushing the library, brushing the table, and brushing the field information. Table right-click function - query table, quickly produce temporary hive script for data viewing, copy table name and delete table, that is, copy table fields. The right-click function of the table can view the table structure, and can display the field details of the table, table details, table partition information, etc. +![](./images/scriptis_database.png) +## UDFs and functions +The UDF function is convenient for users to classify and display UDFs, and users can manage personal functions. The configuration management of UDF has been moved to the Linkis management console. For related configuration, please refer to the related documents of Linkis UDF. + +## HDFS +After Linkis is installed, each user is provided with an HDFS path by default to store user resource files. Scriptis will display the HDFS folder of the user. You can right-click to add, delete, and modify the folder. At the same time, the files in this path can also be managed through the right-click function. +![](./images/scriptis_hdfs.png) diff --git a/en_US/User_Manual/Super_Administrator_Function.md b/en_US/User_Manual/Super_Administrator_Function.md new file mode 100644 index 0000000..423f068 --- /dev/null +++ b/en_US/User_Manual/Super_Administrator_Function.md @@ -0,0 +1,26 @@ +## workspace concept + +In data development and management, the concept of a workspace is similar to a team. A company (ie, a master account) can create multiple workspaces. A workspace is the basic unit for managing tasks, members, and assigning roles and permissions. Collaborative development, operation and maintenance, analysis, etc. in one workspace. + +The organization's master account defaults to the organization's workspace administrator user, which is responsible for managing the organization's workspace management console, planning the workspace structure, and creating and deleting workspaces. + +Super administrators can click [Create Workspace] on the administrator homepage to create a workspace. + +![Create a workspace](images/Create_a_workspace.png) + +#### Workspace Type: + +Workspaces are divided into two categories: project-oriented and department-oriented: + +- Project-oriented workspace participation members are vertical structures and can come from different departments; +- Department-Oriented Workspace Participating members are horizontally structured and come from the same department. + +In the project-oriented workspace, members of any department can be added; + +In a department-oriented workspace, only members of this department can be added. + +#### Workspace management: + +The default role of a workspace creator is owner, who can delegate the management of the workspace to one or more administrators. + +Only the administrator of the workspace can enter the [Workspace Management] module to manage the basic information and permission information of the workspace accordingly. \ No newline at end of file diff --git a/en_US/User_Manual/User_Documentation.md b/en_US/User_Manual/User_Documentation.md new file mode 100644 index 0000000..72d971b --- /dev/null +++ b/en_US/User_Manual/User_Documentation.md @@ -0,0 +1,26 @@ +## DSS User Document + +## motivation +    DSS1.x version is a milestone version. It has carried out a lot of optimization and reconstruction on the basis of DSS0.x. Due to the limited space, it can only cover the basic use process of DSS, and more details of operation and use. I hope to work with community partners. Improve and optimize, if you have any questions or suggestions in the process of use, you can contact the relevant community open source personnel of WeBank's big data platform at any time. We are committed to creating a better one-stop big data suite and contributing to the big data open source ecosystem. the power of. + +## foreword +    The DSS1.x version has refactored and optimized the front-end page interaction. This document is the DSS user manual, which covers the basic usage process of DSS1.0. For more details on operation and usage, please refer to the documentation of each module. +    User documentation is mainly divided into the following documents: +1. [Scriptis usage documentation]() +2. [Workflow usage document]() +3. [Data service usage document]() +4. [Linkis console user manual]() + +## Management Module Introduction +## Login to the homepage +    For the convenience of users, the system defaults to the Linux deployment user of Linkis to log in. For example, Linkis and DSS deployed with hadoop can log in directly through user: hadoop, password: hadoop (the password is the username). After entering the DSS front-end address, 127.0.0.1:8088 Enter the user name and password: hadoop hadoop to log in. The login page provides the DSS user access permission verification function. +![](./images/loginpage.png) +* Note: If you want to support multi-user login, DSS user login depends on Linkis, which needs to be configured in the configuration of linkis-GateWay. Linkis-GateWay supports LDAP by default. * + +## workspace +    Go to the workspace page to create and manage workspaces. Workspace is the top-level concept of DSS. For example, a workspace can be a department, a business line, or an organization, which is used to manage data applications, including personnel, projects, or components. With the role permissions of the workspace administrator, you can manage the workspace and control the components and personnel permissions of the workspace. +![](./images/workspace.png) + +## Engineering Management +    After entering the corresponding workspace, it will jump to the homepage of the project. On the homepage of the project, you can create a project. In actual development and production, projects are often used to manage and develop a type of data application, including workflow, single task, etc. The projects under each workspace are isolated from each other. In practical applications, it is an ideal way to divide a project for each data application. +![](./images/project.png) \ No newline at end of file diff --git a/en_US/User_Manual/Workflow_Usage_Documentation.md b/en_US/User_Manual/Workflow_Usage_Documentation.md new file mode 100644 index 0000000..dac653d --- /dev/null +++ b/en_US/User_Manual/Workflow_Usage_Documentation.md @@ -0,0 +1,100 @@ +DSS-Workflow usage documentation +------ + +## Introduce +        Data workflow (Workflow) refers to the transformation of data extraction and transformation into "the automation of part or the whole of the business process in the computer application environment", which can bring great convenience to scenarios such as data analysis and processing. +        Compared with interactive analysis, DSS workflow greatly simplifies data processing tasks. The advantages of workflow can be simplified into the following three points: + +1. It can automatically run batches to build the intermediate table data table, which is convenient for data use. +2. Automatically start running after the data is ready, and batch data processing regularly. +3. Workflow automation displays data into reports and data output. + + +## Introduction to Workflow Features +        Entering the DSS workflow editing page, you can develop the workflow by dragging and dropping. The execution order of each node needs to be mentioned in the node connection control. If you need to debug a related node, after configuring the node, perform debugging of the node to determine whether the task configuration is correct. +        SQL, HQL and other script nodes, after double-clicking to enter the editor, you can click Execute to run, and you can check whether the script is correct through the results (the entire workflow will not be triggered). After editing a workflow, click the Execute button directly on the page to execute the workflow in real time. This step can check whether the entire workflow can be fully run. +        All flow nodes are placed in the blank area on the right by dragging and dropping. After placing the workflow node on the interface, when the mouse moves over a single node, four dots will appear in the flow node. Click the dot to draw a line. It can then be connected to another flow node, and the direction and order of the connections determine the order in which the workflow nodes are executed. +![](./images/workflow.png) +![](./images/runworkflow.png) +        After the workflow development and debugging is completed, click Publish, and the workflow will submit the workflow to Schedulis. Find the project in Schedulis and set the scheduling parameters to schedule the workflow. + +## Introduction to Workflow Nodes + +### Spark node +         It supports sql, pyspark, and scala to execute spark tasks. When using it, you only need to drag the node to the workbench and write the code. +### Hive node +         The hive node supports the execution of hive tasks in SQL mode. When using it, you only need to drag and drop the node to the workbench and write the hivesql code. +### python node +         The python node supports executing python tasks. When using it, you only need to drag and drop the node to the workbench and write python code. +### shell node +         The shell node supports executing shell commands or running scripts. When using it, you only need to drag and drop the node to the workbench and write shell commands. +### jdbc node +         The jdbc node supports running sql commands in jdbc mode. When using it, you only need to drag and drop the node to the workbench and write the sql. **Note that you need to configure the jdbc connection information in the linkis console management console in advance.** + +### signal node +         The EventSender node is used to send information and send a piece of information event to the EventReceiver. + +         Common scenarios include upstream and downstream dependencies between projects, and upstream and downstream information dependencies between workflows. For example, a workflow node of workflow B depends on some information of a workflow node of workflow A (such as status information, that is, node A can execute successfully before node B can start executing), eventSender supports the following parameters: + +```xml +1. msg.type: used to specify the type of Job, SEND is used to send messages, RECEIVE is used to receive messages +2. msg.sender: Specifies the sender of the message, which needs to be defined in the format of ProjectName@WFName@JobName +3. msg.topic: Specify the topic of the message. It is recommended to use the following format: first-level classification code+_+second-level classification code+_+third-level classification code +4. msg.name: Specifies the message name, which is user-defined +5. msg.body: specify the content of the message, it can be empty if no content is sent +6. **Note: msg.type cannot be changed to SEND by default, msg.sender, msg.topic, msg.name are required** +``` + +E.g： +```xml +msg.type=SEND +msg.sender=project01@flow@job01 +msg.topic=bdp_tac_test +msg.name=TestDynamicReceive +msg.body=${msg.mycontent} +``` +         The EventReceiver node is used to receive the message sent by the eventSender, and store the content of the received message in the context of the workflow. Subsequent nodes will find the information according to the prefix and use it as a custom variable. The eventReceiver supports the following parameters : + +```xml +1. msg.type: used to specify the type of Job, SEND is used to send messages, RECEIVE is used to receive messages +2. msg.receiver: Specifies the receiver of the message, which needs to be defined in the format of projectname@jobname@rtxname +3. msg.topic: Specify the topic of the message. It is recommended to use the following format: first-level classification code+_+second-level classification code+_+third-level classification code +4. msg.name: Specifies the message name, which is user-defined +5. query.frequency: Since active polling is used to receive messages, the number of queries during wait.time +6. max.receive.hours: The longest reception time, in hours +7. msg.savekey: used to save the key value of the message content. For multiple receiving jobs in a single flow, you need to specify different msg.savekey to save the message content. The default value is msg.body. Subsequent jobs can use this key value to get the message content. +8. only.receive.today: If it is true, it can only receive messages from the day the job is started +9. Note: msg.type cannot be changed to RECEIVE by default, msg.receiver, msg.topic, msg.name are required +``` + +### DataCheck node： +         The DataCheck node is used to detect whether the data is ready. It can determine whether a table or partition in the hive library exists. If it exists, it will be executed downstream. It plays a very important role in data-dependent tasks and is used to replace the previous verbal agreement. Good time to start running. + +         dataCheck supports the following parameters： +``` +1. source.type: dependent data source +2. check.object: The name of the dependent data For example: data.object.1=dbname.tablename{partitionlist} +3. max.check.hours: Describe the waiting time of the task, the unit is hours +4. job.desc: Append multi-source information configuration +``` + +### SendEmail node +         The SendEmail node is generally used as the last node of the workflow, which is used to send the result information in front of the workflow. It supports sending forms, texts, DashBoard, Display, pictures, etc. The user directly selects the workflow node that they want to send when using it. That is, at present, it needs to be used with the third-party nodes DashBoard and Display of Visualis in the DSS workflow. +         sendEmail supports the following parameters: +``` +Type: supports node, text, file, link +Email subject: Specify the email list to mention +Send item: the specific content to send, for example: if the type is node, select the node here +Associated approval form: whether the email has been approved, if not, it will not be sent +Other basic mail properties: To, Cc, Bcc +``` + +### connection node +         The role of the Connector node is to adjust the workflow layout as a connection between nodes and nodes, and it does not have special functions. + +### child workflow node +         The Subflow node allows you to embed a sub-workflow in a workflow. When the parent workflow is published, the sub-workflow will also be published synchronously, but when the parent workflow is executed in real time, the execution of the sub-workflow will be skipped. +         If you want to execute a sub-workflow, double-click the sub-workflow node to enter the sub-workflow editing page for execution. + +### third-party node +         The third-party node needs to configure the AppConn installation package according to the corresponding third-party system, and use it after inserting the relevant fields of the AppConn table. diff --git a/en_US/User_Manual/images/Create_Departments_And_Users.png b/en_US/User_Manual/images/Create_Departments_And_Users.png new file mode 100644 index 0000000..fed6c16 Binary files /dev/null and b/en_US/User_Manual/images/Create_Departments_And_Users.png differ diff --git a/en_US/User_Manual/images/Create_a_workspace.png b/en_US/User_Manual/images/Create_a_workspace.png new file mode 100644 index 0000000..d8ac84a Binary files /dev/null and b/en_US/User_Manual/images/Create_a_workspace.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Cycle_Instance_Completion.png b/en_US/User_Manual/images/Schedule_Center/Cycle_Instance_Completion.png new file mode 100644 index 0000000..156346e Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Cycle_Instance_Completion.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Edit_Scheduled_Tasks.png b/en_US/User_Manual/images/Schedule_Center/Edit_Scheduled_Tasks.png new file mode 100644 index 0000000..e144a80 Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Edit_Scheduled_Tasks.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Page_Overview.png b/en_US/User_Manual/images/Schedule_Center/Page_Overview.png new file mode 100644 index 0000000..22c63e6 Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Page_Overview.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Process_Status_Statistics.png b/en_US/User_Manual/images/Schedule_Center/Process_Status_Statistics.png new file mode 100644 index 0000000..7153518 Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Process_Status_Statistics.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Process_Status_Statistics_By_Date.png b/en_US/User_Manual/images/Schedule_Center/Process_Status_Statistics_By_Date.png new file mode 100644 index 0000000..f8facb6 Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Process_Status_Statistics_By_Date.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Process_Time_Ranking.png b/en_US/User_Manual/images/Schedule_Center/Process_Time_Ranking.png new file mode 100644 index 0000000..ff37340 Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Process_Time_Ranking.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Run_A_Workflow_On_A_Schedule.png b/en_US/User_Manual/images/Schedule_Center/Run_A_Workflow_On_A_Schedule.png new file mode 100644 index 0000000..c1a719f Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Run_A_Workflow_On_A_Schedule.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Run_The_Workflow.png b/en_US/User_Manual/images/Schedule_Center/Run_The_Workflow.png new file mode 100644 index 0000000..54e42b7 Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Run_The_Workflow.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Task_Instance.png b/en_US/User_Manual/images/Schedule_Center/Task_Instance.png new file mode 100644 index 0000000..ad55bac Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Task_Instance.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Timed_Page.png b/en_US/User_Manual/images/Schedule_Center/Timed_Page.png new file mode 100644 index 0000000..78c0987 Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Timed_Page.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/View_Logs.png b/en_US/User_Manual/images/Schedule_Center/View_Logs.png new file mode 100644 index 0000000..bf1baac Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/View_Logs.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Workflow_Definition.png b/en_US/User_Manual/images/Schedule_Center/Workflow_Definition.png new file mode 100644 index 0000000..ed9af3e Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Workflow_Definition.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Workflow_Instance.png b/en_US/User_Manual/images/Schedule_Center/Workflow_Instance.png new file mode 100644 index 0000000..c5476b3 Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Workflow_Instance.png differ diff --git a/en_US/User_Manual/images/Schedule_Center/Workflow_Instance_And_Success_Rate_Statistics.png b/en_US/User_Manual/images/Schedule_Center/Workflow_Instance_And_Success_Rate_Statistics.png new file mode 100644 index 0000000..75bb1d4 Binary files /dev/null and b/en_US/User_Manual/images/Schedule_Center/Workflow_Instance_And_Success_Rate_Statistics.png differ diff --git a/en_US/User_Manual/images/Super_Administrator_Function.png b/en_US/User_Manual/images/Super_Administrator_Function.png new file mode 100644 index 0000000..de12dd6 Binary files /dev/null and b/en_US/User_Manual/images/Super_Administrator_Function.png differ diff --git a/en_US/User_Manual/images/apiservicepage.png b/en_US/User_Manual/images/apiservicepage.png new file mode 100644 index 0000000..3030bbc Binary files /dev/null and b/en_US/User_Manual/images/apiservicepage.png differ diff --git a/en_US/User_Manual/images/createapiservice.png b/en_US/User_Manual/images/createapiservice.png new file mode 100644 index 0000000..037525e Binary files /dev/null and b/en_US/User_Manual/images/createapiservice.png differ diff --git a/en_US/User_Manual/images/createapiservice_param.png b/en_US/User_Manual/images/createapiservice_param.png new file mode 100644 index 0000000..3703b42 Binary files /dev/null and b/en_US/User_Manual/images/createapiservice_param.png differ diff --git a/en_US/User_Manual/images/loginpage.png b/en_US/User_Manual/images/loginpage.png new file mode 100644 index 0000000..89e1fd2 Binary files /dev/null and b/en_US/User_Manual/images/loginpage.png differ diff --git a/en_US/User_Manual/images/modifyapiservice.png b/en_US/User_Manual/images/modifyapiservice.png new file mode 100644 index 0000000..babde90 Binary files /dev/null and b/en_US/User_Manual/images/modifyapiservice.png differ diff --git a/en_US/User_Manual/images/postman1.png b/en_US/User_Manual/images/postman1.png new file mode 100644 index 0000000..f4e22b9 Binary files /dev/null and b/en_US/User_Manual/images/postman1.png differ diff --git a/en_US/User_Manual/images/postman2.png b/en_US/User_Manual/images/postman2.png new file mode 100644 index 0000000..a46eac6 Binary files /dev/null and b/en_US/User_Manual/images/postman2.png differ diff --git a/en_US/User_Manual/images/project.png b/en_US/User_Manual/images/project.png new file mode 100644 index 0000000..65cdd3a Binary files /dev/null and b/en_US/User_Manual/images/project.png differ diff --git a/en_US/User_Manual/images/runworkflow.png b/en_US/User_Manual/images/runworkflow.png new file mode 100644 index 0000000..b8f7b77 Binary files /dev/null and b/en_US/User_Manual/images/runworkflow.png differ diff --git a/en_US/User_Manual/images/scriptis.png b/en_US/User_Manual/images/scriptis.png new file mode 100644 index 0000000..88205d2 Binary files /dev/null and b/en_US/User_Manual/images/scriptis.png differ diff --git a/en_US/User_Manual/images/scriptis_database.png b/en_US/User_Manual/images/scriptis_database.png new file mode 100644 index 0000000..6ed7ef8 Binary files /dev/null and b/en_US/User_Manual/images/scriptis_database.png differ diff --git a/en_US/User_Manual/images/scriptis_hdfs.png b/en_US/User_Manual/images/scriptis_hdfs.png new file mode 100644 index 0000000..1ec6fb1 Binary files /dev/null and b/en_US/User_Manual/images/scriptis_hdfs.png differ diff --git a/en_US/User_Manual/images/scriptis_summary.png b/en_US/User_Manual/images/scriptis_summary.png new file mode 100644 index 0000000..2bff56c Binary files /dev/null and b/en_US/User_Manual/images/scriptis_summary.png differ diff --git a/en_US/User_Manual/images/scriptis_workspace.png b/en_US/User_Manual/images/scriptis_workspace.png new file mode 100644 index 0000000..82383cb Binary files /dev/null and b/en_US/User_Manual/images/scriptis_workspace.png differ diff --git a/en_US/User_Manual/images/scriptis_workspace_dir.png b/en_US/User_Manual/images/scriptis_workspace_dir.png new file mode 100644 index 0000000..90a55c0 Binary files /dev/null and b/en_US/User_Manual/images/scriptis_workspace_dir.png differ diff --git a/en_US/User_Manual/images/scriptis_workspace_file.png b/en_US/User_Manual/images/scriptis_workspace_file.png new file mode 100644 index 0000000..37aaaa0 Binary files /dev/null and b/en_US/User_Manual/images/scriptis_workspace_file.png differ diff --git a/en_US/User_Manual/images/useapiservice.png b/en_US/User_Manual/images/useapiservice.png new file mode 100644 index 0000000..7414f23 Binary files /dev/null and b/en_US/User_Manual/images/useapiservice.png differ diff --git a/en_US/User_Manual/images/workflow.png b/en_US/User_Manual/images/workflow.png new file mode 100644 index 0000000..43dac2f Binary files /dev/null and b/en_US/User_Manual/images/workflow.png differ diff --git a/en_US/User_Manual/images/workspace.png b/en_US/User_Manual/images/workspace.png new file mode 100644 index 0000000..66f1ccc Binary files /dev/null and b/en_US/User_Manual/images/workspace.png differ diff --git a/en_US/Using_Document/DSS_Interface_Summary.md b/en_US/Using_Document/DSS_Interface_Summary.md new file mode 100644 index 0000000..b4dbdaf --- /dev/null +++ b/en_US/Using_Document/DSS_Interface_Summary.md @@ -0,0 +1,1921 @@ +### DSS Interface Summary + +#### FlowEntranceRestfulApi + +> Base Info + +##### Interface Name：Workflow execution + +**Interface Path：**`POST` /api/rest_j/v1/dss/flow/entrance/execute + +> Request Param + +**Body:** + +````json +{ + "isReExecute":"" //must Added parameter, this parameter needs to be added when re-execution fails +} +```` + +> Return Parameter + +```json +{ + "status":"", //code status + "message":"", //info + "data":{ + "taskID":"", + "execID":"" + } +} +``` + + + +> Base Info + +##### Interface Name：Get workflow status + +**Interface Path：**`POST` /api/rest_j/v1/flow/entrance/{id}/status + +> Request Param + +**Body:** + +````json +{ + "id":"", //must + "taskID":"" +} +```` + +> Return Parameter + +````json +{ + "status":"", //code status + "message":"", //info + "data":{ + "taskID":"", + "execID":"" + } +} +```` + + + +> Base Info + +##### Interface Name：end workflow + +**Interface Path：**`GET` /api/rest_j/v1/dss/flow/entrance/{id}/killWorkflow + +> Request Param + +**Body:** + +````json +{ + "taskID":"" //must +} +```` + +> response parameter + +````json +{ + "massage":"", + "data":{ + "result":"", + "errorInfo":"" + } +} +```` + +--- + +#### FremeworkReleaseRestful + +> Base Info + +##### Interface Name：Get task progress + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/release/getReleaseStatus + +> Request Param + +**Body:** + +````json +{ + "dssLabel":"", //must dssLabel + "releaseTaskId":"" //must Query's release id +} +```` + +> Response Parameter + +````json +{ + "massage":"", + "data":{ + "status":"", + "errorMsg":"" + } +} +```` + + + +> Base Info + +##### Interface Name：Submit a release task + +**Interface Path：**`POST` /api/rest_j/v1/dss/framework/release/releaseOrchestrator + +> Request Param + +**Body:** + +```json +{ + "orchestratorId":"", //must Orchestration mode id + "orchestratorVersionId":"", //must Orchestration mode version id + "comment":"", //must Post a note + "dssLabel":"" //must dssLabel +} +``` + +> Response Parameter + +````json +{ + "massage":"", + "data":"" +} +```` + +--- + +#### ApiServiceTokenResfulApi + +> Base Info + +##### Interface Name：token query + +**Interface Path：**`GET` /api/rest_j/v1/dss/apiservice/tokenQuery + +> Request Param + +**Body:** + +```json +{ + "apiId":"", + "user":"", + "status":"", + "startDate":"", + "endDate":"", + "currentPage":1, + "pageSize":10 +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "queryList":[ + "id":"", + "apiId":"", + "apiVersionId":"", + "publisher":"", + "user":"", + "applyTime":"", + "duration":"", + "reason":"", + "ipWhitelist":"", + "status":"", + "caller":"", + "accessLimit":"", + "token":"", + "applySource":"" + ] + } +} +``` + + + +> Base Info + +##### Interface Name：Refresh the application form + +**Interface Path：**`GET` /api/rest_j/v1/dss/apiservice/approvalRefresh + +> Request Param + +**Body:** + +```json +{ + "approvalNo":"" +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "approvalStatus":"" + } +} +``` + +--- + +#### ContextServiceRestful + +> Base Info + +##### Interface Name：table query + +**Interface Path：**`POST`/api/rest_j/v1/dss/workflow/tables + +> Request Param + +**Body:** + +```json +{ + "contextID":"", //ID + "nodeName":"", //node name + "labels":{ + "route":"" + } +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "tables":[] + } +} +``` + + + +> Base Info + +##### Interface Name：column query + +**Interface Path：**`POST`/api/rest_j/v1/dss/workflow/columns + +> Request Param + +**Body:** + +```json +{ + "contextID":"", + "nodeName":"", + "contextKey":"", + "labels":{ + "route":"" + } +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "columns":[] + } +} +``` + +--- + +#### DSSDictionaryRestful + +> Base Info + +##### Interface Name：data dictionary - get by key + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/workspace/getDicList + +> Request Param + +**Body:** + +```json +{ + "workspaceId":"", //must workspace id + "dicKey":"", + "parentKey":"" //parent key +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "list":[{ + "id":0, + "workspaceId":0, //workspace id + "parentKey":"", //parent key + "dicName":"", //dictionary name + "dicNameEn":"", //dictionary english name + "dicKey":"", //key Equivalent to encoding, the space starts with w_, and the project is p_ + "dicValue":"", //the value corresponding to the key + "dicValueEn":"", //The value corresponding to the key (English) + "title":"", //title + "titleEn":"", //title(english) + "url":"", + "urlType":1, //url type: 0-internal system, 1-external system; default is internal + "icon":"", //icon + "orderNum":1, //order + "remark":"", + "createUser":"", + "createTime":"", + "updateUser":"", + "updateTime":"" + }] + } +} +``` + + + +> Base Info + +##### Interface Name：get data dictionary + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/workspace/getDicSecondList + +> Request Param + +**Body:** + +```json +{ + "workspaceId":"", //must + "dicKey":"", //key Equivalent to encoding, the space starts with w_, and the project is p_ + "parentKey":"" +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "list":[], + "mapList":{} + } +} +``` + +--- + +#### DSSFrameworkOrchestratorRestful + +> Base Info + +##### Interface Name：Equivalent to encoding, the space starts with w_, and the project is p_ + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/orchestrator/createOrchestrator + +> Request Param + +**Body:** + +```json +{ + "orchestratorName":"", //must Orchestration name + "orchestratorMode":"", //Orchestration patterns such as workflow, composition orchestration, etc. + "orchestratorWays":[], //must Orchestration way + "orchestratorLevel":"", //Orchestration Importance Levels + "dssLabels":[], + "uses":"", //Orchestration use + "description":"", //must + "projectName":"", + "workspaceName":"" +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "orchestratorId":1 + } +} +``` + + + +> Base Info + +##### Interface Name：Query all orchestration schemas + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/orchestrator/getAllOrchestrator + +> Request Param + +**Body:** + +```json +{ + "workspaceId":"", //must + "projectId":"", //must + "orchestratorMode":"", //orchestrator type, such as workflow, compositional orchestrator, etc. + "orchestratorLevel":"", //Orchestration Importance Levels + "dssLabels":{"route":""} //dssLabels is passed in through the front end, mainly used for current environment information +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "page":[{ + "id":1, //primary key ID + "workspaceId":"", //workspace id + "projectId":"", //project id + "orchestratorId":1, //Orchestration mode id (workflow, the orchestratorId returned by calling the orchestrator service) + "orchestratorVersionId":1, //Orchestration mode version id (workflow, the orchestratorVersionId returned by calling the orchestrator service) + "orchestratorName":"", //orchestrator Name + "orchestratorMode":"", //Orchestration mode, the value obtained is dic_key(parent_key=p_orchestratorment_mode) in dss_dictionary + "uses":"", + "description":"", + "createUser":"", + "createTime":"", + "updateUser":"", + "updateTime":"", + "orchestratorWays":[], + "orchestratorLevel":"", //Orchestration Importance Levels + "flowEditLockExist":false //Whether the workflow edit lock is exited + }] + } +} +``` + + + +> Base Info + +##### Interface Name：Modify the orchestrator mode + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/orchestrator/modifyOrchestrator + +> Request Param + +**Body:** + +```json +{ + "id":"", //must + "orchestratorName":"", //must orchestrator Name + "orchestratorMode":"", //Choreography type, such as workflow, compositional choreography, etc. + "orchestratorLevel":"", //Orchestration Importance Levels + "uses":"", //Orchestration use + "description":"", //must describe + "orchestratorWays":[], //must orchestrator Ways + "dssLabels":"" //Current environmental information +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "orchestratorId":1 + } +} +``` + +> Base Info + +##### Interface Name：delete orchestrator mode + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/orchestrator/deleteOrchestrator + +> Request Param + +**Body:** + +```json +{ + "id":"" //must +} +``` + +> Response Parameter + +```json +{ + "massage":"" +} +``` + + + +> Base Info + +##### Interface Name：Get a list of orchestration importance levels + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/orchestrator/orchestratorLevels + +> Request Param + +**Body:** + +```json +{} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "orchestratorLevels":["HIGH","MEDIUM","LOW"], + } +} +``` + +--- + +#### DSSMigrateRestful + +> Base Info + +##### Interface Name：Import old DSS project + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/release/importOldDSSProject + +> Request Param + +**Body:** + +```json +{ + "file":"" //must MultipartFile +} +``` + +> Response Parameter + +```json +{ + "massage":"" +} +``` + + + +> Base Info + +##### Interface Name：Import the workflow exported from the dss1.0 interface + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/release/importworkflow + +> Request Param + +**Body:** + +```json +{ + "workspaceName":"", //must workspace Name + "projectName":"", //Create without project name + "projectUser":"", // Project user, the interface will ensure that this user is added to the permission list + "flowName":"", + "file":"" //Import file, MultipartFile type +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "flowId":"", + "bmlVersion":"" + } +} +``` + + + +> Base Info + +##### Interface Name：Export SQL file + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/release/exportOrcSqlFile + +> Request Param + +**Body:** + +```json +{ + "outputFileName":"", //Default exportOrc + "charset":"utf-8", //encoding default utf-8 + "outputFileType":"zip", // output file type default zip + "projectName":"", //project Name + "orchestratorId":"", //orchestrator Id + "orcVersionId":"", //orchestrator version number ID + "addOrcVersion":false, //Whether to increase the version number + "labels":"" +} +``` + +> Response Parameter + +```json +{} +``` + + + +> Base Info + +##### Interface Name：Batch export all workflows under the project to the specified local directory + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/release/exportallflow + +> Request Param + +**Body:** + +```json +{ + "workspaceName":"", + "projectName":"", + "pathRoot":"" +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "count":0, + "location":"", + "serviceInstance":"" + } +} +``` + +--- + +#### DSSSideInfoRestful + +> Base Info + +##### Interface Name：get sidebar + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/workspace/getSideInfos + +> Request Param + +**Body:** + +```json +{ + "workspaceID":"" +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "presentations":[{ + "name":"", + "title":"", + "type":"", + "contents":[{ + "name":"", + "title":"", + "url":0, //url type: 0-internal system, 1-external system; default is internal + "icon":"" //icon is the icon representing the Content, if it is empty, there is no + }] + }] + } +} +``` + +--- + +#### DSSWorkspacePrivRestful + +> Base Info + +##### Interface Name：Get the workspace menu + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/exportOrcSqlFile + +> Request Param + +**Body:** + +```json +{ + "workspaceId":"" +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "workspaceMenuPrivs":{ + "workspaceId":1, + "menuPrivVOS":[{ + "id":"", + "name":"", + "menuPrivs":[{}] + }], + "DSSWorkspaceComponentPrivVO":[{ + "id":"", + "name":"", + "componentPrivs":[{}] + }], + "DSSWorkspaceRoleVO":[{ + "roleId":"", + "roleName":"", + "roleFrontName":"" + }] + } + } +} +``` + + + +> Base Info + +##### Interface Name：Get workspace settings + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/getWorkspaceHomepageSettings + +> Request Param + +**Body:** + +```json +{ + "workspaceId":"" //工作workspace id +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "homepageSettings":{ + "roleName":"", + "homepageName":"", + "homepageUrl":"", + "roleId":1, + "roleFrontName":"" + } + } +} +``` + + + +> Base Info + +##### Interface Name：Update role menu permissions + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/workspace/updateRoleMenuPriv + +> Request Param + +**Body:** + +```json +{ + "menuId":"", + "workspaceId":"", + "menuPrivs":"" //menu permissions +} +``` + +> Response Parameter + +```json +{ + "message":"" +} +``` + + + +> Base Info + +##### Interface Name：Update component permissions + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/workspace/updateRoleComponentPriv + +> Request Param + +**Body:** + +```json +{ + "componentId":"", + "workspaceId":"", + "componentPrivs":"" //component permissions +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "updateRoleComponentPriv":"" + } +} +``` + +--- + +#### DSSWorkspaceRestful + +> Base Info + +##### Interface Name：Create a workspace + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/workspace/createWorkspace + +> Request Param + +**Body:** + +```json +{ + "workspaceName":"", + "department":"", + "description":"", + "tags":"", + "productName":"" +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "workspaceId":1, + "workspaceName":"" + } +} +``` + + + +> Base Info + +##### Interface Name：Get department list + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/listDepartments + +> Request Param + +**Body:** + +```json +{} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "departments":[{ + "id":1, + "name":"", + "frontName":"" + }] + } +} +``` + + + +> Base Info + +##### Interface Name：Get a list of workspaces + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/getWorkspaces + +> Request Param + +**Body:** + +```json +{} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "workspaces":[{ + "id":1, + "name":"", + "tags":"", + "department":"", + "description":"", + "product":"" + }] + } +} +``` + + + +> Base Info + +##### Interface Name：Get user workspace + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/getWorkspaceHomePage + +> Request Param + +**Body:** + +```json +{ + "micro_module":"" //module name +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "workspaceHomePage":{ + "workspaceId":1, + "roleName":"", + "homePageUrl":"" + ] + } +} +``` + + + +> Base Info + +##### Interface Name：Get an overview of the workspace + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/getOverview + +> Request Param + +**Body:** + +```json +{ + "workspaceId":1 +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "overview":{ + "title":1, + "description":"", + "dssDescription":"", + "videos":[{ + "id":"", + "title":"", + "url":"" + }], + "demos":[{}] + ] + } +} +``` + + + +> Base Info + +##### Interface Name：Flush the workspace cache + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/refreshCache + +> Request Param + +**Body:** + +```json +{} +``` + +> Response Parameter + +```json +{ + "message":"" +} +``` + +--- + +#### DSSWorkspaceRoleRestful + +> Base Info + +##### Interface Name：Get all roles in the workspace + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/getWorkspaceRoles + +> Request Param + +**Body:** + +```json +{ + "workspaceId":"" +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "workspaceRoles":[{ + "roleId":1, + "roleName":"", + "roleFrontName":"" + }] + } +} +``` + + + +> Base Info + +##### Interface Name：Create a workspace role + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/workspace/addWorkspaceRole + +> Request Param + +**Body:** + +```json +{ + "workspaceId":1, + "roleName":"", + "menuIds":[], + "componentIds":[] +} +``` + +> Response Parameter + +```json +{ + "message":"" +} +``` + +--- + +#### DSSWorkspaceUserRestful + + + +> Base Info + +##### Interface Name：Create workspace user + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/getWorkspaceUsers + +> Request Param + +**Body:** + +```json +{ + "workspaceId":1, + "pageNow":1, //current page default 1 + "pageSize":20, //Quantity per page The default display is 20 + "department":"", + "userName":"", + "roleName":"" +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "roles":"", + "workspaceUsers":[{ + "id":"", + "name":"", + "roles":[], + "department":"", + "office":"", + "creator":"", + "joinTime":"" + }], + "total":100 + } +} +``` + + + +> Base Info + +##### Interface Name：Create a workspace for all users + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/getWorkspaceUsers + +> Request Param + +**Body:** + +```json +{} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "users":{ + "editUsers":[], + "accessUsers":[], + "releaseUsers":[] + } + } +} +``` + + + +> Base Info + +##### Interface Name：Whether the user exits the workspace + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/existUserInWorkspace + +> Request Param + +**Body:** + +```json +{ + "workspaceId":1, + "queryUserName":"" +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "existFlag":true + } +} +``` + + + +> Base Info + +##### Interface Name：Add workspace user + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/workspace/existUserInWorkspace + +> Request Param + +**Body:** + +```json +{ + "workspaceId":1, + "userName":"", + "roles":[], + "userId":"" +} +``` + +> Response Parameter + +```json +{ + "message":"" +} +``` + + + +> Base Info + +##### Interface Name：Update workspace user + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/workspace/updateWorkspaceUser + +> Request Param + +**Body:** + +```json +{ + "workspaceId":1, + "userName":"", + "roles":[], + "userId":"" +} +``` + +> Response Parameter + +```json +{ + "message":"" +} +``` + + + +> Base Info + +##### Interface Name：delete workspace user + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/workspace/updateWorkspaceUser + +> Request Param + +**Body:** + +```json +{ + "workspaceId":1, + "userName":"", + "roles":[], + "userId":"" +} +``` + +> Response Parameter + +```json +{ + "message":"" +} +``` + + + +> Base Info + +##### Interface Name：get all users + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/listAllUsers + +> Request Param + +**Body:** + +```json +{} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "users":{ + "id":"", + "username":"", + "department":"", + "office":"" + } + } +} +``` + + + +> Base Info + +##### Interface Name：Get workspace id based on username + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/workspace/getWorkspaceIdByUserName + +> Request Param + +**Body:** + +```json +{ + "userName":"" +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "userWorkspaceIds":"" + } +} +``` + +--- + +#### OrchestratorIERestful + +> Base Info + +##### Interface Name：Orchestrate file uploads + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/orchestrator/importOrchestratorFile + +> Request Param + +**Body:** + +```json +{ + "projectName":"", + "projectID":1, + "labels":"", //Environmental information + "files":"" //MultipartFile type, supports multiple files +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "importOrcId":1 + } +} +``` + + + +##### Interface Name：Arrangement file export + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/orchestrator/exportOrchestrator + +> Request Param + +**Body:** + +```json +{ + "outputFileName":"exportOrc", //output filename default exportOrc + "charset":"utf-8", //encoding default utf-8 + "outputFileType":"zip", //output file type default zip + "projectName":"", //Construction name + "orchestratorId":"", + "orcVersionId":"", + "addOrcVersion":"", + "labels":"" //Environmental information +} +``` + +> Response Parameter + +```json +{} +``` + +--- + +#### OrchestratorRestful + +> Base Info + +##### Interface Name：Roll back a workflow version + +**Interface Path：**`GET`/api/rest_j/v1/dss/framework/orchestrator/rollbackOrchestrator + +> Request Param + +**Body:** + +```json +{ + "orchestratorId":1, + "version":"", + "projectId":1, + "projectName":"", + "labels":"" //Environmental information +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "newVersion":"" + } +} +``` + + + +> Base Info + +##### Interface Name：Open Orchestration Mode + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/orchestrator/openOrchestrator + +> Request Param + +**Body:** + +```json +{ + "workspaceName":1, + "orchestratorId":"", + "labels":{ + "route":"" + }, +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "OrchestratorOpenUrl":"", + "OrchestratorVo":"" + } +} +``` + + + +> Base Info + +##### Interface Name：Get all version numbers in orchestration mode + +**Interface Path：**`POST`/api/rest_j/v1/dss/framework/orchestrator/getVersionByOrchestratorId + +> Request Param + +**Body:** + +```json +{ + "orchestratorId":"" //must +} +``` + +> Response Parameter + +```json +{ + "message":"", + "data":{ + "list":[] + } +} +``` + +--- + +#### FlowRestfulApi + +> Base Info + +##### Interface Name：Add subflow node + +**Interface Path：**`POST`/api/rest_j/v1/dss/workflow/addFlow + +> Request Param + +**Body:** + +```json +{ + "name":"", + "workspaceName":"", + "projectName":"", + "version":"", + "description":"", + "parentFlowID":"", + "uses":"", + "labels":{ + "route":"" + }, + "userName":"" +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "flow":{ + "id":1, + "name":"", + "state":true, //Represents published and unpublished + "source":"", + "description":"", + "createTime":"", + "creator":"", + "isRootFlow":true, + "rank":1, + "projectID":1, + "linkedAppConnNames":"", + "dssLabels":"", + "flowEditLock":"", //Workflow Edit Lock + "hasSaved":true, //ture indicates that the workflow has never been saved and is ignored when publishing + "uses":"", + "children":[], + "flowType":"", + "resourceId":"", + "bmlVersion":"" + } + } +} +``` + + + +> Base Info + +##### Interface Name：Publish workflow + +**Interface Path：**`POST`/api/rest_j/v1/dss/workflow/publishWorkflow + +> Request Param + +**Body:** + +```json +{ + "workflowId":"", + "comment":"", + "labels":{ + "route":"" + }, + "dssLabel":"", + "orchestratorId":1, + "orchestratorVersionId":1 +} +``` + +> Response Parameter + +```json +{ + "massage":"" +} +``` + + + +> Base Info + +##### Interface Name：Get publish task status + +**Interface Path：**`POST`/api/rest_j/v1/dss/workflow/getReleaseStatus + +> Request Param + +**Body:** + +```json +{ + "releaseTaskId":1 //must +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "status":"" + } +} +``` + + + +> Base Info + +##### Interface Name：Update the Base Info of the workflow, excluding updating Json, BML version, etc. + +**Interface Path：**`GET`/api/rest_j/v1/dss/workflow/get + +> Request Param + +**Body:** + +```json +{ + "flowID":1, //must + "isNotHaveLock":true +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "flow":{ + "id":1, + "name":"", + "state":true, //Represents published and unpublished + "source":"", + "description":"", + "createTime":"", + "creator":"", + "isRootFlow":true, + "rank":1, + "projectID":1, + "linkedAppConnNames":"", + "dssLabels":"", + "flowEditLock":"", //Workflow Edit Lock + "hasSaved":true, //ture indicates that the workflow has never been saved and is ignored when publishing + "uses":"", + "children":[], + "flowType":"", + "resourceId":"", + "bmlVersion":"" + } + } +} +``` + + + +> Base Info + +##### Interface Name：Delete workflow + +**Interface Path：**`POST`/api/rest_j/v1/dss/workflow/deleteFlow + +> Request Param + +**Body:** + +```json +{ + "id":1, //must + "sure":true, + "labels":{ + "route":"" + } +} +``` + +> Response Parameter + +```json +{ + "massage":"" +} +``` + + + +> Base Info + +##### Interface Name：Workflow save interface, if the workflow Json content changes, it will update the Json content of the workflow + +**Interface Path：**`POST`/api/rest_j/v1/dss/workflow/saveFlow + +> Request Param + +**Body:** + +```json +{ + "id":1, //must + "json":"", + "workspaceName":"", + "projectName":"", + "flowEditLock":"", + "isNotHaveLock":true, + "labels":{ + "route":"" + } +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "flowVersion":"", + "flowEditLock":"" + } +} +``` + + + +> Base Info + +##### Interface Name：Workflow edit lock update interface + +**Interface Path：**`GET`/api/rest_j/v1/dss/workflow/updateFlowEditLock + +> Request Param + +**Body:** + +```json +{ + "flowEditLock":"" +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "flowEditLock":"" + } +} +``` + + + +> Base Info + +##### Interface Name：Get extra toolbars + +**Interface Path：**`POST`/api/rest_j/v1/dss/workflow/getExtraToolBars + +> Request Param + +**Body:** + +```json +{ + "projectId":1, + "workflowId":"", + "labels":{ + "route":"" + } +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "extraBars":[{ + "name":"", + "url":"", + "icon":"" + }] + } +} +``` + +--- + +#### NodeRestfulApi + +> Base Info + +##### Interface Name：HU + +**Interface Path：**`GET`/api/rest_j/v1/dss/workflow/listNodeType + +> Request Param + +**Body:** + +```json +{ + "projectId":1, + "workflowId":"", + "labels":{ + "route":"" + } +} +``` + +> Response Parameter + +```json +{ + "massage":"", + "data":{ + "extraBars":[{ + "name":"", + "url":"", + "icon":"" + }] + } +} +``` + + + + + + + + + + + + + + + + + + + diff --git a/en_US/Using_Document/Scriptis/Scriptis_User_Tests1_Scala.md b/en_US/Using_Document/Scriptis/Scriptis_User_Tests1_Scala.md new file mode 100644 index 0000000..dc4f843 --- /dev/null +++ b/en_US/Using_Document/Scriptis/Scriptis_User_Tests1_Scala.md @@ -0,0 +1,82 @@ +# DSS User Test Example 1：Scala + +The purpose of DSS user test samples is to provide a set of test samples for new users of the platform to familiarize themselves with the common operations of DSS and to verify the correctness of the DSS platform + +![home](../../Images/Using_Document/Scriptis/home.png) + +## 1.1 Spark Core（entry function sc） + +In Scriptis, SparkContext is already registered for you by default, so just use sc directly: + +### 1.1.1 Single Value operator (Map operator as an example) + +```scala +val rddMap = sc.makeRDD(Array((1,"a"),(1,"d"),(2,"b"),(3,"c")),4) +val res = rddMap.mapValues(data=>{data+"||||"}) +res.collect().foreach(data=>println(data._1+","+data._2)) +``` + +### 1.1.2 Double Value operator (union operator as an example) + +```scala +val rdd1 = sc.makeRDD(1 to 5) +val rdd2 = sc.makeRDD(6 to 10) +val rddCustom = rdd1.union(rdd2) +rddCustom.collect().foreach(println) +``` + +### 1.1.3 K-V operator (reduceByKey operator is an example) + +```scala +val rdd1 = sc.makeRDD(List(("female",1),("male",2),("female",3),("male",4))) +val rdd2 = rdd1.reduceByKey((x,y)=>x+y) +rdd2.collect().foreach(println) +``` + +### 1.1.4 Execute the operator (the above collect operator is an example) + +### 1.1.5 Read files from hdfs and do simple execution + +```scala +case class Person(name:String,age:String) +val file = sc.textFile("/test.txt") +val person = file.map(line=>{ + val values=line.split(",") + + Person(values(0),values(1)) +}) +val df = person.toDF() +df.select($"name").show() +``` + + + +## 1.2 UDF function test + +### 1.2.1 function definition + + + +```scala +def ScalaUDF3(str: String): String = "hello, " + str + "this is a third attempt" +``` + +### 1.2.2 register function + +function-》personal function-》Right click to add spark function=》The registration method is the same as the regular spark development + +![img](../../Images/Using_Document/Scriptis/udf1.png) + +## 1.3 UDAF function test + +### 1.3.1 Jar package upload + +​ Develop a udaf function for averaging on idea, package it into a jar (wordcount) package, and upload the dss jar folder. + +![img](../../Images/Using_Document/Scriptis/udf2.png) + +### 1.3.2 register function + +function-》personal function-》Right click to add spark function=》The registration method is the same as the regular spark development + + ![img](../../Images/Using_Document/Scriptis/udf-3.png) \ No newline at end of file diff --git a/en_US/Using_Document/Scriptis/Scriptis_User_Tests2_Hive.md b/en_US/Using_Document/Scriptis/Scriptis_User_Tests2_Hive.md new file mode 100644 index 0000000..00e188e --- /dev/null +++ b/en_US/Using_Document/Scriptis/Scriptis_User_Tests2_Hive.md @@ -0,0 +1,148 @@ +# DSS User Test Example 2：Hive + +The purpose of DSS user test samples is to provide a set of test samples for new users of the platform to familiarize themselves with the common operations of DSS and to verify the correctness of the DSS platform + +![home](../../Images/Using_Document/Scriptis/home.png) + +## 2.1 Data warehouse building table + +Enter the "Database" page, click "+", enter the table information, table structure and partition information in turn to create a database table: + +![hive1](../../Images/Using_Document/Scriptis/hive1.png) + +![img](../../Images/Using_Document/Scriptis/hive2.png) + +Through the above process, create the department table dept, the employee table emp and the partitioned employee table emp_partition respectively. The table creation statement is as follows: + +```iso92-sql +create external table if not exists default.dept( + deptno int, + dname string, + loc int +) +row format delimited fields terminated by '\t'; + +create external table if not exists default.emp( + empno int, + ename string, + job string, + mgr int, + hiredate string, + sal double, + comm double, + deptno int +) +row format delimited fields terminated by '\t'; + +create table if not exists emp_partition( + empno int, + ename string, + job string, + mgr int, + hiredate string, + sal double, + comm double, + deptno int +) +partitioned by (month string) +row format delimited fields terminated by '\t'; +``` + +**Import Data** + +Currently, data needs to be manually imported in batches through the background, and data can be inserted from the page through the insert method + +```sql +load data local inpath 'dept.txt' into table default.dept; +load data local inpath 'emp.txt' into table default.emp; +load data local inpath 'emp1.txt' into table default.emp_partition; +load data local inpath 'emp2.txt' into table default.emp_partition; +load data local inpath 'emp2.txt' into table default.emp_partition; +``` + +Other data is imported according to the above statement. The path of the sample data file is: `examples\ch3` + +## 2.2 Basic SQL syntax test + +### 2.2.1 Simple query + +```sql +select * from dept; +``` + +### 2.2.2 Join connection + +```sql +select * from emp +left join dept +on emp.deptno = dept.deptno; +``` + +### 2.2.3 Aggregate function + +```sql +select dept.dname, avg(sal) as avg_salary +from emp left join dept +on emp.deptno = dept.deptno +group by dept.dname; +``` + +### 2.2.4 built-in function + +```sql +select ename, job,sal, +rank() over(partition by job order by sal desc) sal_rank +from emp; +``` + +### 2.2.5 Partitioned table simple query + +```sql +show partitions emp_partition; +select * from emp_partition where month='202001'; +``` + +### 2.2.6 Partitioned table union query + +```sql +select * from emp_partition where month='202001' +union +select * from emp_partition where month='202002' +union +select * from emp_partition where month='202003' +``` + +## 2.3 UDF function test + +### 2.3.1 Jar package upload + +After entering the Scriptis page, right-click the directory path to upload the jar package: + +![img](../../Images/Using_Document/Scriptis/hive3.png) + +The test example jar is in `examples\ch3\rename.jar` + +### 4.3.2 custom function + +Enter the "UDF Function" option (such as 1), right-click the "Personal Function" directory, and select "Add Function": + +![hive4](../../Images/Using_Document/Scriptis/hive4.png) + +Enter the function name, select the jar package, and fill in the registration format and input and output format to create a function: + + ![hive5](../../Images/Using_Document/Scriptis/hive5.png) + +![hive6](../../Images/Using_Document/Scriptis/hive-6.png) + +The obtained function is as follows: + +![img](../../Images/Using_Document/Scriptis/hive7.png) + +### 4.3.3 SQL query with custom function + +After completing the function registration, you can enter the workspace page to create a .hql file to use the function: + +```sql +select deptno,ename, rename(ename) as new_name +from emp; +``` diff --git a/en_US/Using_Document/Scriptis/Scriptis_User_Tests3_SparkSQL.md b/en_US/Using_Document/Scriptis/Scriptis_User_Tests3_SparkSQL.md new file mode 100644 index 0000000..40dd1d0 --- /dev/null +++ b/en_US/Using_Document/Scriptis/Scriptis_User_Tests3_SparkSQL.md @@ -0,0 +1,61 @@ +# DSS User Test Example 3：SparkSQL + +The purpose of DSS user test samples is to provide a set of test samples for new users of the platform to familiarize themselves with the common operations of DSS and to verify the correctness of the DSS platform + +![home](../../Images/Using_Document/Scriptis/home.png) + +## 3.1 RDD and DataFrame conversion + +### 3.1.1 RDD to DataFrame + +```scala +case class MyList(id:Int) + +val lis = List(1,2,3,4) + +val listRdd = sc.makeRDD(lis) +import spark.implicits._ +val df = listRdd.map(value => MyList(value)).toDF() + +df.show() +``` + +### 3.1.2 DataFrame to RDD + +```scala +case class MyList(id:Int) + +val lis = List(1,2,3,4) +val listRdd = sc.makeRDD(lis) +import spark.implicits._ +val df = listRdd.map(value => MyList(value)).toDF() +println("------------------") + +val dfToRdd = df.rdd + +dfToRdd.collect().foreach(print(_)) +``` + +## 3.2 DSL syntax style implementation + +```scala +val df = df1.union(df2) +val dfSelect = df.select($"department") +dfSelect.show() +``` + +## 3.3 SQL syntax style implementation (entry function sqlContext) + +```scala +val df = df1.union(df2) + +df.createOrReplaceTempView("dfTable") +val innerSql = """ + SELECT department + FROM dfTable + """ +val sqlDF = sqlContext.sql(innerSql) +sqlDF.show() +``` + +​ \ No newline at end of file diff --git a/en_US/Using_Document/User_login_authentication_system.md b/en_US/Using_Document/User_login_authentication_system.md new file mode 100644 index 0000000..11d2c6f --- /dev/null +++ b/en_US/Using_Document/User_login_authentication_system.md @@ -0,0 +1,129 @@ +# DataSphere Studio User login authentication system + +> DSS only provides an administrator account by default, and user login authentication relies on Linkis' LDAP user login authentication system. +> This article will introduce the user login authentication methods currently supported by Linkis in detail. + +## 1. Basic introduction + +1. The user name of the DSS super administrator is the deployment user name. If the deployment user is hadoop, the user name and password of the administrator are hadoop/hadoop. The specific user can be found in [DSS single-machine deployment document](../Installation/Deployment/DSS&Linkis One-click deployment document stand-alone version.md) view. + +2. The user login authentication of DSS relies on the user login authentication system of Linkis. In addition to administrators, Linkis also supports the following user login authentication methods: + + - Token login method + - Proxy user mode + - Access to SSO single sign-on + +## 2. Token login method + +This method is used for third-party systems to access Linkis and DSS. + +When the third-party system calls the Linkis and DSS background interfaces, the login can be skipped directly through the token mode. + +Specify the following parameters in `linkis/linkis-gateway/conf/linkis.properties`: + +```properties + # Open token mode + wds.linkis.gateway.conf.enable.token.auth=true + # Specify the token configuration file + wds.linkis.gateway.conf.token.auth.config=token.properties +``` + +In the `linkis/linkis-gateway/conf` directory, create a `token.properties` file with the following content: + +```properties + # The format is as follows： + ${TOKEN_NAME}=${USER1},${USER2} + # E.g： + AZKABAN=* +``` + +TOKEN_NAME refers to the tokenId assigned to the third-party system, and the following value is the user who can skip login. If all requests from the system are fully trusted, it can be directly equal to *, indicating full authorization. + +When a third-party system requests DSS and Linkis, write the following two parameters in the request header or cookie: + +```json + { + "Token-Code": "${TOKEN_NAME}", + "Token-User": "${USER}" + } +``` + +## 3. Proxy user mode + +This method allows the login user to be different from the user who actually uses DSS. The main function is to control that the user must be a real-name user when logging in, but when the big data platform is actually used, it is a non-real-name user. + +Specify the following parameters in `linkis/linkis-gateway/conf/linkis.properties`: + +```properties + # Turn on proxy mode + wds.linkis.gateway.conf.enable.proxy.user=true + # Specify proxy configuration file + wds.linkis.gateway.conf.proxy.user.config=proxy.properties +``` + +In the `linkis/linkis-gateway/conf` directory, create a `proxy.properties` file with the following content: + +```properties + # The format is as follows： + ${LOGIN_USER}=${PROXY_USER} + # E.g： + enjoyyin=hadoop +``` + +If the existing proxy mode does not meet your needs, you can also modify it manually:`com.webank.wedatasphere.linkis.gateway.security.ProxyUserUtils`。 + +## 4. Access SSO single sign-on + +Accessing your company's SSO single sign-on system is a bit more complicated. + +First, you need to enable SSO single point authentication function, please specify the following parameters in `linkis/linkis-gateway/conf/linkis.properties`: + +```properties + wds.linkis.gateway.conf.enable.sso=true +``` + +Then you need to implement the SSOInterceptor interface: + +```scala +trait SSOInterceptor { + + /** + * If the SSO single sign-on function is enabled, after the front-end jumps to the SSO login page and the login is successful, it will jump back to the DSS home page again. At this time, the DSS front-end requests the gateway again. + * The gateway will call this method to obtain the user who has logged in with SSO, and then write the user to the cookie to ensure that subsequent requests can be released directly. + * You need to implement this method to return the username through Request. + * @param gatewayContext + * @return + */ + def getUser(gatewayContext: GatewayContext): String + + /** + * Through the DSS homepage URL, the user generates a redirectable SSO login page URL. + * Requirement: You need to bring requestUrl, so that you can jump back after the SSO login is successful + * @param requestUrl DSS homepage URL + * @return 例如：https://${sso_host}:${sso_port}/cas/login?redirectUrl=${requestUrl} + */ + def redirectTo(requestUrl: URI): String + + /** + * When the user logs out, the gateway will call this interface to ensure that after the gateway clears the cookie, the SSO single sign-on will also clear the login information + * @param gatewayContext + */ + def logout(gatewayContext: GatewayContext): Unit + +} +``` + +Type your SSO implementation class into a jar package and put it in the `linkis/linkis-gateway/lib` directory. + +Linkis provides two ways to load your SSO implementation class: + +Declaring the SSO implementation class as a Spring bean requires you to simply annotate the class name with `@Component`. + +Specify the following parameters in `linkis/linkis-gateway/conf/linkis.properties`: + +```properties + #Please specify as your SSO implementation class + wds.linkis.gateway.conf.sso.interceptor=com.webank.wedatasphere.linkis.gateway.security.sso.SSOInterceptor +``` + +Restart linkis-gateway, SSO single sign-on will take effect. \ No newline at end of file diff --git a/en_US/Using_Document/Workspace_User_Manual.md b/en_US/Using_Document/Workspace_User_Manual.md new file mode 100644 index 0000000..bd95483 --- /dev/null +++ b/en_US/Using_Document/Workspace_User_Manual.md @@ -0,0 +1,29 @@ +# 1.Introduction to Workspace Role Permissions +       In a project-oriented workspace, members of any department can be added; in a department-oriented workspace, only members of this department can be added.
+       **management desk：** +- The default role of a workspace creator is owner, who can delegate the management of the workspace to one or more administrators.
+- Only the administrator of the workspace can enter the [Administrator] module to manage the basic information and permission information of the workspace accordingly.
+- The default roles of the workspace include owner, administrator, development user, operation and maintenance user, analysis user, operation user, data service user, and guest.
+ - Owner: has the highest authority, has the operation authority of all functions in the workspace, and can delete the workspace; + - Administrator: has the operation authority of all functions in the workspace; + - Development users: have permissions for data access, data quality, application development, test environment task release, production environment task submission, etc.; + - Operation and maintenance users: have permissions for data quality, resource allocation, test environment task release, production task approval and release, etc.; + - Analysis users: have data access, data analysis and other permissions; + - Operational users: have data analysis and other permissions; + - Data service users: have data access, data service and other rights; + - Guest: has read-only access to all modules in the workspace, cannot edit or execute; + +# 2.Workspace page introduction +       The workspace page contains four parts: **Project List**, **Application Development Process**, **Menu**, and **FAQ**.
+- Project list: After users enter dss, they can choose different workspaces to create their own projects. Selecting different workspaces will display the created project list. +![](../Images/Using_Document/workspace/ws_img1.png) +- Application development process: It includes **requirements** (unavailable), **design** (unavailable), **development**, **debugging**, **production**, click on the corresponding function button to jump to the corresponding function or view the corresponding example +![](../Images/Using_Document/workspace/ws_img2.png) +- Menu: Click UDF management, you can enter the linkis management console for resource management, you can add functions or add UDFs +![](../Images/Using_Document/workspace/ws_img3.png) + +# 3.Create project +       In the project space list, click Create Project, you can configure and add new projects according to your own needs +![](../Images/Using_Document/workspace/ws_img5.png) +       The basic information of the project can be modified by clicking the management button of the project, only the administrator can delete the project +![](../Images/Using_Document/workspace/ws_img6.png) \ No newline at end of file diff --git "a/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DSS&Linkis\344\270\200\351\224\256\351\203\250\347\275\262\346\226\207\346\241\243\345\215\225\346\234\272\347\211\210.md" "b/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DSS&Linkis\344\270\200\351\224\256\351\203\250\347\275\262\346\226\207\346\241\243\345\215\225\346\234\272\347\211\210.md" index e672fab..1387219 100644 --- "a/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DSS&Linkis\344\270\200\351\224\256\351\203\250\347\275\262\346\226\207\346\241\243\345\215\225\346\234\272\347\211\210.md" +++ "b/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DSS&Linkis\344\270\200\351\224\256\351\203\250\347\275\262\346\226\207\346\241\243\345\215\225\346\234\272\347\211\210.md" @@ -393,7 +393,7 @@ HIVE_PASSWORD=xxx |-----------------|----------------|----------------------------------------|-------------------| | Schedulis | Schedulis0.7.0 | [Schedulis部署](https://github.com/WeBankFinTech/Schedulis/blob/master/docs/schedulis_deploy_cn.md) | [Schedulis AppConn安装](SchedulisAppConn插件安装文档.md)| | Visualis | Visualis1.0.0 | [Visualis部署](https://github.com/WeBankFinTech/Visualis/blob/master/visualis_docs/zh_CN/Visualis_deploy_doc_cn.md) |[Visualis AppConn安装](https://github.com/WeBankFinTech/Visualis/blob/master/visualis_docs/zh_CN/Visualis_appconn_install_cn.md)| - | Exchangis | Exchangis1.0.0 | [Exchangis部署](https://github.com/WeDataSphere/Exchangis/blob/master/docs/zh_CN/ch1/exchangis_deploy_cn.md) | [Exchangis AppConn安装](https://github.com/WeBankFinTech/Exchangis/blob/master/docs/zh_CN/ch1/exchangis_appconn_deploy_cn.md) | + | Exchangis | Exchangis1.0.0 | [Exchangis部署](https://github.com/WeBankFinTech/Exchangis/blob/master/docs/zh_CN/ch1/exchangis_deploy_cn.md) | [Exchangis AppConn安装](https://github.com/WeBankFinTech/Exchangis/blob/master/docs/zh_CN/ch1/exchangis_appconn_deploy_cn.md) | | Qualitis |Qualitis0.9.2 | [Qualitis部署](https://github.com/WeBankFinTech/Qualitis/blob/master/docs/zh_CN/ch1/%E5%BF%AB%E9%80%9F%E6%90%AD%E5%BB%BA%E6%89%8B%E5%86%8C%E2%80%94%E2%80%94%E5%8D%95%E6%9C%BA%E7%89%88.md) |[Qualitis AppConn安装](https://github.com/WeBankFinTech/Qualitis/blob/master/docs/zh_CN/ch1/%E6%8E%A5%E5%85%A5%E5%B7%A5%E4%BD%9C%E6%B5%81%E6%8C%87%E5%8D%97.md) | | Prophecis | Prophecis0.3.2 | [Prophecis部署](https://github.com/WeBankFinTech/Prophecis/blob/master/docs/zh_CN/QuickStartGuide.md) | [Prophecis AppConn安装](https://github.com/WeBankFinTech/Prophecis/blob/master/docs/zh_CN/Deployment_Documents/Prophecis%20Appconn%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) | | Streamis | Streamis0.2.0 | [Streamis部署](https://github.com/WeBankFinTech/Streamis/blob/main/docs/zh_CN/0.2.0/Streamis%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) | [Streamis AppConn安装](https://github.com/WeBankFinTech/Streamis/blob/main/docs/zh_CN/0.2.0/development/StreamisAppConn%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) | diff --git "a/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DSS1.0.1\345\210\2601.1.0\345\215\207\347\272\247\346\226\207\346\241\243.md" "b/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DSS1.0.1\345\210\2601.1.0\345\215\207\347\272\247\346\226\207\346\241\243.md" index 38735f4..d1d02c4 100644 --- "a/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DSS1.0.1\345\210\2601.1.0\345\215\207\347\272\247\346\226\207\346\241\243.md" +++ "b/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DSS1.0.1\345\210\2601.1.0\345\215\207\347\272\247\346\226\207\346\241\243.md" @@ -1,6 +1,6 @@ # DataSphere Studio 1.0.1 升级到 1.1.0 使用文档 -###升级步骤主要分为： +### 升级步骤主要分为： - 服务停止 - 执行数据库升级脚本 - dss部署目录替换为新版本包 @@ -95,6 +95,16 @@ wds.dss.appconn.checker.development.ignore.list=workflow,sendemail wds.dss.appconn.checker.project.ignore.list= ``` +并替换为新版本的restful、mybatis配置： +```properties +##restful +wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.dss.framework.workspace.restful,com.webank.wedatasphere.dss.framework.project.restful,com.webank.wedatasphere.dss.framework.release.restful,com.webank.wedatasphere.dss.framework.appconn.restful,com.webank.wedatasphere.dss.framework.admin.restful +##mybatis +wds.linkis.server.mybatis.mapperLocations=classpath*:com/webank/wedatasphere/dss/framework/workspace/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/application/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/framework/project/dao/impl/*Mapper.xml,classpath*:com/webank/wedatasphere/dss/framework/appconn/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/framework/release/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/framework/admin/xml/impl/*.xml +wds.linkis.server.mybatis.typeAliasesPackage=com.webank.wedatasphere.dss.application.entity,com.webank.wedatasphere.dss.common.entity,com.webank.wedatasphere.dss.framework.workspace.bean,com.webank.wedatasphere.dss.framework.project.entity,com.webank.wedatasphere.dss.framework.appconn.entity,com.webank.wedatasphere.dss.framework.release.entity,com.webank.wedatasphere.dss.framework.admin.pojo.entity +wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.dss.framework.workspace.dao,com.webank.wedatasphere.dss.application.dao,com.webank.wedatasphere.dss.framework.project.dao,com.webank.wedatasphere.dss.framework.appconn.dao,com.webank.wedatasphere.dss.framework.release.dao,com.webank.wedatasphere.dss.framework.admin.xml +``` + #### 5. 服务启动 OK，到现在可以启动dss新版本的服务了，在**dss部署目录下**执行命令启动所有服务： diff --git "a/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DolphinScheduler\346\217\222\344\273\266\345\256\211\350\243\205\346\226\207\346\241\243.md" "b/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DolphinScheduler\346\217\222\344\273\266\345\256\211\350\243\205\346\226\207\346\241\243.md" index c4321d3..2b3b0de 100644 --- "a/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DolphinScheduler\346\217\222\344\273\266\345\256\211\350\243\205\346\226\207\346\241\243.md" +++ "b/zh_CN/\345\256\211\350\243\205\351\203\250\347\275\262/DolphinScheduler\346\217\222\344\273\266\345\256\211\350\243\205\346\226\207\346\241\243.md" @@ -26,6 +26,7 @@ ## 3. 配置和部署 +### 3.1 appconn配置与安装 - 将 `dolphinscheduler-appconn.zip` 插件安装包，放置到如下目录并进行解压。 ```shell script @@ -33,9 +34,7 @@ unzip dolphinscheduler-appconn.zip ``` -- 配置参数 - -请按需修改 `appconn.properties` 的配置参数。 +- 配置参数，请按需修改 `appconn.properties` 的配置参数。 ```shell script cd ${DSS_HOME}/dss/dss-appconns/dolphinscheduler @@ -51,7 +50,7 @@ wds.dss.appconn.ds.admin.token= # 【请参考】目前只适配了 DolphinScheduler 1.3.X. wds.dss.appconn.ds.version=1.3.9 -# 用于配置 dss-dolphinscheduler-client 的 home 路径，可为具体路径，具体请参考第 4 大步骤 +# 用于配置 dss-dolphinscheduler-client 的 home 路径，可为具体路径，具体请参考 4.2 小节 wds.dss.appconn.ds.client.home=${DSS_DOLPHINSCHEDULER_CLIENT_HOME} # this property is used to add url prefix, if you add a proxy for dolphinscheduler url. @@ -68,13 +67,14 @@ sh install-appconn.sh # 该脚本为交互式的安装方案，您只需要按照指示，输入字符串 dolphinscheduler 以及 dolphinscheduler 服务的 ip 和端口，即可以完成安装 ``` -请注意：dolphinscheduler 的 ip 不要输入 `localhost` 或 `127.0.0.1`，请输入真实 IP。 +**请注意：dolphinscheduler 的 ip 不要输入 `localhost` 或 `127.0.0.1`，请输入真实 IP。** -#### 3.1.1 - 放入 dss-dolphinscheduler-token.jar 到 dss-framework-project 的 lib 下 +### 3.2 修改jar包 +#### 3.2.1 将 dss-dolphinscheduler-token.jar 放入到 dss-framework-project 的 lib 下 这个 Jar 包的作用是提供 `/api/rest_j/v1/dss/framework/project/ds/token` 接口，用于免密请求 DolphinScheduler 的接口。 -Jar 包获取方式：DSS 编译后从 `plugins/dolphinscheduler` 目录中可以获取： +Jar 包获取方式：DSS 编译后从 `plugins/dolphinscheduler` 目录中可以获取或 [点我下载](https://osp-1257653870.cos.ap-guangzhou.myqcloud.com/WeDatasphere/DolphinScheduler/dss-dolphinscheduler-token-1.1.0.jar) ![img_9.png](../Images/安装部署/DolphinschedulerAppConn部署/img_9.png) @@ -84,11 +84,11 @@ Jar 包获取方式：DSS 编译后从 `plugins/dolphinscheduler` 目录中可 sh sbin/dss-daemon.sh restart project-server ``` -#### 3.1.2 放入 dolphinscheduler-prod-metrics.jar +#### 3.2.2 将 dolphinscheduler-prod-metrics.jar 放入到 DolphinScheduler 的 lib 目录 这一步是将 DolphinScheduler 的自定义接口实现 Jar 包添加到 DolphinScheduler 服务的 lib 目录，并重启 DolphinScheduler 服务使之生效。 -Jar获取方式：从 DSS 编译后的 plugins 目录下有 dolphinscheduler 相关插件包，如图： +Jar获取方式：从 DSS 编译后的 plugins 目录下有 dolphinscheduler 相关插件包或 [点我下载](https://osp-1257653870.cos.ap-guangzhou.myqcloud.com/WeDatasphere/DolphinScheduler/dolphinscheduler-prod-metrics-1.1.0.jar) ![img_6.png](../Images/安装部署/DolphinschedulerAppConn部署/img_6.png) @@ -104,11 +104,9 @@ sh bin/start-all.sh ``` -#### 3.2 修改 DSS 的 nginx 配置，加入 /dolphinscheduler 路径的请求匹配规则。 - -这一步是由于运维中心页面的前端，会直接调用 DolphinScheduler 服务的接口请求数据（`/dolphinscheduler` URI 路径前缀）， +### 3.3 修改 DSS 的 nginx 配置，加入 /dolphinscheduler 路径的请求匹配规则。 -所以需要将请求转发到 DolphinScheduler 服务。 +这一步是由于运维中心页面的前端，会直接调用 DolphinScheduler 服务的接口请求数据（`/dolphinscheduler` URI 路径前缀），所以需要将请求转发到 DolphinScheduler 服务。 ```shell script vim /etc/nginx/conf.d/dss.conf @@ -129,7 +127,7 @@ location /dolphinscheduler { sudo nginx -s reload ``` -#### 3.3. 配置 前往调度中心 的 url +### 3.4 配置 前往调度中心 的 url 修改 `${DSS_HOME}/conf/dss-workflow-server.properties` 配置： @@ -164,36 +162,35 @@ mvn clean install ### 4.2 安装部署 -请先在 `/home/${USER}/.bash_rc` 配置环境变量 `DSS_DOLPHINSCHEDULER_CLIENT_HOME`（如果您在 `appconn.properties` 中指定的是绝对路径而不是该环境变量，则该环境变量也可以不用配置）。 - -将 `DSS_DOLPHINSCHEDULER_CLIENT_HOME` 配置为实际的 `dss-dolphinscheduler-client` 根路径。 - -在 `dss-dolphinscheduler-client` 的根路径进行解压安装，如下： +- 将`dss-dolphinscheduler-client` 插件安装包置入任意目录下，建议放到与`dolphinscheduler`的同级目录xx下，对安装包直接进行解压即完成`dss-dolphinscheduler-client`的安装 ```shell script cd ${DSS_DOLPHINSCHEDULER_CLIENT_HOME} unzip dss-dolphinscheduler-client.zip ``` -解压即可完成 `dss-dolphinscheduler-client` 的安装。 -接着需要修改dss-dolphinscheduler-client中的配置文件conf/linkis.properties的linkis网关的ip和端口： +- 关于 3.1 小节`appconn.properties`中`wds.dss.appconn.ds.client.home`的配置，用户可以配置为`dss-dolphinscheduler-client` 插件的绝对路径，即`xx/dss-dolphinscheduler-client`；抑或是在`/home/${USER}/.bash_rc` 配置环境变量 `DSS_DOLPHINSCHEDULER_CLIENT_HOME`，将其配置为 `dss-dolphinscheduler-client` 实际的根路径 + + +- 接着需要修改`dss-dolphinscheduler-client`中的配置文件`conf/linkis.properties`的linkis网关的ip和端口： ![img_5.png](../Images/安装部署/DolphinschedulerAppConn部署/img_5.png) -## 4. DolphinSchedulerAppConn 的使用 +## 5. DolphinSchedulerAppConn 的使用 -### 4.1 免密跳转 +### 5.1 免密跳转 进入 DSS 的工作空间首页，然后在顶部菜单栏点击跳转到 DolphinScheduler。 ![DolphinScheduler免密跳转](../Images/安装部署/DolphinschedulerAppConn部署/img_13.png) ![img_14](../Images/安装部署/DolphinschedulerAppConn部署/img_14.png) -### 4.2 发布 DSS 工作流到 DolphinScheduler + +### 5.2 发布 DSS 工作流到 DolphinScheduler 点击 DSS 工作流的发布按钮，可将 DSS 工作流一键发布到 DolphinScheduler。 -### 4.3 调度中心使用文档 +### 5.3 调度中心使用文档 更多关于 DSS 调度中心的使用介绍，请参考：[调度中心使用文档](../用户手册/调度中心使用文档.md) \ No newline at end of file