Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-34441] Add guide to submit flink SQL scripts via the operator (using flink-sql-runner-example) #776

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[FLINK-34441] Add guide to submit flink SQL scripts via the operator (using flink-sql-runner-example) #776

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions docs/content/docs/custom-resource/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,38 @@ Alternatively, if you use helm to install flink-kubernetes-operator, it allows y

- Last-state upgradeMode is currently not supported for FlinkSessionJobs

## Flink SQL Jobs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taking another look, this might not be the best place to add this information because it's an overview page and the steps are quite detailed. There is already a section mentioning the examples: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/concepts/overview/#built-in-examples How about creating a dedicated Examples page?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should avoid copying anything over from https://github.com/apache/flink-kubernetes-operator/tree/main/examples/. Instead, we can extract the above linked examples section to a dedicated page and add any extra information we find useful.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comments @mxm ! I agree that we should have a dedicated page for examples, instead of adding more in the overview page. However, I have a few follow-up concerns/questions:

  1. If I don't copy over from the examples' readme, I'd still be needing to write content that's similar to whatever is present in the example's readme. I can try and make it short, while linking to the readme page for more details.

  2. If I create a dedicated Examples page, then I think it shouldn't just have the Flink SQL example, but other examples as well. I will try and write a few more, but not sure if I have enough experience/knowledge to write about all the examples. Can we make this like a running doc where more examples will be added as the need arises?

  3. One concern I have about linking to README pages is that I'm likely going to link to the examples in main branch, and that code may change with later releases in the operator. Would that possibly make the examples in the operator point to an incompatible readme? (I see that the existing links also point to the main branch, so maybe that's okay)

Another common way to deploy Flink jobs is by writing SQL scripts. We have put together [a Flink Java application](https://github.com/apache/flink-kubernetes-operator/tree/main/examples/flink-sql-runner-example#flink-kubernetes-operator-sql-example) that can read Flink SQL scripts and submit the queries using the table API. The user needs to package the job into a JAR and provide the SQL script's path as an argument to the job.

You can clone the [flink-kubernetes-operator](https://github.com/apache/flink-kubernetes-operator.git) repo and follow these steps to deploy a Flink application that can run Flink SQL scripts. These steps assume that you have the Flink Kubernetes Operator installed and running in your environment.

**Step 1**: Build Sql Runner maven project
```bash
cd examples/flink-sql-runner-example
mvn clean package
```

**Step 2**: Add your SQL script files under the `sql-scripts` directory

**Step 3**: Build docker image
```bash
# Uncomment when building for local minikube env:
# eval $(minikube docker-env)

DOCKER_BUILDKIT=1 docker build . -t flink-sql-runner-example:latest
```
This step will create an image based on an official Flink base image including the SQL runner jar and your user scripts.

**Step 4**: Create FlinkDeployment Yaml and Submit

Edit the included `sql-example.yaml` so that the `job.args` section points to the SQL script that you wish to execute, then submit it.

```bash
kubectl apply -f sql-example.yaml
```

***Note:*** *While the included SqlRunner should work for most simple cases, it is not expected to be very robust or battle tested. Please test with your SQL scripts to make sure the queries are parsed and executed correctly. If you find any bugs or limitations, feel free to open Jira tickets and bugfix PRs.*

## Further information

- [Job Management and Stateful upgrades]({{< ref "docs/custom-resource/job-management" >}})
Expand Down