[SPARK-51338][INFRA] Add automated CI build for connect-examples

[vicennial]

What changes were proposed in this pull request?

Adds a build step for the connect-examples directory (specifically the server-library-example project) in the CI steps (build_and_test.yml)

Why are the changes needed?

No existing automated CI build atm

Does this PR introduce any user-facing change?

No

How was this patch tested?

CI run in this PR

Was this patch authored or co-authored using generative AI tooling?

Generated-by: o3-mini-high

[hvanhovell]

@HyukjinKwon can you take a look?

[HyukjinKwon]

If we want to run this with Java 21 (and other scheduled builds), we would need to add this in https://github.com/apache/spark/blob/master/.github/workflows/build_java21.yml too as an example, .e.g., "connect_examples": "true"

[HyukjinKwon]

Can we use build/sbt instead? We use SBT in the PR builder

[HyukjinKwon]

Looks good otherwise

[LuciferYang]

I've noticed a rather peculiar issue here. It seems that the connect-examples project is dependent on a released version of Spark, which means we can only update to a new version after it has been officially released.

For example, when Spark 4.0.0 is released, the version of this project in the published v4.0.0 tag won't be Spark 4.0.0, but rather some previously released version. Am I right about this? If I'm wrong, please correct me.

If I'm right, doesn't that seem a bit strange? Therefore, I still believe that connect-examples module should be placed outside of the Spark Repo, so that we can update and release it in another codebase after Spark 4.0.0(Or other official versions) is out.

spark/connect-examples/server-library-example/pom.xml

Lines 19 to 42 in f40bf4d

	<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>org.apache.connect.examples.serverlibrary</groupId>
	<artifactId>spark-server-library-example</artifactId>
	<version>1.0.0</version>
	<packaging>pom</packaging>
	<modules>
	<module>common</module>
	<module>server</module>
	<module>client</module>
	</modules>
	<properties>
	<maven.compiler.source>17</maven.compiler.source>
	<maven.compiler.target>17</maven.compiler.target>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	<scala.binary>2.13</scala.binary>
	<scala.version>2.13.15</scala.version>
	<protobuf.version>3.25.4</protobuf.version>
	<spark.version>4.0.0-preview2</spark.version>
	</properties>
	</project>

What do you think about this? @HyukjinKwon @hvanhovell @dongjoon-hyun @cloud-fan @srowen @yaooqinn

[LuciferYang]

I think we should discuss #50187 (comment) clearly before merging

[srowen]

Can the examples module simply point to SNAPSHOT versions like everything else in the build? the main branch code is always pointing at unreleased code, but on release, those SNAPSHOT versions are changed to the current release version.

[LuciferYang]

Can the examples module simply point to SNAPSHOT versions like everything else in the build? the main branch code is always pointing at unreleased code, but on release, those SNAPSHOT versions are changed to the current release version.

Yes, if it can be done this way, I think it's ok.