[SPARK-53479][PS] Align == behavior with pandas when comparing against scalar under ANSI

[xinrong-meng]

What changes were proposed in this pull request?

• Ensure == returns a nameless Series when comparing with another Series/Index, but preserves the name for scalar comparisons.
• Add test cases to compare with np.nan

Why are the changes needed?

Part of https://issues.apache.org/jira/browse/SPARK-53389

Does this PR introduce any user-facing change?

No, the feature is not released yet.

For example,

Before

>>> psdf['int'] == 'x'
0    False
1    False
dtype: bool

After

>>> psdf['int'] == 'x'
0    False
1    False
Name: int, dtype: bool

which follows native pandas

>>> pdf['int'] == 'x'
0    False
1    False
Name: int, dtype: bool

How was this patch tested?

Unit tests

Commands below passed

 1037  SPARK_ANSI_SQL_MODE=true  ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_num_ops NumOpsTests.test_comparison_dtype_compatibility"
 1038  SPARK_ANSI_SQL_MODE=false  ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_num_ops NumOpsTests.test_comparison_dtype_compatibility"
 1039  SPARK_ANSI_SQL_MODE=true ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_num_ops NumOpsTests.test_eq"
 1040  SPARK_ANSI_SQL_MODE=false ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_num_ops NumOpsTests.test_eq"
 1041  SPARK_ANSI_SQL_MODE=false ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_num_ops NumOpsTests.test_ne"
 1042  SPARK_ANSI_SQL_MODE=true ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_num_ops NumOpsTests.test_ne"

Was this patch authored or co-authored using generative AI tooling?

No

[xinrong-meng]

@ueshin @zhengruifeng @HyukjinKwon may I get a review please?

[xinrong-meng]

Merged to master thank you!