Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: handle compare_vals turn into str when parse IgnoreColumnType #50205

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

phakawatfong
Copy link

when I parse ignoreColumnType=True when call the function, it cast all values to be 'str' which causing the comparison between val1 and val2 to be failed.
for example

val1 = 1505.76189560
val2= 1505.761896
assertDataFrameEqual(source_df, databricks_df, ignoreColumnType=True,)

I will convert the original data into string with this clause first

if ignoreColumnType:
    actual = cast_columns_to_string(actual)
    expected = cast_columns_to_string(expected)
.
.
compare_vals(val1, val2)

it will become string comparison instead of compare by using the tolerance equation
'1505.76189560' != '1505.761896'

Hence assertError, even if the atol and rtol already assigned

What changes were proposed in this pull request?

add more elif case to convert str to float and parse into tolerant calculations

Why are the changes needed?

I think it might not be the only way to fix it, but just to be your ideas Thank you !

Does this PR introduce any user-facing change?

No

How was this patch tested?

Test on my local data source, comparing 2 dataframes
by patch the code into my local env.

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions bot added the PYTHON label Mar 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant