[EPIC]: Recon Normalizer #2107
Description
Is there an existing issue for this?
- I have searched the existing issues
Category of feature request
Reconcile
Problem statement
We frequently observe that due to the data format and layout, there are inconsistencies when we compare them between two systems.
Introduce a normaliser which provides standardised rules to help match the data, which in turn can be overwritten by User Transformation.
By applying normalisation based on datatypes, we eliminate the need for many default transformations and user transformation requirements that exist today.
For Example:
- [FEATURE]: Automatic decimal number mapping (Avoid type casting source columns) while validating NUMBER datatype with precision in Oracle #1174
- [FEATURE]: Cast to original data type when using derived tables sampling queries #1195
- False Negative Example [BUG]: Number without scale in Oracle vs bigint in Databricks - Reconcilation is treating all records as "mismatch" #1202
- Encoding [MSSQL] Investigate how to handle special characters #1619
- Table, Column Identifier Normaliser(Column names, Table Names) [FEATURE]: Support case sensitivity for reconcile #1941
- [BUG]: Reconcile, false mismatches , incomplete data in reconcile.details #2057 Improve oracle support in reconcile #2081
Proposed Solution
proposed solution #2099
Additional Context
No response