SQL data pipelines are cleaning addresses when it has no address field
Description
CAPL - Story default text according to the team DoR (Definition of Ready)
01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
02 - PROBLEM (WHAT'S THE CURRENT PROBLEM SCENARIO OR PAIN TO BE RESOLVED?):
Normal data pipelines are generating this log entry:
Metrics: [TableRowToJson rejectedCount: 111503, TableRowToJson addressCleansing: 223006]Specifically:
addressCleansing: 223006
The Data Model:
- It does not have the field
mdmAddress. - The pipeline does not define the specific field indicating to clean address.
Temporary table related to the pipeline I shared here:
select *
from `carol-5cdd6773b30644ff851b.5cdd6773b30644ff851b4d4e4d878fcc.tt_temp_task_apinvoice_2023-11-29_c194f64ae734493a8ea05bd28f0bc826`
limit 10Result: https://docs.google.com/spreadsheets/d/1weRfI6S3_uJsd_xfpSXkhTR0VYs3JjGt0iYDIazOsdM/edit?usp=sharing
03 - GOAL (DESCRIBE THE PROPOSED SOLUTION):
04 - WHO CAN USE THIS FEATURE (USER ROLES):
05 - ASSETS (FIGMA LINKS, RELEVANT DOCUMENTATION LINKS, JSON EXAMPLES, ETC):
06 - ACCEPTANCE CRITERIA:
- Validate if we are triggering the address cleansing for cases it is not needed.
- It may have costs impacts.
- In case we are not, the log is misleading the user.
Activity
Details
Priority
Highest
This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.
This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.
This issue was automatically transitioned to TESTED & MERGED, as its PR was just merged into develop branch in Github. PR Approved by glaucioscheibel,jwmoraes.
Github user jwmoraes has just approved a PR (added as Shard Assignee in this Jira issue).
feat: https://totvslabs.atlassian.net/browse/CAPL-5135#icft=CAPL-5135 - No RESOLVE_GEOLOCATION was triggering enrich address
The card has been validated by the QA team.
Tenant with Address Cleasing
mdmAdress with Enable Geocoding
__resolveGeolocationas trueTenant with Address Cleasing
mdmAdress with Enable Geocoding
__resolveGeolocationas falseTenant with Address Cleasing
mdmAdress with Enable Geocoding
__resolveGeolocationas trueTenant with Address Cleasing
mdmAdress with Enable Geocoding
__resolveGeolocationas trueThis issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.
This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.
feat: https://totvslabs.atlassian.net/browse/CAPL-5135#icft=CAPL-5135 - No RESOLVE_GEOLOCATION was triggering enrich address
This issue was automatically transitioned to IN PROGRESS, as its branch was just created in Github.
CAPL-5135-address
@MARCOS STUMPF ,
@Geny Isam Hamud Herrera ,
This issue was planned to be delivered until 2024-01-23. You can check that by consulting the issue in the Due Date field.
Dates already planned for this issue: 2024-01-23
If External Issue Link field is filled, customer was also informed on JIRA TOTVS.