Redefine the meta tag for AuditID, TaskID, batchId, and batchSequenceId on golden records

Description

PRDE - Story default text according to the team DoR (Definition of Ready)

01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
02 - PROBLEM (WHAT'S THE CURRENT PROBLEM SCENARIO OR PAIN TO BE RESOLVED?):

When deploying the issue (

) related to improve the traceability storing the AuditId and TaskID on Golden Records, we noticed some impacts on production.

The impacts are related to some pipelines exploring specific rules not allowing Carol to inject the fields mdmStagingAuditID, mdmTaskId, mdmBatchId, and mdmBatchSequenceId from staging area:

  • Sometimes the pipeline is reading data from data model (the field auditID has different names for staging and golden: mdmAuditId, mdmStagingAuditId).
  • Some pipelines depends on shared data (view), as defined previously, it does not allow select * so new fields should be manually redefined.
  • Some pipelines are limiting the fields from staging tables for optimization.
    • For these cases (clockin also) we suggested using the -metadata- on the first moment we limit the staging fields (first CTE).

We need to define a new version for the metadata fields:

  • -metadata-v2-
  • -metadataNoId-v2-

03 - GOAL (DESCRIBE THE PROPOSED SOLUTION):
04 - WHO CAN USE THIS FEATURE (USER ROLES):
05 - ASSETS (FIGMA LINKS, RELEVANT DOCUMENTATION LINKS, JSON EXAMPLES, ETC):
06 - ACCEPTANCE CRITERIA:

  • We should explore the usage of the fields mdmStagingAuditID, mdmTaskId, mdmBatchId, and mdmBatchSequenceId on Golden records through new metadata fields version, avoiding impact on production and allowing customers to review their own pipelines.
    • -metadata-v2-
    • -metadataNoId-v2-