When provisioning customer / dev tenants (not unified) the BQ provisioning should generate the mdmID

Description

PRDE - Story default text according to the team DoR (Definition of Ready)

01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
02 - PROBLEM (WHAT'S THE CURRENT PROBLEM SCENARIO OR PAIN TO BE RESOLVED?):

We are not investing on our old consolidation flow. The old consolidation flow is the one that uses Batch Dataflow process to rebuild the parquet files.

03 - GOAL (DESCRIBE THE PROPOSED SOLUTION):

To avoid using the consolidation flow, we can redefine the mdmID when importing the data from parquet to bigquery (the dataload flow) and calculate the mdmID again for the imported data.

We should keep all other data as they are, including the counter.

Adjusted Flow: [https://app.diagrams.net/#G1x8g20kW0rUK8PpDyE16xAimsaFVGE2aX#

{"pageId"%3A"A6-Zbxc90aw5eNSou2SU"}

|https://app.diagrams.net/#G1x8g20kW0rUK8PpDyE16xAimsaFVGE2aX#%7B%22pageId%22%3A%22A6-Zbxc90aw5eNSou2SU%22%7D]

Tab: PX-07

04 - WHO CAN USE THIS FEATURE (USER ROLES): tenantAdmin, OrgAdmin
06 - ACCEPTANCE CRITERIA:

  • Provision flow covering the regeneration of mdmID for customer tenants (not Unified tenant).
    • Not unified tenant does not have CDS data, so we can’t provision it again.
  • The process that copy data from D-1 to Carol, after a schema change that required a table recreation should also generate again the mdmID for the reimported data.

Update:
The mdmId regeneration need to be done only for staging tables.

Activity

Automation for Jira 2 January 2024, 19:32 Jira Internal Users

@Robson Thanael Poffo ,
@Geny Isam Hamud Herrera , @Pedro Buzzi , @Daniel Faustino , @Pedro Buzzi

This issue was planned to be delivered until 2024-01-22. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2024-01-22, 2024-01-02

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.

Automation for Jira 20 December 2023, 16:17 Jira Internal Users

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

Robson Thanael Poffo 20 December 2023, 01:19 Jira Internal Users

@Pedro Buzzi , vamos usar essas tenants para o teste:

tenant6efd5ff6df5e11eaadb60a58646001f6
tenant6f395f207953499f81b4fec062f0f973
tenant6f5a7614f6be11eab8bf0a586461422d
tenant6fb7af7b9d2d11eb886c0a58646140ca
tenant6fd841afa3774b76a8178c0bfe1c661e
Pedro Buzzi 19 December 2023, 13:03 Jira Internal Users

Flag added

Waiting for list of tenants to apply the rule

Automation for Jira 15 December 2023, 12:55 Jira Internal Users

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

CAPL-5168 Done feat: calculate mdmId provision and table recreation

Automation for Jira 11 December 2023, 20:35 Jira Internal Users

@Robson Thanael Poffo ,
@Pedro Buzzi ,
@Geny Isam Hamud Herrera ,
This issue was planned to be delivered until 2024-01-01. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2024-01-01

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.