Issue Type Icon CAPL-5136

Carol App Installation: allow to materialize the staging tables applying the crosswalk and adding new fields that belongs to the Carol App

Description

PRDE - Story default text according to the team DoR (Definition of Ready)

01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
02 - PROBLEM (WHAT'S THE CURRENT PROBLEM SCENARIO OR PAIN TO BE RESOLVED?):

Today we have Carol Apps that can’t migrate to Unified Tenants, and for these tenants we should have a way to allow to install the Carol App applying the defined crosswalk and adding new fields the Carol App is defining.

We also need to apply a default behavior when installing a Carol App avoiding to break existing Carol Apps on customer’s tenant.

03 - GOAL (DESCRIBE THE PROPOSED SOLUTION):

The installation of a Carol App ({{/api/v1/tenantApps/

{carolAppId}

/install}}) should consider a default process of update the staging table instead of apply a new schema dropping existing columns.

The rules we should follow:

  • Add a new validation if the staging table does not exists, create it (without consider the parameter forceMaterializeRestricted.
  • If the parameter force (forceMaterializeRestricted):
    • True:
      • Only add new columns the Carol App has additionally to the fields the customer’s staging table have defined.
      • Do not remove existing columns that are not part of the snapshot that is updating the existing staging.
      • Update the crosswalk (primary key) of the staging table as defined on the Carol App snapshot for the specified staging table. This is a replace action (not append).
    • False:
      • Skip the staging table materialization (as today).
  • In case the field type of the carol app tenant and customer tenant is different, it should register a warning message on Carol App installation task’s log informing:
    • The connector “A“ and staging table “Y” has the field name “X” with the type “string” and the app has it defined as “integer”. The schema change was applied to add any new field available on the Carol App for this staging table.
  • As today we don’t allow staging table to change the cluster and partitioning.
  • Please check
    CAPL-4596:PX07 - As tenant admin I should install a Carol App without impacting global ...Canceled
    for more refence.

Flow:

04 - WHO CAN USE THIS FEATURE (USER ROLES): Tenant Admin
06 - ACCEPTANCE CRITERIA:

  • Follow rules defined above (The rules we should follow🙂.

Related issues:

CAPL-4596:PX07 - As tenant admin I should install a Carol App without impacting global ...Canceled

  • This card will be cancelled (replaced) by this one here ( PRDE-3125 Done ).

CAPL-4585:PX07 - As tenant admin I should install a Carol App without impacting global ...Done

  • This issue has the goal to allow the Carol Admin to define Global Connectors that will not materialize the staging tables.

Attachments

image-20240130-...
264.8 KiB
image-20240130-...
182.3 KiB
image-20240130-...
439.2 KiB
image-20240130-...
129.6 KiB
image-20240130-...
213.2 KiB
image-20240130-...
166.3 KiB
image-20240129-...
396.5 KiB
image-20240129-...
73.5 KiB
image-20240129-...
380.8 KiB
image-20240129-...
305.5 KiB
image-20240129-...
647.1 KiB
image-20240129-...
338.6 KiB
image-20240129-...
476.3 KiB
image-20240129-...
329.7 KiB
image-20240126-...
400.5 KiB
image-20240126-...
379.4 KiB
image-20240126-...
381.3 KiB
image-20240126-...
360.5 KiB
image-20240126-...
352.6 KiB
image-20240126-...
398.8 KiB
image-20240126-...
111.9 KiB
image-20240126-...
169.4 KiB
image-20240126-...
361.5 KiB
image-20240126-...
334.2 KiB
image-20240126-...
93.3 KiB
image-20240126-...
168.6 KiB
image-20240126-...
354.0 KiB
image-20240126-...
343.3 KiB
image-20240126-...
348.4 KiB
image-20240126-...
347.7 KiB
image-20240126-...
352.1 KiB
image-20240126-...
356.5 KiB
image-20240126-...
258.0 KiB
image-20240126-...
415.2 KiB
image-20240126-...
83.7 KiB
image-20240126-...
358.4 KiB
image-20240126-...
326.7 KiB
image-20240126-...
127.3 KiB
image-20240126-...
316.8 KiB
image-20240126-...
312.4 KiB
image-20240126-...
416.5 KiB
image-20240126-...
354.9 KiB
image-20240126-...
266.6 KiB
image-20240126-...
94.5 KiB
image-20240126-...
76.4 KiB
image-20240104-...
189.7 KiB
image-20231208-...
243.5 KiB
image-20231206-...
95.6 KiB

Activity

Show:
Automation for Jira 6 February 2024, 20:39 • Jira Internal Users

This issue was automatically transitioned to WAITING DEPLOY, as its linked QA regression issue has just reached WAITING DEPLOY status (PR was just merged into master branch in Github).

Robson Thanael Poffo 31 January 2024, 23:51 • Jira Internal Users

Ok, thank you.

Automation for Jira 31 January 2024, 20:49 • Jira Internal Users

This issue was automatically transitioned to TESTED & MERGED, as its PR was just merged into develop branch in Github. PR Approved by glaucioscheibel,wsneto.

Gabriel DAmore Marciano 31 January 2024, 20:30 • Jira Internal Users

Flag removed

I’m removing the flag 🚩 that was initially blocked by

CAPL-5410:Allow unified tenant to materialize tables independent of the connector model...Done
. After a discussion on our backlog review (24-01-31), it was agreeded that we’re going to move with this card to production even though the other ( CAPL-5410 Done ) is not ready yet. This is because we have benefits coming from this implementation that justifies the deployment ASAP. What we should have caution is to NOT create RESTRICTED ConnectorModels until we deploy the
CAPL-5410:Allow unified tenant to materialize tables independent of the connector model...Done
c/c @Geny Isam Hamud Herrera @Robson Thanael Poffo @Bruno Furtado @MARCOS STUMPF

Gabriel DAmore Marciano 30 January 2024, 18:16 • Jira Internal Users

Flag added

BlockedBy:

CAPL-5410:Allow unified tenant to materialize tables independent of the connector model...Done

MARCOS STUMPF 30 January 2024, 14:09 • Jira Internal Users

Testing scenario regarding this thread https://totvscarol.slack.com/archives/C20SEQ867/p1706555245979589?thread_ts=1706539291.492159&cid=C20SEQ867

Connector xpto STG product on Dev Tenant DEVgijoe

Table Schema:

Install Carol App from Dev Tenant Devemacao

Table Schema (connector xpto stg product) from Dev Tenant Devemacao

The connector model xpto is linked to connector xpto as RESTRICTED

Logs from carol app installation task

No materialization occurs as expected

MARCOS STUMPF 29 January 2024, 19:01 • Jira Internal Users

Novo teste no cenário A3 que apresentou falha contemplando as considerações abaixo.

How to test atualizados com as considerações.

Evidências

A3

Creating | Updating a RESTRICTED Connector Model

Linking the RESTRICTED Connector Model with connector on Dev tenant

Added new field (newColumn3) to STG product on DEV tenant

Created new Carol App version (1.0.6)
New field NOT materialized (newColumn3) on UNIF tenant after App Update (from 1.0.5 to 1.0.6)? 🟢


New field NOT materialized (newColumn3) on CUSTOMER tenant after App Update (from 1.0.4 to 1.0.6) 🟢

Questions marker:

  1. Should Unified apply the same concept of non-materialization that the customer receives?
  2. There may be changes between schemas in the CUSTOMER and UNIFIED tenants if the installation of any version is skipped.
    For example, in version 1.0.5 installed in the UNIFIED tenant, there were schema changes that were applied due to the fact that the model connector was of the NORMAL type. The CUSTOMER tenant did not install this version and in the meantime there was a new version 1.0.6 released and installed in the UNIFIED one with the RESTRICTED connector model for this same connector.
    In this case, when installing version 1.0.6 in the CUSTOMER tenant, it will not receive the changes applied in version 1.0.5 previously installed in the UNIFIED one (eg. PK change). Can we have side effects and is this expected behavior?
Automation for Jira 29 January 2024, 15:41 • Jira Internal Users

Bom dia pessoal voltado aos testes do , em alinhamento com , segue o retorno da primeira analise feita frente ao findings. cc
• Connector nlp RESTRICTED connector foi encontrado, porem o mesmo nao foi configurado com o connector da tenant customer, justificando do por que materializou a staging as it is.
• Espera-se que a tenant unified sempre tenha a materializacao realizada? Nao localizei esse AC no card, sendo assim, creio que seja um topico de outro card

Sent by Slack - back-end - Douglas Coimbra Lopes

MARCOS STUMPF 26 January 2024, 23:21 • Jira Internal Users

Testes concluídos do time de produto.

Sandbox: https://gijoecomandos.qarol.ai/carol-org/admin/tenants

Resumo:

Evidências

A1

First Installation


Connector materialization

Staging materialization

Data Model materialization

A2

Updating App with changes on STG usertable | DM mdmuser schemas
Changes:
Added newColumn to STG & DM
Removed Groups from STG & DM

STG schema before App Update on Customer

STG schema after App Update on Customer (🟢 for groups / 🟢 for newColumn )

DM schema

A3

Creating | Updating a RESTRICTED Connector Model

Added new field (newRestrict) to STG usertable on DEV tenant
Created new Carol App version
New field materialized (newRestrict) on UNIF tenant after App Update 🟢

New field materialized (newRestrict) on CUSTOMER tenant after App Update 🔴 

A4

PK (login) for usertable | PK (id) for product before update on Dev Tenant


Added new field (company) to existing PK on usertable STG into nlp (RESTRICTED) for DEV Tenant
Added new field (newColumn) to existing PK on product STG into xpto connector (NORMAL) for DEV Tenant
Created a new Carol App version on DEV Tenant
Updated the new App version on UNIF Tenant
New PK (login, company) materialized on usertable STG into UNIF Tenant 🟢

New PK (id, newColumn) materialized on product STG into UNIF Tenant 🟢

New PK (login, company) materialized on usertable STG into CUSTOMER Tenant 🟢

New PK (id, newColumn) materialized on product STG into CUSTOMER Tenant 🟢

A5

XPTO connector model become RESTRICTED

Creating a new connector model with the same name of an already existent (XPTO - RESTRICTED) 🟢

XPTO connector model become NORMAL

Creating a new connector model with the same name of an already existent (XPTO - NORMAL) 🟢

The connector XPTO already exists on DEV tenant DEVemacao

Creating a new connector XPTO on a DEV tenant {{DEVgijoe }}🟢

B1 / B2

RESTRICTED connector xpto

snapshot field dropped from STG product
Install forceApp with forceMaterializeRestricted = TRUE on UNIFIED and CUSTOMER tenants
Materialized STG product with schema changes: new PK (id, barcode) and new field (newColumn) on UNIFIED tenant 🟢

Materialized STG product with schema changes: new PK (id, barcode) and new field field (newColumn) on CUSTOMER tenant 🟢

B3 / B4

NORMAL connector nlp

Install forceApp with forceMaterializeRestricted = TRUE on UNIFIED and CUSTOMER tenants
Materialized STG usertable with schema changes: new PK (login, company) and new field (newColumn) on UNIFIED tenant 🟢

Materialized STG usertable with schema changes: new PK (login, company) and new field (newColumn) on CUSTOMER tenant 🟢

Automation for Jira 22 January 2024, 20:01 • Jira Internal Users

@MARCOS STUMPF ,
@Jonathan Willian Moraes , @André Pereira de Oliveira , @Glaucio Scheibel , @wilson.souza

This issue was planned to be delivered until 2024-02-12. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2024-01-23, 2024-02-12

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.

wilson.souza 18 January 2024, 14:53 • Jira Internal Users

Regression validated.

wilson.souza 18 January 2024, 14:53 • Jira Internal Users

Card validated by QA team.

Automation for Jira 18 January 2024, 14:53 • Jira Internal Users

Github user wsneto has just approved a PR (added as Shard Assignee in this Jira issue).

feat: CAPL-5136 Done add rule when update staging schema - connector model restricted

Automation for Jira 10 January 2024, 15:20 • Jira Internal Users

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

Automation for Jira 9 January 2024, 18:06 • Jira Internal Users

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

feat: CAPL-5136 Done add rule when update staging schema - connector model restricted

Automation for Jira 2 January 2024, 20:56 • Jira Internal Users

@MARCOS STUMPF ,
@Geny Isam Hamud Herrera ,

This issue was planned to be delivered until 2024-01-22. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2024-01-22

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.

Done

Details

Priority

Priority: HighestHighest

Assignee

André Pereira de Oliveira

Reporter

MARCOS STUMPF

Labels

CAROL_BACKLOG_REVIEWDEMOPLANNEDPLATFORM_LEADERSHIP_REVIEWPROD_IMPACTREADYRELEASE_2

Due Date

12 February 2024

Fix versions

CAPL_3.99

Components

CAROL_APP | INSTALATION

Created

6 December 2023, 16:06

Updated

19 April 2024, 10:16

More fields

Created: 6 December 2023, 20:06 Updated: 19 April 2024, 15:16