[MVP 1] As tenant admin on Unified Tenant I should be able to trigger the copy data for the whole customer tenant or a specific tenant for a set of tables

Description

PRDE - Story default text according to the team DoR (Definition of Ready)

01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
02 - PROBLEM (WHAT'S THE CURRENT PROBLEM SCENARIO OR PAIN TO BE RESOLVED?):

Sometimes to fix missing data on unified we need to run a copy data process, today we force the uninstall and install again since it dispatches the copy data tasks.

03 - GOAL (DESCRIBE THE PROPOSED SOLUTION):

Allow the Tenant Admin on Unified Tenants to start the copy data for staging tables / connectors listed on manifest file. The tenant admin should be able to run a full copy data (for all staging tables / connectors specified on manifest file) or specifying a list of connectors and staging tables.

04 - WHO CAN USE THIS FEATURE (USER ROLES): Tenant Admin on Unified tenant.

06 - ACCEPTANCE CRITERIA:

  • Tenant Admin on Unified tenant should be able to trigger the copy data for 1 tenant (whole tenant considering the unified manifest pipeline or specifying a list of connectors and staging tables).
  • The endpoint will be responsible to run the validation:
    • We should return a bad request (400) IF:
      • The tenantId of the request does not exist.
      • At least one of the tables choosen does not exist on the unfied pipeline manifest. [NOT MAPPED TABLE]
      • At least one of the table choosen does not exist on the customer tenant. [TABLE DOES NOT EXIST ON CUSTOMER SIDE]
  • On this MVP1, we’re are going to keep the same behavior as we have today: The number of staging tables selected to be copied to unified will be the number of tasks created of COPY_DATA. This means, no parent task will be created to manage the copy data nor the new task architecture will be used.

Activity

Automation for Jira 16 April 2024, 13:58 Jira Internal Users

Esta issue foi automaticamente movida para REGRESSION, pois o PR foi mergeado na branch QA no Github.

Automation for Jira 16 April 2024, 13:58 Jira Internal Users

Esta issue foi automaticamente movida para REGRESSION, pois o PR foi mergeado na branch QA no Github.

Automation for Jira 15 April 2024, 21:36 Jira Internal Users

@MARCOS STUMPF ,
@Jonathan Willian Moraes , @Renan Schroeder , @Cindy de Araujo Soares Moore , @Douglas Coimbra Lopes , @Gabriel DAmore Marciano

Este issue foi planejada para ser entregue até 2024-05-06. Você pode confirmar consultando o campo Due Date desta issue.

Data já planejadas para esta issue: 2024-04-15, 2024-05-06, 2024-03-25

Se o campo External Issue Link estiver preenchido com o link de uma issue válida no Jira Produção o cliente também será notificado no Jira Produção.

Automation for Jira 12 April 2024, 14:38 Jira Internal Users

Esta issue foi automaticamente movida para TESTED & MERGED, pois o PR foi mergeado na branch Develop no Github. PR aprovado por cindysoares,douglascoimbra.

Automation for Jira 12 April 2024, 14:37 Jira Internal Users

Github usuário douglascoimbra aprovou um PR e foi adicionado como Shared Assignee nesta issue.

feat: Copy customer staging tables to unified tenant

Automation for Jira 9 April 2024, 15:20 Jira Internal Users

Esta issue foi automaticamente movimentada para QA REVIEW, pois o PR foi aprovado no Github.

Automation for Jira 9 April 2024, 03:39 Jira Internal Users

Card revalidado pelo time de QA. Pendente apenas code review

Mensagem editada no Slack - plataforma-carol-internal - Douglas Coimbra Lopes

Automation for Jira 9 April 2024, 03:39 Jira Internal Users

Card revalidado pelo time de QA. Pendente apenas code review

Mensagem enviada pelo Slack - plataforma-carol-internal - Douglas Coimbra Lopes

Douglas Coimbra Lopes 9 April 2024, 02:29 Jira Internal Users

INVALID TENANT PERFORMING COPY DATA

image-20240408-143752.png

UNIFIED PERFORMING THE CORRECT ACTION

image-20240408-144454.png

image-20240408-144425.png

NEW TABLE

Automation for Jira 8 April 2024, 17:53 Jira Internal Users

:rotating_light: Cenario encontrado no
• Ao passar nome de connector invalido, o endpoint retorna de que a staging nao foi encontrada.
• Connector invalido informado: `dlp`
```{
"applicationErrorCode": null,
"errorCode": 400,
"errorMessage": "The following staging tables (dlp:purchaseorder) were not found on customer tenant 98679303dd914a1f9f981cb98bd622a3",
"possibleResponsibleField": "stagingTables"
}```

Mensagem enviada pelo Slack - plataforma-carol-internal - Douglas Coimbra Lopes

Renan Schroeder 8 April 2024, 16:43 Jira Internal Users

@Cindy de Araujo Soares Moore já identificamos o problema. O request do Douglas não seguiu o contrato do endpoint que exigia que o parâmetro fosse um JSONList em vez de um JSONObject, como ele fez. Tentei debugar e o request não chega nem no método da implementação do endpoint pra ver se poderia melhorar e implementar mais alguma validação pra ajudar.

Cindy de Araujo Soares Moore 8 April 2024, 16:41 Jira Internal Users

@Douglas Coimbra Lopes essa operação de cópia de dados só é permitida a partir da unif. Customers não irão disparar essa operação. Inclusive, eu vi que o @Renan Schroeder fez essa validação de login em uma unif, acho que o problema aqui é que a mensagem não aparece na resposta do usuário. Tá vindo só 500 sem dizer qual foi o problema.

Douglas Coimbra Lopes 8 April 2024, 15:50 Jira Internal Users

@Renan Schroeder If we try to use a customer token to perform copy data, the platform is not informing correctly the unified should perform this action. Instead, a 500 error is displayed

image-20240408-134853.png

1

2

3

4

5

6

7

8

9

curl -X 'POST' \
  'https://totvsimvertedpeep.qarol.ai/api/v3/admin/tenantsAdmin/copyStagingTablesToUnified/98679303dd914a1f9f981cb98bd622a3' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer 88e9c831885a4dd8b2a134ff3f085bb4' \
  -d '{
  "connectorName": "nlp",
  "stagingName": "product"
}'
Automation for Jira 5 April 2024, 21:18 Jira Internal Users

Esta issue foi automaticamente movimentada para REVIEW, pois o PR (não DRAFT e não WIP) foi criado no Github.

feat: Copy customer staging tables to unified tenant

Automation for Jira 5 April 2024, 21:15 Jira Internal Users

@MARCOS STUMPF ,
@Renan Schroeder , @Cindy de Araujo Soares Moore , @Douglas Coimbra Lopes

Flag foi removida, pois você acabou de alterar o status desta issue.

Gabriel DAmore Marciano 3 April 2024, 15:24 Jira Internal Users

Vamos alinhar sobre as expectativas.

Automation for Jira 25 March 2024, 20:30 Jira Internal Users

@MARCOS STUMPF ,
@Jonathan Willian Moraes , @Renan Schroeder , @Douglas Coimbra Lopes

This issue was planned to be delivered until 2024-04-15. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2024-04-15, 2024-03-25

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.

Automation for Jira 22 March 2024, 21:24 Jira Internal Users

Card validado pelo time de QA. Pendente apenas o code review

Mensagem enviada pelo Slack - plataforma-carol-internal - Douglas Coimbra Lopes

Automation for Jira 21 March 2024, 15:09 Jira Internal Users

Encontrei um cenario ao tentar rodar o endpoint do .
Update topico resolvido :white_check_mark:
• Ao rodar o endpoint nao passando nenhum informacao de staging, eh gerado apenas a task de copy data de uma das tabelas
• Evidencias + sandbox no card.

Mensagem editada no Slack - plataforma-carol-internal - Douglas Coimbra Lopes

Automation for Jira 21 March 2024, 15:07 Jira Internal Users

:rotating_light:Encontrei um cenario ao tentar rodar o endpoint do .
• Ao rodar o endpoint nao passando nenhum informacao de staging, eh gerado apenas a task de copy data de uma das tabelas
• Evidencias + sandbox no card.

Mensagem enviada pelo Slack - plataforma-carol-internal - Douglas Coimbra Lopes

Douglas Coimbra Lopes 21 March 2024, 15:04 Jira Internal Users

CUSTOMER TRYING TO RUN THE ENDPOINT

NON EXISTING PIPELINE

NON EXISTING CAROL APP

CUSTOMER OUT OF THE ALLOWED LIST

STAGING THAT IS NOT PART OF THE SQL PIPELINE

VALID STAGING

Automation for Jira 20 March 2024, 20:20 Jira Internal Users

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

feat: Copy customer staging tables to unified tenant

Automation for Jira 14 March 2024, 15:08 Jira Internal Users

@MARCOS STUMPF ,
@Renan Schroeder ,

This issue was planned to be delivered until 2024-03-25. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2024-03-25

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.