Issue Type Icon CAPL-5504

SQL Processing: it should not create the field on "elasticsearch mapping" when the SQL Processing return a field that does not exist on the tenant - local tenant

Description

PRDE - Story default text according to the team DoR (Definition of Ready)

01 - STAKEHOLDER (PERSON THAT CAN VALIDATE AND ANSWER QUESTIONS):
02 - PROBLEM (WHAT'S THE CURRENT PROBLEM SCENARIO OR PAIN TO BE RESOLVED?):

  • Tenant has DM with RT enabled.
  • We should run a SQL Processing task with the parameter to store data on RT.
  • When the SQL Processing returns fields that the tenant and data model do not have, the process to store the data on ES is creating the fields on ES.
    • This flow should not create fields automatically.

03 - GOAL (DESCRIBE THE PROPOSED SOLUTION):
04 - WHO CAN USE THIS FEATURE (USER ROLES):
05 - ASSETS (FIGMA LINKS, RELEVANT DOCUMENTATION LINKS, JSON EXAMPLES, ETC):
06 - ACCEPTANCE CRITERIA:

  • When running the SQL Processing task, when the tenant has the RT enabled, the task should remove any field not available on the output Data Model to avoid creating them automatically.
    • Observed cases created the new field as string for cases we were expecting to see it as integer
  • This scenario is observed only for tenants with RT enabled (and the SQL Task should have the parameter to save data in RT).
  • This card is covering only the scope for the local tenant, not the fan-out process that happens when processing data on unified tenants.

Attachments

image-20240214-...
189.1 KiB
image-20240214-...
283.9 KiB
image-20240214-...
187.4 KiB
image-20240214-...
145.8 KiB
image-20240214-...
229.3 KiB
image-20240214-...
192.7 KiB
image-20240214-...
115.8 KiB
image-20240214-...
175.7 KiB
image-20240214-...
177.5 KiB
image-20240214-...
173.7 KiB
image-20240214-...
116.4 KiB
image-20240214-...
357.0 KiB
image-20240214-...
179.7 KiB
image-20240214-...
210.7 KiB
image-20240214-...
333.9 KiB
image-20240214-...
276.9 KiB
image-20240214-...
360.9 KiB
image-20240214-...
203.3 KiB
image-20240214-...
337.9 KiB

Activity

Show:
Automation for Jira 14 February 2024, 19:10 • Jira Internal Users

This issue was automatically transitioned to WAITING DEPLOY, as its PR was just merged into master branch in Github.

Automation for Jira 14 February 2024, 19:08 • Jira Internal Users

Github user douglascoimbra has just approved a PR (added as Shard Assignee in this Jira issue).

feat: https://totvslabs.atlassian.net/browse/CAPL-5504#icft=CAPL-5504 - Adjust TableRow with only valid columns

Automation for Jira 14 February 2024, 19:04 • Jira Internal Users

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

Douglas Coimbra Lopes 14 February 2024, 18:50 • Jira Internal Users

@Geny Isam Hamud Herrera @Gabriel DAmore Marciano Card has been validated by the QA team. It is pending only the code review.

Douglas Coimbra Lopes 14 February 2024, 18:47 • Jira Internal Users

ORCHESTRATOR WITH THE CUSTOMER WITH THE FIELD

NO MAPPING CREATED

INSTALLING NEW CAROL APP WITH THE FIELD ADDED TO THE DATAMODEL

FIELD CREATED AND WITH DATA

ORCHESTRATOR WITH THE TENANT WITH THE FIELD

FIELD CREATED

Automation for Jira 14 February 2024, 18:32 • Jira Internal Users

Cenario encontrado para o card
• Update. Cenario jah acontece na master :chavo:
• GCP:
• Evidencias no card. Estou a disposicao para maiores detalhes
• Ao Executar SQL processing pelo ORCHESTRATOR, a task apresenta uma msg de warning de que o pipeline nao foi encontrado.

Edited on Slack - platform-internal - Douglas Coimbra Lopes

Automation for Jira 14 February 2024, 18:32 • Jira Internal Users

Cenario encontrado para o card
• Update. Cenario jah acontece na master :chavo:
• GCP:
• Evidencias no card. Estou a disposicao para maiores detalhes
• Ao Executar SQL processing pelo ORCHESTRATOR, a task apresenta uma msg de warning de que o pipeline nao foi encontrado.

Edited on Slack - platform-internal - Douglas Coimbra Lopes

Automation for Jira 14 February 2024, 17:41 • Jira Internal Users

:alert: Cenario encontrado para o card
• GCP:
• Evidencias no card. Estou a disposicao para maiores detalhes
• Ao Executar SQL processing pelo ORCHESTRATOR, a task apresenta uma msg de warning de que o pipeline nao foi encontrado.

Edited on Slack - platform-internal - Douglas Coimbra Lopes

Douglas Coimbra Lopes 14 February 2024, 17:40 • Jira Internal Users

UNIFIED FLOW

NO MAPPING CREATED FOR TENANT DOUGCUSTOMER

MAPPING CORRECTLY CREATED FOR THE CUSTOMER WITH THE DM FIELD

Automation for Jira 14 February 2024, 17:39 • Jira Internal Users

:alert: Cenario encontrado para o card
• GCP:
• Ao Executar SQL processing pelo ORCHESTRATOR, a task apresenta uma msg de warning de que o pipeline nao foi encontrado.

Edited on Slack - platform-internal - Douglas Coimbra Lopes

Automation for Jira 14 February 2024, 17:38 • Jira Internal Users

:alert: Cenario encontrado para o card
• Ao Executar SQL processing pelo ORCHESTRATOR, a task apresenta uma msg de warning de que o pipeline nao foi encontrado.

Sent by Slack - platform-internal - Douglas Coimbra Lopes

Automation for Jira 14 February 2024, 17:19 • Jira Internal Users

Boa tarde pessoal. Voltado a issue de erro voltado ao mapping de ES de campos criados indevidamente por SQL processing, o card abaixo obtem a correcao para fluxos Unified e ORCHESTRATOR. (Em validacao pelo time de QA)
• Voltado ao card poderiam por favor efetuar a criacao do card CAPL para adicionah-lo na Sprint? cc

Edited on Slack - back-end - Douglas Coimbra Lopes

Automation for Jira 14 February 2024, 17:19 • Jira Internal Users

Boa tarde pessoal. Voltado a issue de erro voltado ao mapping de ES de campos criados indevidamente por SQL processing, o card abaixo obtem a correcao para fluxos Unified e ORCHESTRATOR. (Em validacao pelo time de QA)
• Voltado ao card podemos efetuar a criacao do card CAPL para adicionah-lo na Sprint por favor? cc

Edited on Slack - back-end - Douglas Coimbra Lopes

Automation for Jira 14 February 2024, 17:18 • Jira Internal Users

Boa tarde pessoal. Voltado a issue de erro voltado ao mapping de ES de campos criados indevidamente por SQL processing, o card abaixo obtem a correcao para fluxos Unified e ORCHESTRATOR. (validado por QA)
• Voltado ao card podemos efetuar a criacao do card CAPL para adicionah-lo na Sprint por favor? cc

Sent by Slack - back-end - Douglas Coimbra Lopes

Douglas Coimbra Lopes 14 February 2024, 12:41 • Jira Internal Users

@Geny Isam Hamud Herrera When a new field is added to the DM Schema and an SQl processing is executed, the new field is not being added to the mdmdatamodelGolden mapping.

field created newfieldpurchase

1

curl --location 'http://localhost:9209/master-index-sample/mdmpurchaseorderGolden/_search'

1

dmField newfield
Douglas Coimbra Lopes 14 February 2024, 12:39 • Jira Internal Users

SQL processing executed with an unexisting field for the datamodel

Field/Value not propagated on ES

PROCESS PIPELINE

Automation for Jira 14 February 2024, 00:51 • Jira Internal Users

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

feat: https://totvslabs.atlassian.net/browse/CAPL-5504#icft=CAPL-5504 - Adjust TableRow with only valid columns

Automation for Jira 13 February 2024, 21:59 • Jira Internal Users

This issue was automatically transitioned to IN PROGRESS, as its branch was just created in Github.

CAPL-5504-remove-invalid-columns

Automation for Jira 13 February 2024, 17:25 • Jira Internal Users

Message thread link on #red-phone channel:

https://totvscarol.slack.com/archives/C03NT4US9J9/p1707845130900159

Automation for Jira 13 February 2024, 17:25 • Jira Internal Users

@Robson Thanael Poffo ,
@Geny Isam Hamud Herrera ,

This issue was planned to be delivered until 2024-03-01. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2024-03-01

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.

Done

Details

Priority

Priority: HighestHighest

Assignee

Geny Isam Hamud Herrera

Reporter

Robson Thanael Poffo

Labels

CAROL_BACKLOG_REVIEWPROD_IMPACTRED_PHONE

Due Date

5 March 2024

Fix versions

CAPL_4.00

Components

SQL_PIPELINES | DATA_PROCESSING

Created

13 February 2024, 14:24

Updated

30 July 2024, 15:21

More fields

Created: 13 February 2024, 17:24 Updated: 30 July 2024, 18:21