SQL Processing: it should not create the field on "elasticsearch mapping" when the SQL Processing return a field that does not exist on the tenant - local tenant
Description
PRDE - Story default text according to the team DoR (Definition of Ready)
01 - STAKEHOLDER (PERSON THAT CAN VALIDATE AND ANSWER QUESTIONS):
02 - PROBLEM (WHAT'S THE CURRENT PROBLEM SCENARIO OR PAIN TO BE RESOLVED?):
- Tenant has DM with RT enabled.
- We should run a SQL Processing task with the parameter to store data on RT.
- When the SQL Processing returns fields that the tenant and data model do not have, the process to store the data on ES is creating the fields on ES.
- This flow should not create fields automatically.
03 - GOAL (DESCRIBE THE PROPOSED SOLUTION):
04 - WHO CAN USE THIS FEATURE (USER ROLES):
05 - ASSETS (FIGMA LINKS, RELEVANT DOCUMENTATION LINKS, JSON EXAMPLES, ETC):
06 - ACCEPTANCE CRITERIA:
- When running the SQL Processing task, when the tenant has the RT enabled, the task should remove any field not available on the output Data Model to avoid creating them automatically.
- Observed cases created the new field as string for cases we were expecting to see it as integer
- This scenario is observed only for tenants with RT enabled (and the SQL Task should have the parameter to save data in RT).
- This card is covering only the scope for the local tenant, not the fan-out process that happens when processing data on unified tenants.
Activity
Show:
Details
Priority
HighestAssignee
Geny Isam Hamud Herrera
Reporter
Robson Thanael Poffo
Labels
CAROL_BACKLOG_REVIEWPROD_IMPACTRED_PHONE
Due Date
5 March 2024
Fix versions
CAPL_4.00
Components
SQL_PIPELINES | DATA_PROCESSING
Created
13 February 2024, 14:24
Updated
12 March 2024, 21:25
More fields
Created: 13 February 2024, 18:24
Updated:
13 March 2024, 01:25
This issue was automatically transitioned to WAITING DEPLOY, as its PR was just merged into master branch in Github.
Github user douglascoimbra has just approved a PR (added as Shard Assignee in this Jira issue).
feat:
CAPL-5504 Done - Adjust TableRow with only valid columns
This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.
@Geny Isam Hamud Herrera @Gabriel DAmore Marciano Card has been validated by the QA team. It is pending only the code review.
ORCHESTRATOR WITH THE CUSTOMER WITH THE FIELD
NO MAPPING CREATED
INSTALLING NEW CAROL APP WITH THE FIELD ADDED TO THE DATAMODEL
FIELD CREATED AND WITH DATA
ORCHESTRATOR WITH THE TENANT WITH THE FIELD
FIELD CREATED
Cenario encontrado para o card
• Update. Cenario jah acontece na master :chavo:
• GCP:
• Evidencias no card. Estou a disposicao para maiores detalhes
• Ao Executar SQL processing pelo ORCHESTRATOR, a task apresenta uma msg de warning de que o pipeline nao foi encontrado.
Edited on Slack - platform-internal - Douglas Coimbra Lopes
Cenario encontrado para o card
• Update. Cenario jah acontece na master :chavo:
• GCP:
• Evidencias no card. Estou a disposicao para maiores detalhes
• Ao Executar SQL processing pelo ORCHESTRATOR, a task apresenta uma msg de warning de que o pipeline nao foi encontrado.
Edited on Slack - platform-internal - Douglas Coimbra Lopes
:alert: Cenario encontrado para o card
• GCP:
• Evidencias no card. Estou a disposicao para maiores detalhes
• Ao Executar SQL processing pelo ORCHESTRATOR, a task apresenta uma msg de warning de que o pipeline nao foi encontrado.
Edited on Slack - platform-internal - Douglas Coimbra Lopes
UNIFIED FLOW
NO MAPPING CREATED FOR TENANT DOUGCUSTOMER
MAPPING CORRECTLY CREATED FOR THE CUSTOMER WITH THE DM FIELD
:alert: Cenario encontrado para o card
• GCP:
• Ao Executar SQL processing pelo ORCHESTRATOR, a task apresenta uma msg de warning de que o pipeline nao foi encontrado.
Edited on Slack - platform-internal - Douglas Coimbra Lopes
:alert: Cenario encontrado para o card
• Ao Executar SQL processing pelo ORCHESTRATOR, a task apresenta uma msg de warning de que o pipeline nao foi encontrado.
Sent by Slack - platform-internal - Douglas Coimbra Lopes
Boa tarde pessoal. Voltado a issue de erro voltado ao mapping de ES de campos criados indevidamente por SQL processing, o card abaixo obtem a correcao para fluxos Unified e ORCHESTRATOR. (Em validacao pelo time de QA)
• Voltado ao card poderiam por favor efetuar a criacao do card CAPL para adicionah-lo na Sprint? cc
Edited on Slack - back-end - Douglas Coimbra Lopes
Boa tarde pessoal. Voltado a issue de erro voltado ao mapping de ES de campos criados indevidamente por SQL processing, o card abaixo obtem a correcao para fluxos Unified e ORCHESTRATOR. (Em validacao pelo time de QA)
• Voltado ao card podemos efetuar a criacao do card CAPL para adicionah-lo na Sprint por favor? cc
Edited on Slack - back-end - Douglas Coimbra Lopes
Boa tarde pessoal. Voltado a issue de erro voltado ao mapping de ES de campos criados indevidamente por SQL processing, o card abaixo obtem a correcao para fluxos Unified e ORCHESTRATOR. (validado por QA)
• Voltado ao card podemos efetuar a criacao do card CAPL para adicionah-lo na Sprint por favor? cc
Sent by Slack - back-end - Douglas Coimbra Lopes
@Geny Isam Hamud Herrera When a new field is added to the DM Schema and an SQl processing is executed, the new field is not being added to the mdmdatamodelGolden mapping.
field created{{ newfieldpurchase}}
SQL processing executed with an unexisting field for the datamodel
Field/Value not propagated on ES
PROCESS PIPELINE
This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.
feat:
CAPL-5504 Done - Adjust TableRow with only valid columns
This issue was automatically transitioned to IN PROGRESS, as its branch was just created in Github.
CAPL-5504-remove-invalid-columns
Message thread link on #red-phone channel:
https://totvscarol.slack.com/archives/C03NT4US9J9/p1707845130900159
@Robson Thanael Poffo ,
@Geny Isam Hamud Herrera ,
This issue was planned to be delivered until 2024-03-01. You can check that by consulting the issue in the Due Date field.
Dates already planned for this issue: 2024-03-01
If External Issue Link field is filled, customer was also informed on JIRA TOTVS.