SQL Processing: efficiency not working for scenarios where the customer tenant does not have a newer carol app version installed

Description

PRDE - Bug default text according to the team DoR (Definition of Ready)

01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
02 - PROBLEM (WHAT'S THE ISSUE?):

Task: https://totvstechfin.carol.ai/tenant51c55f3f6f9d11eb84540a586460016e/carol-ui/tasks/activity/28d1464ba1c04f32b2ec396f25ea34e4?p=1&ps=25&sort=dateUpdated&order=DESC&filters=%5B%7B%22hideInternal%22:%22false%22%7D%5D

It is not enabling the SQL efficiency.

AppName: techfinoffbalance
Pipeline: rskphone
Data Model: rskphone

Pipeline on Unified tenant: https://totvstechfindev.carol.ai/protheusmaisnegunif/carol-ui/pipelines/list/techfinoffbalance/rskphone

Cases impacted: https://console.cloud.google.com/logs/query;query="Error while trying to save pipeline ";cursorTimestamp=2024-01-10T15:24:06.300479120Z;duration=P1D?project=labs-app-mdm-production

I noticed it when enabling the apps “techfinoffbalance“, “techfinmndatasul“ and “winthormaisnegapp“.

https://cloudlogging.app.goo.gl/BQoiZx2oLSMSWb8k8

03 - STEPS TO REPRODUCE (STEP (1...N), VIDEO, SCREENSHOTS, LOGS FOLDER, HEARTBEAT, ETC. – IF IS NOT POSSIBLE TO REPRODUCE EXPLAIN THE REASON):

  1. Carol App released.
  2. Created the unified.
  3. Installed the carol app on unified.
  4. The Tenants don’t have the connection on tenant document pointing to the unified.
    1. This is causing the tenant to do not locate the unified tenant, carol app and pipeline when starting the pipeline.
  5. Generated the SQL Tasks - they are not processing.
The app has several versions.

Most versions have no Unified Tenant defined.

When processing the pipeline Carol is getting the wrong version, that has no unified tenant, generating the error.

We need an app with multiple versions before defining the unified tenant so eventually when running the pipeline Carol will get the version without unified tenant specified.

04 - LINKS (ADD A LINK TO THE BUG OR TO THE TENANT):
05 - EXPECTED BEHAVIOR (LIST THE EXPECTED BEHAVIORS TO CONSIDER THIS BUG AS DONE):

  • Efficiency working for cases described on above steps.
    • Today impacting some apps related to Techfin, one example are the Mais Negócio apps (Protheus, Datasul, Winthor).

Activity

Automation for Jira 5 February 2024, 14:04 Jira Internal Users

This issue was automatically transitioned to WAITING DEPLOY, as its linked QA regression issue has just reached WAITING DEPLOY status (PR was just merged into master branch in Github).

Automation for Jira 31 January 2024, 19:19 Jira Internal Users

This issue was automatically transitioned to WAITING DEPLOY, as its linked QA regression issue has just reached WAITING DEPLOY status (PR was just merged into master branch in Github).

Automation for Jira 31 January 2024, 14:14 Jira Internal Users

This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.

Automation for Jira 31 January 2024, 13:59 Jira Internal Users

This issue was automatically transitioned to TESTED & MERGED, as its PR was just merged into develop branch in Github. PR Approved by rfschroeder.

Automation for Jira 29 January 2024, 12:09 Jira Internal Users

Github user olivandre has just commited and issue was sent back to the REVIEW column.

Douglas Coimbra Lopes 26 January 2024, 14:32 Jira Internal Users

@André Pereira de Oliveira The card has been validated by the QA team. It is only awaiting its turn for the release queue

cc @Jonathan Willian Moraes @Renan Schroeder @Gabriel DAmore Marciano

Douglas Coimbra Lopes 26 January 2024, 14:32 Jira Internal Users

ORCHESTRATOR Generic Optimization DEACTIVATED PASSED

ORCHESTRATOR Generic Optimization DEACTIVATED - > BEING ACTIVATED PASSED

NO ERRORS FOUND

NEW TASK

Automation for Jira 25 January 2024, 19:28 Jira Internal Users

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

Automation for Jira 25 January 2024, 18:43 Jira Internal Users

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

fix: https://totvslabs.atlassian.net/browse/CAPL-5288#icft=CAPL-5288 efficiency not working search tenant app unified

Automation for Jira 25 January 2024, 18:27 Jira Internal Users

@MARCOS STUMPF ,
@Douglas Coimbra Lopes , @André Pereira de Oliveira , @Douglas Coimbra Lopes , @Gabriel DAmore Marciano , @Renan Schroeder

Flag was removed since you have just transitioned the issue status/column.

Automation for Jira 25 January 2024, 14:59 Jira Internal Users

:rotating_light: Bug identificado no :
• A otimizacao generica esta ativa na tenant unificada, porem ao rodar orchestrator na customer, a msma nao eh ativada. (customer nao tem vinculo com a unified)
• GCP Error + Evidencias no cad

Edited on Slack - platform-internal - Douglas Coimbra Lopes

Automation for Jira 25 January 2024, 14:39 Jira Internal Users

:rotating_light: Bug identificado no :
• A otimizacao generica esta ativa na tenant unificada, porem ao rodar orchestrator na customer, a msma nao eh ativada. (customer nao tem vinculo com a unified)
• Detalhes + Evidencias no cad

Sent by Slack - platform-internal - Douglas Coimbra Lopes

Douglas Coimbra Lopes 25 January 2024, 14:39 Jira Internal Users

@André Pereira de Oliveira The pipeline version has Generic Optimization activated, however, when the ORCHESTRATOR SQL Processing task is executed, the Generic Optimization is not being applied to the task

Commit id: 8566690ae2c1a54bedb92bccc3dd07500769e8b2

GCP Error: https://cloudlogging.app.goo.gl/vExqWKFimNAyKDHv8

CURL

curl --location 'https://totvsbabayaga.qarol.ai/api/v3/bigQuery/processQuery?entityTemplateName=mdmproduct&saveBigQuery=true&clearBigQuery=false&saveCds=false&clearCds=false&sendSubscriptions=false&clearRealtime=false&deduplicateResults=true&sendRealtime=true&useDataflow=false&checkExistsDataToProcess=true&createIfRunningOrQueuedTasks=false&fanOut=false&pipelineName=newunifproduct_main&carolAppName=mysqlapp' \
--header 'User-Agent: orchestrator-nodejs/1.0.0' \
--header 'Authorization: df75d74809f94551b8bed53320eaaafa' \
--header 'Content-Type: text/plain' \
--data 'SELECT
stg.id as mdmbarcode,
stg.productname as mdmname,
stg.category as mdmdescription,
stg.productprice as mdmunitvalue,
CONTAINS_SUBSTR(stg.productname, '\''banan'\'') AS mdmDeleted
--metadata-v2--
from stg_nlp_product as stg
where 1=1
--timestamp-- and stg.mdmCounterForEntity > {{start_from}}
--tenantIds-- and mdmTenantId IN {{tenantIds}}'

Douglas Coimbra Lopes 25 January 2024, 14:36 Jira Internal Users

NEW TABLE

Automation for Jira 25 January 2024, 14:18 Jira Internal Users

Bug identificado no
Topico Resolvido :white_check_mark:
• Ao atualizar um parametro para desativar a otimizacao generica, ao rodar task Orchestrator, a plataforma ainda executa com a otimizacao
• Detalhes + evidencias no card

Edited on Slack - platform-internal - Douglas Coimbra Lopes

Automation for Jira 25 January 2024, 14:12 Jira Internal Users

:rotating_light: Bug identificado no
• Ao atualizar um parametro para desativar a otimizacao generica, ao rodar task Orchestrator, a plataforma ainda executa com a otimizacao
• Detalhes + evidencias no card

Sent by Slack - platform-internal - Douglas Coimbra Lopes

Douglas Coimbra Lopes 25 January 2024, 14:07 Jira Internal Users

DEV TENANT ORCHESTRATOR

UPDATING PIPELINE THROUGH CHECKOUT

DM WITH THE GENERIC OPTIMIZATION OFF

Automation for Jira 25 January 2024, 13:01 Jira Internal Users

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

Automation for Jira 24 January 2024, 21:32 Jira Internal Users

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

fix: https://totvslabs.atlassian.net/browse/CAPL-5288#icft=CAPL-5288 efficiency not working search tenant app unified

Automation for Jira 22 January 2024, 19:01 Jira Internal Users

@MARCOS STUMPF ,
@Jonathan Willian Moraes , @André Pereira de Oliveira , @Gabriel DAmore Marciano , @Renan Schroeder

This issue was planned to be delivered until 2024-02-12. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2024-01-23, 2024-02-12

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.

Automation for Jira 16 January 2024, 14:38 Jira Internal Users

@MARCOS STUMPF ,
@Gabriel DAmore Marciano , @André Pereira de Oliveira , @Renan Schroeder

This issue was planned to be delivered until 2024-01-23. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2024-01-23

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.

Automation for Jira 16 January 2024, 13:08 Jira Internal Users

@MARCOS STUMPF ,
@André Pereira de Oliveira , @Renan Schroeder

This issue was planned to be delivered until 2024-02-06. You can check that by consulting the issue in the Due Date field.

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.