SQL processing tasks with Efficiency enabled are not finding pipeline related to the Carol App
Description
Person of contact
- @Bruno Furtado
- @Robson Thanael Poffo
Problem
The SQL processing tasks launched by the orchestrator with the data validation flag enabled are unable to find the Carol App (and the latest version of the app is installed on the tenant). Tasks print an error log and remain running, not finishing data processing.
The impact is that we have had customers since October 17th without viewing new records in the Data Model.
Steps to reproduce
- Orchestrator launch the SQL process task with data validation flag enabled and the right app name (clockinapp).
- Customer tenant receive this task, start to process the data but at the final step it prints that was not possible to find the carol app clockinapp. The app is installed there.
- All these data were not sent to Data Model.
Links
- Task with error: https://fazendasbartira.carol.ai/fazendasbartirap/carol-ui/tasks/activity/0c2cb5a6b5014d59aeaace3a81ef5cef
- Dev tenant with the Carol App: https://clockin.carol.ai/clockin/carol-ui/carol-app-dev/b3482829b21e46cd996e9c40e90148b6/overview
- Unified tenant: https://clockin.carol.ai/clockinunif/
- Pipelines manifest: https://github.com/totvslabs/sql-pipelines-clockin/blob/main/pipelines.json
- List of environments with the problem: https://cloudlogging.app.goo.gl/7SK1ZipEt9ifAvuy9
More details
- The manifest is right (DM is there with the right staging tables).
- Customer is using the latest version of the Carol App (clockinapp, 12.1.23090011).
- SQL proccess was running well until we enable the sql data validation.
- It is impacting all the SQL process are using data validation.
- There are 6 envs with this problem, all of them are not linked to unified.
- When we check the mdmUnifiedTenantsMapping, it is empty for all of the envs.
- I tried to link but I receive a 400 with success false and I believe it is happening because of that:
Expected behavior
- Run the SQL process with the efficiency enabled without problems.
- Tenants have the requirement filled:
- Associated with the carol app
- The SQL Task has the carol app name defined
- The SQL Task has the pipeline name defined
- The SQL task has the flag to check for new data.
- After the fix, make sure all the tenants impacted are running the processes well.
- Tenants have the requirement filled:
- In case of any problem getting the pipelines, we should register an warning and run the task without efficiency enabled.
Activity
Show:
Details
Priority
HighestAssignee
Emerson Venancio
Reporter
Bruno Furtado
Labels
CAROL_BACKLOG_REVIEWRED_PHONE
Due Date
11 December 2023
Fix versions
CAPL_3.96
Components
SQL_PIPELINES | DATA_PROCESSING
Created
20 November 2023, 13:40
Updated
21 December 2023, 14:41
More fields
Created: 20 November 2023, 17:40
Updated:
21 December 2023, 18:41
This issue was automatically transitioned to DONE after being deployed to production on Github.
https://github.com/totvslabs/mdm/releases/tag/v4.86.0
No associated issue on JIRA TOTVS BRAZIL.
This issue was automatically transitioned to WAITING DEPLOY, as its PR was just merged into master branch in Github.
Github user douglascoimbra has just approved a PR (added as Shard Assignee in this Jira issue).
fix: Getting pipeline from unified type
TOPIC PASSED
TESTING A PIPELINE OUT OF THE SQL OPTIMIZATION PASSING THE PIPELINE NAME FROM THE SQL OPTIMIZATION APP
NO ERRORS ON THE SQL PROCESSING TASK
This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.
This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.
fix: Getting pipeline from unified type
@Bruno Furtado ,
@Geny Isam Hamud Herrera , @Emerson Venancio ,
This issue was planned to be delivered until 2023-12-11. You can check that by consulting the issue in the Due Date field.
Dates already planned for this issue: 2023-11-20, 2023-12-11
If External Issue Link field is filled, customer was also informed on JIRA TOTVS.
Apenas reportando que dos 6 ambientes impactados, apenas 1 era produtivo e a situação dele já foi resolvida: retiramos o ambiente de modo dev, linkamos ao unificado e o reprocessamento funcionou conforme o esperado.
Quanto aos demais casos que não são produtivos, vamos deixar de usar o processamento otimizado neste momento enquanto as decisões sobre o melhor caminho a ser tomado para essa task são tomadas (talvez processar de forma não otimizada e alertar que a otimização na task).
Message thread link on #red-phone channel:
https://totvscarol.slack.com/archives/C03NT4US9J9/p1700498753224719