SQL processing optimization is not being active for pipelines that do not produce new records on first execution
Description
CAPL - Bug default text according to the team DoR (Definition of Ready)
01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
@Renan Schroeder
02 - PROBLEM (WHAT'S THE ISSUE?):
SQL processing optimization is not being active for pipelines that do not produce new records. In some cases there will be some staging tables that has low frequency of intake records received in Carol, or even an unique intake received for static sources, that isn't updated in source anymore.
In these cases, Carol will not enable SQL processing optimization because there is an inconsistency in the metrics check for a given DataFlow job, preventing the pipeline execution from being marked in the pipeline_execution table.
03 - STEPS TO REPRODUCE (STEP (1...N), VIDEO, SCREENSHOTS, LOGS FOLDER, HEARTBEAT, ETC. – IF IS NOT POSSIBLE TO REPRODUCE EXPLAIN THE REASON):
- Run a SQL processing task to a pipeline that never received a record (or create a pipeline with cron expression to run tasks in a given frequency of time - ex: each 5 min).
- After task conclusion, check if a new record was inserted in
pipeline_execution. It won’t create a new record. - When the second task runs, notice that there will not be a Task Log warning whether or not there are new records to process, indicating that optimization is not yet enabled.
04 - LINKS (ADD A LINK TO THE BUG OR TO THE TENANT):
05 - EXPECTED BEHAVIOR (LIST THE EXPECTED BEHAVIORS TO CONSIDER THIS BUG AS DONE):
- After SQL processing task conclusion, a new record need to be inserted in
pipeline_execution, even no records was processed by pipeline. - When the second task runs, notice that there will be a Task Log warning whether or not there are new records to process, indicating that optimization is already enabled.
Activity
Details
Priority
Highest
This issue was automatically transitioned to DONE after being deployed to production on Github.
https://github.com/totvslabs/mdm/releases/tag/v4.86.1
No associated issue on JIRA TOTVS BRAZIL.
This issue was automatically transitioned to WAITING DEPLOY, as its PR was just merged into master branch in Github.
Github user douglascoimbra has just approved a PR (added as Shard Assignee in this Jira issue).
fix: https://totvslabs.atlassian.net/browse/CAPL-5058#icft=CAPL-5058 SQL processing optimization is not being active for pipelines that do not produce new records
This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.
@Renan Schroeder The card has been validated by the QA team. It is pending only the code review. cc @Gabriel DAmore Marciano @Geny Isam Hamud Herrera
A SINGLE INTAKE REQUEST HAS BEEN MADE:
AWAITING THE NEXT TASK TO BE REPROCESSED
SQL OPTIMIZATION WORKING ON THE NEXT TASK
UNIFIED + ORCHESTRATOR SCENARIOS OK
This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.
fix: https://totvslabs.atlassian.net/browse/CAPL-5058#icft=CAPL-5058 SQL processing optimization is not being active for pipelines that do not produce new records
This issue was automatically transitioned to IN PROGRESS, as its branch was just created in Github.
CAPL-5058-SQL-processing-optimization-for-low-frequency-stagings-intake
@Renan Schroeder ,
@Gabriel DAmore Marciano ,
@Geny Isam Hamud Herrera ,
This issue was planned to be delivered until 2023-12-11. You can check that by consulting the issue in the Due Date field.
Dates already planned for this issue: 2023-12-11
If External Issue Link field is filled, customer was also informed on JIRA TOTVS.
Message thread link on #red-phone channel:
https://totvscarol.slack.com/archives/C03NT4US9J9/p1700575163286289