[SQL Processing] Error inserting records in BQ with DATETIME columns
Description
DAEN - Bug default text according to the team DoR (Definition of Ready)
01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
@Breno Zipoli Monteiro Papa
02 - PROBLEM (GIVE MORE DETAILS HERE ABOUT SCENARIO OR PAIN TO BE RESOLVED?):
After deploy
CAPL-4724: [Dataflow] Change to STORAGE_API_AT_LEAST_ONCE where we use FILE_LOADSDone
In short, task ends with success, generates valid data but dataflow reports warns and errors while inserting data:
https://cloudlogging.app.goo.gl/Rke9FGXDLRqhyD7Y8
https://cloudlogging.app.goo.gl/ZqovTwENfyAY4DdVA
Red phone reason:
- Data isn’t available on unified tenant.
More details:
https://totvscarol.slack.com/archives/C0199CU1BFD/p1700742267945429
03 - ORGANIZATION / TENANT ACCESS URL (REQUIRED ONLY FOR CAROL):
04 - VERSION (2C OR CAROL):
05 - PRODUCT (WHAT IS THE PRODUCT?):
06 - DATABASE (REQUIRED ONLY FOR 2C - WHAT IS THE CUSTOMER DATABASE?):
07 - FORWARDED INVITATIONS (ENVIRONMENT ADMINS):
08 - APP VERSION:
09 - OBSERVATIONS:
10 - WHAT ADDITIONAL INFORMATION COULD YOU PROVIDE?
11 - STEPS TO REPRODUCE (STEPS 1 TO N INCLUDING SCREENSHOTS, LOGS FOLDER, HEARTBEAT, ETC):
12 - EXPECTED BEHAVIOR (WHAT SHOULD BE THE DEFAULT BEHAVIOR LINKED WITH THE STEPS ABOVE):
Activity
Details
Priority
Highest
This issue was automatically transitioned to DONE after being deployed to production on Github.
No associated issue on JIRA TOTVS BRAZIL.
This issue was automatically transitioned to DONE after being deployed to production on Github.
https://github.com/totvslabs/mdm/releases/tag/v4.89.2
No associated issue on JIRA TOTVS BRAZIL.
This issue was automatically transitioned to WAITING DEPLOY, as its PR was just merged into master branch in Github.
Github user douglascoimbra has just approved a PR (added as Shard Assignee in this Jira issue).
fix: https://totvslabs.atlassian.net/browse/CAPL-5088#icft=CAPL-5088 Fix in SQL Processing to ignore case of field names
Regression & OK + Manual tests Are OK for the branch cc @Gabriel DAmore Marciano @Cindy de Araujo Soares Moore
This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.
:rotating_light: Deixei um cenario encontrado em tenants SQL processingOnly:false ao processarmos tasks de SQL processing qdo a task tem `savetoRealtime:true`
Detalhes no card . Para o BQ nao popula qdo o DM field eh Date Only.
:saluting_face: Jah confirmado que na master nao acontece.
Edited on Slack - platform-internal - Douglas Coimbra Lopes
:rotating_light: Deixei um cenario encontrado em tenants SQL processingOnly:false ao processarmos tasks de SQL processing qdo a task tem `savetoRealtime:true`
Detalhes no card
Sent by Slack - platform-internal - Douglas Coimbra Lopes
Many date formats sent by intake
@Cindy de Araujo Soares Moore When we reprocess the task for SQL processing only false tenants, few fields are being set as NULL on the RT layer
https://totvstevegump.qarol.ai/newtenantdougas/carol-ui/tasks/activity/19ab1e89e4a341fc84626a753f7b63f5?p=1&ps=25&sort=dateUpdated&order=DESC&filters=%5B%7B%22hideInternal%22:%22false%22%7D%5D
@Cindy de Araujo Soares Moore
When we run a SQL Processing with the Query below, the records for RT are getting NULL on Explore, except by the field from the staging PK.
FOR BQ THE DATE ONLY FIELD IS NOT BEING POPULATED
https://github.com/douglascoimbra/test-sample-pipeline/blob/main/kameoproduct.csql
LOOK EDITOR’S EVIDENCE
Message thread link on #red-phone channel:
https://totvscarol.slack.com/archives/C03NT4US9J9/p1701213681608609
@Robson Thanael Poffo ,
@Geny Isam Hamud Herrera ,
This issue was planned to be delivered until 2023-12-11. You can check that by consulting the issue in the Due Date field.
Dates already planned for this issue: 2023-12-11
If External Issue Link field is filled, customer was also informed on JIRA TOTVS.