[Dataloss] NullPointer ao tratar exceção do BigQuery no fluxo de streaming
Description
Problema:
Ao tratar uma exceção do BigQuery e adicionar o AuditId, está acontecendo um NullPointerException
java.lang.NullPointerException:
at com.totvslabs.mdm.pipeline.cds.bigquery.BigQueryExtendedErrorHandler.reportStats ( com/totvslabs.mdm.pipeline.cds.bigquery/BigQueryExtendedErrorHandler.java:70 )
at com.totvslabs.mdm.pipeline.cds.bigquery.BigQueryExtendedErrorHandler.processElement ( com/totvslabs.mdm.pipeline.cds.bigquery/BigQueryExtendedErrorHandler.java:64 )https://cloudlogging.app.goo.gl/TbGyJT4S14rTFAEW6
Segue o erro que originou o problema [https://console.cloud.google.com/logs/query;aroundTime=2024-09-23T18:12:12.596Z;cursorTimestamp=2024-09-23T18:11:18.111Z;duration=PT1M;query=resource.type%3D"dataflow_step"
severity%3D"ERROR"
timestamp%3D"2024-09-23T18:11:18.111Z"
insertId%3D"3236755810545597460:164287:0:661401780"?project=labs-app-mdm-production|https://console.cloud.google.com/logs/query;aroundTime=2024-09-23T18:12:12.596Z;cursorTimestamp=2024-09-23T18:11:18.111Z;duration=PT1M;query=resource.type%3D%22dataflow_step%22%0Aseverity%3D%22ERROR%22%0Atimestamp%3D%222024-09-23T18:11:18.111Z%22%0AinsertId%3D%223236755810545597460:164287:0:661401780%22?project=labs-app-mdm-production]
{
"insertId": "3236755810545597460:164287:0:661401780",
"jsonPayload": {
"message": "Error inserting into BigQuery: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=fc99a0caabd541cab3e114dcce16224a, projectId=carol-fc99a0caabd541cab3e1, tableId=ingestion_stg_rm_carol_hr_pffinanc}} - {\"errors\":[{\"debugInfo\":\"\",\"location\":\"_ingestiondatetime\",\"message\":\"Invalid datetime string \\\"2024-09-23T17:47\\\"\",\"reason\":\"invalid\"}],\"index\":113}. Row: {\"mdmDeleted\":true,\"ERP\":null,\"Updated_At\":null,\"Organization\":null,\"CODEVENTO\":null,\"VALORORIGINAL\":null,\"mdmConnectorId\":\"a38c217d3e6b40f4869ffdfac1cc45c2\",\"mdmBatchId\":null,\"mdmLastUpdated\":\"2024-09-23T17:46:50.990\",\"mdmTenantId\":\"351b71bdf8144d1cbce6fbcb3578555c\",\"VERBARESILICAO\":null,\"Tenant\":null,\"ORIGEMMOV\":null,\"ANOCOMP\":null,\"HORA\":null,\"ALTERADOMANUAL\":null,\"VERBAFERIAS\":null,\"CODCOLIGADA\":null,\"NROPERIODO\":null,\"VERBADIFERENCA\":null,\"mdmId\":\"9cd6cb0b55c80f41ed290deb30593174\",\"Created_At\":null,\"mdmBatchIdSequence\":null,\"RECMODIFIEDON\":null,\"mdmCounterForEntity\":1727113610990000,\"mdmCounterForEntity__DATETIME__\":\"2024-09-23T17:46:50.990\",\"MESCOMP\":null,\"DTPAGTO\":null,\"mdmEntityType\":\"a38c217d3e6b40f4869ffdfac1cc45c2_hr_pffinanc\",\"RECMODIFIEDBY\":null,\"mdmIdOrigin\":\"9cd6cb0b55c80f41ed290deb30593174\",\"RECCREATEDBY\":null,\"DISSIDIO\":null,\"mdmAuditId\":null,\"RECCREATEDON\":null,\"REF\":null,\"mdmCreated\":null,\"DISSIDIOALTMANUAL\":null,\"CHAPA\":null,\"VALOR\":null,\"__tenant_id__\":\"fc99a0caabd541cab3e114dcce16224a\",\"__tenant_name__\":\"hrplatformunif\",\"_ingestionDatetime\":\"2024-09-23T17:47\"}",
"logger": "com.totvslabs.mdm.pipeline.cds.bigquery.BigQueryExtendedErrorHandler",
"worker": "cds-prod-mdm1-staging-tec-09201144-yr4d-harness-lg9b",
"job": "2024-09-20_11_44_37-8081062267822863648",
"stage": "P36",
"step": "Save Staging to BigQuery/BigQuery Write Staging (STREAMING_INSERTS) - UNIFIED/Handle Failure (STREAMING_INSERTS) - UNIFIED",
"work": "d05ea615e445603b-2bc622cc18888f8a",
"thread": "16660400"
},
"resource": {
"type": "dataflow_step",
"labels": {
"step_id": "Save Staging to BigQuery/BigQuery Write Staging (STREAMING_INSERTS) - UNIFIED/Handle Failure (STREAMING_INSERTS) - UNIFIED",
"region": "us-central1",
"job_id": "2024-09-20_11_44_37-8081062267822863648",
"job_name": "cds-prod-mdm1-staging-techfin-2-50-0-v5",
"project_id": "labs-app-mdm-production"
}
},
"timestamp": "2024-09-23T18:11:18.111Z",
"severity": "ERROR",
"labels": {
"dataflow.googleapis.com/job_name": "cds-prod-mdm1-staging-techfin-2-50-0-v5",
"compute.googleapis.com/resource_id": "3236755810545597460",
"dataflow.googleapis.com/region": "us-central1",
"compute.googleapis.com/resource_type": "instance",
"dataflow.googleapis.com/job_id": "2024-09-20_11_44_37-8081062267822863648",
"compute.googleapis.com/resource_name": "cds-prod-mdm1-staging-tec-09201144-yr4d-harness-lg9b"
},
"logName": "projects/labs-app-mdm-production/logs/dataflow.googleapis.com%2Fworker",
"receiveTimestamp": "2024-09-23T18:11:21.678241247Z"
}O segundo erro foi causado por falta do mdmAuditId
exception: "java.lang.NullPointerException
at com.totvslabs.mdm.pipeline.cds.bigquery.BigQueryExtendedErrorHandler.reportStats(BigQueryExtendedErrorHandler.java:70)
at com.totvslabs.mdm.pipeline.cds.bigquery.BigQueryExtendedErrorHandler.processElement(BigQueryExtendedErrorHandler.java:64)
"AC:
- Não lançar exceção por conta da falta de mdmAuditId no fluxo de registro de erros de inserção no BQ.
Activity
Show:
Details
Priority
HighestAssignee
Renan Fernando Schroeder
Reporter
Rodrigo Bechtold
Labels
PLANNED
Due Date
18 October 2024
Fix versions
CAPL_4.09
Components
CONNECTOR | DATA_INGESTION
Created
23 September 2024, 16:32
Updated
23 October 2024, 17:00
More fields
Created: 23 September 2024, 21:32
Updated:
23 October 2024, 22:00
esses casos se parecem com o que foi relatado em . Verifiquei erros com registros que estão sendo excluídos (`mdmDeleted=true`) e são inserções na unificada que estão dando erro.
Mensagem enviada pelo Slack - gcp-error-reporting - Cindy Soares
@Rodrigo Bechtold ,
@Cindy de Araujo Soares Moore , @Renan Fernando Schroeder ,
Este issue foi planejada para ser entregue até 2024-10-18. Você pode confirmar consultando o campo Due Date desta issue.
Data já planejadas para esta issue: 2024-10-18
Se o campo External Issue Link estiver preenchido com o link de uma issue válida no Jira Produção o cliente também será notificado no Jira Produção.