Response Code 502 in query-polling API requests
Description
Problem
The Job is finishing quickly, but data endpoint keeps returning in execution (and eventually it returns 502).
Possibly related to queries returning no data?
When I run from explore (global/fluigdata) it works OK:
https://totvstechfindev.carol.ai/rhinsightsappdevelopment/carol-ui/data-subscription
Explore freezes when the query returns no data:
PRDE - Bug default text according to the team DoR (Definition of Ready)
01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
@Breno Zipoli Monteiro Papa
02 - PROBLEM (WHAT'S THE ISSUE?):
Often the endpoint below is returning code 502 Bad Gateway, making the Data Subscription dashboard return error to the final user.
curl 'https://totvstechfindev.carol.ai/api-ui/sql-bq/query-polling' \
-H 'accept: application/json, text/plain, /' \
-H 'authorization: xxxxxxx' \
-H 'content-type: application/json' \
--data-raw '{"queryId":"carol-8cd6e43115e9416eb236-explorer-3143507e-185a-4699-8e91-6f6741659a70","page":1}'If we wait a little and try this API request on terminal, the result is OK:
{{{"schema":
{"nAckedCount":"INTEGER","responseMessage":"STRING"},"rows":[],"totalRows":0,"totalRowsPage":0,"lastPage":true,"pageSize":999,"elapsedTimeInSeconds":172.732,"processedMegabytes":251.33,"cache":false}}}
This is not a query performance issue:
carol-8cd6e43115e9416eb236-explorer-3143507e-185a-4699-8e91-6f6741659a70:
This job was executed in 5 seconds of slot time. But took 3 minutes to start executing and this long wait time is what causes the polling to take long.
carol-8cd6e43115e9416eb236-explorer-eba0b382-3eb1-4304-8b4d-0c6551ac5961:
Same scenario.
03 - STEPS TO REPRODUCE (STEP (1...N), VIDEO, SCREENSHOTS, LOGS FOLDER, HEARTBEAT, ETC. – IF IS NOT POSSIBLE TO REPRODUCE EXPLAIN THE REASON):
Try to use the data subscription stats dashboard on any tenant with various filters until error ocurs.
04 - LINKS (ADD A LINK TO THE BUG OR TO THE TENANT):
Test tenant was: https://totvstechfindev.carol.ai/rhinsightsappdevelopment/carol-ui/data-subscription
05 - EXPECTED BEHAVIOR (LIST THE EXPECTED BEHAVIORS TO CONSIDER THIS BUG AS DONE):
Dashboard not breaking if query-polling returns bad gateway or poll less so the error does not happen.
Activity
Details
Priority
Highest
@Robson Thanael Poffo ,
@Pedro Buzzi ,
This issue was planned to be delivered until 2024-01-01. You can check that by consulting the issue in the Due Date field.
Dates already planned for this issue: 2023-12-11, 2024-01-01
If External Issue Link field is filled, customer was also informed on JIRA TOTVS.
@Robson Thanael Poffo ,
@Pedro Buzzi ,
You have just moved the JIRA TOTVS issue https://jiraproducao.totvs.com.br/browse/DLABSCAPL-3660 to the Customer responsability.
Foi feita uma melhoria na alocação de recursos para suportar uma maior demanda de query-polling. ("query-polling” é uma operação realizada todas as vezes que uma query está pendente)
#customer
Problem solved with PR in Infra:
https://github.com/totvslabs/infra/pull/7149
In high demanding moments, the service were not supporting the amount of requests.
I also talked to FE team to decrease the query polling rate when login into Carol
@Robson Thanael Poffo ,
@Pedro Buzzi ,
You have just sent your prior comment as an information to the customer in JIRA TOTVS issue https://jiraproducao.totvs.com.br/browse/DLABSCAPL-3660.
#inform
Informamos que houve uma movimentação interna aqui no TOTVS LABS e que esta ação pode ter solucionado total ou parcialmente o item reportado nesta issue. Pedimos que valide novamente e que, em caso de dúvidas, alinhe com o seu ponto focal aqui no TOTVS LABS (Epic Owner: Eduardo Jacometti De Souza).
@Robson Thanael Poffo ,
@Geny Isam Hamud Herrera ,
Comment sent to JIRA TOTVS Brazil - https://jiraproducao.totvs.com.br/browse/DLABSCAPL-3660:
Informamos que esta issue foi incluída em uma sprint com previsão de entrega até 2023-12-11 (podendo ocorrer retorno antes). Não é necessário retornar esta interação, caso não tenha dúvidas. Qualquer alteração de planejamento será novamente notificada nesta issue.
Atenciosamente,
Equipe TOTVS LABS.
@Robson Thanael Poffo ,
@Geny Isam Hamud Herrera ,
This issue was planned to be delivered until 2023-12-11. You can check that by consulting the issue in the Due Date field.
Dates already planned for this issue: 2023-12-11
If External Issue Link field is filled, customer was also informed on JIRA TOTVS.