API returns timeout when trying to enable short URL authentication

Description

Contact

  • @Bruno Furtado

Problem

When an user tries to update the tenant to enable the short URL authentication, API returns timeout:

  • Organization/tenant: icomon/icomon
curl -X 'PUT' \
  'https://icomon.carol.ai/api/v3/admin/tenants/f2c73dce2636428688b803a3ef729331' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer TOKEN' \
  -d '{ "mdmCdsShortURLAuthentication": true }'

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>

The timeout happens but the feature is enabled. Maybe that's why we are receiving several error logs.

Even disabling the flag, errors still happening.

Notes

It only happens when tenant was a lot of URLs (we tested is small environments and it works well):

  • Organization/tenant: furtado/furtadoclockin
curl -X 'PUT' \
  'https://furtado.carol.ai/api/v3/admin/tenants/2510054dfd434767b443e6315315b4d0' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer TOKEN' \
  -d '{ "mdmCdsShortURLAuthentication": true }'

{
  "success": true
}

Links

Regression link:

Expected behavior

  1. When there are many URLs, the platform must enable authentication without generating errors.

Activity

Automation for Jira 20 March 2024, 13:24 Jira Internal Users

A causa do problema foi identificada, sendo que as tasks `CAROL_PIPELINES_EXECUTION_SUMMARY` receberam `ACCESS DENIED` ao lançarem os jobs de BigQuery para calcular a unicidade de registros por Batch Id + Data Model.

O problema ocorreu devido a uma mudança recente de projeto GCP que seria executado a query de consolidação de Summary no BigQuery pela variável de ambiente `BIGQUERY_OBSERVABILITY_PROJECT_ID='labs-techfin-production'`, que entrou em produção no dia 18/03, junto ao deploy do card .

O time de SRE já realizou a adequação de permissão na service account do projeto GCP utilizado em produção (`labs-techfin-production`) e todas as tasks voltaram a executar de forma correta, sem necessidade de intervenção de código e também sem maiores impactos, exceto o atraso da entrega dos eventos de Summary para alguns batchs.

Mensagem enviada pelo Slack - plataforma-red-phone - Rodrigo Bechtold

MARCOS STUMPF 18 March 2024, 15:12 Jira Internal Users

Olá@Bruno Furtado bom dia, deploy realizado hoje.

Bruno Furtado 15 March 2024, 16:35 Jira Internal Users

Pessoal, entendo que este item deve subir no próximo deploy, correto?

Houve alguma mudança no endpoint (na forma com que chamamos ou recebemos os dados)? Apenas tentando antecipar o tema e orientar o time Clockin.

Obrigado pela atuação e desculpem o incomodo.

Automation for Jira 13 March 2024, 14:26 Jira Internal Users

This issue was automatically transitioned to TESTED & MERGED, as its PR was just merged into develop branch in Github. PR Approved by glaucioscheibel,douglascoimbra.

Automation for Jira 13 March 2024, 14:25 Jira Internal Users

Github user douglascoimbra has just approved a PR (added as Shard Assignee in this Jira issue).

feat: https://totvslabs.atlassian.net/browse/CAPL-5419#icft=CAPL-5419 - ShortURL Authentication processed by task with rate limit

Automation for Jira 11 March 2024, 13:36 Jira Internal Users

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

Automation for Jira 11 March 2024, 13:21 Jira Internal Users

Github user rodrigo-bechtold has just commited and issue was sent back to the REVIEW column.

Bruno Furtado 11 March 2024, 12:08 Jira Internal Users

Olá pessoal, temos uma previsão de data para o deploy dessa alteração?

Moises Jose Soares Filho 8 March 2024, 19:26 Jira Internal Users

@Rodrigo Bechtold Regression validated by the QA team.

cc @Gabriel DAmore Marciano @Douglas Coimbra Lopes

Automation for Jira 4 March 2024, 17:07 Jira Internal Users

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

Automation for Jira 4 March 2024, 16:10 Jira Internal Users

Github user rodrigo-bechtold has just commited and issue was sent back to the REVIEW column.

Automation for Jira 4 March 2024, 14:53 Jira Internal Users

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

Automation for Jira 1 March 2024, 19:01 Jira Internal Users

@MARCOS STUMPF ,
@Pedro Buzzi , @Rodrigo Bechtold , @André Pereira de Oliveira , @Lucas Noetzold

This issue was planned to be delivered until 2024-03-25. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2024-03-01, 2024-03-25

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.

Douglas Coimbra Lopes 1 March 2024, 14:51 Jira Internal Users

@Rodrigo Bechtold Card validated by the QA team. It is pending only the code review.

cc @Gabriel DAmore Marciano

Douglas Coimbra Lopes 1 March 2024, 13:28 Jira Internal Users

RETESTING FIX

NO ERRORS ON GCP:

PUBLIC ACCESS REMOVED

NEW TENANT AFTER REDEPLOY WITH URL FILES

Automation for Jira 1 March 2024, 02:36 Jira Internal Users

Github user rodrigo-bechtold has just commited and issue was sent back to the REVIEW column.

Douglas Coimbra Lopes 29 February 2024, 21:23 Jira Internal Users

@Rodrigo Bechtold The retest has been performed but now the Task is not being created for the tenant with the parameter true

SANDBOX: https://totvsdragonmaster.qarol.ai/newtenatdoug/carol-ui/explore/search/mdmproduct/rt

SANDBOX REPLOYED

GLOBAL USER RUNNING THE MIGRATION

Automation for Jira 29 February 2024, 19:47 Jira Internal Users

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

Automation for Jira 29 February 2024, 13:31 Jira Internal Users

Adiantando um tema de parking lot voltado ao card .
• Uma tenant eh criada e tem o parametro `"mdmCdsShortURLAuthentication": true`
• Roda-se o Update Migration pontuado no card
• O Public Access eh atualizado na base como false na tabela ShortURL, porem o parametro da tenant continua como TRUE. Esse cenario eh esperado, tendo em vista que a plataforma nao suporta atualizar o parametro para false correto?

Edited on Slack - platform-internal - Douglas Coimbra Lopes

Automation for Jira 29 February 2024, 13:31 Jira Internal Users

Adiantando um tema de parking lot voltado ao card .
• Uma tenant eh criada e tem o parametro `"mdmCdsShortURLAuthentication": true`
• Roda-se o Update Migration pontuado no card

Sent by Slack - platform-internal - Douglas Coimbra Lopes

Automation for Jira 29 February 2024, 12:26 Jira Internal Users

:rotating_light: Cenario encontrado no .
• Ao rodar o endpoint de migration, as tenants continuam com parametero de ShortURL Auth configurado como true
• Evidencias e passos no card

Sent by Slack - platform-internal - Douglas Coimbra Lopes

Douglas Coimbra Lopes 29 February 2024, 12:23 Jira Internal Users

@Rodrigo Bechtold

  1. When The SHORTURLmigration endpoint is executed, the tenants are not getting their ShortURL parameter to false.

  2. The Remove SHortURL access task is failing with NPE: https://cloudlogging.app.goo.gl/Rs7sQBWzUpkqmUe66

SANDBOX TASK: https://totvsybagems.qarol.ai/newtenantdoug/carol-ui/tasks/activity/715b896afad34f4dbc4c78aea7e017cc?p=1&amp;ps=25&amp;sort=dateUpdated&amp;order=DESC&amp;filters=%5B%7B%22hideInternal%22:%22false%22%7D%5D

  • Steps:

  • as CarolAdmin/ OrgAdmin, create a tenant

  • as CarolAdmin/ OrgAdmin, update the tenant to be "mdmCdsShortURLAuthentication": true

  • Try to Run the Migration endpoint

  • 1

    2

    3

    4

    5

    curl -X 'POST' \
      'https://globalybagems.qarol.ai/api/v3/admin/migration?migrationName=Update%20ShortURL%20public%20access&featureLevel=ALL&updateLevel=false' \
      -H 'accept: application/json' \
      -H 'Authorization: Bearer cc3d25c9b9804391aa71410cbfe36150' \
      -d ''
  • Check the tenant that is still with the parameter as false

TENANT WITH THE IMAGE

TENANT BEING UPDATED BY CAROL ADMIN WITH PARAMETER TRUE

Automation for Jira 28 February 2024, 19:46 Jira Internal Users

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

feat: https://totvslabs.atlassian.net/browse/CAPL-5419#icft=CAPL-5419 - ShortURL Authentication processed by task with rate limit

Automation for Jira 12 February 2024, 19:19 Jira Internal Users

@MARCOS STUMPF ,
@Gabriel DAmore Marciano , @Lucas Noetzold

This issue was planned to be delivered until 2024-03-04. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2024-03-04

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.