Person of contact

• @Bruno Furtado

Story briefing

As a tenant admin, I would like to dispatch the data from the SQL process with a little more speed. Today the query is running fast but the dispatch process is taking to much time.

Problem

SQL process is taking to much time to dispatch data (BQ + RT layers).

For example, here is a task: https://totvstechfindev.carol.ai/meupostobycarol/carol-ui/tasks/activity/39b15165acd34722a1d7eb3f0dc92a9b

• The query was executed in less than 1 minute.
• The dispatch step took more than 10 minutes.
• The task took about 15 minutes, but only 1 minute was SQL processing.

Another example from Bom Futuro:

Task: https://global.carol.ai/bomfuturo/carol-ui/tasks/activity/60b0dc205c45492bb435a989240490a9?p=1&ps=25&sort=dateCreated&order=DESC&filters=%5B%7B%22hideInternal%22:%22false%22%7D,%7B%22taskType%22:%5B%22BIGQUERY_PROCESS_DATA%22%5D%7D,%7B%22dataModelName%22:%5B%22clockinrecords%22%5D%7D%5D

Stackdriver:https://cloudlogging.app.goo.gl/fzUeZiJ5yTM1Nkgu7

https://cloudlogging.app.goo.gl/R9QSoJhE2vVT6NEYA

Who can use this feature

Tenant admin.

Acceptance criteria

• We should start the Dataflow or cancel the task after 30 minutes waiting for Dataflow to start.
  • If the stackdriver log is related, it seems the task was on waiting mode for another job that already started.
• More on instructions