[Fix Staging] Query endpoint is bringing records events from deleted staging tables
Description
PRDE - Bug default text according to the team DoR (Definition of Ready)
01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
02 - PROBLEM (WHAT'S THE ISSUE?):
- We created a staging table and send records to it.
- A few minutes later the connector was deleted with the staging.
- After that, We’ve created the same connector and staging tables.
- It seems to BE is failing on call DRP TABLE METHOD FROM BQ.
- GCP Query: [https://console.cloud.google.com/logs/query;cursorTimestamp=2022-12-19T21:21:38.918921261Z;query=resource.labels.namespace_name%3D"sandbox-xmas"
timestamp%3D"2022-12-19T21:21:38.804646761Z"
insertId%3D"q2fzwgmfcj2evd8v";summaryFields=:false:32:beginning;timeRange=2022-12-19T21:21:30.689Z%2F2022-12-19T21:22:00.689Z?project=labs-app-mdm-qa|https://console.cloud.google.com/logs/query;cursorTimestamp=2022-12-19T21:21:38.918921261Z;query=resource.labels.namespace_name%3D%22sandbox-xmas%22%0Atimestamp%3D%222022-12-19T21:21:38.804646761Z%22%0AinsertId%3D%22q2fzwgmfcj2evd8v%22;summaryFields=:false:32:beginning;timeRange=2022-12-19T21:21:30.689Z%2F2022-12-19T21:22:00.689Z?project=labs-app-mdm-qa]
Once I access the staging, We can see the previous events with the records in the staging: https://totvscypress.qarol.ai/totvslabs/carol-ui/connectors/sql_1/cypresstable?p=1&ps=25&order=ASC&filters=%5B%7B%22consolidation%22:%22y%22%7D%5D
Pay attention the ingestion table does not show any record. But the query pooling does:
https://totvscypress.qarol.ai/api/v1/bigQuery/query_polling
Ingestion table
03 - STEPS TO REPRODUCE (STEP (1...N), VIDEO, SCREENSHOTS, LOGS FOLDER, HEARTBEAT, ETC. – IF IS NOT POSSIBLE TO REPRODUCE EXPLAIN THE REASON):
- Create a connector (Name for this test: sql)
- Create a staging table (for this test: cypresstable)
- Send records to it
- After a few minutes, delete the connector
- Once the connector is deleted, create the connector and staging using the same previous names
- Check the /
query
endpoint showing previous events
04 - LINKS (ADD A LINK TO THE BUG OR TO THE TENANT):
05 - EXPECTED BEHAVIOR (LIST THE EXPECTED BEHAVIORS TO CONSIDER THIS BUG AS DONE):
- Once the connector is deleted, the staging data/events must also be deleted.