As a Dev User I should be able to control the retention for temporary tables
Description
PRDE - Story default text according to the team DoR (Definition of Ready)
01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
02 - PROBLEM (WHAT'S THE CURRENT PROBLEM SCENARIO OR PAIN TO BE RESOLVED?):
Currently, temporary tables are maintained for 14 days by default in BQ and depending on the volume of daily processing in the environment, this can have a high impact on the platform's costs.
03 - GOAL (DESCRIBE THE PROPOSED SOLUTION):
As a Dev Tenant (Tenant Admin in a Dev Tennat) I should be able to define the retention period for temporary tables related to my pipelines.
I should be able to inform on manifest file the number of days I want to preserve retention for temporary tables.
The platform should define the default retention policy to 1 day.
As a Dev User, I can define on the manifest of pipelines the retention policy for Dev Tenant.
When the manifest is updated on Unif it will define the retention policy for all temporary tables related to the pipelines. This should be valid for pipelines running on Unif and pipelines running on the orchestrator.
04 - WHO CAN USE THIS FEATURE (USER ROLES):
Tenant Admin
05 - ASSETS (FIGMA LINKS, RELEVANT DOCUMENTATION LINKS, JSON EXAMPLES, ETC):
06 - ACCEPTANCE CRITERIA:
- Default retention policy when there is no information on manifest of pipelines defined to 7 days.
- The range allowed in the Dev user retention policy is 3 to 14 days to be defined on the manifest file.
- Allow the retention to be overrided on Dev tennat.
- Allow the retention to be overrided on Unif tennat.
- Pipelines running through
processQuery(orchestrator) we should support a new parameter to receive a value for data retention, following the same rule:- We should not return an error if this parameter is out of the boundaries we assume the default value (7).
- min: 3 days. Any value lower we assume 3 days.
- max: 14 days. Any value higher we assume 14 days.
Activity
Details
Priority
Highest
This issue was automatically transitioned to WAITING DEPLOY, as its linked QA regression issue has just reached WAITING DEPLOY status (PR was just merged into master branch in Github).
This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.
This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.
This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.
This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.
This issue was automatically transitioned to TESTED & MERGED, as its PR was just merged into develop branch in Github. PR Approved by olivandre,douglascoimbra.
Github user douglascoimbra has just approved a PR (added as Shard Assignee in this Jira issue).
feat:
CAPL-5200 Done 
CAPL-5268 Done - Adding BigQuery billing project & Table retention on Pipelines
@Geny Isam Hamud Herrera This card has been validated by the QA team. cc @Jonathan Willian Moraes
RETENTION FIELD FROM PIPELINE EXECUTION
MANUAL PROCESS PIPELINE
processQuery
Github user olivandre has just approved a PR (added as Shard Assignee in this Jira issue).
feat:
CAPL-5200 Done 
CAPL-5268 Done - Adding BigQuery billing project & Table retention on Pipelines
@MARCOS STUMPF ,
@Jonathan Willian Moraes , @Geny Isam Hamud Herrera , @André Pereira de Oliveira , @Jonathan Willian Moraes , @Renan Schroeder
This issue was planned to be delivered until 2024-02-12. You can check that by consulting the issue in the Due Date field.
Dates already planned for this issue: 2024-01-23, 2024-02-12
If External Issue Link field is filled, customer was also informed on JIRA TOTVS.
This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.
This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.
feat:
CAPL-5200 Done 
CAPL-5268 Done - Adding BigQuery billing project & Table retention on Pipelines
@MARCOS STUMPF ,
@Geny Isam Hamud Herrera ,
This issue was planned to be delivered until 2024-01-23. You can check that by consulting the issue in the Due Date field.
Dates already planned for this issue: 2024-01-23
If External Issue Link field is filled, customer was also informed on JIRA TOTVS.