This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.

This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.

This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.

This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.

This issue was automatically transitioned to REGRESSION, as its PR was just merged into qa branch in Github.

This issue was automatically transitioned to TESTED & MERGED, as its PR was just merged into develop branch in Github. PR Approved by rfschroeder,douglascoimbra.

Github user douglascoimbra has just approved a PR (added as Shard Assignee in this Jira issue).

feat: features to _ingestionDatetime at staging schema

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

@André Pereira de Oliveira Card was revalidated by the QA team, it is pending the code review. cc @Gabriel DAmore Marciano

INGESTION DATE TIME BLOCKED FOR DATA MODELS

SNAPSHOT ALSO BEING BLOCKED

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

feat: features to _ingestionDatetime at staging schema

Github user rfschroeder has just approved a PR (added as Shard Assignee in this Jira issue).

feat: features to _ingestionDatetime at staging schema

Github user douglascoimbra has just approved a PR (added as Shard Assignee in this Jira issue).

feat: features to _ingestionDatetime at staging schema

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

@Gabriel DAmore Marciano @André Pereira de Oliveira This card has been validated by the QA team. It is pending only the code review

@Robson Thanael Poffo ,
@Pedro Buzzi , @André Pereira de Oliveira , @Douglas Coimbra Lopes , @Gabriel DAmore Marciano , @Reinaldo Oliveira Machado Junior

This issue was planned to be delivered until 2024-01-01. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2023-11-20, 2023-12-11, 2024-01-01

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

feat: features to _ingestionDatetime at staging schema

Flag removed

Dev working on the fix

This issue was automatically transitioned to QA REVIEW, as its PR was just approved in Github.

BLOCKING INTAKE FOR FLEXIBLE SCHEMA

BLOCKING STAGING SCHEMA CREATION

BLOCKING UPDATE SCHEMA

BLOCKING SNAPSHOT

UI

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

feat: features to _ingestionDatetime at staging schema

@Robson Thanael Poffo ,
@André Pereira de Oliveira , @Douglas Coimbra Lopes

Flag was removed since you have just transitioned the issue status/column.

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

feat: features to _ingestionDatetime at staging schema

Flag added

Question related to V2 tables are existing _ingestionDatetime column

estou com uma dúvida no card , caso já ocorreu o update da staging schema (ou via intake) adicionando a coluna `_ingestionDatetime`. Qual é o comportamento esperado para os próximos updates?

cc:

Sent by Slack - pm-backend - André Oliveira

@Robson Thanael Poffo ,
@André Pereira de Oliveira , @Douglas Coimbra Lopes

Flag was removed since you have just transitioned the issue status/column.

@André Pereira de Oliveira TETSFAILED

• Scenario: It is possible to create the _ingestionDateTime field on the staging table by updating an existing staging schema.

• Staging on this test: emitente_1

• 
• 

• CURL:

• 1

  2

  3

  4

  5

  6

  7

  8

  9

  10

  11

  12

  13

  14

  15

  16

  17

  18

  19

  20

  21

  22

  23

  24

  25

  26

  27

  28

  29

  30

  31

  32

  33

  34

  35

  36

  37

  38

  39

  40

  41

  42

  43

  44

  45

  46

  47

  48

  49

  50

  51

  52

  53

  54

  55

  curl -X 'PUT' \

    'https://totvsdatastomach.qarol.ai/api/v3/staging/tables/emitente_1/schema?connectorId=71883a8e16644e11959762cd2d4a5f7e' \

    -H 'accept: application/json' \

    -H 'Content-Type: application/json' \

    -H 'Authorization: Bearer 8508544d579742f78674ed8d12429b84' \

    -d '{

    "mdmSchemaMapping": {

      "mdmProperties": {

        "_ingestionDatetime": {

          "mdmType": "date"

        },

        "id": {

          "mdmType": "string",

          "mdmHidden": false

        },

        "name": {

          "mdmType": "string",

          "mdmHidden": false

        },

        "phone": {

          "mdmType": "nested",

          "mdmHidden": false

        }

      },

      "mdmHidden": false,

      "mdmPropertyOrdering": [

        "phone",

        "name",

        "id"

      ]

    },

    "mdmStagingMapping": {

      "properties": {

        "newfield": {

          "type": "string"

        },

        "id": {

          "type": "string"

        },

        "name": {

          "type": "string"

        },

        "phone": {

          "type": "nested"

        }

      }

    },

    "mdmCrosswalkTemplate": {

      "mdmCrossreference": {

        "emitente_1": [

          "id"

        ]

      }

    }

  }'

BLOCKING INGESTION DATETIME ON STAGING SCHEMA CREATION: PASSED

UPDATING STAGING SCHEMA

BLOCKING STAGING SCHEMA UPLOAD SNAPSHOT

This issue was automatically transitioned to REVIEW, as its PR (not DRAFT and not WIP) was just created in Github.

feat: features to _ingestionDatetime at staging schema

@Robson Thanael Poffo ,
@Geny Isam Hamud Herrera ,

This issue was planned to be delivered until 2023-12-11. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2023-11-20, 2023-12-11

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.

@Robson Thanael Poffo ,
@Geny Isam Hamud Herrera ,

This issue was planned to be delivered until 2023-11-20. You can check that by consulting the issue in the Due Date field.

Dates already planned for this issue: 2023-11-20

If External Issue Link field is filled, customer was also informed on JIRA TOTVS.

• In case the request to send data, for any role (except carol admin) send the field _ingestionDatetime, the value should be ignored.

  • Staging table without flexible schema

  • Staging table with flexible schema

@Geny Isam Hamud Herrera , the behavior today is that we just ignore the attributes.

I tested the following flows:

• sent mdmConnector: it was overrided by the default connectorID defined on the request.

• sent mdmConnectorIdpppp: it was ignored instead of create the new field.

So, to avoid any issue by connectors sending data using reserved fields (ignored today), we should keep ignoring them.

Flag added

@Robson Thanael Poffo how do you want us to handle the staging flexible for reserved names. Just ignore the attribute reserved or throw an error message?

Today only two tenant has this field created, the one that belongs to Danilo and the one I explored to validate this scenario:

select *

from (select * from `carol-da1c1280ac9b11e68e25.da1c1280ac9b11e68e250401f8d88501.ingestion_stg_fluigdata_stagingtabledata` where captureDate = (select max(captureDate) from `carol-da1c1280ac9b11e68e25.da1c1280ac9b11e68e250401f8d88501.ingestion_stg_fluigdata_stagingtabledata`)) st, unnest((JSON_EXTRACT_ARRAY(fieldsArray, '$.'))) fieldsArray

where lower(JSON_VALUE(fieldsArray, "$.key")) = lower('_ingestionDatetime')

limit 100