Issue Component(s) on Oct 25, 2023, 12:09:53 PM: CONNECTOR | STAGING_TABLE

PRDE - Story default text according to the team DoR (Definition of Ready)

01 - PERSON OF CONTACT (PERSON THAT CAN ANSWER QUESTIONS ABOUT THE PROBLEM):
02 - PROBLEM (WHAT'S THE CURRENT PROBLEM SCENARIO OR PAIN TO BE RESOLVED?):

• Adiciona atributo numa staging table: nome valor tipo string.
• Esperado atributo aparecer na tabela
• Enviamos dados para o atributo.
• Altera schema em uma staging table (atributo valor tipo string => float) pela UI:
  • Esperado:
    • carol remove atributo string
    • carol adiciona atributo float
    • Atributo sem valor definido.
• Carol app altera schema de um atributo:
  • Esperado:
    • Carol remove o atributo string
    • Carol adiciona o atributo float
    • Atributo sem valor definido.

Aparentemente não estamos aplicando esse ajuste na instalação do Carol App.

Questions:

• Shouldn’t the staging fields have the type as a suffix (name@type)?
• Why do we have these two types:
  • mdmSchemaMapping
    • mdmProperties
  • mdmStagingMapping
    • properties
  • mdmPropertyOrdening
• Cases:
  • Avro with the field as string.

03 - STEPS

SCENARIO 1: Updating Staging table

• Create a staging table with a field named as example with a specific type
• Once the staging table field is ready, from UI, or Endpoint, remove the current field and create another one with the same name, but a different type
• Check the notification schema table task failing:
• 
  

• GCP:

SCENARIO 2: Updating Staging table Carol App update

• Create a carol app and
• Create a staging table with a field named as example with a specific type
• Release approve and install the carol app in another tenant
• Create a new carol app version
• From UI, or Endpoint, remove the current field
• create another one with the same name, but a different type
• Release the update carol app version
• Install the new version
• Check the notification schema table task failing:

04 - WHO CAN USE THIS FEATURE (USER ROLES):
05 - ASSETS (FIGMA LINKS, RELEVANT DOCUMENTATION LINKS, JSON EXAMPLES, ETC):
06 - ACCEPTANCE CRITERIA:

Change schema Data should run a drop column for the old field type and create the new field with the new type.

• Validate flows changing the field type on the UI and through Carol App installation.
• That will run:
  • Drop column.
  • Create column.
  • If something fails, they will recreate the table.
  • Everything happens on the same task.
• Add log for all steps.
  • All logs should have full context since there is no guarantee sequence when previewing the logs.
  • Example: dropping the column NAME from table XPTO because origin field type is STRING and the new field type is LONG.
  • Creating new field NAME on table XPTO with type FLOAT.