Today about 70% of our SQL Processing tasks are not processing data, the reason is because the staging area did not receive records to be processed. Study available here.

We can optimize the processing by allowing senders (like Smart Link) to notify Carol that a batch of data has been completely sent to Carol and Carol can start processing the data.

Definitions:
Diagram result: link to diagram

To restrict pipelines to run only after a batch submission is completed, the pipeline manifest must be changed adding the attribute useBatchNotification = true, at root of JSON.
New tables:

• intake_batch_staging_request: information about intake request with batch;
• intake_batch_summary: store batch summary information and last status;
• intake_batch_summary_status_history: history of summary status;
• pipeline: stores information of pipeline to fast retrieval;
• pipeline_staging: pipeline connector and stagings;
• pipeline_process: bridge table to connect: batch, pipeline and taks;