Data Pipeline for analysis on simulated online store for computer components.
Description
Data Pipeline for analysis on simulated online store for computer components. This service will upload information from multiple sources, store data on a Cloud Data Base, clean and enrich it in order to generate granular indicators and insights on Sales results that will be placed on 3 different platforms: Visualization tool, second Cloud database (different provider) and Rest API to serve external apps.
Child issues
XDP-2
Create Workflow Diagram
Done
XDP-4
Python script to scrape Item descriptions and prices from NewEgg portal and then injecting info on Cloud Data Base
Done
XDP-5
Sign up to Lucid Chart
Done
XDP-6
Create the workflow with all the logos and Icons
In Progress
XDP-7
Code the script
Done
XDP-8
Test the result by sending resultant data frame to Google Sheets
Done
XDP-9
Create instance on GCP - Big Query
Done
XDP-10
Send all data Frames to cloud DB instance (Big Query)
In Progress
XDP-11
Create Rest API to enable external users to consume information from our Data Base
In Progress
XDP-12
Outline the Stament of Work Document
Done
XDP-13
Code Python Script to consume information from Census.gov API
In Progress
XDP-14
Code Python Script to scrape sales data from multiple PDF files, clean info and consolidate results on one single CSV file that will be sent to Google Drive
In Progress
XDP-15
Clean tables on Google Sheets and upload data to Cloud DB as tables by using Python Script
In Progress
XDP-16
Configure Big Query Instance, set up data base Model and perform SQL queries to capture insights
In Progress
XDP-17
Set up Power Bi Instance and create visual Dashboard
To Do
XDP-18
Configure Backup SQL data base on Azure
To Do
XDP-19
Update Portfolio Website
In Progress
XDP-20
Code The Script
Done
XDP-21
Enhance the Notebook by adding text to describe steps
Done
XDP-22
Test the script
Done
XDP-23
Enable connection to Big Query
In Progress
XDP-24
Code the script back bone
In Progress
XDP-25
Expand functionality
To Do
XDP-26
Test API with another script
To Do
XDP-27
Collect all Related Data Frame into G-sheets within Google Drive
To Do
XDP-28
Wrangle and perform action to Clean Tables
To Do
XDP-29
Create Python Script to grab Gsheets from Drive Folder, convert them into pandas Data Frames
Done
XDP-30
Make the script to Establish connection with Big Query and send the data frames as Tables
Done
XDP-31
Create instance on Big Query
Done
XDP-32
Test instance with scripts that are already operational
In Progress
XDP-33
Import data from all established sources
In Progress
XDP-34
Configure Joins and Keys according to ERD,
To Do
XDP-35
Perform SQL statements to find and capture insights to feed all outlets for end system
To Do
XDP-36
Create instance on Azure SQL
Done
XDP-37
Test data base and perform basic SQL operations
To Do
XDP-38
Code Script to extract data from Big Query and load data to Azure Backup DB
To Do
XDP-39
Define set of graphics and visual aids that we'll use
To Do
XDP-40
Invoke Big Query connector and configure pipeline
Done
XDP-41
Test visuals
To Do
XDP-42
Enable Power Bi public portal
Done
XDP-43
Update Website with each partial progress
In Progress
XDP-44
Enhance Descriptions
To Do
XDP-45
Make sure items for portfolio project are displayed on the right order
To Do
XDP-46
Sign up for Confluence
Done
XDP-47
Create Template and link it to Jira Project
Done
XDP-48
Update content and wording
In Progress
XDP-49
Attach Jira's Road Map
Done
XDP-50
Publish document on Website
Done
XDP-51
Code the Script
Done
XDP-52
Enhance Notebook comments
Done
Activity
Show:
Details
Priority
Medium
Created: 30 May 2022, 22:44
Updated:
1 June 2022, 15:55