Financial Data Engine - Crawler, Storage, Processing, Data Analysis, GUI

This application will crawl several financial data websites every 6 hours: 1) [login to view URL] 2) [login to view URL] 3) [login to view URL] Scraping program should be written In Python, probably Scrapy or similar. Data to be scraped will be organised as: A) Providers - various types of financial institutions that will have various characteristics including a list of products as well as various other fields describing the company B) Products - there are several different categories of products. There are also lots of fields associated with each product, and there will be several products sold by each provider in each category. A freelancer will to design and implement the MySQL Schema for all of the data above, including all of the various fields to match the scraped data as well as the relations between the tables. This will be based on the format and relations in the data from the three sources above, and will have input from me. The raw data will be put through several programs: 1) Check the scraped data has worked correctly (specific format for each field) 2) If error, then the product (and the changed field) will be listed in an 'Errors' list in the GUI for the user to check the scraper and adjust inputs to fix 3) If the value has changed then the product (and changed field) will be listed in a 'Changes' list in the GUI for the user to track changes in data scraped GUI with 4 x different summary tables, each table to have search, filter and sort functions, and 2 x summary pages (product and provider): 1) List of providers and some summary details 2) List of products and some summary details 3) List of changes to the data since the last scrape (not scraper errors) 4) List of potential scraper errors to be resolved (data error or scraped value much different format to the last scrape based on rules for each field) GUI to have a number of functions that perform several database queries and return the results (if any) to one of the summary pages: 1) List of providers and a set of specified summary details 2) List of products and a set of specified summary details 3) List of changes to the data since the last scrape (not scraper errors) 4) List of potential unresolved errors in recent scrapes (not just changes in field value) Engine to create a fourth set of data recording every provider and product: - Three data sources to be triangulated based on specific criteria that I will specify - Functionality to change source of data for each product or provider