Crawler to extract keywords and more, from a website

Executive summary: We need a script to extract all the text as ords from given websites, sort those keywords in ascending order, remove some irrelevant ones such as 'and', 'the', and associate them with the sentence that those keywords are mentioned in. - We're a company, developing chatbots used in customer service, answering their questions on their website or mobile apps. Think of "Siri for customer service". - Customers, especially smaller ones usually don't want to dedicate time to create questions and answers to their chatbots, so we have a high barrier of entry in terms of setup. It's not easy to sell expecting them to dedicate time to start using their chatbot. We want to overcome this hurdle by developing an automated content crawler. "Content" is basically most frequently used keywords and important sentences that contain these keywords. (No need to create questions, as keywords and sentences associated with them are the building blocks of a chatbot) So, we essentially need a snapshot of the website in terms of what the website is all about. Phases are: 1- A crawler to crawl plain-text-based content from various websites submitted via a form (domain and subdomains) 2- A frequency analysis to list most common keywords in ascending order. (We need a list of stop words (irrelevant words such as 'the', 'a' etc.) so that we can strip them off before concluding the keyword list. 3- List keywords and the sentences that contain those keywords. We might need to remove from this list some entries based on some rules that may make them irrelevant. We'll figure out those rules as we see some results. The remaining content will be the essence of our project. 4- Based on experience so far, we might develop a new phase to further fine tune results. Our goal is to show the finalized content to website owner and let them say "this exactly covers every bit of information about my business" 5- We'll feed those keywords and sentences (answers to those keywords) to a chatbot database. 6- Last but the not least, we'd like to feed a list of websites to execute previous 5 steps automatically, and create the same output for all of them. If you can provide a really simple proof of concept, I could accept your bid the same moment. We need to see that the results generated from this project will be helpful for us to develop chatbots and increase sales. Oppositely, a copy-paste message will not get you the job. It demonstrates that you're a serial bidder, not someone who is willing to go the extra mile to solve a real problem for a business.