A bot/spider to crawl the web for flash and non-mobile optimized websites

$30-250 USD

クローズ

投稿日:

8年近く前

$30-250 USD

完了時にお支払い

We'd like a spider/bot to crawl the web for old sites that need an update. So we'd like the spider to crawl and filter sites according to some criteria: - 100% flash sites - Sites that don't have a mobile-friendly version (maybe looking for viewport media queries in css? ) - Sites that have an old date on the footer , like © 2014 (or lower, to be configurable) IT should output the results to a spreadsheet. So the results should look like: [login to view URL] Flash [login to view URL] No mobile version [login to view URL] older than XXXX Some important things: - Each of the three options should toggle on/off - This should be cascading, meaning first it checks if the site is flash. If it is, it adds the domain to the database and continue crawling. If not flash, it runs the second rule, if has a mobile version. And so forth. - Limit the results according to a domain filter (example: only .[login to view URL] domains, for targeting australian sites only) In your proposals please specify which language are you going to use, how the interface is going to be and provide as much information as possible. Similar projects that you've done in the past are welcome as well. Thanks M+W

Data Mining

PHP

Shell Script

Software Architecture

プロジェクト ID: 10832617

プロジェクトについて

10個の提案

リモートプロジェクト

アクティブ 8年前

お金を稼ぎたいですか？

メールアドレス

Freelancerで入札する利点

予算と期間を設定してください

仕事で報酬を得る

提案をご説明ください

登録して仕事に入札するのは無料です

この仕事に10人のフリーランサーが、平均$248 USDで入札しています

@mituld

Hi I work towards providing reliable, relevant and robust IT solutions at most competitive prices to my customers. I ensure 100% customer satisfaction so lets start Thanks

$237 USD 10日以内

4.8

(322 レビュー)

8.0

@ruipimentel

Olá, Meu nome é Rui Pimentel, mais conhecido recentemente pelo meu envolvimento em projetos de crawling a sites com netflix, nike, supremenewyork e tantos outros trending neste momento. Eu li completamente a descrição do seu projeto e percebo perfeitamente do que necessita, no entanto tenho algumas preocupações que gostava de discutir consigo. Se fosse você a fazer input dos domínios de websites seria um simples approach, mas pelo que percebo que você procura, teremos de recorrer a pesquisas no google, e temo que tal possa ser complicado se não for possível a utilização de vários IPS ou proxies, ou na pior das hipóteses, a necessidade de remover captchas manualmente à medida que vão aparecendo no processo de pesquisa. Aguardo um contacto da sua parte, espero sinceramente poder ajudá-lo a concretizar este projeto, tenho a certeza que ficará bastante satisfeito com o resultado. Cumprimentos, Rui

$147 USD 3日以内

4.9

(35 レビュー)

5.8

@Websitemaestro

Hi, I have read your specification and can definitely help you with your requirements. Please open a line of communication to discuss the project so that i can make it as per your vision and I am ready to start this project. waiting for your quick and positive response. Best regards

$155 USD 10日以内

4.8

(48 レビュー)

5.5

@ankurs13

I went through the description and found the project interesting. I can make a spider for you to run checks on websites (three as per your specs: flash, mobile support and old footer date) and output you a spreadsheet with the list of sites that fail them. I can code that using python (scrapy or anything else) or ruby as per further discussion. The settings can either be command line, given via inputs, via text files, in code or whatever other way suits you. I have previously written crawlers in scrapy and raw python/ruby, most of it is closed source so can't give you a reference.

$250 USD 7日以内

4.7

(5 レビュー)

4.4

@anupkelkar02

I am expert in python for Web scraping , API integration and lexical analysis. Also a good hand in web application designing using dijango and flask. I am a Oracle Certified professional (OCP) with experience in Oracle,MySQL, SQL Server and MongoDB. I can help you with your requirements. Please initiate online chat.. so we can proceed with discussion

$245 USD 7日以内

4.5

(10 レビュー)

4.2

@antyans

Hey. I got acquainted with you provided Info. The project is very interested. I understand perfectly the essence of the task. There is a lot of work experience, have experience in hozhih projects. I would be happy to implement your project into reality. There is a great desire to work. I propose to discuss all the details of your project in more detail, deadlines, budget. I would like to look at the detail technical task. I look forward to our cooperation. I'll wait for your answer in the chat.

$555 USD 14日以内

5.0

(7 レビュー)

3.7

@razvand70

Please see my reviews

$250 USD 5日以内

5.0

(3 レビュー)

3.5

@codeqoc

Hi, I'm a java developer with big experience in web and console application development. Can make this bot using java that will parse pages for that info.

$122 USD 4日以内