AngularJS and Node app to scrape data into MongoDB
€250-750 EUR
完了済み
投稿日: 約8年前
€250-750 EUR
完了時にお支払い
Create an Angular JS app together with a NodeJS app where it's possible to start a scrape data from [login to view URL] and import through a rest API.
Mockup: [login to view URL]
1. Doing sampling
- Add a select box to pick a site to scrape data from (Today only [login to view URL])
- Pick number of ads from [login to view URL] to get per category
- When user clicks Sample button
- Add a row in a mongodb collection called samples (in a database called adminio) With data
site: [login to view URL]
adsPerCategory: *data from form input*
adsTotal: NULL
status: in-progress
- Run in background a script that goes through every category on [login to view URL] and browse to every page and load the number of ads as was defined in "adsPerCategory" in database. For every ad fetch this and map to mongo structure
header -> [login to view URL]
description -> [login to view URL]
currency -> [login to view URL]
price -> [login to view URL]
county -> [login to view URL]
id -> id
photos: [{medium}] -> [login to view URL]: [{src: }]
[login to view URL] -> [login to view URL]
[login to view URL] -> [login to view URL]
age -> [login to view URL] ([login to view URL] - age)
friendlyUrl -> [login to view URL]
publisherName -> [login to view URL]
[login to view URL] -> [login to view URL]
So final mongo structure looks like:
{
site: "[login to view URL]",
adsPerCategory: 500,
adsTotal: 22001,
status: "done",
items: [{
id: 2313,
content: {
title: "TITLE",
body: "TEXT FOR AD",
price: {
amount: 1000.22,
currency: "EUR"
},
location: {
city: "Stockholm"
},
photos: [{
src: "http://..."
}, {
src: "http://..."
}],
category: {
id: 123,
name: "A category"
},
createdAt: "2015-20-22 10:00",
url: "http://...",
},
user: {
name: "NAME",
id: 12345
}
}]
}
- While sampling is running add to table under the form and instead of import button show "Sampling in progress..."
- Once done and if page is reloaded show green button with text "Import to sentinel"
- When clicking on "Import to Sentinel"
- Go to this page [login to view URL]
- Let user supply a token.
- When click on "Import all 3000 ads" start background job that reads the ads from mongodb and insert them into a REST API with same structure as in mongodb. Only 1000 ads per time so insert 1000, sleep 1 second, insert 1000, sleep 1 second and so on.
- While importing show this [login to view URL]
- When import is done change text to "Success"
Let's discuss more about project to finalize the proper scope with estimated cost and time so ping me over the freelancer chat.
I am myself developer so you will directly work with me. No mediators. No managers. No subcontractors.
Please check my my recent work for the technical expertise along with reviews & feedback on my profile page.
Hello
I am very interested with your project
I have good skills and have many experiences in AngularJS
My ranking is TOP 5th in freelancer.com and have enough skills to do your project perfectly
I wish to work for you
Thanks
KCHG
Hello,
I'm a professional programmer for web programming with php language to build the system website,
Besides, I'm also expert in MySQL , HTML,HTML5,CSS, JS
I'm always top in Vietnam freelancer
https://www.freelancer.com/freelancers/Vietnam/all/
https://www.freelancer.com/u/scriptphp87.html
I'm expert development with Angular JS, Node Js, MongoDB
Would you like discuss more with me?
would you like show me your site to need do?
I'm understand clearly your requirement and i'm interested your project
I'm ready to do it now
Nice to meet you
Should you require further assistance, feel free to contact me.
Thank you
Hope you are doing great. I have checked the given detail of this project and I am interested to provide you my services. I have more than 5 years experience in providing professional website development services and worked with almost every type of project. So this is an easy one for me as well.
What I need is to finalize clear requirements (that play a big role in success of every project) and that needs your few minutes to discuss over the project with me.
Let me know your availability and let’s get this project done!
Hi sir,
I am scraping expert, I have did too many similar projects, please check my feedback then you will know.
Can you tell me more details? then I will provide demo data for you.
Thanks,
Kimi
Hi there what you need is my expertise in mongo and nodejs.
i have done scrapping almost on Half of Worldwide web including ecommerce giants(Amazon,ebay,craigslist) News Feed, Social media websites, API's.
I develop my own tools based on client requirements with Mulithreading, a Bot with human behaviour and Scrapping Applications with documents parsing.
I Can do PDF Parsing and Capctha ByPass code as well.
Contact me for further details
HI.
We did a similar project, in python, that was able to crawl any website. I think nodejs is also good for creating such app and we can help you with this project.
If you are interested let me know and we can discuss in details how we will do it and what we will offer you.
Thx, hope we will collaborate.
Hello,
I have studied about your requirements and ready to assist you to work over them. I am a cross platform developer having proficiency in MEAN STACK, .net framework, SQL & MY SQL for more than 5 years of experience having expertise in building scalable yet responsive websites which are compatible with all devices.
I really appreciate if we have further discussions over this project via chat or skype.
Best Regards,
Siddharth tiwari