Find Jobs
Hire Freelancers

Big data project

$30-250 AUD

クローズ
投稿日: 2年近く前

$30-250 AUD

完了時にお支払い
In this project, you will develop an Oozie workflow to process and analyze a large volume of flight data. • Instructions: 1. Form a project team of four students (including yourself). 2. Install Hadoop/Oozie on your AWS VMs. 3. Download the Airline On-time Performance data set (flight data set) from the period of October 1987 to April 2008 on the following website: [login to view URL]:10.7910/DVN/HG7NV7 4. Design, implement and run an Oozie workflow to find out a. the 3 airlines with the highest and lowest probability, respectively, of being on schedule; b. the 3 airports with the longest and shortest average taxi time per flight (both in and out), respectively; and c. the most common reason for flight cancellations. • Requirements: 1. Your workflow must contain at least three MapReduce jobs that run in fully distributed mode. 2. Run your workflow to analyze the entire data set (total 22 years from 1987 to 2008) at one time on two VMs first and then gradually increase the system scale to the maximum allowed number of VMs for at least 5 increment steps, and measure each corresponding workflow execution time. 3. Run your workflow to analyze the data in a progressive manner with an increment of 1 year, i.e. the first year (1987), the first 2 years (1987-1988), the first 3 years (1987-1989), …, and the total of 22 years (1987-2008), on the maximum allowed number of VMs, and measure each corresponding workflow execution time. • Submission (all in a zipped file: [login to view URL]): 1. A [login to view URL] text file that lists all the commands you used to run your code and produce the required results in a fully distributed mode 2. An [login to view URL] text file that stores the final results from all the runs 3. The source code of your MapReduce programs (including the JAR files) and any other programs you might have developed and included in the workflow 4. The Oozie workflow XML file 5. A project report in PDF that includes: a. A diagram that shows the structure of your Oozie workflow b. A detailed description of the algorithm you designed to solve each of the problems c. A performance measurement plot that compares the workflow execution time in response to an increasing number of VMs used for processing the entire data set (22 years) and an in-depth discussion on the observed performance comparison results d. A performance measurement plot that compares the workflow execution time in response to an increasing data size (from 1 year to 22 years) and an in-depth discussion on the observed performance comparison results Read Less
プロジェクト ID: 33639738

プロジェクトについて

7個の提案
リモートプロジェクト
アクティブ 2年前

お金を稼ぎたいですか?

Freelancerで入札する利点

予算と期間を設定してください
仕事で報酬を得る
提案をご説明ください
登録して仕事に入札するのは無料です
この仕事に7人のフリーランサーが、平均$303 AUDで入札しています
ユーザーアバター
Hi there! I have vast experience in Data Science and Hadoop, and I am confident that I can produce quality content for you that will be of interest to your readers. I am available for a full-time writing position, and I am happy to work on a freelance basis if that is what you prefer. I look forward to hearing from you soon! Thank you for your time. Thanks & regards - Wilfred
$100 AUD 1日以内
0.0 (0 レビュー)
0.0
0.0
ユーザーアバター
Hi Btw I already did exactly same project for one of the airlines company in Europe. I have around 20+ years of professional experience in IT which includes work experience in Big Data, Hadoop ecosystem related technologies. we can start working on it as soon as possible. Thank you
$1,000 AUD 10日以内
0.0 (0 レビュー)
0.0
0.0
ユーザーアバター
big data expert/Hadoop Am ready to develop an Oozie workflow to process and analyze a large volume of flight data. please send me a message to discuss further details about this project in chat. thank you
$222.22 AUD 6日以内
0.0 (0 レビュー)
0.0
0.0
ユーザーアバター
Hi, I have +5 years of experience dealing with machine learning algorithms and worked on multiple projects in this field, I absolutely can do your project as you like. Please contact me to discuss more. Have a nice day
$140 AUD 7日以内
0.0 (0 レビュー)
0.0
0.0

クライアントについて

EGYPTのフラグ
Cairo, Egypt
4.9
39
お支払い方法確認済み
メンバー登録日:10月 25, 2018

クライアント確認

ありがとうございます!無料クレジットを受け取るリンクをメールしました。
メールを送信中に問題が発生しました。もう一度お試しください。
登録ユーザー 投稿された仕事の合計
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
プレビューを読み込み中
位置情報へのアクセスが許可されました。
あなたのログインセッションの有効期限がきれ、ログアウトされました。もう一度ログインしてください。