distinguishing gender and filtering emails using hdfs/nltk/python.

Problem Statement 1: We are thinking to take the live streaming data such as twitter tweets and analyze the tweet, differentiate the male and female tweets and represent them graphically. Generally, names ending with a, e and i are most likely the female names. So, we'd use this concept to distinguish male and female names. Instructions: Extract data from twitter. (using flume) Store it in text file (store the extracted files in hdfs) Do sentiment Analysis using nltk (use the files in hdfs to do sentiment analysis using nltk) Distinguish the gender by the twitter names Create graphical report using visualization Store visualization report in hdfs using sandbox Note: steps 3,4,5 should be written in python Problem Statement 2: Generally, people get hesitated when they are likely to see a spam email in their mailbox as they are expecting an important email. Emails consisting of words such as "won", "rewards", "lottery", "lucky" are mostly spam emails. So we'd like to use this concept and filter emails. We are planning to use Gmail data for streaming. Instructions: Extract data from a selected gmail account Store it in hdfs using sandbox Extract the text data from sandbox and analyze the data using nltk Here filteration should be done using keywords such as won, rewards, lottery, jackpot, rebate, mailin rebate, lucky, winner and distinguish the email as spam or not Use the python and nltk to visualize the filtered data Using sandbox save data in hdfs. Conditions: If the mail contains the words won lottery, won jackpot, lucky winner, winner of the day are possibly 100% spam If the mail contains keywords lucky, rebate is possibly 90% spam If the mail contains keywords mailin rebate, rewards are 80% spam Implementation: We will 1. Collect the live streaming data 2. Write into a text file 3. Give the text file as input to python mapper and reducer 4. Use nltk, python for visualization of data. Note: Project should be completed in 3 days. Execution should be shown using teamviewer. Project ID: 12250178 Skills Required Python, Natural Language, Hadoop