Saturday, October 11, 2014

How to install Hadoop Standalone / Pseudo Distributed mode 2.X.X on Mac with OS X Mavericks


                                                          Image source: http://www.javacodegeeks.com/2011/05/hadoop-soft-introduction.html

I was searching for a complete tutorial on installing Hadoop on Mac and play around with it. There are resources on installing Hadoop with "HomeBrew" which is the missing package manager in Mac ;). But i do not want to offload all the configuration burden to it as i need to learn this from top to bottom. I played with some and here are the configuration steps i followed.

1) You need to download and extract Hadoop Binary. I used Hadoop 2.5.1 which is the latest at the moment.

http://www.apache.org/dyn/closer.cgi/hadoop/common/

2) Extract the binary and lets called the location as HADOOP_HOME

eg: /Users/user1/software/hadoop-2.5.1

3) Add HADOOP_HOME and JAVA_HOME  as path variables to your system. You can add them to
bashrc or bash_profile.

You can add them by issuing following commands.


$vim ~/.bash_profile

Add following entries and change paths according to your machine's configurations. 


export JAVA_HOME=$(/usr/libexec/java_home)
export HADOOP_HOME=/Users/user1/software/hadoop-2.5.1
export PATH=$PATH:$HADOOP_HOME

and then reload the configurations.


$source ~/.bash_profile
(Follow these steps if you need to run in Pseudo Distributed mode. If you do continue you will have to add input files to the HDFS and then download output files from the HDFS too.)

4) Navigate to HADOOP_HOME and change following files as below.

etc/hadoop/core-site.xml:
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>
etc/hadoop/hdfs-site.xml:
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

5) Setup passphraseless ssh

Now check that you can ssh to the localhost without a passphrase:
$ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
 $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

(You may need to enable remote-login in system preferences --> sharing if you have not enabled it later to login through ssh.)
6) Starting the Hadoop in Standalone mode.
 
Navigate to $HADOOP_HOME

Format the filesystem:

 $ bin/hdfs namenode -format


Start NameNode daemon and DataNode daemon:

$ sbin/start-dfs.sh



The hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs).
Browse the web interface for the NameNode; by default it is available at:

So good luck with all your map reduce jobs. :)

82 comments:

  1. excellent article! really helped me to get started on a mac.

    ReplyDelete
  2. Thanks a lot! Worked after breaking my head for ages!

    ReplyDelete
  3. Nice post.. Helped me a lot !!! Thanks Shelan..

    ReplyDelete
  4. I am really glad that it was helpful.

    ReplyDelete
  5. Thanks for giving the installation steps of haddop..Seo training in Chennai

    ReplyDelete
  6. Hadoop is one of the best tool which is used to handle the big data in the IT industy and it is the fastest growing field in information technology.
    hadoop training in Chennai | hadoop training chennai

    ReplyDelete
  7. Well Said, you have furnished the right information that will be useful to anyone at all time. Thanks for sharing your Ideas.
    Node JS training in chennai | Node JS training institute in chennai

    ReplyDelete
  8. Nice interesting information on the latest arrived technology which helped me to get update according to the recent trends.
    Salesforce Training in Chennai | Salesforce Course in Chennai

    ReplyDelete
  9. Great blog ... explanation and installation procedure are very clear and step by step so easy to understand.. thanks a lot for sharing this blog to us

    hadoop training | big data training | hadoop training in chennai

    ReplyDelete
  10. After reading this blog i very strong in this topics and this blog really helpful to all... explanation are very clear so very easy to understand... thanks a lot for sharing this blog

    hadoop training in chennai velachery | big data training in chennai velachery

    ReplyDelete
  11. You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...

    Android App Development Company

    ReplyDelete
  12. great and nice blog thanks sharing..I just want to say that all the information you have given here is awesome...Thank you very much for this one.
    Web Design Development Company
    Web design Company in Chennai
    Web development Company in Chennai

    ReplyDelete
  13. it is really amazing...thanks for sharing....provide more useful information...
    Mobile app development company

    ReplyDelete
  14. I wondered upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.

    Fitness SMS
    Salon SMS
    Investor Relation SMS

    ReplyDelete
  15. I wondered upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.
    iOS App Development Company

    ReplyDelete
  16. This article is very much helpful and i hope this will be an useful information for the needed one. Keep on updating these kinds of informative things...
    snapho

    ReplyDelete
  17. I'm puzzled with lots of exercises. I was afraid I could not do the right time despite my hard work. I need a support person.
    http://run3play.com

    ReplyDelete
  18. This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post.
    Informatica Training in Chennai
    Python Training in Chennai
    AngularJS Training in Chennai

    ReplyDelete
  19. It was awesome to read your blog, This Great content and useful to me.Web Development Company in Chennai

    ReplyDelete
  20. Hello! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community in the same niche. Your blog provided us useful information to work on. You have done an outstanding job.
    Best AWS Training in Chennai | Amazon Web Services Training in Chennai

    AWS Training in Bangalore | Amazon Web Services Training in Bangalore

    Amazon Web Services Training in Pune | Best AWS Training in Pune

    ReplyDelete
  21. Good Post! Thank you so much for sharing this pretty post , it was so good to read and useful to improve my knowledge as updated one, keep blogging

    HTML5 Training in Chennai
    HTML Training

    ReplyDelete
  22. Pleasant Tips..Thanks for Sharing….We keep up hands on approach at work and in the workplace, keeping our business pragmatic, which recommends we can help you with your tree clearing and pruning in an invaluable and fit way.
    angularjs Training in chennai
    angularjs Training in chennai

    angularjs-Training in tambaram

    angularjs-Training in sholinganallur

    angularjs-Training in velachery

    ReplyDelete
  23. Wonderful article, very useful and well explanation. Your post is extremely incredible. I will refer this to my candidates...
    python interview questions and answers | python tutorials

    ReplyDelete
  24. This blog is the general information for the feature. You got a good work for these blog.We have a developing our creative content of this mind.Thank you for this blog. This for very interesting and useful.
    Java training in Marathahalli | Java training in Btm layout

    Java training in Jaya nagar | Java training in Electronic city

    ReplyDelete
  25. I enjoy what you guys are usually up too. This sort of clever work and coverage! Keep up the wonderful works guys I’ve added you guys to my blog roll.
    iosh course in chennai

    ReplyDelete
  26. I love the way you write Business Analytics Online Course and share your niche! Very interesting and different! Keep it coming!

    ReplyDelete
  27. Great blog thanks for sharing The world around is changing at turbo speed. With digital marketing companies booming up at every corner, it can be hard to decide which is the best place for you to begin your online marketing journey. If you are based in Chennai, the answer is plain simple - Adhuntt Media has the best team that cover all your branding needs - SEO, Graphic Design, Logo Design, Social Media Marketing, Google Ads, Competitor Analysis and much more.
    digital marketing company in chennai

    ReplyDelete
  28. Nice blog thanks for sharing Buy unique easy to maintain indoor plants and exotic garden plants all at the same place - Karuna Nursery - the best nursery garden in Chennai. With our exclusive collection of plants and special deals on bulk orders - you know you are at the right place.
    plant nursery in chennai

    ReplyDelete
  29. Excellent blog thanks for sharing Shopping for international cosmetics in Chennai? Look no further, the Pixies Beauty Shop has got your back with hundreds of international brands to choose from, that too with the best value.
    beauty Shop in Chennai


    ReplyDelete
  30. Very useful blog thanks for sharing IndPac India the German technology Packaging and sealing machines in India is the leading manufacturer and exporter of Packing Machines in India.

    ReplyDelete
  31. I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.
    AWS training in chennai | AWS training in annanagar | AWS training in omr | AWS training in porur | AWS training in tambaram | AWS training in velachery

    ReplyDelete
  32. Thanks for sharing such information. This is really helpful for me. you can also visit our blog
    Banquets in Indore

    ReplyDelete
  33. This is one of the best content for this topic and this is very useful for me. Thank you!
    Unix Training in Chennai
    Linux Training in Chennai
    Linux Course in Chennai

    ReplyDelete
  34. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

    Python Institutes in Pune
    Python Classes in Pune

    ReplyDelete
  35. Strategies help create the main framework and provide marketers the big picture. Tactics involve the implementation part of the marketing plan. These two elements must work together in synchrony for direct automation to produce results. https://updigital.ca

    ReplyDelete