Saturday, October 11, 2014

How to install Hadoop Standalone / Pseudo Distributed mode 2.X.X on Mac with OS X Mavericks


                                                          Image source: http://www.javacodegeeks.com/2011/05/hadoop-soft-introduction.html

I was searching for a complete tutorial on installing Hadoop on Mac and play around with it. There are resources on installing Hadoop with "HomeBrew" which is the missing package manager in Mac ;). But i do not want to offload all the configuration burden to it as i need to learn this from top to bottom. I played with some and here are the configuration steps i followed.

1) You need to download and extract Hadoop Binary. I used Hadoop 2.5.1 which is the latest at the moment.

http://www.apache.org/dyn/closer.cgi/hadoop/common/

2) Extract the binary and lets called the location as HADOOP_HOME

eg: /Users/user1/software/hadoop-2.5.1

3) Add HADOOP_HOME and JAVA_HOME  as path variables to your system. You can add them to
bashrc or bash_profile.

You can add them by issuing following commands.


$vim ~/.bash_profile

Add following entries and change paths according to your machine's configurations. 


export JAVA_HOME=$(/usr/libexec/java_home)
export HADOOP_HOME=/Users/user1/software/hadoop-2.5.1
export PATH=$PATH:$HADOOP_HOME

and then reload the configurations.


$source ~/.bash_profile
(Follow these steps if you need to run in Pseudo Distributed mode. If you do continue you will have to add input files to the HDFS and then download output files from the HDFS too.)

4) Navigate to HADOOP_HOME and change following files as below.

etc/hadoop/core-site.xml:
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>
etc/hadoop/hdfs-site.xml:
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

5) Setup passphraseless ssh

Now check that you can ssh to the localhost without a passphrase:
$ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
 $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

(You may need to enable remote-login in system preferences --> sharing if you have not enabled it later to login through ssh.)
6) Starting the Hadoop in Standalone mode.
 
Navigate to $HADOOP_HOME

Format the filesystem:

 $ bin/hdfs namenode -format


Start NameNode daemon and DataNode daemon:

$ sbin/start-dfs.sh



The hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs).
Browse the web interface for the NameNode; by default it is available at:

So good luck with all your map reduce jobs. :)

44 comments:

  1. excellent article! really helped me to get started on a mac.

    ReplyDelete
  2. Thanks a lot! Worked after breaking my head for ages!

    ReplyDelete
  3. Nice post.. Helped me a lot !!! Thanks Shelan..

    ReplyDelete
  4. I am really glad that it was helpful.

    ReplyDelete
  5. Thanks for giving the installation steps of haddop..Seo training in Chennai

    ReplyDelete
  6. Hadoop is one of the best tool which is used to handle the big data in the IT industy and it is the fastest growing field in information technology.
    hadoop training in Chennai | hadoop training chennai

    ReplyDelete
  7. Well Said, you have furnished the right information that will be useful to anyone at all time. Thanks for sharing your Ideas.
    Node JS training in chennai | Node JS training institute in chennai

    ReplyDelete
  8. Nice interesting information on the latest arrived technology which helped me to get update according to the recent trends.
    Salesforce Training in Chennai | Salesforce Course in Chennai

    ReplyDelete
  9. Great blog ... explanation and installation procedure are very clear and step by step so easy to understand.. thanks a lot for sharing this blog to us

    hadoop training | big data training | hadoop training in chennai

    ReplyDelete
  10. After reading this blog i very strong in this topics and this blog really helpful to all... explanation are very clear so very easy to understand... thanks a lot for sharing this blog

    hadoop training in chennai velachery | big data training in chennai velachery

    ReplyDelete
  11. This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post.
    Informatica Training in Chennai
    Python Training in Chennai
    AngularJS Training in Chennai

    ReplyDelete
  12. It was awesome to read your blog, This Great content and useful to me.Web Development Company in Chennai

    ReplyDelete
  13. Good Post! Thank you so much for sharing this pretty post , it was so good to read and useful to improve my knowledge as updated one, keep blogging

    HTML5 Training in Chennai
    HTML Training

    ReplyDelete
  14. Pleasant Tips..Thanks for Sharing….We keep up hands on approach at work and in the workplace, keeping our business pragmatic, which recommends we can help you with your tree clearing and pruning in an invaluable and fit way.
    angularjs Training in chennai
    angularjs Training in chennai

    angularjs-Training in tambaram

    angularjs-Training in sholinganallur

    angularjs-Training in velachery

    ReplyDelete
  15. This blog is the general information for the feature. You got a good work for these blog.We have a developing our creative content of this mind.Thank you for this blog. This for very interesting and useful.
    Java training in Marathahalli | Java training in Btm layout

    Java training in Jaya nagar | Java training in Electronic city

    ReplyDelete
  16. I love the way you write Business Analytics Online Course and share your niche! Very interesting and different! Keep it coming!

    ReplyDelete
  17. Great blog thanks for sharing The world around is changing at turbo speed. With digital marketing companies booming up at every corner, it can be hard to decide which is the best place for you to begin your online marketing journey. If you are based in Chennai, the answer is plain simple - Adhuntt Media has the best team that cover all your branding needs - SEO, Graphic Design, Logo Design, Social Media Marketing, Google Ads, Competitor Analysis and much more.
    digital marketing company in chennai

    ReplyDelete
  18. Excellent blog thanks for sharing Shopping for international cosmetics in Chennai? Look no further, the Pixies Beauty Shop has got your back with hundreds of international brands to choose from, that too with the best value.
    beauty Shop in Chennai


    ReplyDelete
  19. I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.
    AWS training in chennai | AWS training in annanagar | AWS training in omr | AWS training in porur | AWS training in tambaram | AWS training in velachery

    ReplyDelete
  20. Thanks for sharing such information. This is really helpful for me. you can also visit our blog
    Banquets in Indore

    ReplyDelete
  21. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

    Python Institutes in Pune
    Python Classes in Pune

    ReplyDelete
  22. Strategies help create the main framework and provide marketers the big picture. Tactics involve the implementation part of the marketing plan. These two elements must work together in synchrony for direct automation to produce results. https://updigital.ca

    ReplyDelete
  23. The blog you shared is very good. I expect more information from you like this blog. Thank you. Ziyyara Edutech’s online English language classes in Kuwait are designed to provide a comprehensive and immersive learning experience, with a strong emphasis on spoken English proficiency.
    For more info visit English language Class in kuwait

    ReplyDelete