Saturday, October 11, 2014

How to install Hadoop Standalone / Pseudo Distributed mode 2.X.X on Mac with OS X Mavericks


                                                          Image source: http://www.javacodegeeks.com/2011/05/hadoop-soft-introduction.html

I was searching for a complete tutorial on installing Hadoop on Mac and play around with it. There are resources on installing Hadoop with "HomeBrew" which is the missing package manager in Mac ;). But i do not want to offload all the configuration burden to it as i need to learn this from top to bottom. I played with some and here are the configuration steps i followed.

1) You need to download and extract Hadoop Binary. I used Hadoop 2.5.1 which is the latest at the moment.

http://www.apache.org/dyn/closer.cgi/hadoop/common/

2) Extract the binary and lets called the location as HADOOP_HOME

eg: /Users/user1/software/hadoop-2.5.1

3) Add HADOOP_HOME and JAVA_HOME  as path variables to your system. You can add them to
bashrc or bash_profile.

You can add them by issuing following commands.


$vim ~/.bash_profile

Add following entries and change paths according to your machine's configurations. 


export JAVA_HOME=$(/usr/libexec/java_home)
export HADOOP_HOME=/Users/user1/software/hadoop-2.5.1
export PATH=$PATH:$HADOOP_HOME

and then reload the configurations.


$source ~/.bash_profile
(Follow these steps if you need to run in Pseudo Distributed mode. If you do continue you will have to add input files to the HDFS and then download output files from the HDFS too.)

4) Navigate to HADOOP_HOME and change following files as below.

etc/hadoop/core-site.xml:
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>
etc/hadoop/hdfs-site.xml:
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

5) Setup passphraseless ssh

Now check that you can ssh to the localhost without a passphrase:
$ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
 $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

(You may need to enable remote-login in system preferences --> sharing if you have not enabled it later to login through ssh.)
6) Starting the Hadoop in Standalone mode.
 
Navigate to $HADOOP_HOME

Format the filesystem:

 $ bin/hdfs namenode -format


Start NameNode daemon and DataNode daemon:

$ sbin/start-dfs.sh



The hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs).
Browse the web interface for the NameNode; by default it is available at:

So good luck with all your map reduce jobs. :)

26 comments:

  1. excellent article! really helped me to get started on a mac.

    ReplyDelete
  2. Thanks a lot! Worked after breaking my head for ages!

    ReplyDelete
  3. Nice post.. Helped me a lot !!! Thanks Shelan..

    ReplyDelete
  4. I am really glad that it was helpful.

    ReplyDelete
  5. Thanks for giving the installation steps of haddop..Seo training in Chennai

    ReplyDelete
  6. Hadoop is one of the best tool which is used to handle the big data in the IT industy and it is the fastest growing field in information technology.
    hadoop training in Chennai | hadoop training chennai

    ReplyDelete
  7. Excellent post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    salesforce training in chennai | salesforce training institute in chennai

    ReplyDelete
  8. Well Said, you have furnished the right information that will be useful to anyone at all time. Thanks for sharing your Ideas.
    Node JS training in chennai | Node JS training institute in chennai

    ReplyDelete
  9. Nice interesting information on the latest arrived technology which helped me to get update according to the recent trends.
    Salesforce Training in Chennai | Salesforce Course in Chennai

    ReplyDelete
  10. Thanks for sharing this informative content which provided me the required information about the latest technology.
    Salesforce training in Chennai | Salesforce course in Chennai

    ReplyDelete
  11. شخدمات شركة امانى المملكة لخدمات التنظيف شركة مكافحة حشرات بالخرج
    مما لاشك ان الحشرات هى المصدر الاثاثى للقلق بالمنزل من اجل ذالك تقدم شركة امانى المملكة اكبر فروع بالمملكة لمكافحة الحشرات لابادة الحشرات بافضل المبيدات الفعالة المسجلة من وزارة الشصحة السعودية وخدماتن تعه المملكة ومنها افضل شركة مكافحة حشرات بخميس مشيط
    التى توفر العديد من الخدمات المتخصصين بمنطقة عسير لمكافحة وابادة الحشرات المنزل ومن الخدمات الاخرى الاخرة التى تقدمها شركة امانى المملكة شركة مكافحة حشرات حدائق بابها
    من افضل الشركات بالمملكة من حيث شركة رش مبيدات بالباحة
    ليس لها تأثير على الاشجار ولدينا افضل شركة تنسيق حدائق
    ومن الخدمات ايضا شركة مكافحة النمل الابيض بخميس مشيط
    وهى من الخدملت التى تقدمها الشركة على اعلى مستوى وعروض خاصة باسعار لعملائنا لاننا الان على وشك دخول فصل الصيف التى تكثر فية الحشرات وبالاخص النمل الابيض فان شركة مكافحة الحشرات بخميس مشيط تقدم ايضا افضل العروض والخصومات لعملائها باقل الاسعار ومن خدماتن لمكافحة الحشرات بالمملكة: افضل شركة مكافحة حشرات بابها

    ReplyDelete
  12. Great blog ... explanation and installation procedure are very clear and step by step so easy to understand.. thanks a lot for sharing this blog to us

    hadoop training | big data training | hadoop training in chennai

    ReplyDelete
  13. After reading this blog i very strong in this topics and this blog really helpful to all... explanation are very clear so very easy to understand... thanks a lot for sharing this blog

    hadoop training in chennai velachery | big data training in chennai velachery

    ReplyDelete

  14. This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharng this information,this is useful to me...
    Android training in chennai
    Ios training in chennai

    ReplyDelete
  15. Thanks for posting useful information.You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...Really it was an awesome article...very interesting to read..please sharing like this information......
    Web Design Development Company
    Mobile App Development Company

    ReplyDelete
  16. Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
    Web Design Company
    Web Development Company

    ReplyDelete
  17. You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...

    Android App Development Company

    ReplyDelete
  18. great and nice blog thanks sharing..I just want to say that all the information you have given here is awesome...Thank you very much for this one.
    Web Design Development Company
    Web design Company in Chennai
    Web development Company in Chennai

    ReplyDelete
  19. it is really amazing...thanks for sharing....provide more useful information...
    Mobile app development company

    ReplyDelete
  20. I wondered upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.

    Fitness SMS
    Salon SMS
    Investor Relation SMS

    ReplyDelete
  21. I wondered upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.
    iOS App Development Company

    ReplyDelete
  22. These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.
    Texting API
    Text message marketing
    Digital Mobile Marketing
    Sms API
    Sms marketing

    ReplyDelete