Handy Mac OSX shortcuts to work with Terminal

Clean up the line: You can use Ctrl+U to clear up to the beginning.

Clean up the line: Ctrl+A Ctrl+K to wipe the current line in the terminal 

Cancel the current command/line: Ctrl+C.

Recall the deleted command: Ctrl+Y

Go at the beginning of the line: Ctrl+A

Go at the end of the line: Ctrl+E

Remove the forward words for example, if you are middle of the command: Ctrl+K

Remove characters on the left, until the beginning of the word: Ctrl+W

To clear your entire command prompt: Ctrl + L

Toggle between the start of line and current cursor position: Ctrl + XX

Source : http://stackoverflow.com/questions/9679776/how-do-i-clear-delete-the-current-line-in-terminal

How to Remote Debug Standalone Hadoop

  When you run your map reduce applications you may have hiccups here and there and may need to have a look inside. If you need to remote debug rather going through logs and figuring out what went wrong following is the procedure.

I am using Intellij Idea as the IDE but for other IDE's process is similar.

1) In Intellij Idea go to Run > Edit Configuration and then click on "+" . And then add Remote for "Remote Debugging"

2) You will have following window after clicking on Remote. You can change the port you are using for remote debugging in this panel.

3) Open your Hadoop Root folder and navigate to etc/hadoop-env.sh in your editor. At the bottom of the file add the following line. (Make sure to have the port you given for IDE configuration as the address)

export HADOOP_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005"

Now you can start hadoop in standalone mode and it will wait until you attach your IDE to debug process to resume.

How to correct "ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)" on Mac OS X

I tried to install mysql using home-brew. Everything was successful But could not connect to the server. Obviously server have not started.

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)

Sigh..

Following command rescued me..

mysqld stop
mysql.server start

I found it in this stack overflow answer.

http://stackoverflow.com/questions/15016376/cant-connect-to-local-mysql-server-through-socket-homebrew

How to install Hadoop Standalone / Pseudo Distributed mode 2.X.X on Mac with OS X Mavericks

                                                          Image source: http://www.javacodegeeks.com/2011/05/hadoop-soft-introduction.html

I was searching for a complete tutorial on installing Hadoop on Mac and play around with it. There are resources on installing Hadoop with "HomeBrew" which is the missing package manager in Mac ;). But i do not want to offload all the configuration burden to it as i need to learn this from top to bottom. I played with some and here are the configuration steps i followed.

1) You need to download and extract Hadoop Binary. I used Hadoop 2.5.1 which is the latest at the moment.

http://www.apache.org/dyn/closer.cgi/hadoop/common/

2) Extract the binary and lets called the location as HADOOP_HOME

eg: /Users/user1/software/hadoop-2.5.1

3) Add HADOOP_HOME and JAVA_HOME  as path variables to your system. You can add them to

bashrc or bash_profile.

You can add them by issuing following commands.

$vim ~/.bash_profile

Add following entries and change paths according to your machine's configurations. 

export JAVA_HOME=$(/usr/libexec/java_home)
export HADOOP_HOME=/Users/user1/software/hadoop-2.5.1
export PATH=$PATH:$HADOOP_HOME

and then reload the configurations.

$source ~/.bash_profile

(Follow these steps if you need to run in Pseudo Distributed mode. If you do continue you will have to add input files to the HDFS and then download output files from the HDFS too.)

4) Navigate to HADOOP_HOME and change following files as below.

etc/hadoop/core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

etc/hadoop/hdfs-site.xml:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

5) Setup passphraseless ssh

Now check that you can ssh to the localhost without a passphrase:
$ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
 $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

(You may need to enable remote-login in system preferences --> sharing if you have not enabled it later to login through ssh.)

6) Starting the Hadoop in Standalone mode.

 
Navigate to $HADOOP_HOME

Format the filesystem:

 $ bin/hdfs namenode -format

Start NameNode daemon and DataNode daemon:

$ sbin/start-dfs.sh

The hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs).
Browse the web interface for the NameNode; by default it is available at:

NameNode - http://localhost:50070/

So good luck with all your map reduce jobs. :)

References :

http://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-common/SingleCluster.html#Standalone_Operation

Castle of the Moors(Castelo dos Mouros) - Hill top Medieval Castle

This is another weekend trip as last one. It seems there are plenty of places to be covered in Portugal which is an amazing country with lots of beautiful people and places.

 http://blog.shelan.org/2014/09/cabo-da-roca-western-most-cape-of.html

So all the EMDC students got together for a weekend out. We decided to pay a visit to Sintra without knowing what exactly we would we be going to explore or it would offer. We knew that it had lots of attractions and that was all what we needed to know.

 You can reach there using viva gem card which is valid for Metro and train (My previous blog post explains about it) . We travelled from Alameda to Rossio Metro station and then changed into the train to Sintra. It took around 1 hour to reach there and it was a different city with mountains all around and more green sights. You can see lots of tourists visiting there too.

We thought of going to the Castle of the Moors and then visit other places after that. But we had marginal time even to return. That place was amazing and a worth place to kill your time in the weekend. It took around 2 hours or more to go to the top. We hiked on our own and if you are bit lazy
you can get a Hop on Hop off bus. To be frank we thought that option better too had we known that earlier :). But you will burn some calories (a lot i would say while enjoying the nature) so it is okay if you are settled for it.

There are two castles on the top and one of them was closed (It closes around 4.30 pm after summer) so we had to choose Castle of the Moors which closes at 8.00pm. You need to pay Euro 7.50 per person for the entry ticket. This place was amazing and well maintained. Below pics will proof that to you for sure. You can enjoy a breath taking view from top of the Castle. And this castle was a Medieval Castle which built couple of centuries ago during 8th/9th Century. So it was pretty old one which poses a great historical value too.

On our way back six of us got into two groups due to bad timing and three of us almost lost on the way. Luckily we had Google maps and found a way to Sintra Train station. Luckily
That path was much shorter but steeper than the path we climbed. Ask Google the path he will tell you obviously :).

Enjoy my pics.

Want to learn anything faster ?

 We are in a world with lot of learning opportunities. But how many stuff do we really learn ? What holds us back in learning new things faster?

Learning how to learn is the greatest learning of all. I happened to read interesting reading from this blog [1] which had an amazing link mentioning about a guy who learned MIT 4 year course within 1 year.

It starts with

"My friend Scott Young recently finished an astounding feat: he completed all 33 courses in MIT’s fabled computer science curriculum, from Linear Algebra to Theory of Computation, in less than one year. More importantly, he did it all on his own, watching the lectures online and evaluating himself using the actual exams. (See Scott’s FAQ page for the details of how he ran this challenge.)"

http://calnewport.com/blog/2012/10/26/mastering-linear-algebra-in-10-days-astounding-experiments-in-ultra-learning/

When i was going little bit deeper in the subject i found another useful resource in one of the ted talks about the time it takes to learn a new thing. Amazingly it is only twenty hours :). Yes it is less than a day to acquire a new skill.

But...

You need to do it in four steps. Four simple steps to rapid knowledge acquire.

1) Deconstruct the new skill.

     divide and conquer small chunks of skill in new skill. Because most of the big skills are set of smaller skills

2) Learn enough to self correct.

 He depicts the most common error we do in learning which is trying to learn everything and start practicing later. But Ideally what you should do is to learn the essential or more obvious things so you
can self correct your self.

3) Remove practice barriers.

The most common barrier i can think of is "Facebook" ;)

4) Practice atleast 20 hours.

"The major barrier to learn new thing is not intellectual it is emotional"

So start practicing what you love to acquire..!!!

[1] http://kkpradeeban.blogspot.pt/2012/11/optimism-is-selection-of-nature.html

Cabo Da Roca - Western Most Cape of Europe

 It is a new country for me and it is full of new places to travel. Luckily we have one friend at the apartment who knows the area pretty well. So we did not have much trouble in locating the places to visit but if you travel by public transport following information would be useful for you.

  If you are near IST you need to travel from Alameda Metro station to Cais do Sodré station which is at the end of Green line. Then you need to get the Train from Cais do Sodré station to Cascais. To travel up to Cascais you can obtain a travel card from Metro station which covers all 4 forms of transportation including Bus, Metro, Train and Ferry. You have different options but it is better to buy  a viva via gem card if you are a non frequent traveller. It is reusable and reloadable card.

source :http://www.metrolisboa.pt/eng/customer-info/viva-viagem-card

Loading this card with 10 Euros would be enough as we experienced and you can top up it from train stations or metro stations conviniently. It is advisable to keep 5,10 Euro notes as sometimes it will issue receipt for balance which you should claim at train stations if you try to recharge from a train station. Believe me it is a hassle :(.

 You should spend some time at Cascais as it is a beautiful area with lots of tourist and local activities. We stayed there before moving to Cabo da Roca.

                  This was truly a beautiful beach...

From Cascais you need to get 403 bus operated by scottrub. It is usually has one hour frequency. It will go via Cabo da Roca to Sentra and you have to get down from the Cabo da Roca on the way. These buses does not have display boards for the stops but Cabo da Roca is a special stop and you know once you arrive there and just enjoy the breathtaking views out side on the way from the bus. (Believe me it is a special stop ;) )

Cabo da Roca is the western most point of the Europe mainland. You will be surely amazed how picturesque the view is. You will lost in your thoughts for a moment when you reach the top. But make sure not to fall out from the top to Atlantic ocean while taking selfies.. Cos' it is a shame that you cannot tell how magnificent it was to you friends :P

New window just opened ...!

  It has been a week since i have became a student once more in my life. I like the experience of recreating the feelings that i had when i was a student ago two years back. Since now this blog is full of technical content and i did not have much time to blog much about my other experiences. But now this is a new life so thought of doing the same to my blog. Lets see how the new story unfolds...

I am an EMDC ( European Masters in Distributed Computing) student at IST (Instituto Superior Técnico) and KTH (Royal Institute of Technology) in Sweden. This offering is great not only because of the content it offers but the cultural experience you will gain. It is a bonus..

During the first week I was fortunate to visit some of the very beautiful places in Lisbon. I should say the coastal area is really fantastic even though i am coming from a place with full of beaches covering my beautiful island Sri Lanka. I will post more about that place in another post with more details.. and of course with pictures. Till then Just relax with the view out of my apartment...

Error Recovery Mechanisms in WSO2 ESB with Store and Forward

 You may be in a situation where you need to send messages reliably to an endpoint. There are couple of ways you can achieve that. WSO2 ESB provides store/forward pattern with a higher level of reliability and error recovery mechanisms. This post aggregate some of the comprehensive articles into a single piece.

• http://wso2.com/library/articles/2011/10/implementing-store-forward-messaging-patterns-wso2esb-part-1/
• http://wso2.com/library/articles/2011/10/implementing-store-forward-messaging-patterns-wso2esb-part-1/
• http://techfeast-hiranya.blogspot.de/2012/01/wso2-esb-tips-tricks-08-message-stores.html
• http://charith.wickramaarachchi.org/2012/05/another-message-redelivery-pattern-with.html