Side Notes

never finished…

Lambda Jam 2013

This is my impressionistic non-canonical irregular Clojuresque-Erlangish notes on Lambda Jam conference which took place in Chicago on July 8–10, 2013. These notes are pretty long, and I don’t split them on purpose. If you read them all, you should become overwhelmed and overloaded with the information. Only this way you can feel the same I felt on the last day of the conference :)

Sleeping Barber in Erlang

After working on Sleeping Barber problem at Coding Dojo I decided to implement it in Erlang. Because Erlang is the perfect language for these sorts of problems, many people have alredy solved it in many different ways. I found at least three groups of solutions: 1) direct implementation using message passing between processes, 2) OTP solution using gen_server, and 3) something resembling object-oriented approach.

Strangely enough, I didn’t see any solution based on FSM. That is odd because this is the first thing coming to my mind when I hear this problem. To fill the gap, I’m going to solve it using gen_fsm, which is a standard OTP behaviour for FSM.

To Be or Not to Be… Entrepreneur

Rod Johnson, creator of Spring, programmer that became entrepreneur:

One thing I think that you really need to be careful of as well, particularly if you, like me, are a programmer, is don’t get carried away writing code. Typically in my experience anyone who is a good programmer is pretty passionate about it, love writing code, get addicted to the process of writing code, fell pretty good about their code basis. As soon as you get down that path you are not thinking straight anymore and now you are increasing your emotional investment, you are having lots of fun writing interesting code and you are no longer in a place mentally where you are going to be trying to find some reason that you shouldn’t write that code. That has been a big lesson for me that the quicker I get to coding, the longer it takes me to ask the kind of questions I should ask upfront.

Linus Torvalds, creator of Linux, programmer that still programs:

I’m interested in programming. When business started happening, I didn’t get into this for the business side. I wanted to do programming. When other people started selling Linux, I said “Yes! Now I can avoid caring about that side too.”

I never had to deal with single business related thing in Linux ever. Seriously, I had to deal with lot of other things but business related things… I got queries and I just said “I don’t care.”

Joe Armstrong at MostlyErlang

The main guest at today’s MostlyErlang podcast is Joe Armstrong, the co-inventor of Erlang, a wise man, and a brilliant speaker. I listened to the podcast few times, and every time enjoyed it. I highly recommend it to all programmers regardless whether they are using Erlang or not.

While we are waiting for the oficial transcript, this post provides some highlights. I put here the quotes I found particularly interesting or funny.

Concurrency

I’ve got a blog on my GitHub, and I thouhgt I’d put two different types of articles there and see what the response is. One theme I’m pursuing is “explaining to 5-year-old” theme, and the other is “technically complicated” theme.

Then he describes the “concurrency vs parallelism” picture for 5 y.o.

That blog entry got 25,000 people read it, and they got 10 responses and quite a lot of criticism actually from people who know what concurrency and parallelism is. And then I compare it to some other articles I’ve written that technically deeper. There you got only few hundred people read it.

I noticed it too. All those people, who responded with long blogs about concurrency, missed the point: This picture was for kids, and for kids it’s a pretty good picture. Adults should go and read the technical papers, but they didn’t.

Stuck With Your First Programming Language

Barbara Liskov in her keynote presentation:

I’m a little dismayed what’s happened in programming languages. And the reason I’m dismayed is because on the one hand we have the programming languages that experts use — I’m thinking of Java and C# but you can name bunch of others. And the problem with these languages is — they are powerful, you can build big systems in them — but they aren’t very good for beginners. And what happens in MIT, and I think this is happening across the US anyway, is the people are no longer using those languages in their introductory programming language courses, because the amount of craft you have to go through in order to write the little print-loop is just too much, and the students just get lost in the process. So they’ve been switching to languages like Python.

Python is very simple and nice when you start to use it. But you don’t get too far down the road, if you me, before you discover it has no data abstraction at all. That’s not good because big programs require modularity and encapsulation, and you’d like the language that supports that. So my question is: Can we find a language that will work for both communities? And the reason it matters is because a lot of kids start off programming in their first programming language and that’s it. They may have to change eventually, but they are building huge systems in languages that are really ill-suited for this. So it would be good if the language they started off with is the one they can grow with, and would be good for building big programs as well as small ones.

Simple Zookeeper Cluster

Sometimes I need to run ZooKeeper ensemble on my development box to test my application on the production-like environment. I found that recreating the whole ensemble from scratch is much faster than cleaning it up using ZooKeeper CLI tool. To automate this process I created a bash script which I want to share in this blog post. I hard-coded all the paths in the script using my regular conventions. You might need to change them to yours — it should be fairly straightforward.

Before you can use the script, you need to install ZooKeeper on your box. That’s what I did on my machine

$ cd /opt
$ sudo mkdir zookeeper
$ sudo chown -R andrey:admin zookeeper
$ cd zookeeper
$ wget http://apache.mirror.rafal.ca/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz
$ tar xf zookeeper-3.4.5.tar.gz
$ rm zookeeper-3.4.5.tar.gz
$ ln -s zookeeper-3.4.5 zookeeper

In the end you should have a ZooKeeper installed in /opt/zookeeper/zookeeper directory.

Now download, chmod, and run the script. It will create the following files

/opt/zookeeper/zookeeper/cluster
├── server1
│   ├── conf
│   │   ├── log4j.properties
│   │   └── zoo.cfg
│   ├── data
│   │   └── myid
│   └── logs
├── server2
│   ├── conf
│   │   ├── log4j.properties
│   │   └── zoo.cfg
│   ├── data
│   │   └── myid
│   └── logs
├── server3
│   ├── conf
│   │   ├── log4j.properties
│   │   └── zoo.cfg
│   ├── data
│   │   └── myid
│   └── logs
├── start.sh
└── stop.sh

This is the minimum configuration for 3-node ensemble (cluster), which is recommended for production. To start the cluster, run the following command

$ cd /opt/zookeeper/zookeeper
$ cluster/start.sh

Check the log files to see if the cluster is successfully started

$ tail -f cluster/server{1,2,3}/logs/zookeeper.out

When the cluster is up and running, you can test your application. After you are done, shutdown the cluster using the following command

$ cluster/stop.sh
$ ps -ef | grep java

To recreate a clean cluster, just run the script again

$ ./zookeeper-init-ensemble.sh

RabbitMQ, ActiveMQ, ZeroMQ, HornetQ

Warning: In this post I’m going to compare RabbitMQ, ZeroMQ, ActiveMQ, and HornetQ. The basis of the comparison is not the performance, or the scalability, or any other serious feature. The comparison is done purely based on the popularity of those systems. Therefore, if you came here to see some performance metrics, you will be disappointed — there is none in this post.

Note: To calculate popularity, I’m going to use MongoDB and Python, so if you don’t care about message brokers, but you want to see some examples of MongoDB scripts, this post might be interesting to you.

Code Retreat 2012

Yesterday was the Global Day of CodeRetreat. Software engineers around the globe met together to learn from each other.

There were several sessions where people were pair-programming Conway’s Game of Life.

Each session you had to choose a new partner, so that both of you can learn something new.

During the first session my partner and I decided to implement the Game in Java, mainly because it was the language she was most comfortable with. We implemented the procedural solution using two-dimensional array and nested loops. At that moment that was the only solution I could think of. The main challenge was to cover all edge cases and fix all ArrayIndexOutOfBoundsExceptions. Java is fairly verbose language, and with nested loops and if-else statements the final solution was pretty hard to read. You can see here how it might look like.

First session was a warmup, during which most people realized that programming arrays is a tedious work. For the second session my new partner suggested an object-oriented approach, where you would operate on Cell objects that would encapsulate coordinates on the grid. In this case you move the game logic from the grid to the cell, making it easier to calculate a new state. This was my first acquaintance with C#. Interesting language — basically, Java with lambdas. Here is an example of C# implementation. Our solution was very similar.

While the first session’s data structure was array of booleans, on the second session it was replaced by a list of objects. The next step would be to relax the data structure even further. We decided to experiment with un-ordered set of coordinate pairs. For language we chose Clojure. Although we didn’t finish the implementation, by the end of the session we had a clear picture how to solve the problem in functional style.

On the fourth session the facilitators put an interesting constraint: the coding must be done in absolute silence. That was the most amazing experience of the day. Before we started I thought we couldn’t accomplish much without talking. As it turned out, we could. The key of effective silent coding is to use the tools which both partners are familiar with. In our case we both were advanced users of Vim, and we knew Lisp languages. Our Clojure implementation was based on map/filter/reduce approach and spanned 20 lines of code. After the session Leo showed me Christophe Grand’s 7-line solution based on list comprehensions. It is so wonderful that I want to post it here

life.clj
1
2
3
4
5
6
7
8
(defn neighbours [[x y]]
  (for [dx [-1 0 1] dy (if (zero? dx) [-1 1] [-1 0 1])]
    [(+ dx x) (+ dy y)]))

(defn step [cells]
  (set (for [[loc n] (frequencies (mapcat neighbours cells))
             :when (or (= n 3) (and (= n 2) (cells loc)))]
         loc)))

For the last session we chose Erlang. Because we already knew how to implement the functional solution, that was just an exercise of translating Clojure code into Erlang. Unfortunately we didn’t find an equivalent of frequencies function in the standard library, so we implemented it ourselves. Other than that, the Erlang code is almost identical to Clojure.

life.erl
1
2
3
4
5
6
7
8
9
10
11
12
neighbours({X, Y}) ->
    [{X + DX, Y + DY} || DX <- [-1, 0, 1], DY <- [-1, 0, 1], {DX, DY} =/= {0, 0}].

step(Cells) ->
    Nbs = lists:flatmap(fun neighbours/1, sets:to_list(Cells)),
    NewCells = [C || {C, N} <- dict:to_list(frequencies(Nbs)),
                     (N == 3) orelse ((N == 2) andalso sets:is_element(C, Cells))],
    sets:from_list(NewCells).

frequencies(List) -> frequencies(List, dict:new()).
frequencies([], Acc) -> Acc;
frequencies([X|Xs], Acc) -> frequencies(Xs, dict:update_counter(X, 1, Acc)).

Summary

During this day I learnt a lot: new language, new abstractions, new techniques, new ways of communication, new ideas. I met bunch of smart people. I was so overwhelmed with all this cool stuff that I had to write this blog post to offload it from my head.

If you are a programmer and you’ve never been to CodeRetreat, I strongly encourage you to do it next year. It’s an exciting experience.

I want to thank all the people who organized and participated in this event.

Photo Credits

  • Michael DiBernardo [1]
  • Kunal Gupta [2]
  • Carlo Barrettara [3]

Flexible Language

I’ve been learning Lisp for few years now, and every Lisp book I read keeps saying that Lisp is a flexible language that you can extend to the degree when it fits naturally to your domain. It’s easy to say, but what exactly does this phrase mean? After all, when you program in your non-Lisp language, don’t you modify it for your domain problem? I’ve been thinking about it for a long time, and only recently I started to understand what flexibility really means. There is a difference between using the language and changing the language to solve a problem. In this post I will try to show the difference based on a simple example.

Problem

Suppose you have a process that listens to a message queue. The messages are just ordinary maps. If the map contains certain keys, one or more handlers must be invoked. Here is a matrix that shows which handler is invoked for which key.

For example, if the map has key a, then DocHandler and AlertHandler need to be called. If it has key b, then NoteHandler and AlertHandler are called. In reality there might be more keys and more handlers, but for simplicity we limit our example to three keys and three handlers.