Lambda Jam 2013 - Side Notes

This is my impressionistic non-canonical irregular Clojuresque-Erlangish notes on Lambda Jam conference which took place in Chicago on July 8–10, 2013. These notes are pretty long, and I don’t split them on purpose. If you read them all, you should become overwhelmed and overloaded with the information. Only this way you can feel the same I felt on the last day of the conference :)

Stuart Sierra — Data, Visibility, and Abstraction

Video: InfoQ

Quotes

QBasic distinguished between subroutines and functions.
Perl is a QBasic of Linux.
Perl provided bunch of abstractions that made my life easier.
Just a few generic data structures can represent pretty much any kind of data.

#!/usr/bin/env perl
 
my $user =
  { name  => "Stuart",
    age   => 15,
    langs => [ "BASIC",
               "C++",
               "Perl" ] };
 
print $user->{'langs'}[2];

Tie::File — Access the lines of a disk file via a Perl array.
XSLT is a homoiconic programming language.
[In XSLT] you can write the entire program as a series of data transformations.

#!/usr/bin/env bash
saxon one.xslt site.xml t1.xml
saxon two.xslt t1.xml t2.xml
saxon three.xslt t2.xml t3.xml
saxon four.xslt t3.xml index.html

Clojure has universal data structures.

(def user {:name "Stuart"
           :age 25
           :langs ["Lisp"
                   "Ruby"
                   "Clojure"]})

I’m frequently suspicious of libraries that use a lot of macros to create their abstractions because it means I can’t see them, I can’t manipulate them with the tools I already have.

(require 'clojure.inspector)
(clojure.inspector/inspect-tree user)

I started to write my programs as a series of data transformations with just one set of side-effects at the very end.

(defn gather-information [state]
  (assoc state :analysis
               (computation (:input state))))
 
(defn make-decision [state]
  (assoc state :response
               (if (condition? (:analysis state))
                   :launch-missile
                   :erase-hard-drive)))
 
(defn take-action [state]
  (case (:response state)
    :launch-missile (launch-missile)
    :erase-hard-drive (erase-hard-drive)))
 
(defn complex-process [initial-state]
  (-> initial-state
      gather-information
      make-decision
      take-action))

I try to pursue abstractions that make programs more visible, make easier to see what the program is doing.

Being abstract is something profoundly different from being vague…

The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.

Edsger Dijkstra

Aditya Siram — Simile-Free Monad Recipes

Slides: GitHub

Non-idiomatic Haskell monad tutorial

Working with monads is a switching between monadic and non-monadic context.

IO

<- operator gets the value from monadic context.
return puts the value into monadic context.

main :: IO ()
main = do
  writeFile "test.txt" "a,b,c,d,e" :: IO ()
  x :: String <- readFile "test.txt" :: IO String
  let upCased :: String = map toUpper x
  y :: String <- return upCased :: IO String
  print y :: IO ()

Reader

Reader = Read-only State + Result
runReader :: Reader Monad -> Read-Only State -> Result
ask extracts the state from the monad for inspection.

Writer

Writer = Append-Only State + Result
runWriter :: Writer Monad -> (Result, Accumulated State)
State is accumulated using tell.

State

State Monad = Mutable State + Result
get, put do what they sound like.
runState :: State Monad -> Initial State -> (Result, New State)
Initial State is required.

Dean Wampler — Copious Data, the “Killer App” for FP

Slides: GitHub
Video: InfoQ

Quotes

It’s hard to implement many algorithms in MapReduce.
MapReduce is very course-grained.
For Hadoop in particularly, the Java API is hard to use.
Hadoop is the EJBs of our time

import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import java.util.*;
 
class WCMapper extends MapReduceBase
    implements Mapper<LongWritable, Text, Text, IntWritable> {
 
    static final IntWritable one = new IntWritable(1);
    static final Text word = new Text(); // Value will be set in a non-thread-safe way!
 
    @Override
    public void map(LongWritable key, Text valueDocContents,
            OutputCollector<Text, IntWritable> output, Reporter reporter) {
        String[] tokens = valueDocContents.toString().split("\\s+");
        for (String wordString: tokens) {
            if (wordString.length > 0) {
                word.set(wordString.toLowerCase());
                output.collect(word, one);
            }
        }
    }
}
 
class Reduce extends MapReduceBase
    implements Reducer<Text, IntWritable, Text, IntWritable> {
 
    public void reduce(Text keyWord, Iterator<IntWritable> valuesCounts,
            OutputCollector<Text, IntWritable> output, Reporter reporter) {
        int totalCount = 0;
        while (valuesCounts.hasNext()) {
            totalCount += valuesCounts.next().get();
        }
        output.collect(keyWord, new IntWritable(totalCount));
    }
}

Use Cascalog

(defn lowercase [w] (.toLowerCase w))
 
(?<- (stdout) [?word ?count]
  (sentence ?s)
    (split ?s :> ?word1)
  (lowercase ?word1 :> ?word)
    (c/count ?count))

Use Spark

object WordCountSpark {
    def main(args: Array[String]) {
        val file = spark.textFile(args(0))
        val counts = file.flatMap(line => line.split("\\W+"))
                         .map(word => (word, 1))
                         .reduceByKey(_ + _)
        counts.saveAsTextFile(args(1))
    }
}

Data problems are fundamentally Mathematics!
Data will drive widespread FP adoption.

Resources

Cascading — an application framework for Java developers to simply develop Data Analytics applications on Hadoop.
Nathan Marz — Introducing Cascalog: a Clojure-based query language for Hadoop. [blog]
Storm — free and open source distributed realtime computation system.
Spark — an open source cluster computing system that aims to make data analytics fast.
Evan Miller — The Mathematical Hacker. [blog]

Jam

A jam is similar to a code retreat, only you work in groups instead of pairs. It starts with a problem description. Then you form a group and work on the problem for three hours. In the end you share your experience with the audience.

The problem of the first jam was Peter Norvig’s Spelling Corrector. I was wandering the room looking for an Erlang group to join when I bumped into this bunch of wonderful people: Bruce Adams, Joe Armstrong, Garrett Smith, Bryan Hunter, and Karl Grzeszczak.

Soon the jam transformed into an Erlang master class from Joe Armstrong. We were watching Joe’s work flow, his way of thinking, learning his tips and tricks, listening to his brilliant comments about Erlang and Haskell.

At some point I remarked how elegant was the function Joe Armstrong just implemented, on which he replied: There is not much intelligence here. It’s all about practice.

Today I watched @joeerl implementing a parallel spell checker in Erlang. That was worth the price of admission alone. Thank you! #LambdaJam @karl_grz

Resources

Source code

Joe Armstrong — Keynote

Systems That Run Forever Self-Heal and Scale

Slides: GitHub
Video: InfoQ

Quotes

I’m not interested in programming languages. I’m interested in solving problems.
Primary goal Erlang was designed for is the fault-talerant computation.
I think it’s a bad idea to design your system for 10 people and then scale it up for 10,000. It’s better to design it for 10M and scale it down.
The difficult part of making reliable system is to make multiple machines work independently in parallel.

Why distributed programming is hard

Difficulty in identifying and dealing with failures.
Achieving consistency in data across processes.
Heterogeneous nature of the components involved in the system.
Testing a distributed system is quite difficult.
The technologies involved in distributed systems are not easy to understand.

Chord algorithm: distributing data across several machines

S1 IP = 235.23.34.12
S2 IP = 223.23.141.53
S3 IP = 122.67.12.23
 
md5(ip(s1)) = C82D4DB065065DBDCDADFBC5A727208E
md5(ip(s2)) = 099340C20A42E004716233AB216761C3
md5(ip(s3)) = A0E607462A563C4D8CCDB8194E3DEC8B
 
Sorted:
099340C20A42E004716233AB216761C3 => s2
A0E607462A563C4D8CCDB8194E3DEC8B => s3
C82D4DB065065DBDCDADFBC5A727208E => s1
 
lookup Key = "mail-23412"
md5("mail-23412") => B91AF709D7C1E6988FCEE7ADF7094A26
 
So the Value is on machine s3 (first machine with Md5 lower than md5 of key)
 
Replica:
md5(md5("mail-23412")) => D604E7A54DC18FD7AC70D12468C34B63
 
So the replica is on machine s1

What if master dies? Paxos — distributed leadership election algorithm. Very complicated algorithm that only few people understand.
The leadership election is solved in OTP (mnesia, gen_leader) and Riak.
You don’t need libraries to write web server. Any fool can write them. But distributed data storage is difficult.

Six rules for building HA systems

Isolation
Concurrency
Failure detection
Fault identification
Live code upgrade
Stable storage

With stable storage you don’t need backups. You need snapshots, because you override the data, but you don’t need backups.
Threads are evil because they share resources.
We already solved the problem with parallel computing (in Erlang). We are working on detecting bottlenecks now.

Resources

Ericsson SGSN-MME.
Rajith Attapattu — 5 reasons why Distributed Systems are hard to program. [blog]
Chord algorithm.
Paxos algorithm.
Leslie Lamport — Paxos Made Simple. [pdf]
Thomas Arts, Koen Claessen, Hans Svensson — Semi-formal development of a fault-tolerant leader election protocol in Erlang. [article]
Concurix — performance tool for Erlang.

Erlang beer with Joe Armstrong

For those who didn't hear, we're meeting up at The Public House - 400 N State St. @gar1t

This was the best part of the first day :)

John Daily — Distributed Programming with Riak Core and Pipe

Slides: GitHub

Distributed programming is hard (clocks, latency, lost messages, servers break). Use Riak.

Resources

Sources: Riak core and pipe.
Eric Redmond — A Little Riak Book.

Tracy Harms — Semantics, clarity, and notation: the benefits of expressions over statements

@kaleidic’s “Benefits of Expressions Over Statements” is gold. Brain is barely keeping up. @bryan_hunter

Expressions condense meaning in space, and eliminate time.
Understanding occurs only when meaning is selected and simplified enough for a mind to think about it.

This is the example @kaleidic is showing now @gazoombo

allmasks=: 2 #:@i.@^ #
firstend=: 1 0 i.&1@E."1 ]
laststart=: 0 1 {:@I.@E."1 ]
noncont=: <@#~ (#~ firstend < laststart)@allmasks
 
   noncont 1+i.4
┌───┬───┬───┬─────┬─────┐
│2 4│1 4│1 3│1 3 4│1 2 4│
└───┴───┴───┴─────┴─────┘
 
   noncont 'aeiou'
┌──┬──┬──┬───┬───┬──┬──┬───┬──┬───┬───┬────┬───┬───┬────┬────┐
│iu│eu│eo│eou│eiu│au│ao│aou│ai│aiu│aio│aiou│aeu│aeo│aeou│aeiu│
└──┴──┴──┴───┴───┴──┴──┴───┴──┴───┴───┴────┴───┴───┴────┴────┘
 
   #noncont i.10
968

Resources

P. J. Landin — The Next 700 Programming Languages. [pdf]
Tracy Harms — J: A Programming Language. [pdf]

Chris Ford — Functional composition

Video: InfoQ
Demo source code: GitHub

Quotes

Western music notation is a DSL designed to be executed on a pecular kind of FSM called the musician.
Sound error correction happening in the brain.
Given the audience, let’s try F sharp blues.
A canon is defined as a series of notes that are accompanied by a functional transformation of themselves.

(defn canon [f notes] (concat notes (f notes)))

These are all pure functions, so we can compose them together. And that’s what composers did long before the lambda calculus was invented.

Clinton Dreisbach — Functional Web Development with Clojure

The Clojure way of web development: lots of loosely coupled libraries.

Ring

(defproject request-echo "0.1.0-SNAPSHOT"
 
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [ring "1.1.8"]]
 
  :plugins [[lein-ring "0.8.3"]]
 
  :ring {:handler request-echo/handler
         :port 3001})

lein ring server

(ns request-echo
  (:require [clojure.pprint :refer [pprint]]))
 
(defn handler [request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body (str "<h1>Request Echo</h1><pre>"
              (with-out-str (pprint request))
              "</pre>")})

Compojure

(require '[compojure.core :refer :all])
(require '[compojure.route :as route])
 
(routes
  ;verb  route   parameters        handler
  (GET   "/"     []                (index-page))
  (GET   "/debts/:person" [person] (person-page person))
  (GET   "/add-debt" []            (add-debt-page))
  (POST  "/add-debt" [from to amount]
         (add-debt-post {:from from, :to to, :amount amount}))
  (route/resources "/")
  (route/not-found "Page not found"))

Hiccup

(require '[hiccup.core :refer [html]])
 
(html [:a.btn         ; element + class or id
       {:href "/go"}  ; map for attributes
       "Click here"]) ; Content
;;=> "<a class="btn" href="/go">Click here</a>"

Cheshire

(require '[cheshire.core :as json])
 
(let [debts (:debts @db)
      balances (debts/balances debts)]
  {:status 200
   :headers {"Content-Type" "application/json"}
   :body (json/generate-string
          {:debts debts
           :balances balances})})
 
(routes
  (POST "/add-debt.json" {body :body}
        (views/add-debt-json db (slurp body))))
 
(defn add-debt-json [db body]
  (json/parse-string body))

Garden

(require '[garden.units :as u :refer [px pt]])
(require '[garden.core :refer [css]])
 
(def default-color "#EFE")
 
[[:body
  {:background-color default-color}]
 
 [:.btn-primary
  {:border-width (px 5)}
  [:&:hover
   {:border-color "black"}]]]
 
(routes
  (GET "/*.css" { {path :*} :route-params}
       (views/css-page-memoized path)))
 
(defn css-page [path]
  (when-let [garden-url (io/resource (str "public/" path ".garden"))]
    (let [garden-data (load-file (.getPath garden-url))]
      {:status 200
       :headers {"Content-Type" "text/css"}
       :body (css garden-data)})))
 
(def css-page-memoized (memoize css-page))

Resources

lib-noir — a set of utilities and helpers for building ring apps.
Luminus — another Clojure web framework.
ClojureScript — Clojure to JS compiler.
Pedestal — another framework.
Liberator — WebMachine in Clojure.

Gerald Sussman — Keynote

Programming for the Expression of Ideas

Slides: GitHub
Video: InfoQ

Gerald Sussman on stage and Joe Armstrong in the front row nodding along. #LambdaJam is a special place. @bryan_hunter

Lagrange’s equations of motion in Leibniz notation (with type violation)

$\frac{\mathrm d}{\mathrm d t} \left(\frac{\partial L}{\partial \dot{q}^i}\right) - \frac{\partial L}{\partial q^i} = 0$

Expanded form (correct but ugly)

$\frac{\mathrm d}{\mathrm d t} \left(\left.\frac{\partial L(t,q,\dot{q})}{\partial \dot{q}}\right|_{\begin{aligned}q &= w(t)\\ \dot{q} &= \frac{\mathrm d w(t)}{\mathrm d t}\end{aligned}}\right) - \left.\frac{\partial L(t,q,\dot{q})}{\partial q}\right|_{\begin{aligned}q &= w(t)\\ \dot{q} &= \frac{\mathrm d w(t)}{\mathrm d t}\end{aligned}} = 0$

Functional form (correct and beautiful)

$D((\partial_2 L) \circ (\Gamma[w])) - (\partial_1 L) \circ (\Gamma[w]) = 0$

where

$(Df)(t) = \left.\frac{\mathrm d}{\mathrm d x} f(x)\right|_{x=t} \qquad \Gamma[w](t) = (t, w(t), Dw(t))$

As Scheme code

(define ((Lagrange-equations Lagrangian) w)
  (- (D (compose ((partial 2) Lagrangian)
                 (Gamma w)))
     (compose ((partial 1) Lagrangian)
              (Gamma w))))
 
(define ((Gamma w) t)
  (up t (w t) ((D w) t)))

We can even generate LaTeX from the Scheme code.

(show-expression
  (((Lagrange-equations (L-harmonic 'm 'k))
    proposed-solution)
   't))

The moral: originally, Lagrange’s equations had missing parameters and a type error. Programming them forced an elegant and effective statement.

Quotes

Computer revolution changed the way we think.
The way we teach students is not the same way we do it ourselves.
I’ve been doing it all my life. My real goal is to transform things that are hard to understand to the things that are easy to understand. Programming was one of the tools of doing that.
Students have to learn simultaneously the language and the culture, as well as the content.
I like to tell students what is going on.
Idioms make reading hard.
Leibniz notation for the derivatives happens to be a disaster.
Let’s get to the General Relativity… I’ll be mercifully short.
The point is, it makes it comprehensible what was badly expressed in the traditional form.
I have no problem of enslaving electronic apparatus.

Resources

Michael Spivak — Calculus. [pdf]
Gerald Jay Sussman, Jack Wisdom — Functional Differential Geometry. [MIT]
Marvin Minsky — Why programming is a good medium for expressing poorly understood and sloppily-formulated ideas. [html]

Erlang cocktail

with Bryan Hunter, Joe Armstrong, and Kevin Scaldeferri

Wonderful ending of the second day.

Dave Thomas — Living in Big Data with Vector Functional Programming

Slides: GitHub

This is the first time I’ve been in the audience of a talk on vector languages. @kaleidic

Extreme cases — that’s what is interesting.
I don’t do Big Data, I don’t have a petabyte in my pocket.
I’m an industrial language pimp.
I’ve got 3 characters on the screen and I have no idea what they’re doing.
What slows languages down is scalars and non-scalars, boxing and unboxing.
How to learn an array language? Slowly.

Pairing is a great way to learn new things.
Most programs, by 2020, will be queries.
There are five error messages, they are all irritating.
WTF (what the function?) error message.
All IDEs are bad. Intellij is the best of worst.

Resources

Emily Bache — An introduction to Array Languages. [blog]
Bryan Cantrilla — Conversation with Arthur Whitney. [article]

Steve Vinoski — Addressing Network Congestion in Riak Clusters

Slides: Dropbox
Video: YouTube
Riak — A distributed highly available eventually consistent highly scalable open source key-value database written primarily in Erlang, built for operational ease.
Riak TCP traffic:
1. Client requests: made to any node in the ring
2. Coordination: node receiving client request coordinates the operation across the owning replicas
3. Gossip: Riak nodes share ring state via a gossip protocol
4. Active Anti-Entropy: nodes actively verify and repair data consistency across the ring
5. Erlang: distributed Erlang nodes form a full mesh and do periodic node availability checks
6. Handoff
TCP incast.
Low Extra Delay Background Transport (LEDBAT).
Micro Transport Protocol (μTP, or uTP).

Resources

Amazon’s Dynamo Paper and Riak.
libutp — The uTorrent Transport Protocol library.
gen_utp — an API and driver for the uTP protocol.

Mahesh Paolini-Subramanya — Finite State Machines. Why the fear?

Slides: Slideshare

Sitting behind @webyrd, @dfried00, Sussman, and @joeerl in @dieswaytoofast’s finite state machine talk. @gazoombo

Write a project in C++ or Java and you’re creating work for 5 other devs. Write it in Erlang and you’re done.
There is a special place in hell for the people who came up with the Oauth 2.0 spec.
Everything is an FSM. Problem with actually modeling this is complexity of large FSMs. Answer is encapsulation.
Design your system as a big FSM that is a collection of little FSMs. Only transition between them as needed.

Sean Cribbs, Chris Meiklejohn — Functional Web Applications with Webmachine

The keyword here is Functional.

Slides: GitHub
Tutorial: GitHub
Webmachine diagram

git clone git@github.com:cmeiklejohn/webmachine-tutorial.git

git checkout hello-world

f(ReqData,State) -> {RetV,ReqData,State}.
iolist()

src/tweeter_resource.erl

init([]) ->
    {ok, []}.
 
to_html(ReqData, State) ->
    {["<html><body>", wrq:get_req_header("host", ReqData), "</body></html>"],
     ReqData,
     State}.

Supervisor: resources, routes, dispatch

git checkout -f load-tweets

src/tweeter_sup.erl

init([]) ->
    ...
    Resources = [tweeter_wm_tweet_resource, tweeter_wm_asset_resource],
    Dispatch = lists:flatten([Module:routes() || Module <- Resources]),
    ...

src/tweeter_wm_tweet_resource.erl

routes() ->
    [{["tweets"], ?MODULE, []}].

src/tweeter_wm_asset_resource.erl

routes() ->
    [{[""], ?MODULE, []}, {['*'], ?MODULE, []}].

Media Types

src/tweeter_wm_tweet_resource.erl

content_types_provided(ReqData, Context) ->
    {[{"application/json", to_json},
      {"application/x-erlang-binary", to_erlang}], ReqData, Context}.
 
to_json(ReqData, Context) ->
    Tweets = [Value || [{_Key, Value}] <- ets:match(tweets, '$1')],
    Content = mochijson2:encode({struct, [{tweets, Tweets}]}),
    {Content, ReqData, Context}.
 
to_erlang(ReqData, Context) ->
    Tweets = [Value || [{_Key, Value}] <- ets:match(tweets, '$1')],
    Content = term_to_binary(Tweets),
    {Content, ReqData, Context}.

$ curl -i -H "Accept:application/x-erlang-binary" "http://localhost:8080/tweets"
HTTP/1.1 200 OK
Vary: Accept
Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
Content-Type: application/x-erlang-binary
 
?llhdavatarmZhttps://si0.twimg.com/profile_images/2536088319/4sl2go65was3o0km520j_reasonably_small.jpeghdmessageCaremad.jlhdavatarm\https://si0.twimg.com/profile_images/3778090444/e4fde2cad4b921cd8c07fcecc0ff2fff_bigger.jpeghdmessagemRubby is over!jlhdavatarmZhttps://si0.twimg.com/profile_images/2536088319/4sl2go65was3o0km520j_reasonably_small.jpeghdmessagemYou boys having a taste?jj

resource_exists

git checkout -f tweet-urls

src/tweeter_wm_tweet_resource.erl

routes() ->
    [{["tweets", tweet_id], ?MODULE, []}].
 
tweet_id(ReqData) ->
    time_from_timestamp(wrq:path_info(tweet_id, ReqData)).
 
resource_exists(ReqData, Context) ->
    case maybe_retrieve_tweet(Context, tweet_id(ReqData)) of
        {true, NewContext} -> {true, ReqData, NewContext};
        {false, Context}   -> {false, ReqData, Context}
    end.

$ curl http://localhost:8080/tweets
{"tweets":[
  {"avatar":"https://si0.twimg.com/profile_images/2536088319/4sl2go65was3o0km520j_reasonably_small.jpeg","message":"Caremad.","id":"1376843311536798"},
  {"avatar":"https://si0.twimg.com/profile_images/3778090444/e4fde2cad4b921cd8c07fcecc0ff2fff_bigger.jpeg","message":"Rubby is over!","id":"1376843311536799"},
  {"avatar":"https://si0.twimg.com/profile_images/2536088319/4sl2go65was3o0km520j_reasonably_small.jpeg","message":"You boys having a taste?","id":"1376843311536800"}]}

$ curl -i http://localhost:8080/tweets/1376843311536798
HTTP/1.1 200 OK
 
{"tweet":{"avatar":"https://si0.twimg.com/profile_images/2536088319/4sl2go65was3o0km520j_reasonably_small.jpeg","message":"Caremad."}}

$ curl -i http://localhost:8080/tweets/42
HTTP/1.1 404 Object Not Found
 
<HTML><HEAD><TITLE>404 Not Found</TITLE></HEAD><BODY><H1>Not Found</H1>The requested document was not found on this server.<P><HR><ADDRESS>mochiweb+webmachine web server</ADDRESS></BODY></HTML>

POST, response header

git checkout -f create-tweets

src/tweeter_wm_tweets_resource.erl

allowed_methods(ReqData, Context) ->
    {['HEAD', 'GET', 'POST'], ReqData, Context}.
 
post_is_create(ReqData, Context) ->
    {true, ReqData, Context}.
 
create_path(ReqData, Context) ->
    case maybe_create_tweet(ReqData, Context) of
        {true, NewContext} ->
            {Id, _} = NewContext#context.tweet,
            Resource = "/tweets/" ++ binary_to_list(time_to_timestamp(Id)),
            NewReqData = wrq:set_resp_header("Location", Resource, ReqData),
            {Resource, NewReqData, NewContext};
        {false, Context} ->
            {"/users", ReqData, Context}
    end.
 
content_types_accepted(ReqData, Context) ->
    {[{"application/json", from_json}], ReqData, Context}.
 
from_json(ReqData, Context) ->
    case maybe_create_tweet(ReqData, Context) of
        {true, NewContext} ->
            {_, Tweet} = NewContext#context.tweet,
            Response = mochijson2:encode({struct, [{tweet, Tweet}]}),
            NewReqData = wrq:set_resp_body(Response, ReqData),
            {true, NewReqData, NewContext};
        {false, Context} ->
            { {halt, 409}, ReqData, Context}
    end.

$ curl -i -X POST -H "Content-Type:application/json" \
      http://localhost:8080/tweets --data @-
{
  "tweet":{
    "message":"testing...",
    "avatar":"https://si0.twimg.com/profile_images/3778090444/e4fde2cad4b921cd8c07fcecc0ff2fff_bigger.jpeg"
  }
}
HTTP/1.1 201 Created
Location: /tweets/1376847941255278
Content-Type: application/json
 
{"tweet":{"message":"testing...","avatar":"https://si0.twimg.com/profile_images/3778090444/e4fde2cad4b921cd8c07fcecc0ff2fff_bigger.jpeg"}}

$ curl -i -X POST -H "Content-Type:application/xml" \
      http://localhost:8080/tweets --data @-
{
  "tweet":{
    "message":"testing...",
    "avatar":"https://si0.twimg.com/profile_images/3778090444/e4fde2cad4b921cd8c07fcecc0ff2fff_bigger.jpeg"
  }
}
HTTP/1.1 415 Unsupported Media Type
Location: /tweets/1376848621307991
Content-Type: text/html
 
<html><head><title>415 Unsupported Media Type</title></head><body><h1>Unsupported Media Type</h1>Unsupported Media Type<p><hr><address>mochiweb+webmachine web server</address></body></html>

Note: create_path is called before content_types_accepted

$ curl -i -X POST -H "Content-Type:application/xml" \
      http://localhost:8080/tweets --data @-
<tweet>
  <message>testing...</message>
  <avatar>https://si0.twimg.com/profile_images/3778090444/e4fde2cad4b921cd8c07fcecc0ff2fff_bigger.jpeg</avatar>
</tweet>
HTTP/1.1 500 Internal Server Error
Content-Type: text/html
 
<html><head><title>500 Internal Server Error</title></head><body><h1>Internal Server Error</h1>The server encountered an error while processing this request:<br><pre>{"create_path not a string",
 {error,
     {...}}}</pre><P><HR><ADDRESS>mochiweb+webmachine web server</ADDRESS></body></html>

ETags, caching, preconditions

git checkout -f etag-tweets

src/tweeter_wm_tweets_resource.erl

generate_etag(ReqData, Context) ->
    {_, NewContext} =  maybe_retrieve_tweets(Context),
    ETag = mochihex:to_hex(erlang:phash2(NewContext#context.tweets)),
    {ETag, ReqData, NewContext}.
 
last_modified(ReqData, Context) ->
    Id = ets:last(tweets),
    {calendar:now_to_datetime(Id), ReqData, Context}.

$ curl -i http://localhost:8080/tweets
HTTP/1.1 200 OK
Last-Modified: Sun, 18 Aug 2013 21:44:50 GMT
ETag: "30008d7"
 
{"tweets":[
  {"avatar":"https://si0.twimg.com/profile_images/2536088319/4sl2go65was3o0km520j_reasonably_small.jpeg","message":"Caremad.","id":"1376860726482913"},
  {"avatar":"https://si0.twimg.com/profile_images/3778090444/e4fde2cad4b921cd8c07fcecc0ff2fff_bigger.jpeg","message":"Rubby is over!","id":"1376860726482914"},
  {"avatar":"https://si0.twimg.com/profile_images/2536088319/4sl2go65was3o0km520j_reasonably_small.jpeg","message":"You boys having a taste?","id":"1376860726482915"}]}

$ curl -i -H 'If-None-Match:"30008d7"' http://localhost:8080/tweets
HTTP/1.1 304 Not Modified
ETag: "30008d7"

$ curl -i -H 'If-Modified-Since:Sun, 18 Aug 2013 21:45:41 GMT' http://localhost:8080/tweets
HTTP/1.1 304 Not Modified
ETag: "30008d7"

Authorization, CSRF

git checkout -f csrf

src/tweeter_wm_tweets_resource.erl

forbidden(ReqData, Context) ->
    {tweeter_security:is_protected(ReqData, Context), ReqData, Context}.

Visual debugger

git checkout -f debugger

src/tweeter_wm_asset_resource.erl

init([]) ->
    wmtrace_resource:add_dispatch_rule("wmtrace", "/tmp"),
    { {trace, "/tmp"}, #context{}}.

ErlyDTL, Dialyzer

Resources

Webmachine: Request Data API
James Hague — A Ramble Through Erlang IO Lists. [blog]
Andrei Neculau — http-decision-diagram

David Nolen — Keynote

Everything I Have Learned I Have Learned From Someone Else

Slides: GitHub
Video: InfoQ

It is practically impossible to teach good programming to students that have had a prior exposure to BASIC: as potential programmers they are mentally mutilated beyond hope of regeneration.

Edsger Dijkstra

Quotes

We sick of the status quo, and there is a resurgence among engineers and excitement around new programming languages.
In Clojure we can pattern match on persistent vectors and hashmaps.

(let [x true
      y true
      z true]
  (match [x y z]
    [_ false true] 1
    [false true _ ] 2
    [_ _ false] 3
    [_ _ true] 4
    :else 5))
 
(match [x]
  [{:a _ :b 2}] 1
  [{:a 1 :b 1}] 2
  [{:c 3 :d _ :e 4}] 3
  :else 4)

The best way to predict the future is to read papers and engineer it.

Papers

Phil Bagwell — Ideal Hash Trees [pdf]
Phil Bagwell, Tiark Rompf — RRB-Trees: Efficient Immutable Vectors [pdf]
William E. Byrd — Relational Programming in miniKanren: Techniques, Applications, and Implementations [pdf]
David C. Bender, Lindsey Kuper, William E. Byrd, Daniel P. Friedman — Efficient representations for triangular substitutions: A comparison in miniKanren [pdf]
William A. Kornfeld — Equality for Prolog [pdf]
Philip Wadler, Stephen Blott — Ho to make ad-hoc polymorphism less ad hoc [ps]
James Cheney, Christian Urban — Nominal Logic Programming [pdf]
Alexey Radul, Gerald Jay Sussman — The Art of the Propagator [pdf]
Craig Chambers, Weimin Chen — Efficient Multiple and Predicate Dispatching [pdf]
Luc Maranget — Compiling Pattern Matching to good Decision Trees [pdf]
Philip Wadler — Views: A way for pattern matching to cohabit with data abstraction [ps]
Sam Tobin-Hochstadt — Extensible Pattern Matching in an Extensible Language [pdf]
Anurag Mendhekar, Gregor Kiczales, John Lamping — Compilation Strategies as Objects [pdf]
Emden R. Gansner, John H. Reppy — A Multi-threaded Higher-order User Interface Toolkit [ps]
Conal Elliott, Paul Hudak — Functional Reactive Animation [pdf]

Books

Chris Okasaki — Purely Functional Data Structures [pdf]
Daniel P. Friedman, William E. Byrd, Oleg Kiselyov — The Reasoned Schemer [MIT]
Peter Van Roy, Seif Haridi — Concepts, Techniques, and Models of Computer Programming [MIT]
Gregor Kiczales, Jim des Rivieres, Daniel G. Bobrow — The Art of the Metaobject Protocol [MIT]
C. A. R. Hoare — Communicating Sequential Processes [pdf]

Resources

core.logic — GitHub
core.match — GitHub

Bonus 1: Joe Armstrong @ CEUG — 26 Years With Erlang

Video: YouTube

If you are an academic you think to develop a programming language in three or four years because that’s the time it takes to get your PhD, then you finish your PhD and the whole world will use it. It doesn’t work like that. You got to do quite few other things.
That’s what we did in 1985, before the Internet: we were creating programming languages.
Version 1.03 lost in the mists of time.
When Prolog program goes wrong, it says No.

uni --->
    case(wait,[
        n => [term,exit],
        h => [hold,
              case(new_call,[
                  connected => multi,
                  n => case(gone_away,[
                           yes => exit,
                           no => [conv,uni]
                       ]),
                  h => [conv,uni]
              ])
             ]
    ]).

This diagrams are nested state machines.
You do not program the abnormal things, you do not make any decisions about how to program the stuff which is outside of the specification. What you do is you crash your program and let somebody else to resolve the problem and put back all the invariants.
It took four days to re-write the whole Erlang.
That’s the entire documentation of Erlang 1.05

h                help
reset            reset all queues
reset_erlang     kill all erlang definitions
load(F)          load erlang file <F>.erlang
load             load the same file as before
load(?)          what is the current load file
what_erlang      list all loaded erlang files
go               reduce the main queue to zero
send(A,B,C)      perform a send to the main queue
send(A,B)        perform a send to the main queue
cq               see queue — print main queue
wait_queue(N)    print wait_queue(N)
cf               see frozen — print all frozen states
cqns             see all equations
cqns(N)          see equation(N)
start(Mod,Goal)  starts Goal in Mod
top              top loop run system
q                quit top loop
open_dots(Node)  opens Node
talk(N)          N=1 verbose, =0 silent
peep(M)          set peeping point on M
no_peep(M)       unset peeping point on M
vsn(X)           erlang vsn number is X

Don’t speculate about performance. Write the program, run it, and measure it.
Robert collected the whole pile of papers on how abstract machines worked. I borrowed this file and took it for a weekend, and I read every single paper from beginning to end… and I understood nothing. Then Monday morning I suddenly woke up and I understood it.
There is no garbage collection in atom table.
The original Erlang movie was made for ISS 90. And we had script!
Robert wanted to buy a train set on the money from the lab.
“Amazing but true! Blindingly fast!”
8 Dec 1995, AXE-N cancelled. 1996 AXD 301 started.
AXD 301 could switch up to 160Gb/sec. For countries of the size of Sweeden you can only sell one. I think British Telecom bought 3 or something. It was great technical success but it didn’t earn any money.
Banning things has interesting consequences. Erlang got open sourced. Four days after Erlang was banned all people who developed it left Ericsson and started their own company Bluetail.
Now DNA of Erlang is spreading through various companies.

Bonus 2: Joe Armstrong @ CEUG — Sherlock’s Last Case

Video: YouTube

“How can you be more efficient programmer? By not programming.”
When you program for 20–30 years, the “make it work” challenge is gone away. It’s more challenging to think what problem you are going to solve.
The problem I’ve been thinking for the last five years is how to organize the data, how to organize ideas.
Value store is a key-value store without keys. How to get the data out of this database?
Sherlock’s Problem: There is $X$, and there are thousands of $Y_i$. Which $Y_i$ is the nearest to $X$?
The categorization problem is extremly difficult. That’s why object-oriented programming is stupid.
Concurrency oriented programming is a physical modelling.
Measures of similarity. TF*IDF.
Naive Bayesian

$P(A|B) P(B) = P(B|A) P(A)$

Normalized compression difference: If $A$ and $B$ are similar then size(compress($A$++$B$) will be wee bit larger than size(compress($A$)). It’s insensitive to choice of compression algorithm.
Idea for IDE: Social programming network. It shows all the people working on the code similar to what you are typing.

Q & A

Macros and include files should be removed from the language.
Maybe atoms should be garbage collected.
The biggest problem in building technical systems is ‘connecting thing together’ problem.
If I need to write a language, it wouldn’t be programming language. I would write a protocol description language.
Principle of observational equivalence.
UBF.
“There are languages that people bitch about, and there are languages that nobody’s using.” — Bjarne Stroustrup
Scale change. Petabyte change is enormously interesting.
We need to make cryptography available to all the people.
Take the data out of the cloud, and put it into your home clouds.
NSA will kill Google and Facebook.

Epilogue

As I leave #LambdaJam I’m particularly happy at my improved sense of being able to think about OO stuff in FP terms. That may serve me well. @kaleidic

Life returning to normal. Have been suffering for PCSD (Post Conference Stress Disorder) — too many ideas in too short time + Jet Lag @joeerl