Stanford’s Machine Learning week 1 - What I’ve been doing?!

You may enjoy this article more if you know Differential Calculus.

This was my feeling during the first 7-minutes video lecture.

I’ve been listening about machine learning and AI for years as a remarkable thing, still being developed by researchers in the best universities of the world and to aimed in a PhD. Still with this idea, I decided to attend Stanford’s Machine Learning course offered on Coursera.

For me, one of its popular definitions reforces this idea of being something magical:

Machine Learning: Field of study that gives computers the ability to learn without being explicitly programmed.
Arthur Samuel (1959)

Knowing enough about machines, I used to ask myself: “How can a computer do something I haven’t explicitly programmed?”. The answer is (at least in the scenario presented by Andrew Ng in the course): it doesn’t. Even without fully understanding all the models presented, I can already come up with probably not so efficient but similar solutions to the same problems.

Well-posed Learning Problem: A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.
Tom Mitchell (1998)

Putting aside the letters which may cause some trouble on understanding it, defines Machine Learning as something more tangible to programmers:

Recording and analyzing results from predefined tasks, the software will group them into satisfactory or not. Satisfactory may suffer another analysis attempting to improve and reach the performance expected.
Irio Musskopf (2014)

Andrew classifies Machine Learning algorithms between supervised and unsupervised learning. Both need to be fed with data to be analyzed.

Supervised learning

It’s when we give right answers together with the data. One of its examples - and the most explored in this week - is the housing price prediction. Based on a database of house characteristics (starting with size in feet), the algorithm should try to answer the question of how much a given house is worth.

In this case, it’s supervised because we already know the answer for some prior data. Two houses of 1000 feet squared were sold by $280,000 and $300,000. For houses of 1500, we might guess something like $400,000. Greater the amount of data fed, more trustable the guesses become. It’s also a problem of regression analysis, because we’re trying to build a continuous line between all possible “feet squared“ units, even having just some answers in the start.

Another possible case for supervised learning is the one called classification. In front of a question asking for yes/no, it classifies the data between the distinct groups.

In the course example, breast cancer is categorized between malignant/benign - using the question “Is malignant?” - according to the tumor size.

My most exciting thought about supervised learning is that we don’t need (sometimes we should, but it isn’t required) to build a fancy formula relating each characteristic with the results. You may build a software to analyze existing data and find relationships between variables and results for you. And there is no magic included in the process.

Linear Regression as a solution for regression problem

Be careful: section with explicit Math.

Two dimensional graph with blue dots dispersed over a red line

Imagining this (axes and blue dots) as a graph plot of the housing prices from our existing data, one possible tool for doing regression analysis is Linear Regression. It will give us a straight line corresponding as near as possible with the training examples, represented by the blue dots.

$$h_\theta(x) = \theta_0 + \theta_1x$$

Hypotheses for the red line must follow the format of a line equation, where \(\theta_0\) and \(\theta_1\) are constants.

To find the best solution for the equation, we must define a function to add a little more meaning for it:

$$J(\theta_0, \theta_1) = \frac{1}{2m} \sum_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)})^2$$

This function stands for giving a weight for how much the line formed with defined \(\theta_0\) and \(\theta_1\) is far from the list of training examples. Unless every each of the blue dots is on the red line, the function will always return a positive number. We aim to minimize as much as possible from this so called “cost function”. To accomplish this job, we are presented to the gradient descent algorithm.

$$\theta_j := \theta_j - \alpha\frac{\partial}{\partial\theta_j} J(\theta_0, \theta_1)$$

The colon equals operator means assignment, where you give a value for the “name” \(\theta_j\).

To accomplish the algorithm’s goal, we run the above statement twice (one for each \(\theta\) we are trying to discover the value). The first will give us the value for \(\theta_0\) and the second of \(\theta_1\).

$$\theta_0 := \theta_0 - \alpha\frac{\partial}{\partial\theta_j} J(\theta_0, \theta_1)$$ $$\theta_1 := \theta_1 - \alpha\frac{\partial}{\partial\theta_j} J(\theta_0, \theta_1)$$

We consider these two assignments a single iteration. To start, take initial values of 0 for \(\theta_0\) and 1 for \(\theta_1\).

The weird \(\frac{\partial}{\partial\theta_j}\) symbol says we should take the derivative of the cost function. In other words, on each assignment of \(\theta_j\), we follow the derivative arrow and “walk” a little more in the direction of the local minimum (the place with 0 slope).

Running one iteration over another will take care of changing the values of both \(\theta\)’s and approximating them of the local minimum. The time to stop the assignments is when the result stay unchanged between two iterations (convergence). At that point, we will have best possible values for \(\theta_0\) and \(\theta_1\). Or, if you prefer, the equation for that red line above.

Unsupervised learning

You may as well feed an algorithm with data and expect it to be grouped in some way. I don’t know how, but I want to split it between meaningful groups. It’s the same learning used in applications like Google News to populate a “related articles” section. Or put it into “business” and not “sports” category.

So far, this kind of learning wasn’t well explored yet, but the Eureka! moment stays with the possibilities. Tagging articles are useful - and may be helpful for unsupervised machine learning - but common people does not know how to use and isn’t a considerable choice for systems with size of those from Google. For the majority of cases, a computer could do a better (considering cost-benefit) job compared to a human.

Reflexões sobre o sistema bancário brasileiro

Esse post surgiu pra comentar rapidamente sobre a startup brasileira Nubank que, se auto intitulando “a nova geração de serviços financeiros no Brasil”, recebeu um investimento series A de US$14.3M na última semana.

Essa pra mim tem sido a ideia bilionária já há algum tempo. Não vejo ninguém realmente apostando sério, com o real objetivo de inovar no ramo. Por isso é fácil ficar animado com algo como o Apple Pay, que apesar de não ser coisa de outro mundo, é um passo pra tornar a interação mais simples e confortável pro usuário.

Qualquer um da geração Y (1980-2000) sabe que o sistema bancário em geral - seja em contas bancárias, pagamentos, empréstimos ou investimentos - não acompanhou as mudanças que ocorreram nas últimas décadas. Por isso fica tão difícil de confiarmos nessas empresas. Muito pouco do processo interno é público, então só aumenta o sentimento de que pagamos muito pra ter um serviço ruim.

existe regulamentação pra abertura de contas bancárias simples, que oferecem poucos serviços “offline” esperando que o cliente prefira trocar o atendimento na agência por uma taxa fixa mais baixa (possivelmente inexistente).

O que impede do banco revelar claramente o porquê de um empréstimo ter sido negado? Engenharia reversa do algoritmo de avaliação de crédito? Open Source tá aí pra mostrar que isso pode ajudar a melhorar o código e atrair clientes interessados em transparência.

Investimentos? Bom, se tu conhecer uma interface de trader (ou qualquer coisa mais simples que use meu dinheiro pra me retorná-lo com juros) que não precisa de ao menos alguns meses de experiência com o mercado financeiro, por favor me dá um toque. Eu nunca encontrei um. Até mesmo pra investimentos (sic) em poupança você precisa fazer um curso. Além de ter uma UX sofrível, o Banco do Brasil (único do qual posso falar já que sou cliente) requer que tu saiba o número da variação da poupança (que só encontrei em uma página de ajuda ao pesquisar no Google). Me pede confirmações de ações sem mostrar um resumo do que estou aprovando e pede que eu volte pra a página anterior pra lembrar.

Esforços do mercado em atualizar a tecnologia geram resultados como conexão do Facebook com minha conta do Banco do Brasil ou adição de favoritos e amigos num dashboard que pouco conhece o meu perfil de usuário e oferece apenas uma pequena área pra que eu possa listar as ações que geralmente executo no Internet Banking. Sem falar que pra configurá-lo tenho que instalar Java, rodar num browser (às vezes até SO) específico; tudo em troca de “segurança”.

Tenho certeza que sistemas bancários são responsáveis por muita inovação no ramo de software (em machine learning e bancos de dados, por exemplo) que não é exposta à todos, mas todo o resto - design, atendimento e transparência - deixa a desejar. Falta melhorar toda a experiência e focar no usuário das próximas gerações, e não as atuais…

Sem falar que gostaria muito de ver Bitcoins se tornando um meio de pagamento mais acessível e saindo do mundo nerd (geraria regulamentação, mas também poderia ajudar a estabilizá-lo). Mas isso já pode ser utopia demais.

Ruby numeric types: do’s and do not’s

In most programming languages, we have a whole range of numeric types. Ruby provide us 5 different of them:

  • Fixnum
  • Bignum
  • Rational
  • Float
  • BigDecimal

Why?

Diagram showing the relationship between Natural, Integers, Rational and Irrational numbers

As everything in the world, we usually group numbers in different sets by their characteristics. The most common - and probably all you had contact while developing software - are the following: Natural numbers, Integers, Rationals and Irrationals.

Integers

Natural numbers are those, starting by 1 in a sequence that keep adding 1. Thus, 1, 2, 3, 4, 5, …. Integers are these same numbers but also includes this same summation to the opposite direction, 0, -1, -2, -3, -4, …. Ruby has a representation for this set: the abstract class Integer. Its concrete brothers are Fixnum and Bignum.

> 10.class
=> Fixnum

When you give an Integer literal (something that looks like an Integer) to the interpreter, the language will try to fit it into a Fixnum instance.

But sometimes this can’t happen. We have memory limit, since Ruby does not allow Fixnum’s bigger than a native machine word[1] (which varies between processors).

> integer = 2 ** (1.size * 8 - 2) - 1
=> 4611686018427387903
> integer.class
=> Fixnum
> integer = (integer + 1).class
=> Bignum
> (-integer - 2).class
=> Bignum

Isn’t the most daily task to deal with numbers greater than 4 quintillion, so you should be fine with Fixnum. If you change the value of a variable to a number out of the supported range (greater or lesser), Ruby will take care of freeing the memory used to hold the value and store it in another place. This last will be treated as an infinite allocation, so you may use it for basically any integer number you are able to come up.

[1]: Ruby core team is already taking care of defining an easier way to get maximum and minimum numbers fitting into a Fixnum object. Take a look: https://bugs.ruby-lang.org/issues/7517

Do’s and do not’s

It is hardly to expect numbers out of the range of Fixnum’s, or any smaller type for Integers provided by your language or database. If you’re using Ruby, just let it take care of the instantiation and use it in the most efficient way. Performing operations between Fixnum’s and Bignum’s will work fine and (after) will test for the possibility of using the simpler class for the result of calculations. In other words, the subtraction of two Bignum's could result in a Fixnum, for instance.

Also, when using Integer literals in Ruby you can arbitrarily put underscores between the digits. It’s way easier to understand 129_990 than 129990.

Rationals

Unfortunately or not, we can’t represent everything just with Integers. Take the Integer 1 and divide it by 2. You can see the result in two different ways: simply 1/2 (as a valid operation but not capable of fitting into Integers set) or 0.5. Both of them are called “Rational”, since can be perfectly expressed by a ratio, or a division between Integers.

Ruby does provide a way to represent and perform calculations between Rational numbers, through the Rational class. In contrast to Fixnum/Bignum, Ruby will always assume you are expecting a Rational instance after calling mathematical functions using Integers and Rationals.

> Rational(1, 2)
=> (1/2)
> rational = Rational(1, 2) + Rational(1, 2)
=> (1/1)
> rational.class
=> Rational
> rational.to_i
=> 1

Rational numbers can also assume the same “Ruby form” as Irrationals.

Do’s and do not’s

Using the Rational class is especially useful when extremely important to have exact results but you don’t know about the form of numbers you are handling. If it’s likely to appear something that does not fit into Integers, enjoy the Rational class.

> 0.5r
=> (1/2)

If you’re going to have hard coded rationals, Ruby also helps you with literals. Write the number in the decimal form (that with the dot) and append a r.

Do not use it when having trouble with readableness and precision is not so important. If you are not sure if it is, assume it truth and later fix it. Your code can be modified, but precision may cause data to be lost forever.

Irrationals

Not all numbers can be expressed by ratios. But the same truth brings us good news: it doesn’t matter. When calculating the value of pi, we usually don’t need more than a couple of decimal places (even the real pi having infinite).

> Math::PI
=> 3.141592653589793
> Math::PI.class
Float

For representing Irrationals (and also Rationals) in Ruby, we have Floating Point numbers with its own arithmetic.

> (2.0 - 1.1) == 0.9
=> false
> (2.0 - 1.1)
=> 0.8999999999999999
> (2.0 - 1.1 + 1.1) == 2.0
=> true

This is a territory where even experienced software developers learned to not ask themselves so much. If you have some patience, you can understand better this set of numbers in IEEE 754.

WHY??!!! It’s so simple… in what world 2.0 - 1.1 would be different than 0.9 and how this thing got implemented in virtually every computer known?!

The reason is: should not matter! Floating point numbers, kindly called “Float” (homonym to its Ruby representation) imposes a limit for any number that could be really big or even infinite, but you don’t care so much about precision. For your use, doesn’t matter if the result is 0.8999999999999999 or 0.9. Both will get you happy and thanking this precious and big calculator you call “computer”.

> float = 0.00000000000000000000000000000000000000000000000000000000000009
=> 0.0
> float.class
Float
> float.zero?
=> true

Use any number with a dot as decimal place separator and Ruby will please you with a Float instance.

This happens because the standard defined by IEEE in 1985 (and in use by Ruby on its internals) stores the number in a limited precision. Even if you type 1 billion of decimal places, it won’t keep more than 15.

> 2.0 - 0.0000000000000006
=> 1.9999999999999993

In commands like 2.0 - 0.0000000000000006, you use a literal for the floating point number 2.0 and another for 1.1. The first will be stored as 2^10 (2 squared to the 10th power) and the other… well, ask Wolfram|Alpha. Converting the number to base 2 (from base 10 we are used to) turns it into a really big problem. The calculation were made using two Float’s, so you expect a Float back, right? Ruby rounded the number as imagined to be acceptable for you. Since the 4 in the end doesn’t fit in its precision of 15 decimal places, it decided to turn the 4 into a 3.

If you need a little more precision but still doesn’t have to be exact as Rational, Ruby gives you the class BigDecimal. This last (internally) converts numbers to base 10000 (way harder to run “out of space”) and arbitrary precision.

> BigDecimal('500e1000')
=> #<BigDecimal:7fc9c387e0d0,'0.5E1003',9(18)>
> BigDecimal('500e1000').to_f
=> Infinity

BigDecimal is the only number class without a literal form. It expects you to provide a string to initialize one. In my previous example, I gave the number I represented by 500 raised to the 1000th power. Trying to convert it to a Float, I get a whole new thing: Infinity. Ruby tried to fit it into its base/precision but noticed was out of range. “OK, this seems a valid number, but I can’t deal with it. It’s more than I can even imagine. It’s infinity!” Even not really being infinity, for the precision-careless Float, it is.

> 1 / 0.0
=> Infinity
> -1 / 0.0
=> -Infinity

The same occurs for out-of-range calculations. “I tried as far as I can, but it’s still something greater”.

> 0 / 0.0
=> NaN
> Float::INFINITY / Float::INFINITY
=> NaN
> 0 * Float::INFINITY
=> NaN

For calculations Float doesn’t even know how to start, it introduces the special value NaN, from “Not a Number”.

Do’s and do not’s

When 2.0 and 1.9999999999999999 means no harm to your user, take Float by hand and go happily walk on the yellow brick road. Otherwise, if you need to take control of roundings, consider other alternatives. Floating-point itself defines mechanisms of rounding, but doing it in a n-th decimal place won’t be a pleasant work. For use with currencies, which is always expected to have (only) 2 decimal places, the Integer 1000 can be interpreted by your business logic as $10.00.

Functionally evaluating functions

This is a direct continuation of “Functionally thinking”. If you don’t start a trilogy by the second movie, why should you skip the first article?!

No questionably, you get a lot of good things by thinking functionally. Your entire algorithm becomes a set of functions, beautifully combined to compute results. The programmer tells the computer to do one thing, composing it by using a set of functions. One after another, the software will complete its cycle by outputting the result in a predefined way, like saving a file in the disk, showing something in a screen or performing a request.

Functional programming does not come just with new cool principles from Lambda Calculus and Mathematics. We also need to understand what are the effects caused by giving to functions so much importance and the benefits we get from it. In other words: more new cool principles.

let sumArgs x y = x + y
let times10 x = x * 10
times10 . sumArgs 1 $ 2

I have just one goal here: finish this letting you know why the last line of code above outputs 30 when evaluated by a Haskell compiler and how awesome this is.

Don’t be hasty

Do you remember that languages following principles of Lambda calculus (or “functional languages”) are lazy, right? They won’t process anything until it’s really needed. I mean… REALLY needed. This usually envolve some kind of IO, like showing the result in your screen.

Let’s ask GHCi - our Haskell interactive shell - to apply the function map into a List containing three other Lists:

> map maximum [[1,2,3], [20,10,30], [300,200,100]]
[3,30,300]
  • Function: map
  • 1st parameter: maximum
  • 2nd parameter: [[1,2,3], [20,10,30], [300,200,100]]

For every element of the second parameter ([1,2,3], [20,10,30] and [300,200,100]), map applies its first parameter. And as you might imagine, maximum is a function.

> :t maximum
maximum :: Ord a => [a] -> a

Typing :t <something> in GHCi, we get the properties of expressions in the language. This expression we call “maximum” takes a List of a's - call it “anything” - as argument and returns a single element of this same type. Since a is a downcased word, it does not refer to any predefined type. BUT, must implement the Ord class, saying that can be ordered in some way.

Functions are just another element

But… hey, have you noticed? Come back with me to that map:

map maximum [[1,2,3], [20,10,30], [300,200,100]]

maximum is a function. Something that does a computation based on parameters. And we are using as a parameter to the map function. Not its result after called, but the function itself. This is what in formal ways is called “Higher-order function”: in the most higher level of the language - the same way we use may use the integer 1 - we can use functions. There’s nothing so magical about them, we can pass functions as parameters or even return one in a function.

Talking about functions using functions, we can do this pretty easily.

> let applyWith10 aFunction = aFunction 10

Up there you can see the definition of applyWith10. Applies any function with the integer 10.

> let sumWith2 x = x + 2
> applyWith10 sumWith2 -- has the same result of evaluating `sumWith2 10`
12

Or pass one as parameter to another:

> let complexCalculation = 4 + 8
> sumWith2 complexCalculation -- has the same result of evaluating `sumWith2 4 + 8`
14

Partial application

Hold yourself, you don’t need to leave your job (already) to start playing with Haskell just because you learned a dozen of new things. Because I have more.

> let sumThreeNumbers x y z = x + y + z
> sumThreeNumbers 2 4 8
14
> let sumTwoNumbersWith2 = sumThreeNumbers 2
> sumTwoNumbersWith2 4 8
14
> let sumNumberWith6 = sumTwoNumbersWith2 4
> sumNumberWith6 8
14

We first define a function receiving 3 parameters. If you apply it with less than the complete number of parameters, Haskell will automatically return a function ready to complete the computation with the remaining parameters.

> :t sumTwoNumbersWith2
sumTwoNumbersWith2 :: Integer -> Integer -> Integer
> :t sumNumberWith6
sumNumberWith6 :: Integer -> Integer

What the type definitions of these intermediate functions mean? sumTwoNumbersWith2 takes 2 integers as parameters and returns another integer. sumNumberWith6, by itself, takes just 1 integer and returns another. And what do you get from it? Modularity with no sweat. If you think really well on defining interfaces to your functions, like giving good names and receiving nothing different than the needed as parameters, you can share the same function across more blocks of a project, partially applying it and getting just the necessary parts of the algorithm.

Look mum, no parenthesis!

You’ve been seeing that I didn’t use parenthesis to apply functions. Neither on their definition.

You can try using them, but they won’t accept being used for this, lowered to a position where they aren’t welcome.

> let sumThreeNumbers(x y z) = x + y + z

:2:21: Parse error in pattern: x
> let sumThreeNumbers (x y z) = x + y + z

:3:22: Parse error in pattern: x
> let sumThreeNumbers x y z = x + y + z
> sumThreeNumbers(2 4 8)

:5:1:
    No instance for (Num a0) arising from a use of `sumThreeNumbers'
    The type variable `a0' is ambiguous
    Possible fix: add a type signature that fixes these type variable(s)
    Note: there are several potential instances:
      instance Num Double -- Defined in `GHC.Float'
      instance Num Float -- Defined in `GHC.Float'
      instance Integral a => Num (GHC.Real.Ratio a)
        -- Defined in `GHC.Real'
      ...plus three others
    In the expression: sumThreeNumbers (2 4 8)
    In an equation for `it': it = sumThreeNumbers (2 4 8)

:5:17:
    No instance for (Num (a1 -> a2 -> a0)) arising from the literal `2'
    Possible fix:
      add an instance declaration for (Num (a1 -> a2 -> a0))
    In the expression: 2
    In the first argument of `sumThreeNumbers', namely `(2 4 8)'
    In the expression: sumThreeNumbers (2 4 8)

:5:19:
    No instance for (Num a1) arising from the literal `4'
    The type variable `a1' is ambiguous
    Possible fix: add a type signature that fixes these type variable(s)
    Note: there are several potential instances:
      instance Num Double -- Defined in `GHC.Float'
      instance Num Float -- Defined in `GHC.Float'
      instance Integral a => Num (GHC.Real.Ratio a)
        -- Defined in `GHC.Real'
      ...plus three others
    In the first argument of `2', namely `4'
    In the first argument of `sumThreeNumbers', namely `(2 4 8)'
    In the expression: sumThreeNumbers (2 4 8)

:5:21:
    No instance for (Num a2) arising from the literal `8'
    The type variable `a2' is ambiguous
    Possible fix: add a type signature that fixes these type variable(s)
    Note: there are several potential instances:
      instance Num Double -- Defined in `GHC.Float'
      instance Num Float -- Defined in `GHC.Float'
      instance Integral a => Num (GHC.Real.Ratio a)
        -- Defined in `GHC.Real'
      ...plus three others
    In the second argument of `2', namely `8'
    In the first argument of `sumThreeNumbers', namely `(2 4 8)'
    In the expression: sumThreeNumbers (2 4 8)
> sumThreeNumbers (2 4 8)

:6:1:
    No instance for (Num a0) arising from a use of `sumThreeNumbers'
    The type variable `a0' is ambiguous
    Possible fix: add a type signature that fixes these type variable(s)
    Note: there are several potential instances:
      instance Num Double -- Defined in `GHC.Float'
      instance Num Float -- Defined in `GHC.Float'
      instance Integral a => Num (GHC.Real.Ratio a)
        -- Defined in `GHC.Real'
      ...plus three others
    In the expression: sumThreeNumbers (2 4 8)
    In an equation for `it': it = sumThreeNumbers (2 4 8)

:6:18:
    No instance for (Num (a1 -> a2 -> a0)) arising from the literal `2'
    Possible fix:
      add an instance declaration for (Num (a1 -> a2 -> a0))
    In the expression: 2
    In the first argument of `sumThreeNumbers', namely `(2 4 8)'
    In the expression: sumThreeNumbers (2 4 8)

:6:20:
    No instance for (Num a1) arising from the literal `4'
    The type variable `a1' is ambiguous
    Possible fix: add a type signature that fixes these type variable(s)
    Note: there are several potential instances:
      instance Num Double -- Defined in `GHC.Float'
      instance Num Float -- Defined in `GHC.Float'
      instance Integral a => Num (GHC.Real.Ratio a)
        -- Defined in `GHC.Real'
      ...plus three others
    In the first argument of `2', namely `4'
    In the first argument of `sumThreeNumbers', namely `(2 4 8)'
    In the expression: sumThreeNumbers (2 4 8)

:6:22:
    No instance for (Num a2) arising from the literal `8'
    The type variable `a2' is ambiguous
    Possible fix: add a type signature that fixes these type variable(s)
    Note: there are several potential instances:
      instance Num Double -- Defined in `GHC.Float'
      instance Num Float -- Defined in `GHC.Float'
      instance Integral a => Num (GHC.Real.Ratio a)
        -- Defined in `GHC.Real'
      ...plus three others
    In the second argument of `2', namely `8'
    In the first argument of `sumThreeNumbers', namely `(2 4 8)'
    In the expression: sumThreeNumbers (2 4 8)

Besides those creepy error messages, we’re already aware about this aspect: we don’t need and won’t be using parenthesis to enclose parameters for a function. Take your time to digest the idea.

How many times have you been in trouble on understanding where is the start/ending of each expression enclosed by parenthesis? What would you do if, from today, you woudn’t need them anymore? Maybe have more time to actually solve problems coding? Ya, that’s sound a good idea for everyone involved in a project.

Side effects of cleaning syntax

Let’s think about a use case more practical.

-- Since this block is intended to seem like more real world code,
-- I am using type declaration even when not required.
-- You will need to save it in a *.hs file and load it in a GHCi session
-- by using `:l `
platformFee = 0.03
paymentGatewayFee = 0.01
contributionsAmounts = [1000.0, 2300.0, 100.0, 5100.0, 800.0]

netAmount :: Double -> Double
netAmount grossAmount = (1.0 - platformFee - paymentGatewayFee) * grossAmount

In Neighborly's context, a crowdfunding platform, we have many contributions to projects. After a campaign is finished, we need to handle the project owner takeout. But first, we need to calculate the net amount of the campaign, subtracting platform (platform owners have the feature of keeping a percentage of received contributions) and payment gateway fees.

At first, we could think of using the above functions in this way:

> netAmount sum contributionsAmounts

:8:1:
    Couldn't match expected type `[Integer] -> t0'
                with actual type `Double'
    The function `netAmount' is applied to two arguments,
    but its type `Double -> Double' has only one
    In the expression: netAmount sum contributionsAmounts
    In an equation for `it': it = netAmount sum contributionsAmounts

:8:11:
    Couldn't match expected type `Double' with actual type `[a0] -> a0'
    In the first argument of `netAmount', namely `sum'
    In the expression: netAmount sum contributionsAmounts
    In an equation for `it': it = netAmount sum contributionsAmounts

But, as we already learned, no parenthesis are allowed. This is trying to apply the function netAmount with 2 parameters: the function sum and the list contributionsAmounts, which is wrong.

> let grossAmount = sum contributionsAmounts
> netAmount grossAmount
8928.0

We can give a name to the intermediate step of calculate the gross amont summing all contributions’ amounts. But why, if is going to be referenced just in the following line? We can surely improve this.

> netAmount $ sum contributionsAmounts
8928.0

The only thing special about the $ function is its infix purpose. Unlike functions with alphanumeric characters, $ was made to receive one parameter before its name and another after.

> :t ($)
($) :: (a -> b) -> a -> b

Evaluating netAmount $ sum contributionsAmounts has almost the same result as netAmount sum, with the difference that says “parameters after sum must be passed to sum function, not netAmount”. You might have a vague idea why this function is called “function application operator”.

> x 1 2 $ y 3 4 5

The above line will evaluate the expression y 3 4 5 and use the result as third parameter of the x function.

Summing up: composition nirvana

Usually payment gateways asks for whole numbers when dealing with money amounts. The reason is that, if you trigger a transaction with the value 120.222 should he complete with 22 or 23 cents? Or just consider the integral part (120)?

To complete this action, we will need to “compose” a function. In other words, use the result of a function as parameter to another.

> truncate . (* 100) . netAmount $ sum contributionsAmounts

You just used function application to get the sum of all contributions’ amounts, got the net amount from it, composed it using a partially applied function *, multiplying the number by 100 and truncated it.

> truncate ((netAmount (sum contributionsAmounts)) * 100)

At a higher level, you’re just doing this: applying some functions cleverly.

If this wasn’t already enough to consider Haskell a well designed language, I have more. Much more!

Mr. Burns touching fingers of one hand to another

Functionally thinking

Let’s refresh our minds with some concepts you learned in high school and may have forgotten.

In Mathematics, we have something cool that is used to play with numbers. Given one, we operate over it and return another.

100 (---) 103.5

For instance, we could input the number 100 and receive 103.5 back.

10, 42 (---) 105

Is even allowed to use more than one number in the input. We will use this pair of numbers, transform then the way we want and return the result. Since it’s just a matter of transformation, inputting the same values to this bubble will always output the same result. We call this bubble “function”.

Function is a basic concept when considering the origin of computers. These, created to give faster results for problems hard or slow to solve manually. If you had the chance to decrypt a message between two allies in a war, reading the original message fast were a matter of survival, helping you to prepare your army before enemy’s attack. You input some apparently random characters to the decryption function - which will be responsible to transform it - and receive back a readable message. The exact kind of function you learned in school, but with letters in the place of numbers.

No Pain, No Gain

No, it isn’t easy to learn anything on Mathematics. But there is a reason for this.

∠A + ∠B + ∠C = 180˚, ∀ ΔABC

Math aims to define concepts and use them to build larger structures. How about the above statement? Do you understand it? Translated to English, it’s something like “For all triangles ABC, the sum of angles A, B and C is equal to 180 degrees”.

ABC triangle, where ∠A + ∠B + ∠C = 180˚, ∀ ΔABC

Should be already enough to get the idea, but to get even more from this translated definition, you must know what “triangle”, “angle” and “degree” is and how these concepts relate to each other.

To better understand the large, you should know about concepts used in its construction. You end up getting a lot of information with a less words.

λ calculus (or just Lambda calculus if you don’t want to sound cool)

In 1924 and 1930, we had a major work made by two mathematicians toward the goal of making functions more powerful than ever. Moses Schönfinkel was the first, answering the question that, yes, we can create functions that depend lesser and lesser of external variables. This motivated Haskell Curry in his study to show that even functions with multiple arguments could be made combining single argument functions.

In 1936, Alonzo Church was the responsible of creating the whole new field called Lambda calculus, a notation to define and apply operations using just functions. He proved that every numeric function can be expressed by lambda-terms, or as you might know today by just “lambda”. Think about the following function:

λx. x + 3

This is the notation developed to express functions in this field. In the example, it defines a function receiving one argument - x - and, when evaluated, returns it summed with 3.

But I mentioned he proved we can express ANY numeric function. Which means we can include addition, multiplication, loops and even conditionals. Together with another name you may know better, Alan Turing, he showed that Lambda calculus can accomplish the same computations of a Turing machine. Every set of operations you make in any computer language can be expressed by functions. Both returning the same value.

Relation with computation #1: functions are immutable

In Mathematics, functions are always immutable.

gross_amount(price) = price * 1.03
gross_amount(100) => 103

When we defined this function “gross_amount”, we also defined the relation between the numbers and how we can get to the result. Also in computation.

gross_amount(100) => 103
gross_amount(100) => 103

We can run this function as many times as we want. Nothing will change its result. Calling it with 100 must always return 103, since 100 * 1.03 will always be 103. Wait 30 minutes and try again. 103. Multiplication didn’t changed its rules, so same result.

Even functions made to generate randomic numbers are immutable. To understand how, watch this Numberphile video, because you will understand way better than any explanation I would try to do.

x(y) = 10 * 4
x(10) = 100
x(12) = 120
x(3001.23) = 30012.3

If these functions will always return the same result when called with the same parameters, the computer can remember the last time you called and give the result to you. This is called “Memoization”.

But Irio, I can’t memoize all of my functions.
def store_contributions
  File.write("#{Time.now.to_i}"_contributions", contributions)
end

You are cheating. This is a procedure, not a function. It performs an IO operation, so yes, must be run in a different way.

Relation with computation #2: expressions does not have order

x(y) = 10 * y
y = 7
z(x) = 5 * x + 2

Think about this system of equations. If I want to know z(5), does it matter if I calculate first x, y or z? The result will always be 352, right?

numberOfProjects = 0
contribution = 1000
contributedToEachProject = contribution / numberOfProjects
puts contributedToEachProject

In this case, I don’t need to calculate contributedToEachProject until reaching the 4th line. No one is using, so why should I care? I just to remember that know how to calculate numberOfProjects, contribution and contributedToEachProject. When I ask for contributedToEachProject to do something really useful (printing in the screen), I go to its definition. I will need the contribution value, so I get it. The order of evaluation could be lines 4, 3, 2, 1 without any problem. What means this? I could run all of the three first lines in parallel, using multiple cores of my processor.

Since I don’t evalute expressions until being really needed, we’re going to call this “Lazy evaluation”.

But Irio, sometimes we do need order. I don’t want to detonate a missile and then lauch it. For sure.
a_collection.insert(some_value)
a_collection.remove(42)

If the first line run after the second, we might not have the expected result. I could be adding 42 to a_collection, the 42 wouldn’t be removed as we would expect. The problem is given the shared variable, a_collection.

-- Haskell
a_collection -: insert some_value -: remove 42
# Elixir
a_collection.insert(some_value)
  |> a_collection.remove(42)

Languages like Haskell and Elixir have ways to get around this and define order of operations. One way available is named “Monads" and is and abstraction used in Haskell implementation, for instance.

Paradise #NOT

Unfortunately, programming following these concepts in languages that support them well - called functional languages - won’t be (always) faster than doing your procedural code. Existing commercial processors essentially follow the Turing machine, so asking the processor to do exactly what will end up doing, most of the time, will bring faster results.

Sometimes, like when taking advantage of memoization, will beat times of imperative. But we started using abstrations like Object-oriented because maintaing pure imperative code wasn’t always the best option. With functional programming, we have another paradigm, focused on defining the solution of problems (grounded in its Mathematics’ foundations). We start saying a lot with less and using more the current multiple core processors we already have, making sure we learned what was presented to us more than 70 years ago.

Appendice

Do you want to learn more about Lambda calculus? Study this theoric introduction by Stuart A. Kurtz.

Do you want to learn more about Haskell? Read Learn You a Haskell.

Do you want to learn more about Elixir? The Pragmatic Bookshelf has something for you.

Do you want to learn more about how Mathematics can be applied in software development? Follow me!

Encontro mensal do GURU londrino

Como estava planejando uma viagem para a Europa iniciando em duas semanas, resolvi procurar uma conferência ou mesmo grupo de usuários que me permitisse apresentar algo sobre Ruby. Tenho estudado cálculo lambda e functional programming em geral nos últimos tempos, tenho uma talk simples que eu poderia traduzir para inglês e melhorar baseado no público que o evento tivesse. Por que não?

Fui pro Google e encontrei o LRUG, o GURU londrino. Mandei um e-mail para o organizador explicando que estudo Matemática e tinha alguns comentários acerca de functional programming que seriam úteis pros rubistas.

Hi Irio,

Yeah, sorry, the June meeting is full for talks. The group is quite big and so there’s about a 2-3 month lead time on slots for talks at the moment.

[…]

Agradeço e digo que de qualquer forma estaria por lá.

Ontem eu e o Josemar Luedke (vulgo Josemas) fomos no tal encontro mensal. Essa imagem é o que quero deixar:

128 confirmações de presença no evento

Com 128 confirmações, seguramente tiveram 60 presentes.

O café, água, chá e bolachinha são detalhes. Apesar de terem seguido estritamente o modelo tradicional de apresentação seguida de perguntas, a interação realmente aconteceu no fim de cada palestra com o pessoal virando pro lado e comentando algo interessante. E na ida subsequente ao pub, que pareceu ser rotineira.

Software development needs a Campfire

You will always want the best developers. Or at least those more adequate to your company. They won’t be in your city. The world has 7 billion of people, and each one can have good reasons to not reallocate or travel daily. Forcing it to do it will result in more costs and dissatisfaction with the moving. But don’t get me wrong: someone isn’t accommodated just for deciding to not leave friends.

Limitations. At least geographical distance is not one. We are on the 21st century.

Hiring remote doesn’t necessarily mean closing a deal with Indonesians (but would probably add a lot in culture); a remote employee can be in a close city or even in the peripheral area of the same one. Because, definitely, developers don’t need to be in the same office to do their job

Although it can be seem, the developer job is not just deploy a list of features. He needs a casual environment of constant knowledge exchange. It’s like an office itself, where he can talk about the need of semicolons in JavaScript while drinking a cup of coffee. An abnormal rate of delivered points on Pivotal Tracker can be a sign of a successful project, as exhaustion and introduction of bugs to be noticed just a couple of sprints later. As any professional, he will have a place to learn and cool off when needed. Problems solved joining everyone interested in the project in the same room; no wonder a discussion group on Campfire is called room.

Campfire doesn’t have any witchcraft. It’s just a chat. People send messages that other team members can read. With a differential: not every message needs to be read instantly. When you feel that this is needed, you can use… instant messaging (like Google Talk or Skype), which keep working as always. But what kind of message would you send in a company’s (or team’s) chat? The same sentences said out loud in the office’s room: “Guys, do you know any lib for authorizations compatible with Rails 4.1?”, “I’m using rbenv, but Ruby 2.1 isn’t properly loading with the project”, “What do you think of the design of this autocomplete” or “We can’t send user’s password by e-mail? Can we find any alternative without leaving the page?”. All of them are part of features that a developer could do alone (with or without Google’s help), but why not share?

As any workroom, not everything is about work. The Campfire room is free to talk about the movie watched in the weekend, an article being shared on Twitter or the last video from Freddie Wong. This kind of action will bring the office to anyone distant to it.

A concern to those trying this method is about communication. Although is a really important worry, cannot be a reason to prevent its use. Working physically remote won’t stop - or increase - communication problems, but will make them clearer. Given the both sides of the coin, you can use this to fix existing problems. A person pushing a feature after its estimate without getting help shows someone without pro activity. Being in the same site, he can argue the problem was “demonstrated”; remote, it must be directly reported.

And remember about necessity of instantly reading each message? It will give you a bonus: having a lot of people in the same room will force you to scan messages from those not working directly with you. This process will stand to, in a future point, remember yourself that someone knows something about a given problem that you are having. Will stand to know what happens in the company. In a way that working in site wouldn’t allow you. This will give the flexibility to do your job outside of office’s schedule and even attending to some courses.

But one of the challenges of working with Campfire is using it in the same room (that with four walls). What should be spoken and what goes as text? Each company can find the better solution for itself, but one simple and interesting use is: needs to be heard or can be interesting to someone else? Campfire. Is immediate? Skip instant messaging and poke your co-worker. You will see satisfaction and more concentration in the team.

So, use Campfire. Even if not from 37signals. You can have HipChat, a channel on IRC or any set group on a chat. Tools, as any other, must be used with a purpose.

Desenvolvimento de software precisa de um Campfire

Você vai sempre querer os melhores desenvolvedores. Ou ao menos os mais adequados à sua empresa. Eles não estarão na sua cidade. O mundo possui 7 bilhões de pessoas, e cada uma pode ter os seus bons motivos para escolher não se mudar ou deslocar-se grandes distâncias diariamente. Obrigá-la a isso trará problemas como custos com deslocamento e insatisfação com a mudança física. E não leve a mal: a pessoa não é acomodada apenas por não querer deixar amigos.

Limitações. Ao menos a distância física não é mais uma. Estamos no século XXI.

Contratar pessoas remotas não significa necessariamente fechar contrato com indonésios (apesar de poder te adicionar muito no quesito cultural); um empregado remoto pode estar numa cidade vizinha ou mesmo na periferia do mesmo município. Porque definitivamente desenvolvedores não precisam estar no mesmo espaço físico para desempenharem o próprio trabalho.

Apesar do que pode aparecer à primeira vista, o trabalho de desenvolvedor não é apenas fazer deploy de funcionalidades. Este, precisa de um ambiente de troca de informação constante e descontração. Ele é como o próprio escritório, onde você pode parar para trocar uma ideia sobre a necessidade ou não de ponto e vírgula no JavaScript durante um café. Uma taxa de entrega anormalmente alta de pontos no Pivotal Tracker pode ser sinal de um projeto bem sucedido, mas também de cansaço e introdução de bugs que serão detectados apenas meia dúzia de sprints adiante. Como qualquer profissional, a ideia é que tenha tempo para conseguir esfriar a cabeça e se auto-aperfeiçoar. Problemas estes resolvidos reunindo os interessados na mesma sala; e não é à toa que um grupo de discussão do Campfire é chamado de room.

O Campfire não tem nenhuma magia negra. É apenas um chat. Pessoas digitam mensagens que membros do grupo podem ler. Com um diferencial de uso: nem toda mensagem precisa ser lida instantaneamente. Quando essa necessidade realmente existe, você tem… mensageiros instantâneos (como Google Talk ou Skype), que seguem funcionando perfeitamente. Mas que tipo de mensagem enviaria num chat da empresa (ou da equipe)? O mesmo que diria em voz alta na sala física: “Pessoal, alguém conhece uma biblioteca para ajudar a criar permissões compatível com Rails 4.1?”, “O Ruby 2.1 não é carregado corretamente no meu projeto. Estou usando o rbenv.”, “O que vocês acham dessa aparência para o autocomplete?“ ou “Não podemos enviar a senha do usuário por e-mail. Tenho como contornar isso sem sair da página atual?”. Todas fazem parte de alguma tarefa que um desenvolvedor deveria ser capaz de realizar sozinho (com ou sem ajuda do Google), mas por que não compartilhar?

Como toda sala de trabalho, nem tudo que é ouvido é referente a trabalho. O Campfire é livre para qualquer um falar sobre o filme que assistiu no fim de semana, colocar o artigo da vez sendo compartilhado pelo Twitter e postar o último vídeo do Porta dos Fundos. Esse será o tipo de atitude que trará o ambiente físico do escritório para qualquer pessoa distante dele.

Uma preocupação constante de quem está entrando no mundo do Campfire é a comunicação. Apesar de ser um bom motivo para se repreender à mudança, não pode ser impeditivo. Trabalhar fisicamente remoto não impedirá - nem aumentará - problemas de comunicação, mas os deixarão mais evidentes. São dois lados da mesma moeda que podem ser aproveitados de modos diferente, caso exista interesse. Uma pessoa prolongando uma tarefa enquanto tem dificuldades e não procura ajuda, demonstra a falta de pró-atividade do profissional. Presencialmente poderia ter criado a ilusão que a dificuldade foi “demonstrada”; remoto, ela deve ser reportada diretamente.

E lembra da parte de não precisar ler as mensagens imediatamente? Você leva mais um bônus: se tiver bastante gente na sala, você pode escolher apenas escanear mensagens de determinadas pessoas com quem não está trabalhando diretamente. Esse scanning pode servir para num futuro lembrar que alguém já teve um determinado problema que atualmente te incomoda, por exemplo. Servirá para saber o que acontece na empresa. De um modo que trabalhar presencialmente não permitiria. E de um jeito que permitirá você ter horários mais flexíveis, talvez até para fazer cursos em um turno.

Agora talvez um dos maiores desafios de se trabalhar com Campfire é usá-lo dentro do escritório físico. O que devo falar pessoalmente e o que vai por escrito? Cada empresa pode encontrar o que se encaixa mais com o seu fluxo, mas uma forma interessante de usar é: precisa ou acha que qualquer um pode se interessar? Campfire. É imediato? Pule o mensageiro instantâneo e cutuque o colega diretamente. Você verá ganhos em satisfação e concentração da equipe.

Por isso, use o Campfire. Mesmo que não seja o da 37signals. Você tem o HipChat, um canal do IRC ou um grupo fixo em qualquer chat. Eles são apenas ferramentas, e como qualquer outra, não tem utilidade se não for usada com um propósito.

A necessidade de desenvolver aprendendo

Dificilmente você conseguirá manter desenvolvedores (ou, se preferir, “criadores”, já que não é de apenas programadores que me refiro) instigados no ambiente de trabalho se ele não oferecer formas de sempre aprender coisas novas. Um profissional desses está em constante formação e quer trabalhar em volta de pessoas que saibam mais do que ele, para que todos os dias possa tirar coisas novas. E tão constante quanto as reclamações, vemos gente que sente não estar aprendendo tudo o que poderia. Vindo de empregados de empresas pequenas, como é comum no ramo de tecnologia, é fácil encontrar causas: dificilmente num grupo de 10 pessoas encontraremos gente com grande (seja em dimensão ou profundidade) conhecimento e saiba instigar os outros a elevar o próprio nível.

As tentativas

Sabendo que depois de certo tempo a pessoa pode se achar limitada, a primeira alternativa é recorrer à literatura para conhecer casos passados. Do mesmo modo que um escritor lê obras clássicas para aprender padrões em casos de sucesso, o desenvolvedor quer conhecer técnicas provadas como funcionais. A literatura técnica, seja em qualquer área, não é exatamente barata. Mesmo ganhando 1 ou 2 salários mínimos, isso é um problema contornável, mas você quer ou não apoiar quem está se esforçando para aumentar a qualidade da sua empresa? Para ambos é um investimento baixo e com alta chance de bons lucros.

Muito valorizada também, é direcionar o profissional para cursos. Estes geralmente são fáceis de encontrar e têm preços variados. A empresa paga o empregado em dobro (salário e curso) e é um dos mais arriscados. E exatamente por isso que pode trazer melhor retorno: você está treinando uma pessoa em alguma habilidade necessária para o seu trabalho. Talvez esta seja única na empresa. O risco, caso você não consiga fidelizar o funcionário, de a médio prazo estar evoluindo o time do seu concorrente existe, mas aí valem as conversas que você vai constantemente levar com ele sobre o alcance de suas expectativas.

A terceira, e talvez a mais interessante de incentivar o aprendizado, é o networking. Chame do que quiser, mas conhecer pessoas novas e discutir ideias com gente diferente só tem a adicionar à capacidade da sua empresa. Nesse caso não pode haver medo da concorrência: ai existem pessoas com objetivos semelhantes, seja trabalhando com o mesmo crachá ou não.

Como se não bastasse, esse pessoal costuma levar a sério qualquer conselho bonito que ouvem por ai. Resultado: se passar 2 dias insatisfeito com o que está fazendo, tentará corrigir os problemas e propor soluções. De você, líder, é esperado que aproveite a oportunidade. Daí podem surgir mudanças interessantes e inovadoras.

As alternativas

Para contornar isso, não tem bala de prata: você precisará confiar em quem contratou e ser sincero (como também cobrar a contra-parte) no máximo de aspectos possíveis. Incentive, discuta e apoie.

Quer uma alternativa simples de ser implementada? Rotatividade. Se a empresa possui poucos funcionários, dê a oportunidade para trocas de projeto a cada 3 ou 6 meses. Isso fará bem para o profissional - que estará em um ambiente novo e de maior aprendizado - e para o projeto - que pode ter problemas invisíveis até então, revelados por carne nova no pedaço. Não tem gente nem projeto suficiente para ficar rotando? Use o networking que espero que já tenha feito para trocar um empregado por outro de mesmo nível em uma empresa parceira. Laços entre todos os envolvidos têm grande chance de se reforçarem.

O fim

E esse processo simplesmente não tem fim. Trabalhar com gente que está sempre criando é extremamente disposta a inovar. Aprender é fundamental para que essa capacidade esteja crescendo e tendendo ao infinito.