The iNK blot

Data Engineering Principles

Mon, 25 May 2020 00:00:00 +0000

Here are a set of principles that guide the design and development of systems responsible for transforming data. They come from experience with this general domain but they originated from my experience at Ambiata. I was taught these principles by Russell Aronson, Mark Hibberd and Nick Hibberd, and I am very grateful to them.</p>

Perhaps the fundamental principle is this first one.</p>

Expect data to be poor quality</h3>
Expect data to be poorly formed, to have errors, and work defensively to detect and correct this. Failure to adopt this principle is indicated by sporadic failures in data pipelines with diagnoses that point to unexpected data. These are usually accompanied by complaints, fingers pointed at the source of the data, and judgement passed. The code base ends up with churny fixes that patch the system to cope with these point problems - a whack-a-mole approach if you will.</p>
Here are some examples of poor data quality in a data feed that should be expected and accepted:</p>

Date-formats that change over time</li>
Schemas that change - columns disappear or appear</li>
File formats change (dos, unix)</li>
The size/amount varies suddenly (possibly indicating errors upstream to the system)</li> </ul>
The remainder of the principles are mostly in response to this fundamental principle.</p>
Measure the characteristics of the data</h3>
If we expect the data to be poor quality, it behooves us to gather information about data so that we know what is being ingested and whether it should be processed. Here are some characteristics that are relatively straightforward to collect and store as metadata:</p>

Size</li>
Header</li>
Number of columns</li>
Number of rows</li>
Means, medians, quartiles of numeric data</li>
Uniqueness of entries in col x</li>
Type inference for column x (what primitive type does the column show up as - String, Int, Double or Bool)</li> </ul>
A corollary is that we need steps in the pipeline to act on these measurements. This is usually a stage of the pipeline that has operational responsibility - it typically occurs soon after ingestion/extraction and compares incoming data with measurements from similar events. It raises alerts and/or stops the pipeline from processing data that is deemed suspect.</p>
Normalise and enrich the data</h3>
Have a transformation step to normalize the data to formats and schemas the pipeline can expect. The sorts of normalising transformations that fit this step include:</p>

Standardising file and character encodings (use command line tools like `file</code> and iconv</code>)</li>`
`Transforming different date and time formats</li>`
Standardising separators, padding etc. (this also allows for detection of mangled quoting in CSVs for example)</li> </ul> Maintain provenance information</h3> Information pertaining to the provenance of data in the system enables the ability to trace or backtrack through transformations. It helps with diagnosing issues but also with definitively being able to answer questions about when and from where, data entered any stage of the pipeline. The next principle follows on quite naturally from this.</p> Keep the data immutable</h3> At every step in the pipeline, source data should always be available exactly as it was first encountered. This will address:</p> Concerns around processing corrupt or otherwise mangled data</li> Easier and faster debugging or troubleshooting</li> The ability to re-run any component or stage of the pipeline in isolation</li> </ul> This will mean storing multiple copies of data but the copies should have different contexts (typically corresponding to the stages of the pipeline).</p> Monitor information loss</h3> As data traverses the pipeline it can lose information. This should not be accidental. Consciously allow this to happen, or not.</p> Hopefully these principles are thought provoking and help provide some structure. They are not a complete set by any means - please be in touch if you see errors or have suggestions for improvements. I recommend Mark Hibberd's Lake, Swamp or Puddle: Data Quality at Scale</a> for more on the topic.</p>



Refactoring with Applicative - a small example in Haskell
Sun, 27 May 2018 00:00:00 +0000
Recently I came across a neat little refactoring example that I thought
might prove useful, particularly to those starting out with Haskell.</p>
Setting the scene</h4>
Say you have a file of comma-separated data, where each row is meant
to represent the length, breadth and height of a box. Our task is to calculate
the volume of each box. Simple, right? Well, like in real life, the data is not
complete and sometimes values are missing. We want to handle these cases by
simply not calculating a volume if any of the dimensions of the box are missing.</p>
l,b,h (inputs)      v (our output)
</span>---------------------------------
</span>4,5,1               20
</span>1,,                 -
</span>3,4,                -
</span>1,2,4               8
</span></code></pre>
Maybe volume</h3>
We'll work towards a function to calculate volume - something with a signature like this:</p>
maybeVolume </span>:: Maybe Length -> Maybe Breadth -> Maybe Height -> Maybe Volume
</span></code></pre>
To illustrate the patterns at play in this example let's start with something a bit simpler.
Let's work with Area before we get to Volume.</p>
A great place to start is with some data types.</p>
newtype </span>Length </span>=
</span>  </span>Length</span> {
</span>    </span>getLength </span>:: Int
</span>  } </span>deriving</span> (</span>Eq</span>, </span>Show</span>)
</span></code></pre>
newtype </span>Width </span>=
</span>  </span>Width</span> {
</span>    </span>getWidth </span>:: Int
</span>  } </span>deriving</span> (</span>Eq</span>, </span>Show</span>)
</span></code></pre>
newtype </span>Area </span>=
</span>  </span>Area</span> {
</span>    </span>getArea </span>:: Int
</span>  } </span>deriving</span> (</span>Eq</span>, </span>Show</span>)
</span></code></pre>
So with these types our pure function for area</code> is:</p>
area </span>:: Length -> Width -> Area
</span>area l w </span>=
</span>  </span>Area </span>$</span> (getLength l) * (getWidth w)
</span></code></pre>
Working towards the problem at hand, let's extend this to
a function that handles missing values. Maybe</code></a>
encodes the concept that a value may be missing. Following a simple pattern
matching approach gets us:</p>
marea </span>:: Maybe Length -> Maybe Width -> Maybe Area
</span>marea ml mw </span>=
</span>  </span>case</span> ml </span>of
</span>    </span>Nothing </span>->
</span>      </span>Nothing
</span>    </span>Just</span> l </span>->
</span>      </span>case</span> mw </span>of
</span>        </span>Nothing </span>->
</span>          </span>Nothing
</span>        </span>Just</span> w </span>->
</span>          </span>Just </span>$</span> area l w
</span></code></pre>
This is fine but it is a little verbose and is ripe for some simplification.</p>
Let's use the fact that Maybe</code> is a monad. The bind operation passes through Just,
while Nothing will force the result to always be Nothing. Using do</code> notation we
can rewrite marea</code> like so:</p>
marea' </span>:: Maybe Length -> Maybe Width -> Maybe Area
</span>marea' ml mw </span>= </span>do
</span>  l </span><-</span> ml
</span>  w </span><-</span> mw
</span>  pure </span>$</span> area l w
</span></code></pre>
The shape of marea'</code> is exactly the motivating example for introducing applicative in
McBride and Patterson's, Applicative Programming with Effects</a>.
Using applicative notation gets us to a one-liner.</p>
marea'' </span>:: Maybe Length -> Maybe Width -> Maybe Area
</span>marea'' ml mw </span>=
</span>  area </span><$></span> ml </span><</span>*</span>></span> mw
</span></code></pre>
Armed with these examples, let us return to maybeVolume.</code> We will need a couple
of extra data types.</p>
newtype </span>Breadth </span>=
</span>  </span>Breadth</span> {
</span>    </span>getBreadth </span>:: Int
</span>  } </span>deriving</span> (</span>Eq</span>, </span>Show</span>)
</span></code></pre>
newtype </span>Volume </span>=
</span>  </span>Volume</span> {
</span>    </span>getVolume </span>:: Int
</span>  } </span>deriving</span> (</span>Eq</span>, </span>Show</span>)
</span></code></pre>
And then our pure function for volume</code> is simply:</p>
volume </span>:: Length -> Width -> Breadth -> Volume
</span>volume l w b </span>=
</span>  </span>Volume </span>$</span> (getLength l) * (getWidth w) * (getBreadth b)
</span></code></pre>
Skipping over the simplest version of the function (that uses pattern matching),
here are the monadic and applicative versions of maybeVolume</code></p>
-- monadic approach
</span>mvolume </span>:: Maybe Length -> Maybe Width -> Maybe Breadth -> Maybe Volume
</span>mvolume ml mw mb </span>= </span>do
</span>   l </span><-</span> ml
</span>   w </span><-</span> mw
</span>   b </span><-</span> mb
</span>   pure </span>$</span> volume l w b
</span>
</span>-- and with applicative
</span>mvolume' </span>:: Maybe Length -> Maybe Width -> Maybe Breadth -> Maybe Volume
</span>mvolume' ml mw mb </span>=
</span>    volume </span><$></span> ml </span><</span>*</span>></span> mw </span><</span>*</span>></span> mb
</span></code></pre>
To wrap up, we've seen how to take perfectly good code that uses
pattern matching, to code that uses idiomatic Haskell patterns like
monadic do</code> notation, and composable applicative notation. It's routine
to run into monads and applicatives in Haskell codebases and I found
simple examples like this one to help concretely connect with them.</p>
Highly recommend the McBride and Patterson paper</a>
on applicatives by the way. Same goes for pretty much anything written by Conor McBride.</p>


Postgres JSON Functions From Rails
Sun, 25 May 2014 00:00:00 +0000
Say you are storing JSON in Postgres and need scopes to get records
based on their JSON attributes. This post covers the use of JSON functions
available in Postgres v9.3 to achieve this.</p>
Suppose you have an Order class, containing a JSON blob that describes
the product that the order relates to:</p>
# == Schema Information
</span>#
</span># Table name: orders
</span>#
</span>#  id                                    :integer  not null, primary key
</span>#  customer_id                           :integer
</span>#  product                               :json
</span>#  created_at                            :datetime
</span>#  updated_at                            :datetime
</span>
</span>class </span>Order </span>< </span>ActiveRecord</span>::Base
</span>
</span>end
</span></code></pre>
Suppose product</code> has the following structure:</p>
{
</span>  </span>"name"</span>: </span>"Widget 01"</span>,
</span>  </span>"code"</span>: </span>"W001"</span>,
</span>  </span>"plan"</span>: {
</span>    </span>"name"</span>: </span>"Silver"</span>,
</span>     </span>"terms"</span>: </span>"12 months"
</span>   },
</span>  </span>"created_at"</span>: </span>"2013-12-22 09:27:45 UTC"</span>,
</span>  </span>"updated_at"</span>: </span>"2014-10-21 10:30:40 UTC"
</span>}
</span></code></pre>
And you want a scope on Order</code> to find all orders with a particular
code - something like Order.with_code("W001")</code></p>
If you are using version 9.3 of Postgres, you can get it to do the heavy
lifting for you using its JSON functions</a>
like so:</p>
scope </span>:with_code</span>, </span>-></span>(code) { where(</span>"product ->> 'code' = ?"</span>, code) }
</span></code></pre>
Getting to nested JSON attributes is a bit trickier. Say you want a
scope for all orders with a particular plan name - something like
Order.with_plan_name("Silver")</code></p>
scope </span>:with_plan_name</span>, </span>-></span>(plan_name) { where(</span>"product::json #>> '{ plan,name }' = ?"</span>, plan_name) }
</span></code></pre>
I got to these by looking through Stack Overflow posts, grokking the Postgres
docs on JSON functions and this post</a>
of Avdi Grimm's. As usual, you can't comment here but if you see an error or
want to offer a suggestion for improvement, please get in touch via email.</p>


Handy scopes for associations
Sun, 29 Sep 2013 00:00:00 +0000
If your Rails project has a has-many association and you want a way of finding records that have no associated records, or,
records that have at least one associated record, you might find this post useful.</p>
Suppose you have classes Person</code> and Friend</code> that are associated via a has_many</code> relationship:</p>
class </span>Person </span>< </span>ActiveRecord</span>::Base
</span>  has_many </span>:friends
</span>end
</span>
</span>class </span>Friend </span>< </span>ActiveRecord</span>::Base
</span>  belongs_to </span>:person
</span>end
</span></code></pre>
How do you find persons</code> that have no friends</code>?</p>
    </span>Person</span>.includes(</span>:friends</span>).where(</span>"friends.person_id IS NULL"</span>)
</span></code></pre>
Or that have at least one friend</code></p>
    </span>Person</span>.includes(</span>:friends</span>).where(</span>"friends.person_id IS NOT NULL"</span>)
</span></code></pre>
Using Arel</h4>
To accomplish this with Arel
set up the following scopes on Friend</code> (thanks to @macfanatic</a> for the tip),</p>
class </span>Friend
</span>  belongs_to </span>:person
</span>
</span>  scope </span>:to_somebody</span>, </span>-></span>{ where arel_table[</span>:person_id</span>].not_eq(</span>nil</span>) }
</span>  scope </span>:to_nobody</span>,   </span>-></span>{ where arel_table[</span>:person_id</span>].eq(</span>nil</span>) }
</span>end
</span></code></pre>
And then, Persons who have at least one friend:</p>
    </span>Person</span>.includes(</span>:friends</span>).merge(</span>Friend</span>.to_somebody)
</span></code></pre>
The friendless:</p>
    </span>Person</span>.includes(</span>:friends</span>).merge(</span>Friend</span>.to_nobody)
</span></code></pre>
Note:</em> If you are on Rails 4 don't forget to append references(:friends)</code> to these.</p>

The iNK blot

Data Engineering Principles

Refactoring with Applicative - a small example in Haskell

Maybe volume</h3> We'll work towards a function to calculate volume - something with a signature like this:</p>

Postgres JSON Functions From Rails

Handy scopes for associations