Month: April 2021

Kafka Record Patterns for Data Replication

Imagine going down to your local milkshake bar and signing a contract with the owner so that you could purchase bespoke drinks at a set price. Let's say you agreed on fresh milk with 3.5% fat and one tablespoon of chocolate powder, per 500ml of milk.  Putting that into a table might look like this: PK contract_number start fat_content chocolate_powder 100 12345678 2021-01-01 3.5% 1 tbsp After a few weeks, your tastebuds become a little desensitised and you decide you want to add some more chocolate powder. The owner is agile, so he adjusts the contract, meaning we need to add a few columns in order to track validity: PK contract_number contract_from start end fat_content chocolate_powder 100 12345678 2021-01-01 0001-01-01 2021-01-31 3.5% 1 tbsp 101 12345678 2021-01-01 2021-02-01 9999-12-31 3.5% 2 tbsp Note two things: 1) this table is not normalised and 2) I used a low date (year 0001) and high date (year 9999) for the start of the first row and the end of the last row. In reality we would probably normalise this data. For the sake of this example, I won't because it will make it more readable as I add more information below. The low and high dates are there, so that I can always find data, regardless of the date I use - I don't have to know the contract termination date which is different for every contract, in order to be able to simply ask what the latest recipe is, for a given…

Read more