Apart from the focus on data quality, freshness SLAs and column based restrictions, more so for PII data, another focus should be on establishing the data lineage. Every fact / information ever created in an org should offer insights into it's lineage. One of the problems organizations face is establishing all the clear write paths into the store and documenting the consumer behaviour from the store.

Thanks for sharing. Would recommend switching to a consistent throughput unit to save on the reader's mental bandwidth. The article switches anywhere from Billion / day as the title to per minute (when talking about feature store ingestion) to per second (write capacity units). This makes it hard to grasp the impact. 1 Billion per day sounds like a lot (probably to garner readers' attention), but if you put it simply, it's just 270 K messages/sec. Otherwise, might as well say "How we process 1/3 trillion messages a year"

Hey Nikhil, thanks for reading. That's a good callout. However, for most scenarios, you' want to generate an ID yourself instead of relying on the database sequence, in which case it would be deterministically available in the app memory before you write the record, say a UUID or better ksuid. …

The SQL community is fairly divided on what a “null” means for a column that is a part of a unique index key. MySQL allows it, while some such as SQL Server outright reject it. …

Use Case Analysis Assume you have a use case, where there’s an input of N numbers from the user and you need to fetch the results of an expensive mathematical function f(x) computed on those numbers and return any first K that finish first. Further, the logic of computing the result of the…

Uber engineering blogs are a goldmine that provide solid insights into their architectures. There’s no doubt. Like all big tech firms, they take great pride in the quality of their content. However, there’s something off about how the page is structured. …

Pull request creation, review, comment redressal, re-review and LGTM process carve out a significant chunk of a developer’s bandwidth. There’s no denying the importance of this process. However, this can take time. Let’s discuss how we can reduce the turnaround for the entire process. For this article, let developer be…

We’ve all been there. For our pull requests (PRs), there’s this. Hoping they safely make it through the barbaric eyes of a senior developer. Hoping that we do not get nit picked on some “code” aspects by a pedant. Hoping that we are only judged by the functional completeness of the…

Debezium is a wonderful piece of technology. You can easily start a Debezium connector having followed the basic guide and lo and behold, you have events! However, problems starts to appear when the environment around the initial happy path configuration changes which then breaks your connectors and the downstream consumers. …