Aman Garg – Medium

May 19

Matt Weingarten

Apart from the focus on data quality, freshness SLAs and column based restrictions, more so for PII…

Apart from the focus on data quality, freshness SLAs and column based restrictions, more so for PII data, another focus should be on establishing the data lineage. Every fact / information ever created in an org should offer insights into it's lineage. One of the problems organizations face is establishing all the clear write paths into the store and documenting the consumer behaviour from the store.

1 min read

Apart from the focus on data quality, freshness SLAs and column based restrictions, more so for PII data, another focus should be on establishing the data lineage. Every fact / information ever created in an org should offer insights into it's lineage.

One of the problems organizations face is establishing all the clear write paths into the store and documenting the consumer behaviour from the store.

This not only helps with the right asset ownership but is a boon when it comes to figuring out the right migration and upgrade strategies.

Jan 8

Consuming over 1 billion Kafka messages per day at Ifood

469

felipe volpone

Thanks for sharing.

Thanks for sharing. Would recommend switching to a consistent throughput unit to save on the reader's mental bandwidth. The article switches anywhere from Billion / day as the title to per minute (when talking about feature store ingestion) to per second (write capacity units). This makes it hard to grasp the impact. 1 Billion per day sounds like a lot (probably to garner readers' attention), but if you put it simply, it's just 270 K messages/sec. Otherwise, might as well say "How we process 1/3 trillion messages a year"

1 min read

Thanks for sharing. Would recommend switching to a consistent throughput unit to save on the reader's mental bandwidth.

The article switches anywhere from Billion / day as the title to per minute (when talking about feature store ingestion) to per second (write capacity units).

This makes it hard to grasp the impact. 1 Billion per day sounds like a lot (probably to garner readers' attention), but if you put it simply, it's just 270 K messages/sec. Otherwise, might as well say "How we process 1/3 trillion messages a year"

Having one standardised unit would help in improving the effectiveness of the changes you highlight.

Nov 18, 2021

Nice article @Aman.

Nikhil

Hey Nikhil, thanks for reading.

Hey Nikhil, thanks for reading. That's a good callout. However, for most scenarios, you' want to generate an ID yourself instead of relying on the database sequence, in which case it would be deterministically available in the app memory before you write the record, say a UUID or better ksuid. …

1 min read

Published in Nerd For Tech

·Oct 17, 2021

The “Nullbuster ” workaround for the unique index in SQL

The SQL community is fairly divided on what a “null” means for a column that is a part of a unique index key. MySQL allows it, while some such as SQL Server outright reject it. …

My SQL

7 min read

The “Nullbuster ” workaround for the unique index in SQL

Published in Nerd For Tech

·Mar 14, 2021

Elegant Patterns to execute work concurrently using Completion Service [Java]

Use Case Analysis Assume you have a use case, where there’s an input of N numbers from the user and you need to fetch the results of an expensive mathematical function f(x) computed on those numbers and return any first K that finish first. Further, the logic of computing the result of the…

Executorservice

6 min read

Elegant Patterns to execute work concurrently using Completion Service [Java]

Mar 12, 2021

Decluttering the Uber Tech Blog

Uber engineering blogs are a goldmine that provide solid insights into their architectures. There’s no doubt. Like all big tech firms, they take great pride in the quality of their content. However, there’s something off about how the page is structured. …

Tech Blog

2 min read

Published in Level Up Coding

·Mar 12, 2021

Dissecting why Pull Request merges get delayed

Pull request creation, review, comment redressal, re-review and LGTM process carve out a significant chunk of a developer’s bandwidth. There’s no denying the importance of this process. However, this can take time. Let’s discuss how we can reduce the turnaround for the entire process. For this article, let developer be…

Developer Productivity

10 min read

Dissecting why some Pull Request merges get delayed

Published in Level Up Coding

·Mar 5, 2021

Nitpicks from a triggered senior developer in a code review

We’ve all been there. For our pull requests (PRs), there’s this. Hoping they safely make it through the barbaric eyes of a senior developer. Hoping that we do not get nit picked on some “code” aspects by a pedant. Hoping that we are only judged by the functional completeness of the…

Code Review

6 min read

Published in Level Up Coding

·Feb 5, 2021

Fixing Debezium Connectors when they break on production.

Debezium is a wonderful piece of technology. You can easily start a Debezium connector having followed the basic guide and lo and behold, you have events! However, problems starts to appear when the environment around the initial happy path configuration changes which then breaks your connectors and the downstream consumers. …

Kafka Connect

6 min read

Fixing Debezium Connectors when they break on production.

Published in Nerd For Tech

·Jan 31, 2021

“Hacking” a Hackerrank problem using Python Basics.

This is a story about how I leveraged some of the Python’s lesser known capabilities to my advantage in solving a regular expression problem in a rather esoteric way (as we’ll see) on a popular coding platform called Hackerrank. It’s been 5 years since and I think I can finally…

Python

4 min read