Andreas de Ruiter’s blog

Building a time machine with Cosmos DB and Kusto

2022-07-13T00:00:00+00:00

Kusto, also known as Azure Data Explorer or ADX, is best known for its ability to query and analyze logs. The technology powering Kusto is awesome and can be used for more than just analyzing logs. Another application, which I will discuss in this article, is that you can pair Kusto to Cosmos DB and create a temporal database. A temporal database is one which retains historic data, so that you can easily query data as it was at any moment in the past, like having a time machine! This is great when you need the ability to audit or troubleshoot changes to data over time.

We have a lot of ground to cover, and in this article, I’ll focus on the high-level concepts.

In case you’re not familiar with Kusto, check out this post: Quick Kusto intro for software developers

How to use Kusto as a temporal database, conceptually

By now it should be obvious that Kusto will not replace the existing transactional database in your application, which in this article we’ll assume to be a Cosmos DB collection. You’ll keep using the transactional database as you did in the past, but whenever a row in the transactional database is inserted or updated, you log the new version of it in the Kusto database. So, while your transactional database always has the latest version of each row, the Kusto database contains all versions of all rows.

Of course, you should keep track of deletions too in Kusto. The way to do this is to maintain an extra row attribute in Kusto called ‘isActive’ which you set to true upon ingestion when rows are inserted or updated in the source database. When a row is deleted from the source database, you treat this as a new version of the object, with isActive as false.

So far, this approach allows you to have a temporal database, provided you start with an empty source database. However, it’s more likely you want to implement a temporal database for an existing database. In that case, you should just start by ingesting the existing transactional data to Kusto as inserts, and once that’s completed process any further changes and mentioned before. It goes without saying that your historic queries in Kusto can’t go back to the time before you started recording changes to Kusto!

Implementing a temporal database with Cosmos DB

We now know at a high-level how to implement a temporal database with Kusto. Let’s take a deeper look at how to specifically implement this with Cosmos DB.

Creating a Kusto table to track changes to the Cosmos DB collection

Cosmos DB doesn’t explicitly manage schema. Every row in Cosmos DB is just a Json object (albeit with a couple of built-in system attributes like _ts). Your application manages the schema implicitly through code.

Kusto has a more traditional schema approach, where a database contains tables, and tables have attributes. Luckily string attributes can be up to 1MB in size, so you can store the entire object payload as Json in an attribute called something like ‘data’. The benefit of this approach is that it’s generic, and you don’t need to update the Kusto schema every time your application schema changes.

For practical reasons, you’ll want to add several additional attributes to the Kusto table.

You should add the id attribute as a separate attribute to your Kusto table. Keep in mind that while those IDs are unique to Cosmos DB, they won’t be unique in Kusto. That’s fine, Kusto doesn’t even support the concept of a unique primary key.

As mentioned before, we also need an isActive Boolean attribute on the table to track deletions.

Chances are that you already have an attribute in your Cosmos DB database collection called something like ‘type’, ‘entity’ or ‘class’. With this attribute, your application distinguishes between various types of objects stored in the collection. You should also add this attribute explicitly on the Kusto table. We’ll assume this attribute is named type.

If you have other generic attributes that exist on all your objects in Kusto, such as createdBy and modifiedBy, I recommend you also add those as separate attributes to the Kusto table.

Finally, you need a timestamp attribute on your Kusto table which is the date/time when this row was inserted, updated or soft-deleted.

By adding these attributes, you will be able pre-filter many queries without the need of having Kusto parse the data attribute first. This is important for performance.

Putting it all together, we’ll be creating a Kusto table like so:

.create table MyCollectionTable (
    ['id']: string,
    ['type']: string
    isActive: bool,
    timestamp: datetime,
    ['data']: string)

(I assume you already have created a Kusto cluster in the Azure Portal. My favorite tool for running queries and commands is the web-based portal at https://dataexplorer.azure.com/ )

Updating the Cosmos DB application

Ideally, we could use Kusto with an existing application and transactional database without needing to make any adjustments to it. There are however a couple of caveats with Cosmos DB which requires us to make some modifications to the Cosmos DB application and its schema first.

Update the Cosmos DB application to support soft-delete

What I will describe in this section is a common pattern when using Cosmos DB, so you might already have this in place. In that case, you’re in luck!

As I’ll explain later, we’ll rely on a Cosmos DB feature called “Change Feed” to update our Kusto table whenever anything changes in Cosmos DB. One caveat with Change Feed is that it only triggers when data is created or updated, not when objects are deleted.

Therefore, instead of hard deleting data from your Cosmos DB collection, you should implement a soft-delete mechanism by adding a Boolean attribute to the Json schema in Cosmos DB, which I will suggest you call “isActive”, just like the equivalent attribute in the Kusto table.

Whenever your application creates an object, isActive should be initialized to True, and when it needs to delete the object, your application should set this attribute to False rather than hard deleting the object. Your application should also clear any other attributes in the payload upon deletion, except of course id and the partition key, both of which are immutable attributes in Cosmos DB.

You’ll need to update all the queries in your application and add an additional filter so that only objects where isActive does not equal False are returned (this is more robust than returning objects where isActive equals True).

Implement a timestamp attribute in your Cosmos DB application

When ingesting data to Kusto, it will need to know when the data was last modified so it can initialize the timestamp attribute accordingly. To this end, I recommend you add a timestamp attribute to your data in Cosmos DB if you don’t already have one.

Instead of implementing your own timestamp attribute, however, you could also rely on the _ts system attribute, which is already part of every Cosmos DB object. The downside of _ts however is that it has a 1-second granularity, which can be limiting when you later want to use Kusto to analyze what happened during a chain of events that occurred within a short time span.

Update the data in your Cosmos database

If you implemented soft-delete and/or a timestamp attribute as mentioned above, you need to run a batch process to update the existing data in Cosmos DB to initialize those attributes. The process should set isActive to true for all objects, and you can use the value of _ts to initialize the timestamp attribute for all the already existing objects.

Implement data ingestion of the Cosmos DB collection to Kusto

I mentioned earlier that to use Kusto as a temporal database, you’ll first need to ingest the already existing data in Cosmos DB to Kusto, and from then onwards all the objects as they change. The good news is that using Cosmos DB’s Change Feed, you can do both at once, and doing it in a very reliable way, while the Cosmos DB application remains online!

Cosmos DB change feed works a bit differently than you might expect and understanding this is important!

Let me first explain what Change Feed is not: it’s not a message or event queue, and it’s not a “push” mechanism.

Fundamentally, Change Feed is just a special way to query the Cosmos DB database. This query returns a list of all objects, ordered by modification time, oldest first. The result set is limited to a batch of max 1000 object per call, and to get the next batch, you need to call it again with a continuation token. You could do all of that using an ordinary Cosmos DB query!

What’s special is that when you call it with a continuation token for the next batch, rather than returning the next 1000 objects from the original results, the remaining results are refreshed. It will then return fresh values starting where it left off the last time.

By continuously pulling results this way, you eventually get all objects in the collection, and from that moment on, the result set will just be empty, assuming nothing changes.

It gets interesting when data does change. If an object changes, it will reappear on the list of objects to be returned to the client, even if a previous version of it was already returned before. Since the object was the last one to change, it will be the last on the list. However, once the change feed processor gets around to process it, other objects may have also changed in Cosmos DB, which will then appear after it.

This mechanism allows us to implement a process that keeps another system in sync with the Cosmos DB collection, and that’s exactly what we want to do with Kusto!

There are some interesting implications that you need to keep in mind:

Objects that are deleted from Cosmos DB will not appear in the feed. This makes sense now that you know that Change Feed is just a query, and a query only returns objects that currently exist in the database. Now you understand why we needed to implement soft-delete in the Cosmos DB application!
The objects returned are always the current version. Again, this should be obvious now that you know that Change Feed is just a query and returns data as it exists now.
The time Change Feed lags the source database can vary. Assume there is a huge spike changes to objects in Cosmos DB. The Change Feed client will receive and process those in batches of up to 1000 objects at a time. The lag of how long it takes for those changes to become visible downstream (in Kusto) will increase as objects change faster than the ingestion process can keep up. Later, when the load on Cosmos DB lessons, the lag decreases again as the change feed processes catches up.
A nice consequence of the previous point is that the downstream system is shielded from the spike in Cosmos DB. It just keeps processing a steady stream of objects, max 1000 at a time.
You are not guaranteed to receive all changes to an object. If an object in Cosmos DB is changed, and rapidly changes again before the change feed client received the first change, the first change will appear to disappear. This is important to keep in mind when you query the data in Kusto, though I am yet to encounter a situation where this was a problem.
Changes to objects in Cosmos DB are not guaranteed to arrive in the order in which they occurred. However, within a Cosmos DB partition they do arrive in the right order.

There are several ways to implement a change feed processor. I recommend creating an Azure function with a Cosmos DB trigger, which abstracts a lot of the complexity so that your code just needs to listen for changes. If you wouldn’t know better, you’d think that change feed has a push mechanism that sends you change events!

Here’s some skeleton code for such an Azure function:

[Singleton(Mode = SingletonMode.Listener)]
[FunctionName(nameof(KustoIngestorAsync))]
public async Task KustoIngestorAsync(
    [CosmosDBTrigger(
        databaseName: "%CosmosStoreSettings:DatabaseName%",
        collectionName: "%CosmosStoreSettings:CollectionName%",
        ConnectionStringSetting = "CosmosDbConnectionString",
        LeaseCollectionName = "KustoIngestor",
        LeasesCollectionThroughput = 2000,
        CreateLeaseCollectionIfNotExists = true,
        StartFromBeginning = true,
        MaxItemsPerInvocation = 1000,
        FeedPollDelay = 500,
        LeaseCollectionPrefix = "KustoIngestor")]IReadOnlyList<Document> input)
{
    if (input != null && input.Count > 0)
    {
        try
        {
            await ProcessDocumentsForKustoIngestion(input).ConfigureAwait(false);
        }
        catch (Exception ex)
        {	// Exception handling logic
        }
    }
}

The ProcessDocumentsForKustoIngestion() method would to the actual legwork of calling the Kusto’s ingestion libraries to ingest the data. While ingesting the data, it needs to set the attributes on the Kusto table we created before accordingly:

id is initialized to the Cosmos DB ID of the object
The type attribute is set to the type/entity/class of the object as you manage it in your Cosmos DB schema.
The timestamp is set to the timestamp you added to the Cosmos DB database as I suggested before. If you already had an attribute with the modification date/time, you could of course use that to set the timestamp on the Kusto table.
To be on the safe side, your ingestion code could check if the timestamp attribute in Cosmos DB has a valid value, and if not, initialize it based on _ts (which is a Unix time) instead.
The isActive attribute needs to be set based on the attribute you added to the Cosmos database to support soft delete.
Finally, the data attribute should be set to the Json payload. Keep in mind that the max size is 1MB. If your Cosmos DB collection has objects that exceed that size, you could consider stripping some low-value attributes from the payload as part of the ingestion process.

Implementing ingestion in a robust way is not entirely trivial. For example, when an exception occurs, the data to be ingested needs to be temporarily saved to a persistent store such as Azure Blob Storage, so that ingestion can be retried later. This means you need another Azure Function that periodically checks for items which previously failed and try again.

About the code:

We’re running the function as a singleton. This will make it more robust as it essentially throttles the ingestion of data and prevents concurrency troubles.

Note that we specified StartFromBeginning as true. This means that when we deploy the function, it will first start processing all the existing data in the collection. This is essential as we want to use Kusto as a temporal database.

Querying the temporal database

Assuming everything is in place and data is being ingested to Kusto, we can start querying it.

My favorite tool for this is the web-based Azure Data Explore site at https://dataexplorer.azure.com/. It supports nice editing features, such as context highlighting and auto complete, and it allows you to easily share links to queries.

I’ll show some useful queries on our temporal MyCollectionTable table.

To get a sense of what the data looks like, try the following query, which returns 10 objects somewhat randomly:

MyCollectionTable
| take 10

You can count the number of rows in Kusto like this:

MyCollectionTable
| count

Keep in mind that the object count above is expected to be higher than the number of objects in Cosmos DB, because there can be multiple versions of each object in Kusto. Inactive (deleted) records are also counted.

To query the number of “current”, non-deleted objects in Kusto, use this query:

MyCollectionTable
| summarize arg_max(timestamp, \*) by id // Keep only the newest version of each object
| where isActive // Keep only records which aren't deleted
| count

This function should return the same number of objects as Cosmos DB does, not counting those where isActive is false. The actual count might differ slightly, only because there is a time lag of a couple of minutes from Cosmos DB to Kusto.

The line where we summarize data using the arg_max function is where the magic happens to filter out previous versions of each object. This is a demanding call though, and if possible, you should filter the data already prior to making that call.

A nice bonus is that you can now easily query the data in Cosmos DB using KQL, without being limited by the Cosmos DB’s SQL query language, which supports a very limited SQL dialect.

The following query counts the number of active objects where the type is ‘Company’

MyCollectionTable
| where type == 'Company'
| summarize arg_max(timestamp, \*) by id // Keep only the newest version of each object
| where isActive // Keep only records which aren't deleted
| count

Now let’s try our first historic query. This query returns the number of companies we had in the Cosmos DB database exactly one day ago:

MyCollectionTable
| where timestamp \< ago(1d)
| where type == 'Company'
| summarize arg_max(timestamp, \*) by id // Keep only the newest version of each object
| where isActive // Keep only records which aren't deleted
| count

As you see we have the same query as before, except we now also filter out all data from the last day. Knowing that existing data in Kusto is never updated, this query returns the same results as if we had run it one day ago!

Now suppose we want to know which Company object has changed most often:

MyCollectionTable
| where type == 'Company'
| summarize n=count() by ['id']
| sort by n // default sort order is descending
| take 1

Result:

Id	n
6f2a11e5041a9fd34184c8aab4e5718a	288

As you can see, we have a company object that has changed 288 times!

To see all versions of this company object:

MyCollectionTable
| where id == '6f2a11e5041a9fd34184c8aab4e5718a'
| sort by timestamp // returns newest version on top

To see graphically when changes to this object were made:

MyCollectionTable  
| where id == '6f2a11e5041a9fd34184c8aab4e5718a'  
| make-series n=count() default=0 on timestamp step 7d  
| render timechart

As you can see, something happened in March of the previous year that caused most changes to this company object. With the historic records in Kusto we could further investigate in detail what caused this to happen!

Now for something more sophisticated, let’s see how many active (i.e., non-deleted) Company objects we had in the Cosmos DB database over time:

MyCollectionTable  
| where type == 'Company'  
| sort by id, timestamp asc   
| extend v=row_number(1,id != prev(id))  
| extend delta = iff(v==1,toint(isActive),toint(isActive)-toint(prev(isActive)))  
| sort by timestamp asc  
| extend total = row_cumsum(delta)  
| summarize max(total) by bin(timestamp, 1d)  
| render timechart

Until now, we haven’t really done anything with the payload yet. With the extract_json and todynamc functions in KQL you can extract data from the Json payload, which in our case is loaded as a string in the data attribute.

For my Cosmos DB schema, the ‘createdBy’ value can be retrieved from the payload using the Json path ‘$.data.createdBy.v’. To retrieve a list of people who created company objects and display the number of company objects each of them created, you can use this query:

MyCollectionTable
| where type == 'Company'
| summarize arg_max(timestamp, \*) by id // Keep only the newest version of each object
| where isActive // Keep only records which aren't deleted
| extend createdBy = extract_json('\$.data.createdBy.v', data, typeof(string))
| summarize n=count() by createdBy
| sort by createdBy asc

As before, we could also run this query at a moment in the past by adding a time filter.

Finally, let’s select all the company objects that were created by me:

MyCollectionTable
| where type == 'Company'
| summarize arg_max(timestamp, \*) by id // Keep only the newest version of each object
| where isActive // Keep only records which aren't deleted
| where data has 'Andreas de Ruiter' // pre-filter the data
| extend createdBy = extract_json('\$.data.createdBy.v', data, typeof(string))
| where createdBy == 'Andreas de Ruiter'

Note that the query pre-filters the data before the line where we call extract_json. By leveraging Kusto’s highly efficient full-text search this way, fewer objects need to be parsed by extract_json, thereby speeding up this query.

Final thoughts

I’ve used Kusto as a temporal database in projects for several years and having a lens to the past feels like having super-powers. I’ll end by mentioning a couple of things that I haven’t touched upon yet, but also need careful consideration.

Compliance with privacy laws

The companies we work with need to comply with privacy laws. Moreover, your customers, business partners and employees deserve their privacy to be respected. Effectually this means that applications and databases need to be able to ‘forget’ about them after a given period. In some cases, you want to go even further and prevent sensitive data from being ingested into your Kusto in the first place. Having sensitive data in Kusto means that fewer people may access it, which limits its usefulness.

Kusto supports multiple mechanisms to forget data, including automated retention policies.

The problem is that these retention mechanisms by themselves don’t play well when using Kusto as a temporal database. I’ll explain how you can address this in another article.

Beware of bug fixes that require database updates!

Temporal tables as described in this post have attributes like timestamp and isActive which are essential for the temporal database queries in Kusto to behave correctly. If we design out Cosmos DB application right, we can assume the code to consistently sets these attributes, so what could go wrong here?

That’s a good question, glad you asked!

Let’s say a bug is found in the Cosmos DB application which has nothing to do with the temporal table or Kusto. The application’s engineering team goes ahead and implements a code fix. So far so good, but what if this fix requires data be updated in the Cosmos DB database? To do that, the engineering team creates a script to update the data directly in the Cosmos DB database, bypassing the code which normally updates attributes like timestamp and isActive.

Change feed will see those changes and Kusto will be updated with the updated records accordingly. However, if the developer of the script didn’t think of updating timestamp and isActive flags in Cosmos DB, or if the script hard deletes records in the database all together, the temporal queries will stop to return accurate results. Now we have a lot of work to find the discrepancies and fix the data in Kusto, which is further complicated by the fact that we updating data in Kusto is difficult (see https://andreas-deruiter.github.io/2022/07/12/how-to-update-data-in-kusto.html).

If have seen this happen it the past, especially when the Cosmos DB application developers are not the same people who implemented the Kusto integration.

Quick Kusto intro for software developers

2022-07-13T00:00:00+00:00

Kusto, also known as Azure Data Explorer or ADX, is best known for its ability to query and analyze logs. That sounds a bit boring (as does the ‘Azure Data Explorer’ name), but I found it to be one of those hidden gems that has become one of my favorite Microsoft technologies, right up there with VS Code, Typescript and Excel. Seriously!

At its core, Kusto is just another database system. How does it work, and how does it fundamentally differ from other databases? What makes it so awesome for certain workloads?

Kusto as a cloud-based database

Kusto is essentially a cloud-based database hosted in Azure. In that sense, it’s like Cosmos DB or Azure SQL Server.

You can go to the Azure portal, and with a few clicks create a new Kusto cluster and a database. A Kusto cluster is the equivalent of an Azure SQL Server. It’s called a cluster because Kusto is from the ground up designed to run in parallel on many VMs, which are called nodes. Because it’s a PaaS service, you don’t need to manage those VMs yourself. A Kusto administrator can, however, easily scale up by adding nodes to the cluster.

In case you already use Application Insights or Log Analytics, you already have a Kusto cluster you can work with! These are Microsoft owned services that run on top of Kusto, and that you can query the same way that you can with any Kusto cluster. Of course, you don’t have access to the management commands, but you can run queries the same way as you would on your own cluster.

What makes Kusto different from other databases

Every database system needs to make trade-offs in its design. Kusto is very good at rapidly ingesting and querying huge amounts of both structured and unstructured data. It’s primary use case is to analyze logging data. That data needs to be ingested into the Kusto database before it can be queried. This often happens in real-time, something known as streaming ingestion.

There are also things which Kusto does not do very well, or at all. Most importantly, you can’t update data as it’s an ‘append-only’ database. (There is an indirect way of updating data as you can read in How to update data in Kusto).

Kusto also doesn’t support transactions. Knowing that you can’t update data, which shouldn’t come as a surprise.

You can remove data, although that can sometimes require more steps that you would like. See There are many ways to remove data in Kusto. When to use which?.

These tradeoff means that Kusto doesn’t compete with the traditional SQL or non-SQL databases for many typical workloads. Instead, you’d use it in scenarios where you might otherwise use a technology like Apache Spark. As with Spark, Kusto’s ability to handle huge amounts of data largely stems from the fact that the work is spread across many nodes (computers) in the backend.

A secret sauce in Kusto is the column store technology which also made it to other Microsoft products, including SQL Server and Power BI. This technology seams ideal for Kusto, given how data never changes.

The trade-offs made in Kusto’s design make it ideal for querying logs, the main workload. However, the technology can also be used in other creative ways. One is to use it together with a ‘normal’ database to create a time machine. I will cover this in more detail in Building a time machine with Cosmos DB and Kusto

KQL

When I first learned that Kusto has its own query language, instead of SQL, I was quite upset having to learn yet another query language after having spent countless years building my SQL skills. After spending a bit of time with KQL though, I quickly got to appreciate it, and found it to be superior (and fun to use!) compared to SQL.

One thing I like most about KQL is its fluent syntax. SQL’s language has a very specific structure, for example the WHERE clause needs to come after the SELECT and FROM clause, and before GROUP BY and HAVING. HAVING awkwardly is just another WHERE clause for filtering on aggregated data. KQL on the other hand allows you to use any operator at any point in the query. Any operator is simply a transformation that takes what becomes before it and produces an output. If you know C# LINQ, this must sound familiar! This makes KQL queries highly composable and you don’t need a clumsy construct like Common Table Expressions to perform a series of transformations.

Another benefit of KQL compared to SQL is the much better IntelliSense for it in tooling. Anyone writing SQL queries in SSMS, or another editor will understand what I mean!

While SQL was designed by scientists, KQL appears to be designed by engineers. There are lots of practical features that are clearly based on actual experience using it. Just a small example is that in Kusto, when using ‘sort by’ or ‘order by’, data is by default sorted in descending order. This may seem counter-intuitive at first, but in Kusto you often find yourself sorting data by time and sorting it in descending order just seems to be the right thing most of the time.

A minor downside of KQL – perhaps also the result of being designed by engineers – is that language elements don’t always follow a consistent naming convention, and some statements exist with multiple names. For example, ‘sort by’ and ‘order by’ are the same thing. (One of the two is deprecated, but I always forget which one).

Interacting with Kusto

You can work with Kusto interactively, just like you can with SQL and SSMS. In fact, there are multiple tools to interact with Kusto this way.

My favorite one tool is the browser via https://dataexplorer.azure.com/. You get an amazingly rich experience, including syntax highlighting and IntelliSense, like what you would expect from an IDE. It remembers your sessions, so you can close the browser and open it again later, all your tabs with queries are still there. The best feature is the Share button, which lets you create URLs to queries that you can share.

There are also several client tools you can use. I sometimes like using Azure Data Studio, because of its support for Jupyter notebooks in which you can use KQL. You can also use it together with Python, which is a nice way to create scripts to run sequences of KQL commands.

As a developer you can access Kusto from your application through a REST API. You can use that to query data from a Kusto database, or to ingest data to one. There’s also a connector which you can use with Power Automate (Logic Apps). You can combine this with KQL’s ability to render the output into a chart, which the Power Automate app can then send as an email.

How to update data in Kusto

2022-07-12T00:00:00+00:00

One of the first things you learn when using Kusto (a.k.a. Azure Data Explorer a.k.a. ADX) is: you cannot update data. In fact, you can update data in Kusto, it’s just not as simple as it would be in most database systems. (If you find yourself needing to update data all the time though, then you should consider using another type of database.)

When you need to update some rows of data in Kusto, you could of course simply delete those rows (for by example using the .delete command), and then ingest replacement data back into the table.

There are several problems with that naïve approach, including: it is inefficient, you can mess up data retention if you rely on retention policies, and it is not possible to have the deletion of the old rows and the ingestion of the new rows happen at exactly the same time.

This is where the .replace extents command comes in. It allows you to replace old data with new data, and to do it atomically and as efficiently as possible.

The caveat: this command operates on extents and not on rows. Extents are parts of the table that each contain many rows; when you replace an extent, you need to replace all rows within it, also the rows with good data.

But with a bit of juggling, we can make it work for replacing any set of rows you want. I will show you how! And as a bonus, I will also show you how you can also use it to delete another set of rows at the same time.

First write a KQL query that returns the rows you wish to have updated or deleted.

MyTable
| where someAttribute has 'bad data' // rows we wish to update
    or someAttribute has 'remove this' // rows we wish to remove

Then update the query to return the distinct set of extent IDs for those rows. You will need to call the extent_id() for this. So this query returns a list of extents which eventually need to be replaced.

MyTable
| extend ExtentId = extent_id()
| where someAttribute has 'bad data' // rows we wish to update
    or someAttribute has 'remove this' // rows we wish to remove
| distinct ExtentId

Double-check that the query returns the data you expect it to before continuing.

Tag the extents to be replaced using the .alter-merge extent tags command, in combination with the query from the previous step. In the sample code we use a tag named ‘drop_this’, but it could be any name. In fact, it is a good idea to make it more specific, for example by adding today’s date to the tag name.

.alter-merge extent tags ('drop_this') <|
MyTable
| extend ExtentId = extent_id()
| where someAttribute has 'bad data' // rows we wish to update
    or someAttribute has 'remove this' // rows we wish to remove
| distinct ExtentId

We only tagged the extents, the table data itself is still as it was before. We do this because of efficiency; we only will replace those extents that have ‘poisoned’ data in them. Often, this is only a small subset of all the extents of the table.

Next write a query that returns the corrected rows as they should be, including the good rows for the entire table. Include a filter to remove the rows you want to delete but keeps all the rows you do not want to delete (so the condition is inverted compared to before).

MyTable
| where someAttribute !has 'remove this' // inverted condition
| extend someAttribute = iif(someAttribute has 'bad data'
                ,'corrected data' // corrected data
                ,someAttribute    // leave good data unchanged
                )

Add a filter to the query so that it limits the results to the rows in the extents we tagged before. The query returns the replacement data for the tagged extents.

MyTable
| where extent_tags() has 'drop_this'
| where someAttribute !has 'remove this'
| extend someAttribute = iif(someAttribute has 'bad data'
                ,'corrected data' // corrected data
                ,someAttribute    // leave good data unchanged
                )

Double-check that the query returns the data you expect it to before continuing.

Create a new table with the output of the query using the .set command. You can name the table whatever you like, you’ll just need it temporarily.

One thing you need to consider at this point is what ingestion date/time you want the replacement data to have. This is important when you use a retention policy on your main table. The .set command has a creationTime option that you can use to back-date the ingestion date (not shown in the example below).

.set MyTableWithReplacementData <| // Consider back-dating ingestion time!
MyTable
| where extent_tags() has 'drop_this'
| where someAttribute !has 'remove this'
| extend someAttribute = iif(someAttribute has 'bad data'
                ,'corrected data' // corrected data
                ,someAttribute    // leave good data unchanged
                )

The .set command above is where the heavy lifting happens. The operation might run out of resources and fail. If that happens, you need to divide the data into smaller chunks and do one chunk at a time by adding to the table using .append. The following sample code shows how you would do that by splitting the data into two chunks, but this method can be made to work with any number of chunks.

// For the first chunk we use .set or .set-or-replace to create the table 
.set-or-replace MyTableWithReplacementData <|
MyTable
| where extent_tags() has 'drop_this'
| where someAttribute !has 'remove this'
| where hash(id,2) == 0 // 2 is the total number of chunks. The first one is 0 
| extend someAttribute = iif(someAttribute has 'bad data'
                ,'corrected data' // corrected data
                ,someAttribute    // leave good data unchanged
                )

// In subsequent chunks we use .append to add to the table
.append  MyTableWithReplacementData <|
MyTable
| where extent_tags() has 'drop_this'
| where someAttribute !has 'remove this'
| where hash(id,2) == 1 // The second chunk is 1
| extend someAttribute = iif(someAttribute has 'bad data'
                ,'corrected data' // corrected data
                ,someAttribute    // leave good data unchanged
                )

So far, all we did was just prep-work, nothing changed to our main table yet. Finally, we are ready for the magic to happen. The replace extents command accepts two predicates, the first one identifies the list of extents to be dropped, and the second one the list of replacement extents that should be added back into the table.

.replace extents in table MyTable <|
    {
        .show table MyTable extents where tags has 'drop_this'
    },
    {
        .show tables (MyTableWithReplacementData) extents 
    }

The .replace extents command returns fast because all the heavy lifting of creating the replacement extents already happened before. The actions of dropping the old extents and adding back in the replacement extents are executed simultaneously and atomically.

Do not forget to clean up the MyTableWithReplacementData table, which is now empty.

Have fun but be careful out there!

There are many ways to remove data in Kusto. When to use which?

2022-07-12T00:00:00+00:00

Kusto, also known as Azure Data Explorer or ADX, is best known for its ability to query and analyze logs. The essence is that it allows you to capture, retain and analyze lots of data.

This is great, but how do you get data out of Kusto again? There are many reasons you may want to remove data, including…

Performance
Cost
Cleaning up bad data you ingested
Complying with privacy laws such as GDPR

In Kusto, you can set up a retention policy, delete or purge rows from a table, drop entire tables and more exotic options such as dropping and replacing extents.

This can be a bit bewildering, until you understand how Kusto handles data removal internally.

I’ll first discuss how data removal works in Kusto internally, and then cover all the removal methods and their trade-offs.

How Kusto removes data

Kusto is optimized for ingesting and querying vast amounts of data very rapidly. The way it can accomplish that is at odds with something that is very easy in other types of databases, namely the ability to update and delete data. In fact, Kusto doesn’t even have a way to update data, unless by removing it adding it back again.

Key to understanding why this is, is the concept of extents. Each table is under divided into extents, each of which has a subset of the data. Those extents are managed by multiple computers (nodes) in the data center, allowing Kusto to efficiently distribute the load and achieve a high level of performance.

It’s important to understand that those extents are immutable. After an extent has been created while ingesting data, with will never be updated again.

This means that data is eventually removed at the extent level. The extent contains many rows of data, and all of them eventually get removed together. There are two commands in Kusto which seem to contradict this, .delete and .purge, but we’ll discuss those later.

Extents don’t get removed in one step though. They first get dropped, which means that they are not visible anymore. They still exist though, and sometimes you can get them back if needed.

Dropped extents can hang around for a while, even if you don’t see them.

Eventually dropped extents are erased. Think of this as garbage collection. You have some control over that process though, as you will see later.

With all of this in mind, let’s look at the various commands. We’ll start with the most common ones, and then the more advanced ones, until we end with the most exotic ones.

Setting a retention policy

The most convenient way of deleting old data is by defining a retention policy on a table, materialized view or database in Kusto. By doing so, Kusto automatically takes care of removing data beyond a certain age.

In the policy you define a SoftDeletePeriod. Microsoft guarantees that the data will remain available for this period after ingestion but gives no guarantee how soon it will remove the data once this age has been reached.

Supposedly, this is used to trigger when the extent to which the data exists is dropped. Remember that the extent contains many rows of data, so it will not be dropped before all the rows within it have expired. However, keep in mind that all the rows were ingested during the same day or so, and all the rows have the same retention policy since they belong to the same table. Therefore, the expiration dates of all the rows usually lie within a short time span.

Even after an extent data has been dropped, it still exists for an undefined period in the Microsoft data center.

Setting a data retention policy is a good way to limit the resources and costs, which would increase endlessly if you would never delete any data.

Deleting records using `.delete`

While I mentioned earlier that Kusto handles the deletion of data at the extent level, you can still delete individual rows using the .delete command.

The .delete command does this by remembering which rows were deleted (think of it as a row-level delete flag) and applying an extra filter behind the scenes whenever someone runs a query.

The main use case for the .delete command is to get rid of corrupted data. Say for example you’re using Kusto to analyze IoT data, you could use the .delete command to quickly remove data from a faulty device.

According to Microsoft’s documentation, the deletion process is final and irreversible. It’s therefore a bit misleading that .delete is sometimes referred to as “soft-delete”, given that the data cannot be undeleted.

Note however that the extent to which the deleted rows lived will not be dropped.

The benefits of using the .delete command are that it can be performed on individual rows. It’s very fast and does not require a lot of computing resources.

On the other hand, it does not actually free up any resources, so the total size of the data occupied by your cluster won’t decrease. Moreover, if it’s a concern that the data still exists somewhere in Microsoft’s data center, this command is not for you.

Purging records using `.purge`

Like .delete, .purge is a way to remove individual records from Kusto.

The .purge command differs from .delete in that it is meant to permanently delete rows from a table. The most important use case for this is if you need to comply to GDPR or other privacy laws.

Remember that data removal in Kusto really happens at the extent level. This means that when you use .purge to remove a couple of rows from a table, Kusto first identifies to which extents they are located. Then it will copy all the rows from those extents – except the ones that need to be purged – to a new set of extents. Then it drops the ‘poisoned’ extents and replaces them with the new ones. Finally, it triggers a cleanup of the dropped extents so they will be entirely removed from the Microsoft data center. This last step happens after 5 days at the earliest, and 30 days at latest.

Purging data uses a lot of system resources because large portions of the table will need to be rebuilt. You should expect a significant performance impact and additional usage costs. Microsoft advises you to use this command sparingly.

The .purge command does not initially mark the affected rows as deleted, as .delete does. Therefore, queries run immediately after you issue the .purge command can still return those rows.

The key reason to use this command is when you have user data, and you need to comply with privacy laws like GDPR. It uses many system resources, so it negatively affects cost and performance. The fact that the data is permanently removed might eventually result in a lower cost, but if that’s your main objective, you can better rely on other data removal methods.

Dropping tables using `.drop table` and `.drop tables`

As its name suggests, you can use .drop table(s) to remove entire tables.

The most obvious use case is if you have old data in a table which you don’t need anymore. Of course, this can occasionally happen when system requirements change, or you want to remove old tables you’ve used for system development and testing. It’s a simple and fast way to remove data.

When you drop a table, it also drops the extents within the table. As was mentioned earlier, those extents will continue to exist for a period in the data center (which can be a long time because it depends on the original retention policy).

You can in fact undo dropping a table, which will also recover the extents back, assuming they still exist in the data center.

Clearing table data using `.clear table`

Kusto provides the .clear table command to remove all the data from a given table, without removing the table itself.

.clear table data - Azure Data Explorer | Microsoft Docs

This is like using TRUNCATE TABLE in SQL Server.

According to the documentation, .clear table clears the data of an existing table, including streaming ingestion data. I’m not sure what it means by this. Perhaps that data still sitting in the streaming ingestion pipeline is also flushed?

Dropping extents using `.drop extents`

You can explicitly drop extents from a table. This essentially means that all the rows in those extents are removed, which is a quick operation.

This command can be useful in several ways:

You can use it to quickly remove all the data from a table but retain the empty table itself. This appears like what .clear table does, although there might be a difference in whether they remove data in the ingestion pipeline as well.
You can also use it to get rid of the oldest data in a table by dropping extents by age.

Keep in mind that those dropped extents will continue to exist for a period in the Microsoft data center.

Unlike extents that were dropped as part of .drop table, and which can be recovered using .undo drop table, I don’t know of a way to recover dropped extents which were dropped using .drop extents. But there’s a work-around for this, as you’ll see in the next section.

How to drop extents and still be able to recover them

As I mentioned above, I don’t know of a direct way to recover extents which were removed using .drop extents. There is an indirect approach however!

Instead of using .drop extents, you can do the following:

First create a table, with a schema based on the table from which you wish to delete the extents. You can use the .create table based-on command for this. See .create table based-on - Azure Data Explorer | Microsoft Docs
Then, use the .move extents command to move the extents you wish to remove from the main table to the one you just created. See .move extents - Azure Data Explorer | Microsoft Docs
Then use .drop table to remove the table with the extents to be removed.

You can later recover that table again using .undo drop table, which also brings back the extents.

Updating and deleting data with `.replace extents`

One of the first things you learn when using Kusto is: you can’t update data. Well, with .replace extents you can, and you can also use it to remove data.

I wrote a separate blog article on this:

How to update data in Kusto

The ability to update data in Kusto is exciting, but you can also use this method to remove rows of data, which makes it an third way of removing rows of data, next to using .delete and .purge.

The method is similar to .purge in that the data is not just flagged as deleted, but you remove rows from the extents by rewriting the poisoned extents. Using .replace gives you a bit more control in exchange for a bit more complexity because there are more steps for you to take.

Unlike .purge, .replace does not control when the dropped data is physically removed in the Microsoft data center.

Physically deleting dropped extents in the Microsoft data center

With the exception of .purge, none of the methods discussed so far by themselves give any guarantees on when Microsoft physically removes the deleted data from the data center. Often, it’s good enough not to know about how garbage collection works behind the scenes, but there can be exceptions, such as when you must comply by law to ensure every copy of certain data is removed according to a given timeline or schedule.

.purge is designed to give that guarantee. Kusto, however, also provides another way to influence when the physical data is removed using .clean databases extentcontainers.

Like .purge, when you use .clean databases extentcontainers Kusto waits at least 5 days before physically removing the data. Unlike .purge though, Microsoft’s documentation doesn’t currently state a maximum number of days it might wait.

Another difference is that .purge works at the table level, and .clean databases extentcontainers works on an entire database.

You can track the progress of a .clean databases extentcontainers operation via .show database <database> extentcontainers clean operations. .purge commands do not show up here. This shows that, while there are similarities between both commands, they appear to be independent from each other.

Putting it together

As you saw, there are many ways to remove data in Kusto. It depends on your situation which method to use.

Scenario	What to use
Quickly delete faulty rows of data	`.delete`
Sporadically delete rows of data for legal reasons.	`.purge`
Updating and deleting rows of data	`.replace extents`
Get rid of old tables	`.drop table`
Get rid of all data in a table	`.clear table`
Get rid of old data in a table, manually	`.drop extents`
Get rid of old data in a table, automatically	Define a retention policy
Physically remove data from dropped tables and extents	`.clean database extentcontainers`

Andreas de Ruiter’s blog

Building a time machine with Cosmos DB and Kusto

How to use Kusto as a temporal database, conceptually

Implementing a temporal database with Cosmos DB

Creating a Kusto table to track changes to the Cosmos DB collection

Updating the Cosmos DB application

Update the Cosmos DB application to support soft-delete

Implement a timestamp attribute in your Cosmos DB application

Update the data in your Cosmos database

Implement data ingestion of the Cosmos DB collection to Kusto

Querying the temporal database

Final thoughts

Compliance with privacy laws

Beware of bug fixes that require database updates!

Quick Kusto intro for software developers

Kusto as a cloud-based database

What makes Kusto different from other databases

KQL

Interacting with Kusto

How to update data in Kusto

There are many ways to remove data in Kusto. When to use which?

How Kusto removes data

Setting a retention policy

Deleting records using .delete

Purging records using .purge

Dropping tables using .drop table and .drop tables

Clearing table data using .clear table

Dropping extents using .drop extents

How to drop extents and still be able to recover them

Updating and deleting data with .replace extents

Physically deleting dropped extents in the Microsoft data center

Putting it together

Deleting records using `.delete`

Purging records using `.purge`

Dropping tables using `.drop table` and `.drop tables`

Clearing table data using `.clear table`

Dropping extents using `.drop extents`

Updating and deleting data with `.replace extents`