IndexOutOfRange

MongoDB vs RavenDB

Sun, 01 Dec 2019 00:00:00 +0100

I need a document database. Why? There are areas in my problem generating (How to calculate 17 billion similarities) pet project cookit that are just asking for a document modelling approach.

Most people will say that this is a straightforward problem to solve:

Slap on MongoDB, and you are ready to go.

It is NOT a good idea.

TLTR

This article started as a way to structurize the comparison process but grew a bit more ( just like my previous comparison). Here is a table of content to give you an overview and make it easier to see the sections:

Table of content:

Why slapping on MongoDB isn’t the right idea?
If not MongoDB than what?
MongoDB vs. RavenDB - Fast comparison
- Docker image size
Free on-premise tier
Cloud offerings
- Backups
- Cloud offering Conclusion
Clustering model
Multi-document transactions
- What are multi-document transactions?
- MongoDB vs. RavenDB
Query Language
Indexing
Summary

Why slapping on MongoDB isn’t the right idea?

Slapping on any piece of technology into a system is fundamentally a bad idea. No matter where You read/heard/saw a recommendation of a product, always consider the context of the problem and the recommended solution. You can read about my context here, here and here where I try to calculate 17 billion similarities I need a document database for storing denormalized entries that will:
- Handle a versatile load - background calculations are very computation heavy
- Have a reasonable learning curve - I want to concentrate on delivering value, not get stuck in the docs for the first week
- Will just work - the less my time it needs, the better
- Not bite me in the least expected moment
I worked with MongoDB in the past (not my choice, it was already there), and that wasn’t a pleasant ride. To give some examples of problems we had:
- Querry times for the same query with different parameters (“Steve” was OK, but “Garry” wasn’t) varied by a factor of 100. Up to above 20 seconds on a database that was less than 2GB in size.
- Using Robo 3T as the default database browser felt like being transported into the year 2000. It was slow, clunky, and tended to crash.
MongoDB lacked multi-document transactions. When I used MongoDB, it just announced the support for multi-document transactions. With that said, it wasn’t encouraged because of significant performance penalties.

The lack of multi-document transactionality didn’t seem like a big deal at the start. But after some time, we saw that we were writing more and more code to handle the second, or next, document update fail. In hindsight, this was to expected because:

You can't remove complexity from a system. You can only move it around.

Now MongoDB supports the option to enable multi-document transactions, but it has significant performance degradation implications).

Google also reached the same conclusion. When the database doesn’t have multi-document transactions, people will put this logic into the application code. A company that started the NoSQL movement is not using a relational database - Spanner. Having proper transaction support is cheaper in the long run.

To sum up the problems: My last interaction with MongoDB was about two years ago, so not all of them might still be valid. But just checking if they are present without having something to compare it to is a waste of time. Technology doesn’t stand still, and new ideas are turned into products all the time.

If not MongoDB than what?

The title of the post gave it away a bit, but my contender is RavenDB.

A valid question here might be why am I considering Raven when there are other more popular document databases on the market? A few reasons:

The main man behind Raven is Ayende. Also known as Oren Eini, but I am betting the more people will recognize the first “name” or from his blog (he has been publishing a post a day since 2004!). Why is this important? He has a track record of collaborating and creating very well thought out, performance-orientated projects that still are one of the best in .NET ecosystem.
A by-product of reading Ayende’s blog is the insight into RavenDB development process over the years. What were the critical design decisions and how they matured over time. To point a few:
- Sticking to multi-document transactions from the start
- The effort put into making the database require as little Ops as possible with things like automatically creating indexes based on database usage.
- Running on extremely limited hardware (Raspberry Pi level of limitations) as a test process.
- Very early development of clustering features in the database.
I always wanted to check it out. But for a long time, it never had a free tier, and my curiosity didn’t justify the expense.

So, let’s compare those two.

MongoDB vs. RavenDB - Fast comparison

When looking from a very far distance, those two databases might look very similar. Both are document databases that accept JSON, offer transactions, have indexing options, and scale horizontally. Moving a bit closer shows, in many cases, the opposite approach to those areas in both databases. First, let’s take a very high-level overview of capabilities:

Feature	MongoDB	RavenDB
OS	Windows Linux Mac OS Solaris	Windows Linux Mac OS Raspberry Pi
Official Docker Image	Linux and Windows-based	Linux and Windows-based
Schema	No, but there is schema validation*1	No
Support for types	Partly	Partly
Querry language	custom, more map-reduce like syntax	SQL like RQL
Monitoring	Free console tools or paid OpsManager	free HTTP endpoint free UI (Studio Manager)
SNMP support	In the paid version	In the paid version
Clustering model	Primary-secondary version with Replica Sets or sharding	Multi-primary

	Indexes
Primary indexes	Yes	Yes
Secondary indexes	Yes	Yes
Auto indexes	No*2	Yes
Multiple fields in one index	Yes (Compound Index)	Yes (Map indexes)
Multi-collection indexes	No	Yes
Spatial indexes	Yes	Yes
Hierarchical indexes	No	Yes

	Cloud offering
Fully managed?	Yes	Yes
Clouds supported	AWS Azure GCP	AWS Azure
Consumption plan available	No	No
By hour payment?	Yes	Yes
Sample price for an hour (AWS, N. Virginia, 3 nodes, 4 GB of RAM, 20GB storage, 2vCPU)	$0.20	$0.216 yearly upfront $0.227 yearly no up-front $0.239 on demand
Backups	First 1 GB is free $2.50 per GB/month for the rest	First 1GB is free $1.00 per GB/month for the rest
Backups retention policy	1 day	14 days

	Free tier
Free on-premise	Yes	Yes
	Free Cloud offering
Storage	0.5 GB	10 GB
Memory	Shared RAM	512 MB
CPU	no info	2 vCPU

Legend:

*1 - MongoDB, from version 3.2 has introduced schema validation that allows for some schema verification to be made but doesn’t address the performance implications of lack of a schema.
*2 - it can be done with Mongo Atlas but isn’t available in the free version and doesn’t work out of the box.

Docker image size

This one is a bit neet peaky, but hear me out. I tend to look at the size of the Docker images. You can call me picky, but having a small Docker image affects a few things:

It makes continuous deployment faster and simpler
Smaller containers start faster
My well being. I know that creating a small image requires some effort. This is why a smaller image feels a bit more polished.

What makes this more interesting is that both databases have Windows container images for Docker (yes, those are a thing). The sizes:

	Windows	Linux
MongoDB	6.07GB (`mongo:4.2.0-windowsservercore`)	139.96 MB (`mongo:4.2.0`)
RavenDB	1.48GB (`ravendb/ravendb:4.2.4-patch-42020-windows-nanoserver`)	177.23 MB (`ravendb/ravendb:4.1.9-patch-41022-ubuntu.18.04-x64`)
Difference	MongoDB image is 4.59GB larger	RavenDB is 37,27 MB larger

The Linux base image size difference is negligible, but the Windows size difference is huge. Why this size difference exists? Different base images. MongoDB uses the Windows Core base image. RavenDB uses Windows Nano, which is a stripped version of Windows (Microsoft is aiming for it to be below 400MB). I know Windows Docker images aren’t on the hype train, and probably will never be. But in some cases, they are the only option, and I hoped that the official image would be better.

After some additional digging, MongoDB looked into running on Windows Nano, but they are waiting on multi-stage build support for Windows images. It is a valid argument, but multi-stage builds are available only from version 17.05 of Docker for Linux images, and we used Docker before and managed to bypass this limitation :(

Free on-premise tier

Another area that requires a more in-depth look is the free tier offering. RavenDB, for a long time, discarded the need for a free tier in their product. MongoDB started as a free database and added a paid enterprise version later on.

RavenDB free version comes with [some limitations]((https://ravendb.net/buy):

max cluster size: 3
max cores in a cluster: 3
max cluster memory usage: 6GB

Those limits are a bit of a pain, especially since MongoDB doesn’t have restrictions on the free tier. While I wish those weren’t there, I understand why they are present.

But, the limits aren’t the only thing that is different between the free versions. There is also what is left out between paid and free for both databases.

What is missing from the free version when compared to the paid one?

The main reason why all free databases have a paid version is to offer support and different licensing options for enterprise users. Both databases moved some features to the enterprise tier:

Encryption at the storage level
Snapshot backups
SNMP (Simple Network Management Protocol) support

If I could have one option moved to free, that would be storage level encryption. It could save us from a few data leaks. Now for the differences in missing features.

Paid vs. free comparison.

What features RavenDB limits in the free version?

First of all, RavenDB has an excellent page comparing the versions

Area	Feature name	What it is?	Does it have a MongoDB counterpart?
Replication	Pull Replication	This feature could use a better explanation in the documentation, but Ayendes blog to the rescue. This is a very interesting feature for synchronizing databases/database clusters that are separated or don’t have a permanent connection. RavenDB - please make this more readable and easier to understand.	No counterpart on MongoDB side
Replication	External replication	The ability to replicate the database outside of the cluster. Useful for backup and recovery scenarios.	No counterpart on MongoDB side
Clustering	Dynamic database distribution across the cluster	As I said earlier, RavenDB puts a lot of effort into creating a no-Ops system. And this feature is an excellent example. When one of the nodes in the RavenDB cluster goes down, the cluster manager can distribute the remaining databases to keep a proper number of replicas.	No counterpart on MongoDB side

What MongoDB limits in the free version?

The most important missing features when comparing free MongoDB to its paid version (called MongoDB Enterprise Advanced) are:

Area	Feature name	What it is?	Does it have a RavenDB counterpart?	Comment
Ops	OpsManger	A all-in-one tool for monitoring and managing MongoDB.	For: - metrics RavenDB has a metrics HTTP endpoint exposed by the database itself. - for visual management, RavenDB has Management Studio.
Performance	In-memory storage engine	A in-memory only storage. Without any persistence.	RavenDB has RunInMemory option.	My first thought was that it would work like SQL Server In-Memory Database which would be a killer feature. Without persistence, I don’t see this option as a very useful one.
Visualization	Compas	Kibana-like system for MongoDB	RavenDB has it’s Management Studio which also offers data visualization options, but I didn’t make a direct comparison.

Cloud offerings

Both companies offer a very similar model for providing managed cloud services: Select the cloud provider and virtual machine types, and they take care of the rest. While RavenDB has a very simple configurator, MongoDB Atlas has significantly more knobs and switches. I tried to do my best in doing an apples to apples comparison and here is what I came up with:

Configuration	MongoDB cost	RavenDB cost
Provider: AWS Region: N. Virginia Number of nodes: 3 RAM: 4 GB Storage 20GB CPU: 2vCPU	$0.20/h	$0.239/h on demand $0.227/h yearly no up-front $0.216/h yearly upfront

Provider: AWS Region: N. Virginia Number of nodes: 3 RAM: 16 GB Storage 1000GB CPU: 4vCPU	$1.09/h	$1.109/h on demand $1.056/h yearly no up-front $1.003/h yearly upfront

Take into account that this is comparing only the hardware, not the performance you get for your money.

Backups

Why go deep into backups? Read and see. Both services offer the same pricing strategy:

the first 1GB is free
anything above that is charged per gigabyte a month.

MongoDB also offers Cloud Provider Snapshots but since it doesn’t guarantee consistency of the backup (what is the point in having backups if we don’t know if make sense?) I’m excluding them.

RavenDB is very clear how backups work. And they are mandatory - I like that “we will push You into the pit of success approach”:

full backups are taken every 24 hours
incremental every 30 minutes
full backups are kept for 14 days (again this period can be extended, but not reduces - I like that)

With MongoDB Continous Backups the policy is a bit murky. The backups are kept for 24 hours and allow for recovering in a selected point in time within the last 24 hours (citation taken from the docs). Why is this a murky description? Because it lacks information about the granularity of point in time. Is it a second, minute or 15 minutes? There is also an option to enable Scheduled Snapshots that look like a counterpart to what RavenDB is offering with some differences in numbers:

	MongoDB	RavenDB
Minimum available granularity	1 hour	30 minutes
Minimum retention period	2 days	14 days
Maximum retention period	5 days, or 30 if they are taken every 24 hours	No limit
Price for GB/month	1.5- 2.5$	1$

That is a 1.5$ a GB/month difference. A difference in something that will be a significant part of the overall cost of the service. Why do I have such a big issue with this?

I don’t see any technical reason for such a big difference.
It is pointing users into a straightforward cost-saving measure - have fewer backups. This is the reverse direction we should be pointing users.
I see this pricing as a way to look cheaper than the service is.

Cloud offering Conclusion

In a simple comparison, RavenDB is a bit more expensive. But when we take into account, a proper backup behaviour the total pricing will look differently. As a side note, I will add that I am very happy with the move of database providers to provide managed instances in a selected cloud. It removes, or at least lowers, the need for the company to gain all operation expertise for the selected database. It makes deployments and company politics significantly simpler.

Clustering model

Clustering goals

What is clustering? In short, it is how the database behaves when running on multiple servers. For a more in-depth explanation, read this here.

Before we go into the clustering behaviour lets define the goals we want to accomplish with clustering:

Redundancy - Being able to operate and not lose data when a node in a cluster fails.
Horizontal scaling - Spread the load on multiple machines.

MongoDB vs. RavenDB clustering

Clustering is where those two databases show the opposite end of the design spectrum.

MongoDB - uses replica sets (from version 3.4 up) with database sharding as a method for horizontal scaling
RavenDB - is a multi-primary cluster

Interesting things start when we go deeper.

MongoDB

MongoDB replication

MongoDB used to operate in a Primary-secondary configuration until version 3.4. Now, primary-secondary replication was replaced with replica-sets. What is the difference between replica-sets and primary-secondary replication?

Both modes allow for only one node to accept all operations and, once they are committed data is replicated to the secondary nodes asynchronously. Where the replica sets shine is what happens when the primary node fails. MongoDB replica sets will do an automatic failover with automated primary election from the shard nodes.

Image source: MongoDB documentation

MongoDB horizontal scaling

MongoDB scales horizontally by sharding on the database level. Sharding is splitting the data into ** non-overlapping** sub data sets (called chunks in MongoDB) and hosting each chunk on a separate machine. For such an operation to work, we have to define a sharding key what will determine the server which will host the document.

Sharding can be done in two places:

by the client - the client knows where to put each document.
by the database - the database is responsible for splitting the data, and it should be transparent to the client. MongoDB uses the latter approach.

How does it work from the client’s perspective?

The client connects to an instance of mongos that behaves like a MongoDB database. Mongos is responsible for fetching the shard structure form the Config Server, splitting the execution of operations into appropriate servers and combining them at the end.

This makes mongos an abstraction layer. Let’s look at the benefits and drawbacks of it.

Benefits of database level sharding:

Application agnostic - the client doesn’t know that its queries are shared).
Simplifies application code.

Disadvantages of database level sharding:

If we deploy mongos on the application machine, we will need a beefier application server. Deploying it on a database server that will be used by multiple clients creates a single point of failure and a performance bottleneck.
Sharding key definition can’t be changed. Migrating to a different sharding key specification (selecting different keys to define how data is shared) is, in most cases, not supported.

I don’t blame MongoDB for not providing such an option because it is a very compute and network intensive operation that on more massive databases might not be even feasible.
Last, but not least. I saw a few approaches to abstracting sharding on the database level, and they never worked. They always leaked. Mostly because of performance.

RavenDB clustering

RavenDB replication

RavenDB has a very different approach to clustering. RavenDB cluster is a group of self-organizing nodes that maintains consensus using an implementation of the RAFT algorithm.

What is RAFT and how it works is beyond the scope of this article, but a few things to note:

it is the standard for reaching consensus in distributed systems.

it addresses the problem of a Single Point of Failure in distributed systems.

it has been proven to work.

If You want to know more, an excellent place to start is here

This approach is more complicated, but allows for a few key features:

Dynamic database distribution. The cluster will rearrange which node is storing which database in case of a cluster node failure, or adding a new node to the cluster.
Variable consensus level. By default, the data is assumed as saved when one node accepts it. Then it is replicated to other primary nodes. For special operations, the client can define that he wants to wait for replication to other nodes.

RavenDB horizontal scaling

RavenDB doesn’t support database level sharding. It is on their roadmap and from the design notes, the approach looks very solid. Taking into account that they already have:

dynamic database distribution
RAFT for consensus

We might get auto partitioning like in Reddis. How would it be different? We won’t have to define sharding keys, but let the database distribute the data according to load.

Clustering conclusion

Before I wrap up this section, let me explain my beef with user-defined sharding (also known as content sharding):

data has to be split into non-overlapping datasets which in some cases might not be possible or very hard.
Selecting the right sharding key is very hard and requires an in-depth knowledge of the business domain and future system usage patterns. Because:
Sharding keys can’t be changed easily This applies to MongoDB, Azure CosmosDB and
Sharding keys should distribute the data evenly across the cluster nodes. Otherwise, we end up with a single node that requires vertical scaling (meaning bigger and more expensive machines - the exact thing we wanted to avoid)
Shards have to be monitored. What once was a good sharding key can become a bad one over time.

Will RavenDB non-content based sharding be the end-all designs that will fix all of the above? I’m betting no. But it has the potential to solve most of them. Up to this time, I prefer to stay with sharding done by the client.

Multi-document transactions

What are multi-document transactions?

First, let’s establish what I mean by multi-document transactions. Such a feature has to follow a few rules:

Expose the ability to execute a list of operations (insert, update, delete) on a set of entities.
The changes are applied only if all operations succeed. If at least one fails, no changes are applied.
Until the transaction is committed, the changes aren’t visible to other operations.

So how RavenDB and MongoDB compare in this area?

MongoDB vs. RavenDB

Both support multi-document transactions, but the road to this feature was different. In RavenDB it was a design decision from the start, with MongoDB it was added in version 3.4 with a proper implementation being available just in 4.2. Up to version 4.0, the transaction in MongoDB was limited to the total size of 16 MB. This limitation was removed in version 4.2.

Query Language

From all the differences and features discussed in this post, this one will have the most significant day to day implications.

MongoDB querying language

MongoDB has a very JSON like approach to building querries. The simplest find querry looks like this:

db.inventory.find( { Name: "test" } )

Corresponding to SQL query of this:

Select * from inventory where Name="tests"

Looks OK. But where it starts to fall apart is when we want to express things like operators:

db.inventory.find( { $or: [ { status: "A" }, { qty: { $lt: 30 } } ] } )

This corresponds to this SQL:

SELECT * FROM inventory WHERE status = "A" OR qty < 30

What is nice is the fact that MongoDB managed to keep the language principles simple when dealing with more complex for complex properties in the document. For example this:

db.inventory.find( { "instock": { qty: 5, warehouse: "A" } } )

Will find documents with the instock object equal to { qty: 5, warehouse: "A" }.

Going back to MongoDB for this article made me remember the most common annoyance with writing MongoDB queries - missing or misplaced braces. Now let’s see what RavenDb has to offer.

RavenDB query language

When talking about querying RavenDB, we have to separate two cases - writing in C# and all the other languages. Why? Because RavenDB client leverages C# LINQ expressions (a crossover between SQL and pipes from functional languages) and allows to express queries right in the code.

But not all is lost for not - C# devs. There is RQL which is a mix of SQL and JavaScript looking like this:

from Inventory
where Name = 'test'

The from at the start looks strange, but I can live with this syntax. Now for the second query:

from Inventory
where status = 'A' or qty < 30

For me, it is much cleaner than the one in MongoDB. Time for querying complex properties:

from Inventory
where instock.qty = 5 and instock.warehouse = 'A'

Direct comparison

MongoDB	RavenDB
`db.inventory.find( { Name: "test" } )`	`from Inventory where Name = 'test'`
`db.inventory.find( { $or: [ { status: "A" }, { qty: { $lt: 30 } } ] } )`	`from Inventory where status = 'A' or qty < 30`
`db.inventory.find( { "instock": { qty: 5, warehouse: "A" } } )`	`from Inventory where instock.qty = 5 and instock.warehouse = 'A'`

Query language conclusion

I know this is a matter of taste, but SQL is the most popular language for querying data. Having something similar to it makes the developer more productive faster. Some will say that custom language in MongoDB allows for unique functionalities and that SQL is not well suited for document databases. I agree with this, but only partially. Multiple document databases expose SQL like querying language, and some migrated from custom to syntax to one more similar to SQL without losing unique functionalities.

Indexing

Why is indexing vital in document databases?

We all know that database indexing is essential. For decades they were the most effective ways to optimize relational databases. But there is one very distinctive difference between relational and document databases (MongoDB and RavenDB in our case). The lack of a schema. Why does this matter?

When we query over a non-indexed column in a relational database, it doesn’t have to read the whole file representing the table. It knows the table structure so it can calculate the file offsets and read only the necessary file fragments where this property is stored. Not ideal but works quite good.

When we query over a ** non-indexed** property in a schema-less document database the database doesn’t know the structure. It means that it has to deserialize each document and check if the requested property exists and what is its value. This means a higher disc, CPU and RAM usage.

RavenDB vs. MongoDB indexes

MongoDB indexes

MongoDB has a very standard set of indexes:

Single field - for indexing one field
Compound index - for indexing multiple properties
Multikey index - for indexing array items
Geospatial Index - for GIS operations
Text index - for text searches. It uses Lucene, so you get full-text search like in Elastic Search or Solr.

MongoDB has a limitation that one collection can have only one text index.

As an addition, indexes can have additional properties:

Unique Indexes - for guaranteeing uniqueness of the features
Partial Indexes - they give the ability to filter the documents that will be indexed.
Sparse Indexes - they will only index documents that have the selected property after version 3.2 replaced by Partial Indexes indexes.
TTL Indexes - allows defining an automatic document removal period.

A useful option, but why is it on an index?

It is a very standard and robust set of indexes. Nothing groundbreaking, but OK.

RavenDB indexes

RavenDB has a fundamentally different design approach to indexes. They are represented and declared as a user-defined function. Such a function is passed to RavenDB, and it takes care of the indexing and maintaining the index. This design decision leads to exciting possibilities like:

Combining fields in the index (FirstName + ‘ ‘ + LastName )
Executing calculations (if it can be expressed as a function it can be used in an index)
Aggregating documents before indexing
Having multiple entries in an index for a document - this is very useful when we want to flatten the object structure for indexing.
Perform queries in the indexing function
Indexing hierarchical data. I like this feature because executing hierarchical queries without support from the database is tough. In most cases requires special data modelling.

I wrote more on modelling hierarchical data here.

I am very impressed by the power that RavenDB indexes offer. They are pushing the user into two behaviours:

For complex queries, separate the write model from the read model - CQRS. Something that we all know we should do, but don’t always do.
Always try to query over an index.

Auto indexes

The push to always try to query over an index is supported by the last feature that I will discuss - auto-indexing. RavenDB can monitor the queries and automatically create indexes. It could be a very dangerous option because indexes generate an additional load on the server. But RavenDB also can delete unused indexes. Why do I like this feature? Adding each new component to the system means one additional element to worry about, monitor and maintain. Auto indexing won’t zero this overhead but anything that will decrease it is a welcomed addition.

Indexing comparison

Option	MongoDB	RavenDB
Primary indexes	Yes	Yes
Secondary indexes	Yes	Yes
Auto indexes	No*2	Yes
Multiple fields in one index	Yes (Compound Index)	Yes (Map indexes)
Spatial indexes	Yes (GeoSpatial Index)	Yes (Spatial Data)
Full-text indexes	Yes (Text Index)	Yes (Full-Text Search)
Multi-collection indexes	No	Yes
Hierarchical indexes	No	Yes
Perform calculations or operations on indexes	No	Yes(Map-Reduce Indexes)
Convert data during indexing	No	Yes
Aggregate data in indexes	No	Yes

Legend:

*2 - it could be done with Mongo Atlas but isn’t available in the free version and doesn’t work out of the box

Summary

MongoDB

The good:

Free version without a database or used CPU/memory limits.
Solid documentation

The bad:

The lack of SQL like syntax for query operations makes for a steep learning curve.
Robo3T is better than it used to be, but still lacking.

The ugly:

What is with the MongoDB Atlas backup pricing?
That Windows Docker image could be smaller.

RavenDB

The good:

Multiple indexing options.
SQL-like RQL (Raven Query Language) provides a smooth learning curve.
Raven Studio for querying and managing the database is just a substantial addition to the product. Way better than Robot3T
A clear design goal to push the user towards proper behaviour

The bad:

The limits in the free version could be higher. For me, they won’t be an issue, but I can imagine them deter some people. Maybe a different licencing model?

The ugly:

~~The documentation is lacking and is very uneven how deep is each section.~~
There is an excellent book on the website that goes in-depth and beyond. Also not a fan of the dark theme, but I know I’m in the minority.

So, what will it be?

This comparison is far from exhaustive, but I have some conclusions. I will probably make a performance comparison of those two. For now, RavenDB feels like a solid, well thought out product with even more potential (e.g., clustering) for delivering new features that will build on the previous design decisions.

Tips, tricks, and good practices for Data-Driven Testing. Part 2.

Mon, 09 Sep 2019 00:00:00 +0200

This post is a continuation of a previous tips, tricks, and good practices for Data-Driven Testing entry. This one with more code.

Before we begin

I will be converting this series into an ebook with additional source code and examples. If you want to get it, please subscribe:

1. Reading data files

1.1 Use `SetName`

We know that the objects returned by the function that TestCaseDataAttribute points to will generate tests. The question is, how will they be called? By default, the name is generated from the method name and the arguments provided. It won’t be pretty. But we already have descriptive names for the test case folders. Why not use them? This is where the SetName method on the TestCaseData class comes in. We can change the initial file reading code to something better:

    public static TestCaseData[] GetData()
    {
        var retArray = Directory.GetDirectories("MyTestDataFolder", "*", SearchOption.AllDirectories)
            .OrderBy(a => a)
            .Select(a =>
            {
                var ret = new TestCaseData(JsonConvert.DeserializeObject("testdata.json"));
                ret.ExpectedResult = JsonConvert.DeserializeObject("expectedResult.json");
                ret.SetName(a.Replace("_"," ")); //to make it even more human readable I remove the underscores and replace them with spaces.
                return ret;
            })
            .ToArray();
        return retArray;
    }

This isn’t a huge change, but it saves a lot of time in resizing the test results window to check which file failed.

1.2 Wrap file parsing in a try/catch

Your files will have errors, and parsing them will fail. It might not seem like a problem, but it is a bit of a pain. Why? Exceptions thrown in the TestCaseData generating method don’t show in the test results. What happens is that the tests disappear and only the primary method (which is always the parent of all data generated tests) is marked as failed, or inconclusive. Debugging such problems is also not ideal because if Break on all exceptions isn’t enabled, the exception will get caught by the test runner. Exiting our debug session. My solution to this is to wrap the folder reader in a try/catch block and add the crashed file name there. After the change, our code will look like below:

    public static TestCaseData[] GetData()
    {
        var retArray = Directory.GetDirectories("MyTestDataFolder", "*", SearchOption.AllDirectories)
            .OrderBy(a => a)
            .Select(a =>
            {
                try
                {
                    var ret = new TestCaseData(JsonConvert.DeserializeObject("testdata.json"));
                    ret.ExpectedResult = JsonConvert.DeserializeObject("expectedResult.json");
                    ret.SetName(a.Replace("_", " ")); //to make it even more human readable I remove the underscores and replace them with spaces.
                    return ret;
                }
                catch (Exception ex)
                {
                    throw new Exception($"Error in processing test data for folder: '{a}'", ex);
                }
            })
            .ToArray();
        return retArray;
    }

This again isn’t a huge change, but one that when added at the start will save a few hours in pointless debugging.

1.3 Have a file reader class

This tip is one that will save you from reimplementing the reading logic. Extract the logic of reading the dataset files into a separate reader class (I name it DATA_SET_NAMEReader) and place in the same folder as the dataset folder. It will look like this:

This simplifies the where is that reader question, lowering the effort, and by that making the reuse more probable.

2 Comparing

2.1 Don’t use ExpectedResult for complex objects.

If you read the previous entry, you know that I like using this property. But, as always in IT. It depends. It is handy for simple objects, but for comparing complex ones, I prefer to write my own Asserts.

The other option in NUnit is to use the Returns method on the TestCaseData class. It has the benefits that it returns a TestCaseData class so that we can have tidy fluent calls.

    public static TestCaseData[] GetData()
    {
        var retArray = Directory.GetDirectories("MyTestDataFolder", "*", SearchOption.AllDirectories)
            .OrderBy(a => a)
            .Select(a =>
            {
                try
                {
                    //a bit more compacted and fluent.
                    return new TestCaseData(JsonConvert.DeserializeObject("testdata.json"))
                        .Returns(JsonConvert.DeserializeObject("expectedResult.json"))
                        .SetName(a.Replace("_", " "))
                        ;
                }
                catch (Exception ex)
                {
                    throw new Exception($"Error in processing test data for folder: '{a}'", ex);
                }
            })
            .ToArray();
        return retArray;
    }

2.2 Use a diff tool for comparing

Asserts are great. But when running a large number of tests, there are better ways to compare what we expected with what we got - Diff comparison tools. First, how to save the received results as files and what practices I use:

    public class MyTestSiute
    {
        private string _pathForReceived = @"C:\temp\MyDataDrivenTests_ReceivedResults";
        private bool _saveReceivedResultsToDrive = false; //A single flag that determins if we write the restults to the disk

        [NUnit.Framework.OneTimeSetUp]
        public void TestSuiteSetup()
        {
            // A common source of mistakes is comparing old files. This is why I always delete the whole folder
            if (_saveReceivedResultsToDrive && System.IO.Directory.Exists(_pathForReceived))
                Directory.Delete(_pathForReceived, true);
            Directory.CreateDirectory(_pathForReceived); // And recreate it.
        }

        /*
        Get data method and other helper functions
        */

        [TestCaseSource(nameof(GetData))]
        public void MyDataDrivenTests(MyTestData data)
        {
            /*
            some preparation code
            */
            var received = testObject.TestMethod(data.Data);

            if (_saveReceivedResultsToDrive)
            {
                var receivedFilename = Path.Combine(_pathForReceived, data.TestCaseFolderName, "expected.json"); // We want to have the same folder structure and names in the results and in the received folder
                File.WriteAllText(receivedFilename, JsonConvert.SerializeObject(received,Formatting.Indented)); // Remember about the formating!
            }
        }
    }
}

Once this is in place, we can use Code Compare and its option to compare folders to get this result (blue files are those that differ):

Easy and readable.

Code Compare by default launches in file comparison mode. Switching to directory comparison isn’t that visible, so to save you some time here it is:

3. Advanced testsing scenarios

3.1 One data set, multiple results

In the first post of this series I wrote about testing a gearbox controller. We already had a test case source, so why move to files? When we move the test data to files, we open new options on reusability. What will happen when we want to test a second gearbox controller? We could reuse the same datasets. What to do with the expected result files? Save them in the same folder structure, but use a convention for recognizing for which scenario they are for:

3.2 Multi step testing

Data-driven tests are great not only to verify a single function. Their biggest strength is in verifying the whole process. I wrote previously about how I use data-driven tests extensively in cookit for verifying recipe text parsing. This process is split into multiple data-driven tests:

Determine if HTML is a recipe page.
Extract recipe text, ingredient list, image, etc.
Recognize ingredients.
Build a list of ingredients, with weights, need for the recipe.
Extract cooking time, tags, etc.

All of this can be done in one run. But splitting them makes it easier to understand what precisely broke. Data driven tests are especially easy to break down into more detaild tests because all we have to do is to save some intermediary result to the drive, and use it to feed the next step. No need to create additional test data. We already have it.

4 General

4.1 Copy localon all files in Visual Studio/MSBuild

Before I explain it more let me say that I have mixed feelings with this one. I will explain why, after explaining the problem and the solution.

The problem

By default, MSBuild copies only the dll files into the bin folder. That is a good thing since we don’t want to ship source code with our dll files. But with data-driven tests, this is an annoyance because then our data files aren’t on the same level as the test dlls. This is a problem for two reasons:

Tests are executed with the root folder equal to the test dll root folder. So we will need to traverse to the data folders with ..\..\. Not too elegant and prone to errors.
To speed up the CI pipeline, I often split building and running tests. This way, I can build once and use the build artifacts to run the tests in parallel on multiple agents. Getting the verification faster. The problem is that most build agents only copy the content of the bin directory into the build artifacts.

Both of the above can be changed, but are annoying at first. A much smoother solution would be to copy the data directory into the bin folder path.

The Solution (?)

To copy our data files into the bin folder, we can change the property of each file to Copy if never.

But this approach is wrong because:

we have to set it on each file
when we add a new test data, it won’t be picked up if we don’t set the copy local.
pointing to each file will create a big .csproj file, so merging and conflicts will be harder.
there is a much better solution.

We can set the in the *.csproj to copy the whole folder with a simple change:

  <ItemGroup>
    <None Update="Data\**">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
  </ItemGroup>

This will copy the whole Data folder into the output only if there is a newer version of the file. Awesome!? Not exactly. This solution has one major flaw.

Why I have doubts about this solution?

The biggest problem with this solution is that it will slow down the build (we are adding file copy for each build). Will it be a significant slowdown? For a small number of files and a computer using SSD drives, it probably won’t be noticable. For a large number of files or a machine running a mechanical drive, it can be a disadvantage to consider.

Conclusion

Are those groundbreaking tips? No. But adopting each one of them will make your life easier with data-driven tests.

Tips, tricks, and good practices for Data-Driven Testing. Part 1.

Tue, 03 Sep 2019 00:00:00 +0200

As promised here and here, this will be tips, tricks, and good practices for data-driven tests.

If You want to know more about data-driven testing see my previous posts on this topic.

Here we go.

Before we begin

I will be converting this series into an ebook, or something like that, with additional source code and examples. If You want to get it please subscribe:

Keep data in files.

I showed this in the previous post, but it is the most important tip. Save test case data as files and use the TestCaseDataAttribute to declare the function that will read it and pass as test cases. Something like this:

    public static TestCaseData[] GetData()
    {
        var retArray = Directory.GetDirectories("MyTestDataFolder", "*", SearchOption.AllDirectories)
            .OrderBy(a => a)
            .Select(a =>
            {
                var ret = new TestCaseData(JsonConvert.DeserializeObject("testdata.json"));
                ret.ExpectedResult = JsonConvert.DeserializeObject("expectedResult.json");
                return ret;
            })
            .ToArray();
        return retArray;
    }

Now to te next most important area: folder structure.

Tips on folder structure in Data-Driven Tests:

Since our test data is stored in files and folders, organizing them is important. Below is a sample structure I use to store test data files:

Doesn’t look sophisticated, and isn’t once it is ready. But there are some tips there:

1. Have one `Data` folder in the test project.

Having one central Data folder sounds like an anti-pattern. Let me explain why it isn’t in data-driven tests:

They are easier to locate. To read the test files, we need to know the relative path to them. One that starts at the project root. This means that if we place the test data files deep in the folder structure of the test project, we will need to duplicate that path in the test data path. Making the test more fragile for simple name changes.
Data reuse. Reusing test objects in multiple tests is, in most cases, an anti-pattern. It leads to many people modifying a single object, or a class creating a no man’s land. A part of the test project that isn’t owned by any particular test or a person. In data-driven test data is taken from real-life usage of the system. It isn’t made up, but something that happened. There is no point in modifying it (except for upgrading the object structure). We mostly create a new verification data set.

2. Second level folders - the name of the dataset

This is the folder over which we iterate in the data reading method. The most apparent naming convention is to name it after the test. Sounds reasonable, but leads to one of the anti-patterns in data-driven testing: many test folders, each with a few tests and every test folder with a different file format. How to do it better? I propose naming the test after the dataset that it is.

Name it not what it is used for, but what it is.

For example: In cookit.pl, I am parsing HTML websites and extracting recipe text and ingredients. The main logic is in the Parse method in the RecipeParser class. I initially named the folder RecipeParser_ParseWebsites. When I renamed it to HtmlRecipePages it opened the possibility to use the HTML websites for testing:

HTML validation

encoding detection

recipe text detection

ingredient detection

etc.

Names matter. Use good ones.

3. Third level folders represent the data cases

This level is the easiest to explain. Each bug, test case has its folder. It should also have a name representing the problem that it tests.

4. Don’t be afraid to add additional grouping folders

I don’t like to have many folders in the solution tree, so I group them. If this is also your preference, remember that the grouping key should be simple and easy to understand. Some that I used:

a website domain name
name of the client that encountered the error
application version

Anything will do as long as it is simple to explain and intuitive.

File tips

1. Use proper file types.

JSON is the go-to format because of readability, and ease of import/export. Some rules make working with them easier:

Consistent formatting. As we said, most of our data files will come from system usage. Those files should not be saved with formatting (it is a nontrivial overhead). The formatting should be applied when such file is used in tests. Define a convention, or better yet a tool that will do the formatting. It is good to have them for data files for the same reason why we have them for code files - people will argue over them. I, personally, use the VS Code for formatting.
Consistent encoding. A very similar problem as the one above. It is obvious to use the same encoding for one dataset, but we need to have the same for ALL data files.
Not only JSON. Some data types are just perfect for tabular files formats like CSV, TSV, etc. For reading them, I recommend CsvHelper
Remove not important fields. We don’t always have a perfectly logged object. When using a data file from production, analyze the properties, and remove the unnecessary ones. Remember, tests should be optimized for readability. This also applies to test data.
Anonymize the data. Don’t commit to the repo data files that contain sensitive or personal data. Once you identify the problem anonymize the data file.

JSON tips

1 object in one file. Some tests will take in a collection of elements. Then it is tempting to have one JSON file with an array of objects. Don’t do it. JSON files aren’t easy to merge. Having big files is just asking for trouble. How to deal with this? I introduce a fourth level folder and save each object as a separate file:

Consistent formatting. I know I am repeating myself, but this is important. Have consistent formatting

CSV/TSV tips

Don’t use a comma as the separator. Commas are used in numbers, texts and aren’t too visible when opening the file. What to use, then? I use a tab.
Set the culture explicitly for reading the file. Culture on your machine may be different than the one on the build agent machine, or other developers. Set the culture explicitly and save yourself a lot of debugging. A sample using CsvHelper NuGet package:

var csvReader = new CsvReader(reader)
{
    Configuration =
    {
        CultureInfo = CultureInfo.CreateSpecificCulture("en-US")
    }
};

2. How to handle large files in data-driven tests

Large files or a large number of files can be a problem because they will significantly increase the repo size making Git slower. Here we have two equally good solutions:

2.1. Use Git LFS (Git Large File Storage)

Git LFS was designed to store nontext or large files. Files stored using Git LFS are visible ass regular files in the repository, but changes to them aren’t tracked using the standard git diff mechanism. When cloning the repository with Git LFS, we will only get the latest version of the file, saving us a lot of disk space. Sound awesome. And for most cases it is. But it has some drawbacks:

Requires support on the git server. Git server has to support Git LFS. Most online providers do, but some older on-premise solutions might not.
Requires an extension on every developer machine. Each developer has to install Git LFS. Without the extension, we will see text pointers to Git LFS files.

2.2. Use Git submodules

Git LFS is fantastic, but for text files, we might still prefer the usual git way. What then? The answer is to use Git, but put the files in a separate repository that will only contain data files. Then add that repository to the code repository using a submodule:

git submodule add GIT_REPOSITORY_URL

This call will clone the data repository to a folder inside our standard repository.

When cloning the code repository, we only need to add one flag to the clone call (--recurse-submodules):

git clone --recurse-submodules

This approach has some significant benefits:

Easy to share. Because it is a separate repository, it can be used as a normal repository or used as a submodule in other repositories.
Separate versioning. The code repository links to the specific commit in the data repository. Versioning them separately can be handy when we are getting test case data from an external source and may not be in control of the structure.

This is not the end.

Jekyll doesn’t like long posts (I learned it the hard way with this one), and I also feel this is a good moment to break this post. To be continued …

How to get data for Data-Driven Tests?

Tue, 27 Aug 2019 00:00:00 +0200

The previous post was meant to be an encouragement and a warmup to data-driven testing. This post describes why I love this way of testing. Understanding a simple fact about testing moved me from “Oh, I should write tests” to “I want it all! And I want it now!” and Data-Driven Testing. And the simple truth is:

Before we begin

I will be converting this series into an ebook, or something like that, with additional source code and examples. If You want to get it please subscribe:

Data is the king

Anyone who is a bit interested in machine learning heard statements like data is the king, data is the new oil, etc. The same applies to tests. Having tests provides very little value. Having tests with the right test data provides huge value. Let me explain. Previously I wrote that I don’t like writing tests. Writing tests is one of two:

It is a verification of the quality of the code you produced.
Or can be the verification of the skill in hacking using mocks, stubs, and fakes you poses.

Pick one. I will concentrate on the first option. In it, writing tests is fun and engaging. It requires a lot of rethinking of previous assumptions, has a very short feedback loop, and gives a feeling of satisfaction. Fun and awesome.

The part that drains my mental energy the most is producing test cases and test data. This part is:

boring.
doesn’t require significant thought.
in most cases is copying previous data and changing one variable.
repetitive
doesn’t give any feeling satisfaction

All the above reasons made me try most auto testing approaches in the .net ecosystem with the biggest hopes for Pex But, like most Microsoft products, it died. And for a good reason. It approached producing testing data from the wrong angle. It analyzed the code and found edge cases. Sounds reasonable - this is how we write tests and why we write tests. To test the edge cases. But is it how we should write tests? No! Writing tests this way is just duplicating the conditions from ifs, switches, etc. We end up with tightly coupled, made-up tests.

I am not saying that such tests aren’t necessary. They are useful. But only initially, the first verification. Once we verified that the code is working, or we want to test more complex logic, they become too expensive. Time invested in them produces a minimal gain. It makes no think of the data that will cover all paths. But we need the data to test the code. How to get it?

How to get test data

Where to get better test data? That is simple - from the users. A user of our code is a developer, a tester, or a triggered job that will call it. In this case

User = someone who uses our code.

All we need to do is to log the call data. This functionality can be already implemented or might require some tinkering in our app. Let’s examine a few usages.

Messages

Some architectures are better than others for gathering data for data-driven tests. Message-based ones are near perfect. Most message systems have the notion of a dead letter queue - a queue that contains messages that failed to process (thrown an exception). The dead letter queue makes for a perfect input to our data-driven tests. If we want to have a better correlation of the message and the exception or wish to have all the data in logs, we might do something like this:

public class MyHandler<MyMessage>{
    /*
    initization code etc...
    */
    
    public async Task Handle(MyMessage m){
        try{
            // message handling logic here
        }catch(Exception ex){
            var messageJson = JsonConverter.SerializeObject(m);//the call failed, but we have access to the message that coused the error.
            _logger.LogFailedMessages(messageJson,ex);
            throw ex;//we need to throw to mark the message as failed so it is retries or put into dead letter queue.
        }
    }
}

The above is a very simplified implementation designed to show the idea, so please don’t use it on production. A better way to implement it is to hook this code in the code calling the handlers.

Asynchronous processing

A simplified bit easier to implement approach is to use async processing. I don’t mean the use of async keywords in controllers, but splitting the business process into asynchronous jobs that can be processed later.

Example: We register the user, but create a job that will send the confirmation email.

For this I use Hangfire.

A shameless plug: I wrote a series of articles on Hangfire and how it works.

It makes it even simpler than messaging. Hangfire displays execution parameters with the thrown error in the Hangfire dashboard:

MVC/WebApi

Http calls make it more tricky, but still possible. The same try/catch logic can be implemented in the controller methods directly, but a more general solution is to have a global filter. Something like this:

    public class MyErrorFilter : IActionFilter
    {
        public void OnActionExecuted(ActionExecutedContext context)
        {
            if ( context.Exception != null && !context.ExceptionHandled) {
                // get data from context.HttpContext.Request
            }
        }
    }

The code above is not for production use. Especially since HTTP calls logging requires considering a few additional things:

You have to filter our typical HTTP errors (404, 400, etc.)
If the website is public, you will get a ton of automated scans looking for /phpadmin and some injection attempts. Those have to be filtered out.
HTTP requests, more often than the previous, contain sensitive data such as user and password, cookie values, etc. Don’t log those. Never.

Logging on this controller level has one significant disadvantage when compared to the previous options. Unit and data-driven testing for controllers is hard. How to do it better?

Log on business methods level

The more pinpointed the logging, the better. Ideally, every vital method that has a data-driven test should be logging call data. Here, we might encounter the banana problem.

Trying to serialize HTTPContext, DBContext or anything with lazy loading might result in us trying to serialize the whole application world. Not passing such objects is a good codding practice. Simple and narrowed down classes are better anyway.

Things to consider when logging input parameters for data-driven tests:

Sensitive data. Logs are very often treated with not the same care as the database. Adding input data logging means that they should be. They will contain important data.
GDPR. Adding such logging, we are adding yet another place to monitor for user data.
Data size. It is very easy to create tens of thousands of such logs. Filter them out and fix fast.
Not a silver bullet. This approach requires architecture changes. Just dropping it into an application with global state won’t provide to much value.
Performance. We are logging only in case of a failure. Why care about response timings so much? The user won’t be happy anyways. What if the user is not human and we are getting thousands of such errors a second? Logging is making our situation just worse. Be careful, notify Your Ops team.
Just logging is not enough. Only logging won’t make your system better. Logging is so that it is easier to fix the bugs. Don’t forget about that.

Creating data-driven tests

I have the data now what?

Format it. Remember, tests should be optimized for readability.
Save it to a special directory
Read it and use it in a test.

Remember the TestCaseSourceAttribute from the previous article? Just change it into something like this:

    public static TestCaseData[] GetData()
    {
        var retArray= Directory.GetDirectories("MyTestDataFolder", "*", SearchOption.AllDirectories)
            .OrderBy(a => a)
            .Select(a =>
            {
                var ret = new TestCaseData(JsonConverter.DeserializeObject("testdata.json"));
                ret.ExpectedResult = JsonConverter.DeserializeObject("expectedResult.json");
                ret.SetName(a.FolderName);
                return ret;
            })
            .ToArray();
        return retArray;
    }

Some assumption:

Each test is in a separate folder.
The name of the folder describes the test case.
Each folder has a testdata.json file containing the test data object.
Each folder has an expectedResult.json file containing the expected result object.

I started with the intention to write a tips and tricks entry, but after a few conversations, I saw that a part about gathering data is missing. I promise tips, tricks and good practices are next! If you don’t want to miss it, subscribe to the newsletter.

Refactoring to Data Driven Tests

Tue, 20 Aug 2019 00:00:00 +0200

I am not a big fan of writing tests. I like having them, but I find writing them to be boring. That said, retesting manually is even more annoying, so I write tests. The thought that there has to be a better way, never passed. I tried a few approaches. After some experimentation, I think I have the answer - DDT (Data Driven Testing)

Before we begin

I will be converting this series into an ebook, or something like that, with additional source code and examples. If You want to get it please subscribe:

Establishing the baseline

Before we dig into what Data Driven Tests are, let’s look at some standard, non-DDT tests.

What we will be testing?

There is a fragile line for code examples. Too simple and they lose any business applicability. Too complicated and the domain overwhelms the problem described. Let me know if this problem works.

We will be testing an automatic gearbox controller for a car. This “simple” box decides when to change gear. Our version will get the following inputs:

Gear
Acceleration applied (values in the range of 0 to 1 in our example)
RPM (Rotations Per Minute), so how fast the engine is turning.

The return value is only one - the gear the engine should use. It sounds simple. Let’s see some tests.

The starting point. No Data Driven Tests

    public class GearCalculatorTests
    {
        [Test]
        public void ShouldReduceGear_whenAcceleratingRapidlyOnLowRPM()
        {
            //Arrange
            const int initialGear = 5;
            const double rapidAcceleration = 0.7;
            const int currentRpm = 2000;

            //Act
            var gear = new GearCalculator().Calculate(rapidAcceleration, currentRpm, initialGear);

            //Assert
            Assert.AreEqual(initialGear - 1, gear, "The gear should decrease by one");
        }

        [Test]
        public void ShouldNotReduceGearWhenOnFirstGear_whenAcceleratingRapidlyOnLowRPM()
        {
            //Arrange
            const int initialGear = 1;
            const double rapidAcceleration = 0.7;
            const int currentRpm = 2000;

            //Act
            var gear = new GearCalculator().Calculate(rapidAcceleration, currentRpm, initialGear);

            //Assert
            Assert.AreEqual(initialGear, gear, "The gear should stay the same");
        }

        [Test]
        public void ShouldNotReduceGear_whenAcceleratingNotRapidlyOnLowRPM()
        {
            //Arrange 
            const int initialGear = 5;
            const double nonRapidAcceleration = 0.4;
            const int currentRpm = 2000;

            //Act
            var gear = new GearCalculator().Calculate(nonRapidAcceleration, currentRpm, initialGear);

            //Assert
            Assert.AreEqual(initialGear, gear, "The gear should stay the same");
        }
    }

Refactoring to Data Driven Tests

What are Data Driven Tests?

Not what we have some baseline, let’s define what a Data Driven Test is.

A Data Driven Test is a test where the data is separated from the logic.

It might sound obvious. We don’t mix data with code. Don’t we? Look at the tests above and try to recall if your tests aren’t splashed with a big dose of setup objects. They probably are.

Some people argue saying that this is normal. Test need testing data. I am not saying they don’t. I’m saying that we should not couple test logic and test data so strongly. A few arguments to back up that statement:

Tests should always be optimized for readability. Tests have a lot of boilerplate. Mocks, stubs, test data classes, mocked methods, etc. All this code makes it hard to know what is the boilerplate and what is the needed data. Therefore it makes the tests less readable.
Test cases should always be optimized for readability. Again about readability? Yes, but in a different take. Let’s take two tests testing the same class, but a different path. Let’s say they are a medium size test with ~100 effective lines of code. What will be the difference between those two tests? I’m betting below ten lines of code. That is 10%. Even if the tests are readable, the difference between them is not.

The problem with tests like above is that new ones will be made by copying an old test and changing some lines (making the 10% changes). Lowering readability further on. Now let’s refactor the above tests to Data Driven Tests.

Version 0.1 Refactoring

The first step is to extract the data from the test code. Let’s put in into a function:

private void CalculateAndVerifyGear(double acceleration, int currentRpm, int gear, int expectedGear){
    //Act
    var calculatedGear = new GearCalculator().Calculate(acceleration, currentRpm, gear);

    //Assert
    Assert.AreEqual(expectedGear, calculatedGear);
}

Now, to use it:

    public class GearCalculatorTests
    {
        [Test]
        public void ShouldReduceGear_whenAcceleratingRapidlyOnLowRPM()
        {
            //Arrange
            const int initialGear = 5;
            const double rapidAcceleration = 0.7;
            const int currentRpm = 2000;

            CalculateAndVerifyGear(rapidAcceleration, currentRpm, initialGear, initialGear - 1);
        }

        [Test]
        public void ShouldNotReduceGearWhenOnFirstGear_whenAcceleratingRapidlyOnLowRPM()
        {
            //Arrange 
            const int initialGear = 1;
            const double rapidAcceleration = 0.7;
            const int currentRpm = 2000;

            CalculateAndVerifyGear(rapidAcceleration, currentRpm, initialGear, initialGear);
        }

        [Test]
        public void ShouldNotReduceGear_whenAcceleratingNotRapidlyOnLowRPM()
        {
            //Arrange
            const int initialGear = 5;
            const double nonRapidAcceleration = 0.4;
            const int currentRpm = 2000;

            CalculateAndVerifyGear(nonRapidAcceleration, currentRpm, initialGear, initialGear);
        }
    }

This is the starting point. We will refactor it further on

The refactor has a few problems:

We lost the assert message (it could be added to our version, but most tests have more than one assert. Passing many assert messages will make the test unreadable)
New tests will still be made by copying an old one and changing a few dials. Changes will be more visible, but this still isn’t good enough.
Test data still isn’t a first-class citizen.
When looking at those tests, I get the feeling that there should be a better way.

Version 0.2 - Making test data a first-class citizen

There are a few ways the problems mentioned above can be addressed. We will go over them one by one.

From here on we will use NUnit. If You prefer XUnit or any other testing framework, check the documentation. It probably supports Data Driven Testing.

The first way is to use a TestCase attribute. It allows to pass parameters to tests. After refactoring the code will look like this:

    public class GearCalculatorTests
    {
        [TestCase(0.7, 2000, 5, 4, Description = "When accelerating rapidly on low RPM should reduce gear ")]
        [TestCase(0.7, 2000, 1, 1, Description = "When accelerating rapidly on low RPM should not reduce gear when on first gear ")]
        [TestCase(0.4, 2000, 5, 5, Description = "When accelerating NOT rapidly on low RPM should not reduce gear")]
        public void CalculateAndVerifyGear(double acceleration, int currentRpm, int gear, int expectedGear)
        {
            //Act
            var calculatedGear = new GearCalculator().Calculate(acceleration, currentRpm, gear);

            //Assert
            Assert.AreEqual(expectedGear, calculatedGear);
        }
    }

Yes, that is all. Initial 45 lines of tests now take 15 lines of code and are more readable. Now I see that I could remove the current RPM from the tests data because it never changes.

This approach is better than what we started with, but has some drawbacks:

Attributes can only have parameters that are known on compile time.
We can’t pass complex objects to attributes.
There are no compile-time checks for the wrong number of parameters (Resharper will show a warning).
Values in attributes aren’t that readable.
Having four attributes is OK, but more of them will be messy.
It is hard to read what we are asserting.

Let’s first address the last point.

Version 0.2.5 - Better asserts

To better express what we are asserting, we can use the ExpectedResult from TestCaseAttribute. After refactoring it will look like this:

    [TestCase(0.7, 2000, 5, ExpectedResult = 4, Description = "When accelerating rapidly on low RPM should reduce gear ")]
    [TestCase(0.7, 2000, 1, ExpectedResult = 1, Description = "When accelerating rapidly on low RPM should not reduce gear when on first gear ")]
    [TestCase(0.4, 2000, 5, ExpectedResult = 5, Description = "When accelerating NOT rapidly on low RPM should not reduce gear")]
    public int CalculateAndVerifyGear(double acceleration, int currentRpm, int gear)
    {
        //Act
        return new GearCalculator().Calculate(acceleration, currentRpm, gear);
    }

How this works:

now our test returns a value.
this value will be compared with the value of ExpectedResult

It is a bit better, but we still have to address drawbacks 1-4.

Version 0.3 - TestDataSourceAttribute

We can adress most of the drawbacks by using the TestDataSourceAttribute. This attribute allows defines a method that will generate test cases by creating a TestCaseData objects. Let’s see it in action and explain later:

    public class GearCalculatorTests
    {
        public static TestCaseData[] TestDataSource
        {
            get
            {
                return new[]
                {
                    new TestCaseData(new GearBoxTestData(0.7, 2000, 5))
                        { ExpectedResult = 4, TestName = "When accelerating rapidly on low RPM should reduce gear "},
                    new TestCaseData(new GearBoxTestData(0.7, 2000, 1))
                        { ExpectedResult = 1, TestName = "When accelerating rapidly on low RPM should not reduce gear when on first gear "},
                    new TestCaseData(new GearBoxTestData(0.4, 2000, 5))
                        { ExpectedResult = 5, TestName = "When accelerating NOT rapidly on low RPM should not reduce gear"}
                };
            }
        }

        [TestCaseSource(nameof(TestDataSource))]
        public int CalculateAndVerifyGear(GearBoxTestData data)
        {
            //Act
            return new GearCalculator().Calculate(data.Acceleration, data.Rpm, data.Gear);
        }
    }

    public class GearBoxTestData
    {
        public GearBoxTestData(double acceleration, int rpm, int gear)
        {
            Acceleration = acceleration;
            Rpm = rpm;
            Gear = gear;
        }

        public double Acceleration { get; set; }
        public int Rpm { get; set; }
        public int Gear { get; set; }
    }

A few things to note:

We could also pass parameters the same way as with TestCase, but not using the power of dedicated class would be a waste of potential.
Notice that the method has to be static. Why? Because this generates separate tests for each TestCaseData.

How this works is that a test runner first calls the method defined in TestCaseSourceAttribute and generates tests based on data returned. Each TestCaseData will be seen as a separate test:

Depending on the runner, each test can be debugged separately. They behave just like any other tests.

Is this the end?

No. But this post is long enough as an introduction. Following tips and tricks.

How to choose the partition key

Tue, 13 Aug 2019 00:00:00 +0200

NoSQL PaaS databases like AWS DynamoDB or Cosmos DB offer incredible capabilities in scale, speed, and availability. There is also a dark side to those databases. They will punish anyone greatly for mistakes. And no mistakes are punished more than choosing the wrong partition key. Below is an ORDERED list on how to approach selecting the partition key.

1. Transaction boundary

I can’t stress this enough.

This is the most critical factor. If our transaction boundary is wrong, please don’t even think about using this partition key.

Now to elaborate.

In Cosmos DB and DynamoDB partition is the transaction boundary. An operation performed over records with N partition keys (and so in N partitions) is split into N separate transactions. Each partition transaction can fail or succeed independently with no rollback of the primary transaction.

If some operations fail, the changes to other partitions won’t be rollbacked.

There is no way around this limitation.

Why is this requirement so essential? We have to take into account a few limitations:

Both databases have minimal capabilities for multi-operation transactions (Cosmos DB supports them only in stored procedures, pre/post triggers, user-defined-functions (UDFs)).
We don’t have transaction isolation levels like in relational databases.
There is no option to lock a record for editing.

The above limitations mean that we will encounter a lot more cases when data changed between our operations. To not override data Cosmos DB and DynamoDB uses optimistic locking. It guarantees that the application will know that some changes weren’t applied, but won’t prevent data inconsistency.

Therefore not platform failure, but changes executed between operations will be the main reason why we might want to revert changes.

I wrote how to simulate transaction isolation on noSQL databases in the previous post.

2. Size limitation

The obvious solution to the lack of transactionality over multiple partition keys is to store everything in one partition. Not so fast. Cosmos DB and DynamoDB have a hard limit on partition size - 10 GB. (DynamoDB, CosmosDB). Making matters worse, this is not a limit only on data, but all the things around it like indexes. When the limit is reached, all operations increasing the partition size will fail. As with the transaction boundary, this is a hard limit, and can’t be changed.

3. Partition key values can’t be updated

When creating a table/collection, we have to define a path to the property which value will be used as the partition id. Selecting a property as the partition adds some additional requirements on it. There are some type and size limitations, but most importantly, this property has to be present in every document in the collection/table.

From the application logic, it is just like any other required property. With one exception:

Values of properties designated as partition keys can't be changed.

Why?

Why partition keys values can’t be changed?

Let’s think about it. We know that:

The partition is the boundary of the transaction.
All documents with the same partition key value are in one partition.
The partition contains documents with the same partition key value.

The change of a partition key property value would spawn across two partitions (old, and new). However, that won’t be transactional. For this reason, Cosmos DB throws an exception if we try to change the value of the partition key property.

4. Performance limitation

One partition can’t be split across multiple servers (DynamoDB can split a single partition using a range key, but this is more or less equal to having a more detailed partition key value). This leads to an obvious limitation:

Partitions have a limit on the number of operations a second they can perform.

Since partitions have limited performance capabilities we might want to distribute queries over multiple partitions. In general, this is a good practice but has to be executed with one thing in mind.

NoSQL databases have a named anti-pattern for partitions that are used more than others - hot partitions.

We have to be careful with load distribution because of:

5. Cost

One of the first steps in query execution in Cosmos DB and DynamoDB is to send the query to proper partitions. When we don’t provide a value for the partition key property in the query, the engine executes it on all partitions. Even if the query executed on the partition doesn’t return any results, we will still pay for the used compute resources. Most of those queries should end on index scans. This little expense is multiplied by the number of partitions and might end up a significant one in the end.

A good practice is to design queries so that they always execute on as few partitions as possible.

Conclusion

If anyone expected a straightforward answer on how to choose the partition key, sorry for letting you down. There is no simple answer. As always, it depends. Choosing the partition key is always a trade-off:

Large transaction boundary or the peace of mind of small partitions?
Risk of hot partitions or more expensive queries?

Stop trying to mock the ILogger methods

Mon, 05 Aug 2019 00:00:00 +0200

I’ve seen multiple people struggling with trying to mock the ILogger calls. This post will hopefully save some people from wandering this path. Here it goes:

Please stop trying to mock the ILogger methods from .net core.

Here is why:

It is not designed to be mocked

Most ILogger methods that we use for logging are extension methods - static methods. While mocking them is possible using Prig, or Microsoft Fakes it is not easy or pleasant. Mocking a static method boils down to replacing the code at runtime using the debugger API. It sounds hard because it is hard. Those testing frameworks have additional restrictions that may make them impossible to run on a CI server.

Why test calling a log method?

A log is a technical detail. It should be used for diagnosing problems and not much else. The most common reason I see why people want to mock the ILogger interface is because they are using a logger to write audit data to an audit file using a logger.

It might sound like a good idea, but it is wrong for a few reasons:

Audit data should not be stored in a file. It can be exported to a file, but a file should not be its main storage.
Loggers are optimized for throughput, not consistency. They might lose data.
Writing to a file is not (it can be, but don’t do it) transactional. If we write to a log and then rollback the transaction, the entry will stay in the file.
Most loggers use a buffer or a delayed/async write. Writing to a file is a slow IO operation. Loggers delay it not to slow down the process and write asynchronously in batches.
Calling the logging method does not imply that the entry will be written to a file. Loggers offer multiple ways to filter what will be written to which file. From log levels to disabling some loggers. Additionally, logging configurations are, in most cases, customized for each environment. Making the test meaningless.

You are testing the wrong thing

Even if we could test the ILogger call, we still shouldn’t do it. Let me show an example. Let’s imagine we are logging access to RegisterUser. Something like this:

    public class UserService
    {
        private ILogger _logger;

        public UserService(ILogger logger)
        {
            _logger = logger;
        }

        public bool RegisterUser(RegisterData data)
        {
            _logger.LogInformation($"User:{data.UserName} logged in.");
            // do some cerification logic
        }
    }

It might look reasonable to try to mock the LogInformation of the ILogger to verify if the business process logs registration attempts.

When testing if a business process logs some information, we should be testing the business intent. By testing the ILogger call, we are testing the implementation. Those two are mixed in the code above, causing our problems.

Refactor

If we split the implementation from the intent, our work will be much more straightforward:

    public class UserService
    {
        private ILogger _logger;

        public UserService(ILogger logger)
        {
            _logger = logger;
        }

        public bool RegisterUser(RegisterData data)
        {
            LogRegisterAttempt(userName);
            // do some cerification logic
        }
        
        internal virtual void LogRegisterAttempt(string userName){
            _logger.LogInformation($"User:{userName} tried to register.");
        }
    }

Now for a simple test (used Moq, but any mocking framework will do):

    public class UserServiceTests
    {
        public void RegisterUser_ShouldLogLogginAttempt()
        {
            // Arrange
            var registerData = new RegisterData(){
                /*
                ...
                */
            };
            var userService = new Mock<UserService>(null);
            userService.CallBase = true;
            userService
                .Setup(a => a.LogRegisterAttempt(It.IsAny<RegisterData>()))   
                ;

            // Act 
            userService.Object.RegisterUser(registerData);

            // Assert
            userService.Verify(a => a.LogRegisterAttempt(registerData), Times.Exactly(1));
        }
    }

A few things to note here:

Those who read the code carefully will notice that I am mocking the system under test (my UserService). Let me explain:

Yes, it will work (the method has to be virtual).
Yes, you can do such things.
Yes, it is perfectly OK.
The RegisterUser will be called thanks to setting CallBase = true

Making it even better

An even better idea is to refactor the audit logging function into a class. Since it is fulfilling a business need, so it should encapsulate how it is doing it.

We aren’t testing the same thing!

No, we aren’t. We are missing one test. One that will call LogRegisterAttempt, read the log file and verify if it has the log entry - a simple integration test. I leave it to the reader ;)

Conclusion

Testing is not only writing test code. It is also looking at the existing code and making it better.

If something is hard to test, it means that it is badly written.
or
You are trying to test something that shouldn't be tested in this way.

Why use a service mesh? A few arguments

Mon, 29 Jul 2019 00:00:00 +0200

I will be back to the series about data modeling in noSQL databases, but for now, I want to note down a few arguments that are still hot after an interesting discussion with a friend. The talk boiled down to a simple question:

Should I use a service mesh or use a bare bones Kubernetes?

I know how strange the combination of bare-bones and Kubernetes sounds, but this is the world we are living in :).

I’m standing firmly on the position that service mesh is the only way to go. Here is why:

A service mesh means less code

A correctly implemented synchronous call should have the following policies/rules around it:

getting a JWT token for authentication
timeout for the service we are calling
retry policy with exponential backoff and jitter
circuit breaker (a distributed if possible)
caching
fallback policy on what happens if the service is down
a timeout to wrap the whole thing since it isn’t that simple any more.

Depending on what framework, or language this will be ~200-400 lines of code. For one HTTP/gRPC call! I know they can be applied globally using things like HttpClientFactory in .net, but they still need to be defined somewhere. The argument that “we can define them once and reuse” is false. Those policies will/should differ depending on the business use case. Reuse will be possible, but not to a full extent.

Developers don’t …

Developers don’t care

This title is a read-bate, but let me explain. Developers are not the ones who will get the call when the system crashes. They should be, but in most cases, the on-call person will be someone with “DevOps” in the job title. This creates a week incentive for implementing those rules properly. Most developers have their backlogs full of business features that will take them more than a year to implement. They won’t spend enough time on this. And even if they would:

Developers don’t understand

Let’s take a developer that will want to do it properly. To implement all the policies correctly, he, or she would have to understand all the ways a system might fail. This knowledge can be gained from:

Watching systems fail daily.
Doing post mortem and root cause analysis.
Having access to systems failing where people notice - the production environment. Those are not the tasks of a developer, but an SRE (Service Reliability Engineer).

Developers don’t have the numbers

Let’s imagine a person who wants to do it properly, and will spend a lot of time understanding how a system might fail. They would be still missing some crucial information. The numbers. Almost every policy has some number attached to it:

Policy	Number	Sources
Timeout of the service we are calling	Timeout value	SLA agreement verified using System monitoring
Retry	Number of retries Backoff time Jitter value	System monitoring
Caching	Caching time	HTTP headers System montoring
Circuit breaker	Open state time	System monitoring

Even if the developer acquires those values, they might change over time. What then? Should we redeploy the code to change the timeout value? It doesn’t sound reasonable. The alternative is to have them in a config file, leading do a large and unmaintainable file.

The circuit breaker

The circuit breaker policy is the one that gains the most when done in the layer of a service mesh. For two reasons:

Distributed circuit breaker

Having a circuit breaker is fine. But it is only the beginning of what it can and should do to protect our system from cascading failure.

A standard circuit breaker works in the scope of a single application, not the whole system. It looks at requests going out of the service and monitors the number of errors. If the percentage of errors crosses a particular value, communication to that server is cut for some time. We do it not to escalate failure to other services. When looking from a single service point of view, it is the right solution.

When looking from the point of view of the whole system, the problem is still unresolved. Because each service has to open it’s breaker independently, services will hammer the under stress service. The more reasonable way to do it is to use a distributed circuit breaker. A distributed circuit breaker broadcasts the information about opening and closing a breaker to other services. If a breaker in one service opens, others will be notified about this via the broadcast, and also open. This way, we remove a faulty service from the whole system almost at once.

Implementing a distributed circuit breaker in the application code is not a trivial task and will require the use of some centralized coordinator.

When switching our mindset to a service mesh where we are proxying all incoming requests, the solution becomes very simple. We can break on the proxy — a much simpler solution.

Overload circuit breaker

The circuit breaker pattern is protecting the caller from calling a faulty service. That is great. We fail faster and protect from requests piling in on our input while retrying or waiting on the defected service. Now let’s think about the electric circuit breaker. It works differently. The real one protects from a too high current entering the system. In system architecture terms: from too many requests entering the application. Adding requests to a system under load will only make it worse. With the exception to this:

It is better to fail them earlier and have a chance for the system healing itself. An overload circuit breaker protects from failure happening, not from the consequences of failure. Such a circuit breaker is hard to implement on the application layer, but very simple when using a service mesh.

Get rid of documentation

At least some parts of it. A service mesh can easily give us the actual topology of the system. No more looking at diagrams and asking ourselves:

How outdated is it?

Open Zipkin, Jaeger or Application Insights and see in real-time what service is used in which business process. Easy and, more importantly, accurate. Some might argue that we can have this without a service mesh. Yes, we can. But it is easier to deploy and maintain it with a service mesh.

Monitoring, tracing and centralized logging

Except for reporting business metrics, the application should not care what type of centralized logging system is used. Why should we add a wrapper for measuring execution time to every call? We will end up with something like this :

Timer t = Timer.Start();

_http.Get("http://google.com");

t.Stop()
_metrics.ReportTime("googlecall",t.ElapsedMilliseconds);

Application code isn’t the right place to do it. It can, and should be implemented globally. This way, we will be able to:

guarantee a standard for the naming scheme.
be sure that all requests are logged.
have all applications log to the same system.
have logging independent from application framework and language.
minimize the infrastructure boilerplate that is needed before a team can start delivering value.

Reducing complexity

Granted, we can’t remove complexity. We can only move it to a different place in the system. But moving technical services such as service discovery and API gateways from application code to infrastructure sounds OK to me. They shouldn’t have any business code in them anyways.

Security

Kubernetes, Docker, VPN, and sub-networks give us the possibility to narrow down the list of who can talk to who. A service mesh is going a step forward by monitoring who is talking and what type o traffic is it. What makes it even better is that previously our monitoring would look at packets of TCP/IP or UDP traffic. In a service mesh, we are looking at HTTP traffic. This opens a lot of possibilities and makes some checks simpler.

It just works

How important are the decisions below?:

Eureka or Consul?
Jaeger or Zipkin?
Grafana with InfluxDB or Graphite?

Doesn’t the decision boil down to the question: What works with what we already have? Most service meshes are an opinionated set of tools that work together. That is a good thing. I can live with a bit less system integration code in my backlog. Can You?

What will be the difference in the end?

Going back to the talk that sparked the need for writing this article. The more we discussed, the more I got closer to the essential point:

Let's asume that you didn't use a service mesh.
Then you add all the necessary infrastructure for a distributed system.
How different will it be from a service mesh?

We won’t have a lightweight reverse proxy deployed with every service (probably). But is this such a high overhead versus all the things that a service mesh makes easier?

No flaws?

On the contrary! Service meshes offer even more ways to shoot ourselves in the feet, hands, the head, etc. Like with every technology we will have to spend time to learn it’s pitfalls and how to avoid them. But this is a topic for a different conversation.

Modeling version and temporary state in noSQL databases

Mon, 22 Jul 2019 00:00:00 +0200

The problem of storing draft state and auditing is not limited to noSQL databases, and as previously, below patterns can be applied to SQL modeling. But because noSQL databases are, in most cases, lacking transactionality over multiple partitions, the problem is harder there. Making it more interesting :) Additionally, smart use of neet features of noSQL databases allows for a novel solution.

The problem(s):

Problem 1: Storing temporary state

Most developers cringe when they hear about storing temporary state, but can’t imagine living without Save as draft in Gmail, or git stash :) Give the users what You expect. A different few use-cases :

The user can edit a document and save it as a draft. That draft can be later on committed or discarded.
UI flow requires a page reload when working on a single object. I know that we have SPA applications now, but not everywhere and we might be switching between different systems, or refactoring a legacy system.
Our document validation logic is quite complex, requires external HTTP calls, takes a long time to execute, and sometimes requires retries. Since we are good developers, we don’t run that logic during saving but save as draft, schedule a background task to do the validation. If something fails, we notify the user - a good UX.

All the above can be simplified to a problem of storing not confirmed data. We can solve it in a few ways:

Slap a `State` flag field with two states: `Draft` and `Confirmed` to the object.

This is one of the most common solutions I see in systems. It is effortless, and those are all of its advantages. The list of disadvantages is longer:

Everyone has to remember to add State = Confirmed to every query over this collection/table.
Modeling different objects (DraftOrder and Order) using enums is breaking SOLID principles.
Adding a Status property opens the gates to other states. If we leave opened gates, we can be sure that dragons🐉 will come.
We can’t reuse the required fields, and validation logic since a draft is not a full object. Leading us to objects that are the same but are not the same.

Add a boolean field `IsDraft`.

It might look like a good workaround to not leaving the gates open. But it is even worse because more flag fields will come making the problem of state distributed (current state = SUM(all flag fields)).

A separate table/collection to store the draft object.

We can create a separate table or collection to store draft state. Sounds OK, but again we have some drawbacks here:

When migrating object structure, we have to migrate the structure of the original object and the drafts.
Most noSQL databases guarantee transactionality only in the scope of a single partition. We loose this by splitting the collections.

Problem 2: Entity auditing

In essence: Who changed what and when. A must-have in most financial systems and with time will be implemented in every system that allows for any user data update. Why?

Why did I say that those two problems are very similar? Change the State to VersionNumber add who and when did the change and we have versioning. I know I am oversimplifying but stay with me.

Now for a few well-known patterns:

Storing only the changed fields

Because we want to track changes, we can only store the fields that changed. We will end up with something like this:

FieldName	NewValue	Date	ChangingUser	ChangeGroup
Name	Anna	2019-07-19	Rick Sanchez	10
Surname	Smith	2019-07-19	Beth Smith	10

Where:

FieldName - name of the field what was changed
NewValue - as the name suggests, the new value of the field
Date and ChangingUser - date and the user responsible for the change
ChangeGroup - a unique number combining all changes in one save/update action

The main benefits of this approach:

We can very quickly answer the question who changed a particular field
Since in most cases, we change a small percentage of all properties, this solution saves a lot of space.

While it is very flexible, the flexibility has some drawbacks:

When migrating the object structure, we also have to update FieldName values.
Reading object state in a specific moment in time is quite complicated. Requires reading all past events from the given date and applying them in order.
Auditing adding and removing from lists is not that easy.

Storing full objects with audit data

In this approach, we store the whole changed object (as a blob or in a structured way) and only append audit data to it. The database structure would look something like this:

Id	Original Entity Properties	EntityId	Date	ChangingUser
0	{…}	10	2018-06-19	Rick Sanchez
1	{…}	10	2019-07-19	Evil Morty

Again, there are plusses:

getting the state in time is very easy
very easy to implement
object complexity is not an issue.

And minuses:

we are storing a lot of duplicated data.
getting the information who changed a particular field is a non-trivial task.

Alternative solution

All the above solutions can be implemented in SQL and noSQL databases, but they were designed with the first in mind. When we take into account features that noSQL databases offer:

PartitionKey/PrimaryKey can be assigned to multiple rows/objects/states.
Id/SecondaryKey can be used as a drill-down key to a specific object.
The ability to do commands pre-checks. A pre-check is an if statement. It has two properties:
- is executed in the same transaction before the operation.
- if it returns false, the operation won’t execute.

Having the above allows us for a bit different solution:

Version	CurrentVersion	Entity business attributes
0	2	{…}
1		{…}
2		{…}

Legend:

PartitionKey - primary entity key. It doesn’t change and is the boundary of our transaction.
Version - as the name suggests, the number of the object version. It is our secondary key.
CurrentVersion - this field might look like an unnecessary duplication but isn’t. We use it for optimistic locking. See the processes implementation below.

Additional rules:

Row with Version=0 is always the current row.
There is a unique index on (PartitionKey, Version).
CurrentVersion is the last version that was approved.

Now to the processes. We have two:

Start editing

Calculate the highest version

Select MAX(Version) Where PartitionKey=X

Copy the Version 0 item to a new record with Version = newMax +1

COPY (PartitionKey, Version, CurrentVersion, ....) Values (v0.PartitionKey, newMaxVersion, v0.CurrentVersion, v0 business properties....) 

This is why we have a unique index on (PartitionKey, Version) pair. The newMaxVersion calculation is done in a separate call/transaction, so there is no guarantee that nothing happened in the middle (no new version was created). If a new version was created in the meantime, we will get an error. All we have to do is to recalculate the newMaxVersion and try to do a new insert.

Approve version

Here, we use only one statement:

UPDATE v0.CurrentVersion = newRow.Version, v0.BusiessProp = newRow.BusinessProp IF v0.CurrentVersion = newRow.CurrentVersion

Some explanation to the above:

v0 - the Version = 0 row.
newRow - our new row.

Why does it work?

IF v0.CurrentVersion = newRow.CurrentVersion is a command pre-check mentioned before. If this statement is false, the operation will fail. What it gives us, and why do we have it? We are comparing the CurrentVersion of the current state of the record with the CurrentVersion of our row when we started editing.

When will this fail? Think about it.

Remember that we are updating the CurrentVersion with each approval. It will fail if someone approved a version between us starting editing and approving the version. Without the if we would override changes made by someone else. With it, we will discover a version conflict that has to be resolved.

Another benefit to this approach is ACID like transactionality from starting editing to approving the version.

Conclusion

Yes, noSQL databases are simpler and less powerful in terms of features than their SQL cousins. But data modeling is about looking at the limitations and turning them into advantages.

How to model hierarchical data in noSQL databases

Mon, 15 Jul 2019 00:00:00 +0200

Querying hierarchical data is always where the big boys of SQL shined. And I really mean the big boys part since ~~only Oracle, and Microsoft SQL Server have support for CTE~~ [As Thomas Levesque pointed out in the comments, not only those have support for CTE. Full list on wikipedia] (Common Table Expressions) that allow for executing one SQL statement what will fetch a subtree. There are data modeling approaches that allow for doing hierarchical data reads with noSQL databases and databases without support for CTE.

An Example

The most common and obvious example of hierarchical data is the family tree (we are using parent and child when talking about hierarchical data for a reason). But because of limitations that we will discuss in the end, let’s look at another widespread usage: folder structure.

Modeling the structure in a relational database is very easy and can be done in one table:

Id	Name	ParentId
1	07-Shots	null
2	EX-000	2
3	0000	2
4	Anin	3
5	Cloth	3

names are self-explanatory.

SQL

As I wrote earlier, with SQL, we can use CTE. It will look more or less like this:

WITH Folder_CTE (Id, Name)
AS
(
    -- Select the root
    SELECT 
        rootParent.Id,
        rootParent.Name
    FROM Folder as rootParent
    Where Id= -- The most parent Id from which we want to read the structure

    -- Union the result
    UNION ALL

    -- Select the next level of hierarchy by joining Folder with the already selected elements 
    SELECT 
        Id,
        Name
    FROM  Folder as child
    Join Folder_CTE as parent on child.ParentId = parent.Id -- Join existing with new. 
)
-- Select the whole result
SELECT     
    Id,
    Name,
    Surname
FROM Folder_CTE

What will happen is SQL Server engine will unwind the query fetching results until no new results are returned. It still means doing multiple joins, but the performance penalty isn’t that big.

There are some limits to this approach.

As with most (all to my knowledge) programming languages, there is a limit to this recursion. In case of SQL Server it is 100 by default but can be changed to 32,767. With OLTP (Online Transaction Processing) loads this shouldn’t be a problem. For reporting (OLAP - Online Analytics Processing), you should be denormalizing your data anyway.

Modeling hierarchy in noSQL database or without CTE

Techniques for modeling hierarchy efficiently can be applied in noSQL databases and in SQL databases without the support for CTE.

Application recursion

The first approach is to fetch each individual layer of hierarchy one at a time with the looping done by the application. The initial SQLs will look like this:

SELECT 
    Id,
    Name
FROM Folder

The next SQL will look like this:

SELECT 
    Id,
    Name
FROM  Folder
Where ParentId IN (...) --Ids of parents read in the previous iteration

There are a few problems with this approach:

We will be executing multiple SQL queries
The IN operator has a limit to the number of elements it can have. I remember it being around 2100, but now the documentation mentions many thousands.
The query plan optimizer selects existing plans for procedures with the same text and number of parameters (if you are sending parameter values in the query text please stop reading and fix it). For every laver of hierarchy, we will be changing the number of IN parameters (@p1, @p2, @p3, .... @pN) so every query will have the overhead of the optimizer.

Building the hierarchy in the application

Another approach is to get all the data, build the hierarchy in the app, and then do filtering. If we will be holding the data in the application cache, this is the simplest solution, so let’s not over enginer.

Hierarchy path

The idea of this approach is to extend our object/table with one additional field - HierarchyPath:

{
    "Id":
    "Name":
    "ParentId":
    "HierarchyPath":
}

In this field, we put Ids of our parents (from the furthest away to the closest) separated by some unique character. Like this:

5$18$201$8

For folder structure example from the start of the post, it will look like this:

Id	Name	ParentId	HierarchyPath
1	07-Shots	null
2	EX-000	2	1$
3	0000	2	1$2$
4	Anin	3	1$2$3$
5	Cloth	3	1$2$3$

Where $ plays the role of the separator.

You might say. It’s stupid. Why would this make it any better? We are missing two additional ingredients:

Create a text index on the field.
Use StartsWith (or a right hand like) to search for the structure.

We can’t use a standard index since it will create an index of hashed values. That will not help us since a StartsWith on hashes just doesn’t make any sense. We need to query actual values. This type of indexes are available in most document databases under different names:

In our case, the query will look more or less like this :

SELECT 
    Id,
    Name
FROM Folder
Where HierarchyPath StartsWith "1$"

Using this approach, we can use a single query to select the whole subtree without doing a scan over all the records.

Criticism

When I saw it, it felt like this:

source: Daily Mail.

But the same as with the image, going deeper made me appreciate the solution more. For the plane ducktape I will give it to the experts:

Limitations:

We have the “feeling” part sorted, so let’s talk about the limitations.:

Selecting the identifier

This one isn’t significant, but being aware of. When using string identifiers inputted by users selecting the HierarchyPath separator isn’t that simple since it can’t exist in the Id itself.

Multiple parent scenario.

This limitation is why we didn’t model the parent-child / family tree scenario. In most cases, you have at least two (biological) parents, and the only way to support this is to generate all paths (all 2^N of them where N is the depth of the structure)

Changing the `Id` or the `ParentId`

When we change the Id, or the hierarchy structure, we need to update the HierarchyPath in the whole subtree. Solution for this varies a but for SQL and noSQL databases.

With SQL databases we can use a trigger (I know triggers are bad, but this isn’t business logic but technical, so they are less bad here). The main benefit of doing it this way is that triggers are executed in the same transaction, so we will not lose data integrity.

Even noSQL databases that offer the functionality of a trigger don’t execute them in the same transaction. Here we are left with eventual consistency.

The approach that can be used in both SQL and noSQL implementations is to trigger a background job (AWS Lambda, Azure Function, or a Hangfire job that will go over the data and update the HierarchyPath values.

Starting from the middle

The last, and a quite significant problem is that we can’t effectively ask for substructure that would start in the middle of the tree. When starting from the middle, we can’t use a StartsWith clause, but will have to go with Contains. This will scan all values in the index. It will still be faster than scanning the documents, but not as optimal as StartsWith.

Summary

The solutions above aren’t ideal, but no answer is. As always think about the pros and cons and select the best for your use case.

Lessons learned from making a video course for Pluralsight

Mon, 08 Jul 2019 00:00:00 +0200

So, Yes. I hit a pause button on writing because many things were happening, and as it turns out, you can only squeeze 24 hours in one day. Who knew? However (hopefully) my blog will come back to life. One of the things that kept me busy during this time was realizing one of my goals:

Make a video course for Pluralsight.

And I made it:

Now that it is done, and I had a few months to reflect, a few lessons for those thinking about doing any video course.

Preface

Choosing the course

Before we talk about what I learned, a few words about the course itself. First, how did I choose the topic. I had a few criteria:

I wanted to make a course about something that I feel is very underrated mostly because of lack of awareness in the development community.
I didn’t want to do yet another course about what is driving the hype train currently.
I wanted to make a course about something that I know but could use the course to structure and deepen my knowledge. As the saying goes: if You want to learn something, teach it to someone.

A few topics fitted those criteria:

graph databases
TPL Dataflow
general concepts of data modeling

After much thinking and talking with people from Pluralsight, the decision was made to do a course about TPL Dataflow. It fitted the criteria perfectly:

Not too many devs know TPL Dataflow
I used it a lot when optimizing cookit and calculating 17 billion similarities
It’s a perfect tool for some use-cases. I wanted to have a deeper understanding of how it works.

What is TPL Dataflow?

TPL Dataflow is one of those gems that Microsoft sometimes develops and doesn’t put enough effort to inform the public (similar cases: Reactive Extensions, Orleans).

Apart from the above it’s a nuget package (available for .net framework and .net Core ) that implements the Dataflow Programming Model. Dataflow Programming Model might sound ancient and abstract but just looking at the current implementations: TensorFlow, Apache Beam and Apache Flink gives the domain where it is appliable: highly parallel, throughput orientated workloads.

In a nutshell, the main concepts are:

business logic should not mix with execution logic.
code is encapsulated in blocks that can be linked to each other.
optimize for the throughput, not latency (can be tweaked).
optimize the performance of the whole workflow, not a single step.
data consistency over availability (this also can be changed).

Since this post isn’t about TPL Dataflow or the Dataflow Programming Model I will leave it here. If You want to know more see me the course I guess.

The lessons

I won’t advise on choosing the topic, but let’s assume You have one. What next?

0. Find the key

You have the title, but can’t get to actually making the thing? You might say that this is normal because starting is hard. Which part of starting is hard? Why is it hard?

For me, and reading the lessons learned of other people that made a video course, the biggest stopper for starting was knowing how do I structure the course? Phrasing it differently: how to tell the story?

There are a few approaches:

0.1 The builder

During the course, we build something (in 95% of the cases a website) using some framework and learn along the way. The most significant advantage of this theme is that it is very engaging. Developers want to build stuff. It also gives a purpose and some sense of accomplishment. On the downside, You will be building a yet another typical example (blog, forum, shop, etc.) because choosing an overcomplicated domain will waste to much time on explaining it. Additionally, the building order isn’t the learning order, so You will end up with digressions or using some feature that will be explained later on. Lastly, and for me, it was the biggest drawback, some features of the framework don’t make sense in what You are building. So you either leave them out (not a good idea) or cram them in somehow with little sense.

0.2 Feature centric

This approach concentrates on the features of the framework/topic at hand without actually building anything bigger than a few lines of code. The most significant advantage of this is that we can start from the fundamental parts of the framework and then progress to the more intricate features building on the knowledge gained earlier. This gives a natural, easy to understand progress path. The benefit of not building anything big is that the code samples demonstrating the feature can be more narrowed down to showcase the problem, making them easier to understand. On the downside, it is less rewarding for the viewer, we have less real-life usages, and no fully working solution at the end.

0.3 Problem solver

An interesting approach mitigating the problems of the feature centric approach is to demonstrate the features on tiny problems that don’t work at first. From my experience in mentoring people, doing workshops, and based on Pluralsight review feedback, it is more engaging than just showing things that work out of the box. Troubleshooting issues also make it easier to show the underlying principles of the framework.

0.4 Mix it! And my approach

Those themes are not exclusive. You can switch between them during the course and use multiple at the same time (especially the problem solver).
I went with the feature-centric approach with a lot of problem-solving. This allowed me to do a deep dive into how Dataflow works and show how the basic principles that we discovered in the early clips apply to the advanced features. Such combination allowed me to fulfill one of my goals - teach how it works, not only how to use it.

Choosing the feature-centric approach doesn’t mean that we have to say goodbye to the problem solver. One way we can use it is to extend the same code sample in progressive exercises. We start with something simple, then add problems and features showing how to build more substantial systems. That proved to be a significant time-saver for me since it eliminated the need to explain the starting point with each clip.

1. Plan your work

You have the vision soo start coding? No. Don’t just start doing the course. Planning is important. People will watch it to learn something. Don’t waste their time with chaotic mumbling, or get a per review saying rethink Your life. If You don’t care about viewers time (that is strange on its own. Please rethink it. ) You should care about Your time. The same chart we know from software development:

source: https://deepsource.io/blog/exponential-cost-of-fixing-bugs/

Applies to courses also. Redoing a recorded video because You forgot to say something essential that is needed one hour later is frustrating and very time-consuming.

2. Write a checklist

Write a list of things that You want to talk about. Don’t concentrate on order. There will be time for this later. It might be some high-level stuff or really tiny bits that are interesting. Don’t focus on this. Just dump everything into a text file or a piece of paper.
Once You have it go through it and check if nothing is missing. Give yourself time since doing it properly now will save a lot of time in the future.

3. Write an index first

You have the checklist. Now:

3.1 Group it.

Assign the small bits to some larger topics that are best suited for explaining them. If there is no such topic just create one. Now you should have topics of more or less the same size. Next step is to:

3.2 Add order

Now we need some order in our list. Try to put the main topics in such a way that the viewer will build upon the gained knowledge.
This might sound simple, but there is a problem. We can’t bore someone with the basics for the first 45 minutes because noone will have enough paitience to wait for the usefull stuff. The first question that viewers will ask is: is this useful for me? If no, they will stop watching.
The best approaches I found was to show how to use the framework while demonstraiting the underlying principles in the meantime.Part 3 of my course uses this approach. In each clip, I go through one building block of Dataflow and also show the basics as input and output queues, how to control concurrency, and how messages are delivered. Talking about them separately and without real-life examples would make the basics boring. To be frank, this is the part of the course I am most happy with (except for non-technical issues, but more about them later).

4. Write a script and code

I know that for some people having the list above is enough. They just start live coding and recording. For me, and probably for anyone doing it for the first time, it wasn’t the case. I tried it and the amount of time that went into editing the recorded video and audio to make it watchable was too big. It turns out editing text is easier than editing video. Again who knew? ;) For the n-th course, I will probably be able to do that. But for the first one, I decided to write an exact script that I would record. Another great idea was to write the code samples with the script.
For me, this approach was a blessing. Thanks to it I:

discovered topics that I needed to explain before for some clips.
noticed that in some cases, I need to rearrange the clips.
saw opportunities where I could reuse code from previous exercises in the next ones making the course more connected.
removed the writer’s block. For me, writing code is more natural than writing a script. Starting from the code and writing the script along the way was just more comfortable.
verified my own knowledge. To be honest, there were a few cases, especially with some inner working details, where I thought Dataflow would behave differently. Having code that proved and verified my assumptions eliminated errors.
was able to reuse the audio from different takes to make the final cut.

Writing the script was a significant time investment, but looking back, I would make it once again.

5. Just do it

Having said so much positive stuff about the benefits of writing a script don’t over do it. I made the mistake of spending to much time reading and fixing it.
Write the script, read and fix it once, and move on to recording. Record the audio and video and then fix the script. A few reasons why:

Trust me. You will find way more bugs and issues in the text, and code samples, by watching the recording than by reading the script and looking at the code samples.
Long, complex sentences are acceptable when written, but reading them is a nightmare. Listening also isn’t easy. I had to fix a lot of such sentences.
Doing the whole pipeline now will give You an overall view of what awaits. Every medium has its own does and don’ts. Knowing them soon is very beneficial.
You will gain some estimations on how long each clip is. They shouldn’t be too long or too short. About 7 minutes is perfect.
Start getting used to Your own voice. Most people don’t like how they sound on a recording. This is normal. People around You don’t mind, and You will spend a few hours listening to Yourself. Have it over with.
The first take will be bad. It will go into the trash can. Don’t spend to much time on it. Do it, learn from it and move on.

6. Do audio and video separately

Again, more seasoned trainers do it differently. For me doing the audio when codding was too much to do at once. It leads to many “yyyyyy” and code errors. Doing it separately was easier for me, and I didn’t have to worry about recording the keystrokes.

7. Don’t strive for perfection

You will see more errors, issues, and things to improve in the video than any other viewer. But keep in mind that people watch it for knowledge, not artistic perfection. If the knowledge is there and it is well served You will do just fine.

FAQ

What was the hardest?

Finding a concept of how I wanted the video to be structured and recording audio.

How much time did it take?

I didn’t count exactly but way too much because I had to learn so many things. At the end, doing one clip (audio + codding + slides + cutting + export) took between 1-4 hours. In the beginning, it was closer to 7 hours (mostly because multiple audio takes).

Would you do it again?

Yes. I learned a ton. Not only how to make video courses, but also about Dataflow and I have a ton of killer exercises for my workshops about Application Architecture

Would would you do differently?

I would redo the recording. But after recording it once, I would probably want to redo it once again…

Are you happy with how it turned out?

No. Anyone who knows me knows that I always see things to improve. The same with the course. I would rate myself with 3/5, but viewers on Pluralsight give it 4/5 soo I won’t argue.

What would I change?

How slow I talk. It should be faster. In person, I am a fast talker, and I was concerned that I would be in hard to understand for non-native speakers. I overdid it. But this is what the video playback speed regulation is for. I guess.

Links, further reading etc:

Troubleshooting compiled Azure Functions in F#

Mon, 08 Jan 2018 00:00:00 +0100

As I wrote in the previous post there are two ways to run precompiled .NET code in Azure Functions - .NET 4.6.x or .NET Core. Why did I decide to go with the old .NET runtime? For the current moment, F# on .NET Core does not support type providers (there is a workaround, but I didn’t want to go with it for the current moment). I went to work thinking that it will be a breeze. Just attach the repo and keep on coding. It turned out that I learned more Azure Functions troubleshooting and below is the shortened version.

The problem

The easiest way to run precompiled functions is to attach a repository with a F# project containing the function’s code to Azure Functions. This can be done
from Platform features->Deployment options->Setup

Doing this will trigger code compilation. And to my astoundment, a failure:

Now we have a problem, so let’s investigate.

Build failure investigations

The fist step into investigation should be checking the logs for details:

This shows a MSBuild error:

Command: "D:\home\site\deployments\tools\deploy.cmd"
Handling function App deployment with MSBuild.
Using Msbuild from 'D:\Program Files (x86)\MSBuild-15.3.409.57025\MSBuild\15.0\Bin'.
Nothing to do. None of the projects in this solution specify any packages for NuGet to restore.
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 1/1/2018 11:08:14 AM.
Project "D:\home\site\repository\fsharptest.fsproj" on node 1 (default targets).
D:\Program Files (x86)\MSBuild-15.3.409.57025\MSBuild\Microsoft\VisualStudio\v15.0\FSharp\Microsoft.FSharp.NetSdk.props(3,3): error MSB4019: The imported project "D:\Program Files (x86)\Microsoft SDKs\F#\4.1\Framework\v4.0\Microsoft.FSharp.NetSdk.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk. [D:\home\site\repository\fsharptest.fsproj]
Done Building Project "D:\home\site\repository\fsharptest.fsproj" (default targets) -- FAILED.

Build FAILED.

"D:\home\site\repository\fsharptest.fsproj" (default target) (1) ->
  D:\Program Files (x86)\MSBuild-15.3.409.57025\MSBuild\Microsoft\VisualStudio\v15.0\FSharp\Microsoft.FSharp.NetSdk.props(3,3): error MSB4019: The imported project "D:\Program Files (x86)\Microsoft SDKs\F#\4.1\Framework\v4.0\Microsoft.FSharp.NetSdk.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk. [D:\home\site\repository\fsharptest.fsproj]

    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:00.90
Failed exitCode=1, command="D:\Program Files (x86)\MSBuild-15.3.409.57025\MSBuild\15.0\Bin\MSBuild.exe" "D:\home\site\repository\fsharptest.fsproj" /p:DeployOnBuild=true /p:configuration=Release /p:publishurl="D:\local\Temp\8d55107eb70a8ff"
An error has occurred during web site deployment.
\r\nD:\Program Files (x86)\SiteExtensions\Kudu\69.61204.3166\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd"

The key line is this one:

D:\Program Files (x86)\MSBuild-15.3.409.57025\MSBuild\Microsoft\VisualStudio\v15.0\FSharp\Microsoft.FSharp.NetSdk.props(3,3): 
error MSB4019: 
The imported project "D:\Program Files (x86)\Microsoft SDKs\F#\4.1\Framework\v4.0\Microsoft.FSharp.NetSdk.props" was not found. 
Confirm that the path in the <Import> declaration is correct, and that the file exists on disk. [D:\home\site\repository\fsharptest.fsproj]

It is a standard MSBuild output indicating that a .props (the same as .target. MSBuild doesn’t care about the extension) file is missing. Lets check is the file really missing.

Azure Functions Advanced tools (Kudu)

Azure functions offer a way to look into the underlying machine that is serving our functions - Kudu. It can be accessed in two ways:

using a link https://FUNCTION_APP_NAME.scm.azurewebsites.net/
by clicking on the portal Platform features->DEVELOPMENT TOOLS->Advanced tools (Kudu)

Kudu offers a lot of advanced functionalities (more than I have time to write in this blog post) but today we are interested in accessing the file system. This can be done by clicking the Debug console->CMD/Powershell:

This bings us to the file system, and cmd, explorer:

According to the error message, we are interested in two paths:

MSBuild props: D:\Program Files (x86)\MSBuild-15.3.409.57025\MSBuild\Microsoft\VisualStudio\v15.0\FSharp\Microsoft.FSharp.NetSdk.props
FSharp props: D:\Program Files (x86)\Microsoft SDKs\F#\4.1\Framework\v4.0\Microsoft.FSharp.NetSdk.props

To explore the file system we can use normal cmd commands, or just click using the graphic representation (don’t use the browser back button!).

Kudu file explorer has some glitches:

for reasons unknown to me the only way to go upper than default d:\home is to use the cmd cd .. command

entering the exact path to the folder using cd like: cd D:\Program Files (x86)\MSBuild-15.3.409.57025\MSBuild\Microsoft\VisualStudio\v15.0\FSharp does work in cmd, but the graphic file representation doesn’t refresh

hitting Tab auto-completes the file/folder name (like in normal cmd), but only for the current level files/folders.

After navigating to the first file, its content can be viewed using the pen icon on the left. And it looks like this:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <Import Project="$(MSBuildProgramFiles32)\Microsoft SDKs\F#\4.1\Framework\v4.0\Microsoft.FSharp.NetSdk.targets" />

</Project>

So indeed D:\Program Files (x86)\Microsoft SDKs\F#\4.1\Framework\v4.0\Microsoft.FSharp.NetSdk.props is the file we are looking for.

Locating the problem

Trying to locate the file will end on the D:\Program Files (x86)\Microsoft SDKs\F#\ part of the file since it looks like this:

See what is wrong?

The first prop file (Microsoft.FSharp.NetSdk.props) is pointing to D:\Program Files (x86)\Microsoft SDKs\F#\4.1, but in the F# there is only a 4.0 folder, so MSBuild reports a missing file.

How to solve it?

The fastest, but not the cleanest, solution would be to upload the missing files. While Kudu supports file upload, it is only allowed in specific folders, and this is not one of them :(. The second solution is to use the fact that Azure Functions are hosted like an IIS website and use WebDeploy for deploying a prepared package. How to use VSTS for package creation and build/deployment automation - in the next post.

The missing Azure Functions documentation

Mon, 01 Jan 2018 00:00:00 +0100

Trying to understand how to run code in Azure Functions is not an easy task since this product has evolved on its own and thanks to the rise of .NET Core. This post will give You a history background necessary to understand the documentation and, most of all, all the blog posts talking about Azure Functions.

v1.0 - script files

The initial way to run code in Azure Function was to write a script file in Azure Functions portal editor. Yes, You read correctly - in the portal editor. Later on, deployment support was added, so we could deploy the script files (fsx and csx). It works good, but has some drawbacks:

the compilation is done when the function is called for the first time (or called after an extended inactivity when Azure decided to kill the process)
IntelliSense support is quite clumsy
there is no compilation during development. When working on the code locally compiling the project did nothing since script files are not taken into account when compiling. This means that we are losing a lot of the benefits of a compiled language.

v1.1 - compiled functions

Microsoft addressed most problems with script files with the introduction of compiled functions. This was not a change to how they worked since Azure Functions are hosted in WebJobs, so as IIS application. Compilation can be done two ways:

by deploying project code. When we deploy a project (csproj/fsproj/sln) Azure Functions machine will trigger compilation on the hosted machine, deploy the result to wwwroot as Default Website and run discovery looking for functions declaration (this is done by a config file or special attributes).
by deploying compiled code. This does not mean that we have to store binaries in the repository (although it would work). This means that we can treat Azure Functions as an IIS Website and use the good (?) old Web Deploy that we have been using since ASP.NET (btw. VSTS has an excellent configuration for Azure Functions deployment)

Why should we bother with option two since Azure can handle it? Option one (deploying just code) has some drawbacks:

compilation is limited to .NET 4.6.1 (this is the version installed on the Azure Functions machines)
use of custom steps like Paket or Fake requires manual upload to the hosting machine to tools folder for them to work.
if we want to change the build process we have to change the .deployment and deploy.cmd files and checking them into our repository. So from now on, we are responsible for maintaining and troubleshooting them, not Microsoft. Also, there is no point in looking for documentation for them (but they are quite easy batch scripts).

v2.0 - Azure Functions v2.0

Azure Functions 2.0 (in beta at the time of writing) is Azure Functions running on Linux (the OS can be selected when creating the Function App):

There are some changes/limitations compared to version 1.0:

the OS is Linux
runtime is .NET Core
only C#, F#, JavaScript and Java is supported (Azure functions 1.0 supported Batch, C#, F#, JavaScript, PowerShell, Bash, Python, and TypeScript)
~~there is no consumption plan billing (as of the time of writing, but I’m sure that one will be offered they go to stable)~~ (apparently this is a false assumption - look at the comments)

Of course, there are some benefits:

.NET Core is faster and consumes less memory. It is a significant benefit since in Azure Functions we are paying for execution time and used memory so expect a lower bill.
I suspect Linux offering will be a bit cheaper (VM with Linux tend to be less expensive)

Summary

Azure Functions is evolving fast so make sure to look at the date when project/documentation/blog post was recently updated.

What is the problem with key-value databases and how wide column stores solve it.

Mon, 10 Apr 2017 00:00:00 +0200

Previously I’ve wrote about key-value databases. They are awesome - ultra-fast, simple, can scale almost linear with the number of nodes. So why bother with complicating them?

Well, they have some issues.

Problems with key-value databases

The central concept of a key-value database is that the database doesn’t care what the is value. It may have some assumptions, like Redis, but the structure of the data is not of its interest. This leads to some limitations that can be problematic in some scenarios.

1. Can’t filter on value fields

Quite evident since from the database point of view value is a blob.

2. The whole value is returned

This may not seem as a problem, but remember, key-value databases are chosen for speed. When looking at a flow of retrieving data from a database:

Most of the steps performance is dependent on the size of the transmitted data, not the data used (this is why SELECT * is a sign of someone not giving a f**k about performance).

3. The value can be updated only as a whole

It is a problem because we have to:

get the complete data to the client (see point above)
operate on the entire data
send the whole data back to the database

Sounds not that bad, even good. We want to have the whole object when updating it, right?

Think about those cases:

update users last login date
append the element to the list (like online checkout basket)
change the prices on certain products because of a promotion

So how to solve those problems and not loose all that speed?

Wide column databases

The idea behind it is simple:

Let's structure data (that is the value part) again into key-value pairs.

This is hat we have in key-value databases:

This is how wide column databases represent data:

Having columns allows defining a subset of data we want to return to the client or subset of data that should be updated.

How is it different from a regular table in a relational database?

In most wide column databases, columns are defined on the single item level meaning there is no database-wide schema, which leads us to some interesting features of wide column databases:

Benefits from wide column databases

While wide column databases keep most of the perks of key-value databases mentioned previously they have some additional ones (They don’t have to exist in every implementation of a wide-column database, but most of them have it):

partial operations (add column value, update column value).
data compression. When dealing with sparse data we don’t have to store empty/null values. This way we can save space normally reserved because schema defined them.
WHERE clause. This feature is not that common in this type of databases, but filtering on data starts to appear.

In the next post a sample domain modeling using a wide-column database.

Want unlimited scale and performance? This is where to start

Tue, 28 Mar 2017 00:00:00 +0200

The previous post laid out the most minimum requirements for something to be called a database. While they may be too bare bones for many, there are a lot of databases that don’t fulfill even half of them, and this isn’t stopping from using them on a daily basis.

The last time I’ve looked at files, this time something a bit more complex - key-value databases.

The idea

The idea behind key-value databases is very simple:

Store and retrieve objects based on a key.

So we are saying goodbye to:

tables, columns or ant data typing - everything is a blob of some kind
relations
complex operations

We gave almost everything, what do we gain in exchange?

Why use a key-value database?

Speed

If You are thinking that getting the object from a key-value database is more or less the same as an index lookup in a relational database, keep on reading because You are very wrong, an order of big O wrong.

Indexes in relational databases are implemented using a B-Tree structure that looks like this:

This is one of the most fundamental data structure in modern computer science. If You don’t know it please at least read this Wikipedia page. It will be a time well spend.

While this is a remarkable structure it has to obey the rules of mathematics when it comes to the cost of search:

O(log(n/m))

where:

n - the total number of elements
m - the number of items in a single node.

To contrast it, most key-value databases store the data in memory, so they can use a hashing function to determine the position of the element in the array. It’s cost is:

O(1)

You can’t get any better than this.

Simplicity

If something is simple it breaks less often, is easier to maintain and to reason about. Without this simplicity, we wouldn’t have the next point.

Horizontal scaling

When we introduce the following rules, that most key-value databases implement:

the key is the only way to identify an item
the hash function transforms the key into an integer in a deterministic way
we aren’t doing any, or are limiting the scope, of aggregate operations
update changes the whole value (since the database doesn’t know about the schema of the item). There are some databases that bypassed this rule. For more keep on reading.

Having those assumptions allow scaling such a solution horizontally easily. In the most simplistic way:

hash(key) % NUMBER_OF_SERVERS

While an oversimplification, but some key-value databases use it.

What not to expect from key-value databases

If You have been living in RDMS land here are some things not to look for in key-value databases:

Transactions

It is very rare to see them in key-value databases. Instead, we have atomicity. What is the difference?

Atomicity - means that the operation will execute or not. In short, in the case of failure, we won’t end up with corrupted or partially changed data.
Transactionality - a series of multiple operations that will execute atomicity as one.

Is the lack of transactions a problem? No. Let’s examine the cases when we would use them:

SELECT - no need for transactions. We ask for an element and get it back.
DELETE- single DELETE is atomic, so no problem here. The need could arrive when doing multiple deletes. This case boils down to the fact how did we get the keys of the elements to delete?
- If we stored those keys as a value of another key - go to the complex statements flows.
- If we want to remove values, which keys fulfill some pattern then just retry the delete.
UPDATE - the same as with DELETE.
complex statements flows - the need for it happens when the value of one key contains keys that should be, for example, deleted. This means that there is a relation, and this means that we are trying to simulate an RDMS on top of a key-value database, and it’s not a good idea.

Examples

While those are very simple databases, they differ quite significantly. To show that let’s examine 3 most popular key-value databases according to db-engines.

Memcached

Designed for: cache

I’m starting from the third place and also from the oldest database in the ranking (initial release in 2003). Memcached is not exactly a database since its main feature involves auto deleting data. Think of it as a massive, fixed size, in memory cache. When there is no more memory available Memcached will start removing elements using the LRU algorithm. It starts deleting the least used values until it frees enough memory to store the new value. Memcached doesn’t have any option of persistence to disc, but let’s face it: there is no sense for persistence if the database can delete the data at any time.

Since Memcached is a cache store it has some limits on key and value sizes:

key: up to 250 bytes
value: up to 1MB

Let’s look at the must and should haves:

Must-haves:

[-] ability to reliably persist data - Memcached will auto delete the oldest data, and we are talking about a database that stores everything in memory, so let’s not call it reliable persistence.
[-] ability to reliably retrieve data - If the data wasn’t deleted it will be returned.
ability to delete data - using delete command

Should-haves:

[-] ability to query data - We can’t do any matching on keys.
ability to update data - We can update the whole value. No partial updates.
[-] has transactions - No the case in this databases.

Riak

Designed for: key-value database synchronized across multiple data centers

Riak, an implementation of Amazon Dynamo paper, is an entirely different beast than Memcached and was build to solve other problems. It is also the closes we get to a database in the classical definition. It’s a distributed, cross data center, persistent key-value database aiming for availability, even at the cost of consistency. The above statement contains a lot of information, so let’s decompose.

Data types

Key-value databases treat stored values as blobs, but some of them implement types that have a particular purpose and have separate API. In Riak case its:

Flags - true/false values. Can only be used inside a map type.
Registers - named binaries. Also can only be used inside a map
Counters - as the number suggests incremented integers. Can be used in a map and as a value on its own.
Sets - collection of binary values. Similar to the Counter type can be used on its own and in a map.
Maps - collection of values. Differently than Sets they can contain other data types, even other maps.
HyperLogLog - a probabilistic structure for checking cardinality of a set.

Clustering

Riak supports clustering with tunable consistency. How is it done? Since the cluster is a ring architecture, like this:

Tuning the level of consistency is done by defining how many nodes have to accept the operation before it’s confirmed (default is 3).

Riak goal from CAP is A, and it tries to achieve it by constructing a cluster where:

nodes are organized in a ring
there are no master nodes
any node can accept writes for any key (no master node for a given hash value).
allows concurrent writes on multiple machines for a single key.

All of the above lead to the need to handle conflict resolution resolution. Riak recommends using custom types whenever possible, but it will also work with blob values. Behavior when dealing with conflicts can be configured, and ranges from timestamp, last-write-wins to letting the client decide. One thing to note is that it’s not hard to find people complaining about Riak resurrecting deleted values even days after deletion.

Must-haves:

ability to reliably persist data - This is one of the main points of Riak
ability to reliably retrieve data - Since Riak consistency is tunable this point is also partly passed.
ability to delete data - works, although resurecting deleted items is a reported problem.

Should-haves:

[-] ability to query data - We can’t do any matching on keys, although it supports key searches using Solr.
ability to update data - When using custom types they are possible. With blobs, no.
[-] has transactions - No transaction support.

Redis

Designed for: speed

The number one should not be a shock to anyone. Redis is, as the full name suggests, a REmote DIctionary Server. Over the years it has grown a few functionalities, but it still is a memory key-value dictionary.

Architecture

Redis show that simplicity is speed. Important thing to note is that it is mainly an in-memory store with optional persistence:

point in time snapshot
Append Only File with async writes
both of the above

Building upon the idea that everything is a string Redis exposes an interface for manipulating the values right on the database, without the need to send them to the client.
One last thing to note is that Redis has the option to write Lua scripts.

Data structures

Redis, similarly to Riak implements custom data structure, but nonstructured data is stored as a string, not as binary. Custom data structures are:

Binary-safe string
Lists - collections of strings sorted according to the order of insertion (a linked list)
Sets - collections of unique, unsorted strings.
Sorted sets - similar to Sets but every string has a score (a floating number). Elements are sorted by score, so unlike Sets it is possible to retrieve ranges like top or bottom 10.
Hashes - maps composed of fields associated with values. Both the field and the value are strings.
Bit arrays - allows to set, clear, count and find the first set or unset bit.
HyperLogLogs - the same as in Riak

Clustering

Redis is taking a different approach to clustering than the previous two:

all nodes are connected
values are automatically propagated to multiple servers
has the concept of master and slave datasets
it does not guarantee strong constistency, but it can be achieved using WAIT
it if advised for the client to keep an up to date routing table of the cluster
it can detect not responding and new nodes.
nodes do not proxy requests. This means that if we request a key not present on the current node the server will return MOVED command to the client.
multi-node commands and Lua scripts are limited to near keys (no cross server operations)
replication is asynchronous
only one node accepts a write for a given key

Pub-sub

One feature that is unique to Redis, and to key-value databases in general (since it is not a database feature) is pub sub capability of Redis. With features as full cluster support and pattern matching for subscriptions, it is a first class citizen in Redis features and works quite well.

Must-haves:

[-] ability to reliably persist data - This is not the main goal of Redis. Provided persistence options don’t give 100% certainty of data recovery in case of for example power failure.
ability to reliably retrieve data - Redis has the notion of a master node for a given key and if we don’t mess with the routing table data will be returned reliably.
ability to delete data - just works.

Should-haves:

ability to query data - Redis exposes API to search for keys matching a pattern. What to note is it will return the keys, not the values.
ability to update data - Works with custom types, and with blobs. Since everything is a string Redis provides string operations that are executed on the server and thus don’t require a round trip to the client.
[o] has transactions - As Funbit and Alex pointed out, there are transactions in Redis. In Lua scripts and using the MULTI statement.

Comparison

Option	Memcached	Riak	Redis
Key limits	250 bytes	No limit	No limit
Value limits	1 MB	No limit	512 MB
Persistent	No	Yes	Optional
Transactions	No	No	Yes
Connection protocol	TCP/IP	HTTP	TCP/IP
Key scans	No	Done with Solr	Yes
Scripting	No	No	Yes(Lua)
Data schema	No	Custom types + blob	Custom types + blob
Data stored	binary	binary	string
Licence	BSD 3-clause	Apache 2	BSD 3-clause

Cluster
Cluster info	Client knows all servers in cluster	Client knows some nodes	Client knows the routing table
Cluster architecture	share nothing	ring	all connected
Consistency	Doesn’t apply	Tunable from eventual to strong	No guarantee
Replication	No	Configurable	Async
Multi data center sync	No	Yes	Yes(only Enterprise version)

Run on	Windows/Linux/Unix	Linux	Windows/Linux
Main features	auto deletion of data	multi data center replication	speed/pub-sub
Build for	cache server	Key-value store across multiple data centers	speed

Big thanks to Łukasz Pyrzyk for proofreading and pointing out some mistackes. Go check out his blog

Next, wide column databases.

What is the simplest database?

Mon, 20 Mar 2017 00:00:00 +0100

The world of databases is a fascinating topic. It is very diverse. Many of them are extremely complex systems, but there are also very simple ones. There are the general purpose ones, and ones that do only one thing good, but they do it excelent. Despite all of this we tend to pick them just like we order food in a restaurant:

I’ll take the same as last time. It wasn’t ideal for what I wanted, but I could pick worse.
Hmm… Everyone is taking this one, so I’ll take it also.

Well, maybe it is time to dig deeper into it?

Preface

I’m writing this post in addition to my You are using the wrong database! talk which intention is to show:

how important is picking the right database and how strongly it affects the whole system
How diverse is the world of databases.
Why the most common criterium for choosing: “I’ll use what I know” sucks, putting it lightly.

Definition

Here is where a copy and paste from Wikipedia typically happens, but let’s create a definition based a more look and feel approach and let’s define simple requirements. I divided them into two parts: must-haves and should-haves. Here we go:

Must haves

Those points are what, for me, defines a database. The core functionalities that make a product a database:

ability to reliably persist data - kind of a no-brainer, but this paragraph is just for that.
ability to reliably retrieve data - there is little use for a database that can only accept data but can’t retrieve it.
ability to delete data - the first point defines INSERT, the second a SELECT operation, so all that is missing is a DELETE

Only three points? Yes, but remember, we are talking about the must haves.

Should-haves

So this is where I put all the theoretical features mumbling? Not exactly, here are my should-haves:

ability to query data - how this one differs from the must-haves ability to retrieve data? There we identified the data using a single key. Here we require the capacity to return a list of data that meets certain parameters. Think of it as the conditions on columns in the WHERE clause from SQL.
ability to update data - this, in theory, could be achieved by deleting and inserting a new record, but this implementation fails quite fast when taking into account a bit more advanced features (deleting and inserting would change the auto-generated ID, or could fail if the row was referenced).
has transactions - I am not talking here about the more advanced things like isolation levels or distributed transactions, but the simple fact that if something fails while executing an update the database won’t get corrupted.

The simplest database

Go on have a guess. What is the simplest database that fulfills all the must-haves and almost all should-haves?

Are You sure You know? Think again, but if You are sure

CLICK HERE if You know

File system is the simplest database

Did You get it right? Do You believe it? Let’s got through the lists. Must haves:

ability to reliable persist data - calling fsync will make sure that the file was written to the hard drive and the hard drive is what we think when talking about persistent storage.
ability to reliably retrieve data - You can read the contents of a file.
ability to delete data - once again, a simple delete.

Should haves:

ability to update data - We can update a file right? We can even do a partial update thanks to the ability to start writing from a certain index.
has transactions - A pass? Files systems have transactions? Yes. Windows even has support for distributed transactions. This means that you can combine saving a file with an update in a database on an external server. Anyone who worked with distributed transaction knows that they should be avoided. That said, in some cases, it is not possible.
ability to query data - We can query files by their name, content and directory structure. This is more that some real databases offer.

This post is a preface for a more advanced series covering database mechanics and the variety of database types. If You are interested, subscribe to RSS, or follow on Twitter

Developers perspective on PowerShell Desired State Configuration

Mon, 06 Mar 2017 00:00:00 +0100

Configuration as code movement isn’t anything new and is here to stay. I haven’t had time to actually do a from zero to the desired state configuration script until a few weeks ago. Below are my thoughts after three weeks with PowerShell Desired State Configuration.

What I wanted to do

First, let me describe the goal I’ve set before myself.
I wanted to write a script that would go from a clean Windows machine to a fully configured TeamCity server with SQL Server storage and two build agents running. This may not be the ideal use case for PowerShell DSC, but:

I don’t know a better way to accomplish this with code as configuration. I could use a VM template, but this misses the point of being able to quickly respond to new versions of any of the dependencies.
I like PowerShell, and it is a very powerful tool, and DSC seemed OK for this task

My thoughts

How did it end? It was an up and down journey, sometimes being frustrated, other times astounded by how powerful it is. Here are the main thoughts that I wish someone would tlll me earlier:

You are not in developer land anymore. Different rules apply here!

We, as developers are used to the fact that we are operating on functions doing simple things like write to a file, save to a database, call over HTTP. When doing any system configuration we are operating on a bit higher level: install SQL Server, create w Windows user, install TeamCity build agent…
What it implies? A LOT more things can go wrong, and not just because your code is wrong, but some external assumption were not met.
Here learn to look for causes not only in Your code but in the whole system.

Leaky abstraction and an unmet promise

Take this code:

    WindowsFeature dotNetFramework35{
        Name="NET-Framework-Core"
        Ensure ="Present"
    }

It will check if .NET Framework 3.5 is installed on a machine, if not it will install it. Everything works great. Run it again and it does nothing because the framework is already present.

Now take this code:

    xSQLServerSetup($Node.NodeName){
        DependsOn = "[WindowsFeature]dotNetFramework35","[xMountImage]mountedSql","[xWaitForVolume]waitVolume"
        SetupCredential = $account
        SourcePath = "E:\"
        SourceFolder = ""
        InstanceName = "SQLEXPRESS"
        Features = "SQLENGINE,SSMS,ADV_SSMS"
        SecurityMode = "SQL"
        SQLUserDBDir = "D:\SQL\Data"
    }

The code above will install Microsoft SQL Server with SQL Server Management Studio and SQL Server Profiler. Run it again and it does nothing because SQL Server is present. So what is the fuzz about?

PowerShell DSC promises to be something more than an install script. This suppose to be a tool that will take the current system, sprinkle some magic, wave a wand and transform into what is in the script. And this is where the abstraction starts to get leaky.
Changing the SQLUserDBDir or SecurityMode and running the script again will succede, but won’t change anything. Is it a game-over problem? For me, no, but it was a “there is no Santa” moment for me.

To be precise:

this was the situation in the time when I was using it

it is an x package and what it means it is still experimental.

xSQLServerSetup has 40 parameters that can be set. It is a complex piece of code, so I fully understand that it will take some time to implement all the edge cases

the package is very powerful and I am very happy that it exists. Big thanks to the people behind it.

Use a Virtual Machine

Writing a script on a bare metal machine is the fastest way to throw the computer out of the window. Why?
Let’s take the example above. You run the script, change it, run again - it passes, but without applying the changes. So You uninstall SQL Server and run the script again. This is when the output transforms from white to a wall of red. Why? Because SQL Server doesn’t do a clean uninstall (database catalogs aren’t being removed. This is a good thing). So when trying to run an unsupervised install (no GUI) it fails because the installer asks if it should use this catalog or a new from scratch.

My method of working was to take snapshots of the VM from Hyper-V manager after finishing each step. It takes few seconds to take and restore the snapshot. Highly recommend doing so.

The community is awesome

Before writing anything first google it! The number of packages is really impressive. If I had to point out one thing that really astounded me it would be the community behind DSC. I tried to return the favor and added some changes to cTeamCityAgent allowing to install multiple TeamCity agents.

Also, don’t limit Yourself to packages from the official feed. There are many packages on Github that aren’t available on the feed.

Error messages

A To contrast the best thing with the worst - error messages. They are terrible and in most cases don’t give even a hint what went wrong (especially in the more complex modules). All they say is that the step failed.

Use Chocolatey

If You want to install/need a dependency from some application use Chocolatey module to install it:

    cChocoPackageInstaller installJava{
        Name="jre8"
    }

Yup, that is all that is needed for installing Java Runtime. Simple as that!

Script block is Your last resort

When there is no module to be found You can always fall back to Script block that allows executing any PowerShell. This one unzips TeamCity using 7zip:

Script unpackTeamCityGzip{
    GetScript = { @(Result = "Nie ma tu nic ciekawego") }
    SetScript = { & 'C:\Program Files\7-Zip\7z.exe' e  C:\temp\teamCity.tar.gz -oc:\temp }
    TestScript = { (Test-Path "c:\temp\teamCity.tar") }
    DependsOn = "[cRemoteFile]teamCityTarGzDownload"
}

Intelisense

Packages are self-describing, so IntelliSense works even with new packages. In PowerShell ISE and VS Code.

No restore packages

DSC consists of multiple packages. If You want to use the package it should be included in the script:

Import-DscResource -ModuleName xSQLServer

What annoys me is the fact that there is no way to install referenced packages. To put it simply there is no Nuget - restore. You have to have a separate file that will take care of the installation. Something like this:

Install-PackageProvider -Name "Nuget"
Install-Module xSQLServer 

This is annoying and error prone.

Final thoughts

First and most important: Did I write the script and did it work?
The answer is Yes.
Writing it for the second time would I use PowerShell DSC?
Again Yes.

It has some rough edges, leaky abstractions and there are things to fix, or make better, but it is a good product.

Could not load file or assembly or one of its dependencies.

Mon, 27 Feb 2017 00:00:00 +0100

In most cases .NET manages to solve the DLL hell problem pretty well, but sometimes it all falls apart, and when it does in best case scenario we see something like this:

Could not load file or assembly 'XXXX, Version=X.Y.Z.W, Culture=neutral, PublicKeyToken=eb42632606e9261f' or one of its dependencies. 
The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

The much worst case is this:

The method 'XXXX' was not found on the interface/type 'YYYY, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null'.

This post is an analysis of why this happens and how to diagnose it:

As the old saying goes:

There is a log for that.

In this case, it only needs to be turned on:

Enable assembly binding logging (Fusion log)

Assembly binding is turned on using those registry settings:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion\EnableLog - Type of DWORD
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion\ForceLog - Type DWORD
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion\LogFailures - Type DWORD
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion\LogResourceBinds - Type DWORD
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion\LogPath - Type String

To enable logging the first four should be set to 1 and the last to a existing directory path (it should also end with an \).

It can be done manually with the use of regedit.exe, but since it can be automated, here are the scripts:

Disable Fusion Log Enable Fusion Log

CHECK THE FILES CONTENT BEFORE RUNNING! (they are just text files) DON’T RUN THIS OR ANY OTHER SCRIPT WITHOUT CHECKING. MODIFYING REGISTRY REQUIRES ADMINISTRATOR PERMISSIONS, SO A LOT OF HARM CAN BE DONE WHEN RUNNING A SCRIPT FORM A NOT TRUSTED SOURCE.

After enabling assembly binding logging, there are two ways to continue:

Analyse assembly binding logs with Fuslogvw

Windows has a build-in tool called Fuslogvw.exe. It should be located in several places, but the pattern is: C:\Program Files (x86)\Microsoft SDKs\Windows\<<SDK_VERSION>>\bin\NETFX <<RUNTIME_VERSION>> Tools\. Any version will do since the tool is available from .NET version 1.1, and the version numbers between 4.6.2 and 4.0 differ only in minor version. The tool is simple, so I won’t describe how it works. I personally prefer the other way:

Reading assembly binding log files (Fusion Log)

This is my preferred way to diagnose. After dealing with many Microsoft loggs, I think the assembly binding ones are the best ones. The files are named using a pattern like this:

[assembly name], Version=[assembly version], Culture=[culture], PublicKeyToken=[public token]

so for NLog we will have:

NLog, Version=4.0.0.0, Culture=neutral, PublicKeyToken=5120e14c03d0593c

The best way to understand how .NET runtime loads assemblies let’s look at the logs.

The operation completed successfully.

First an easy one - the one that succeeded.

  
  *** Assembly Binder Log Entry  (25.02.2017 @ 13:44:54) ***

  

  The operation was successful.

  Bind result: hr = 0x0. The operation completed successfully.</a>

  Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll

  Running under executable  d:\src\FusionLogTest\FusionLogRunner\bin\Debug\FusionLogRunner.exe

  

  --- A detailed error log follows.

  === Pre-bind state information ===

  LOG: DisplayName = NLog, Version=3.2.1.0, Culture=neutral, PublicKeyToken=5120e14c03d0593c

  (Fully-specified)

  LOG: Appbase = file:///d:/src/FusionLog/FusionLogRunner/bin/Debug/

  LOG: Initial PrivatePath = NULL

  LOG: Dynamic Base = NULL

  LOG: Cache Base = NULL

  LOG: AppName = FusionLogRunner.exe

  Calling assembly : FusionLogLib, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.

  ===
  LOG: This bind starts in default load context.

  LOG: Using application configuration file: d:\src\private\FusionLog\FusionLogRunner\bin\Debug\FusionLogRunner.exe.Config

  LOG: Using host configuration file: 

  LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.

  LOG: Redirect found in application configuration file: 3.2.1.0 redirected to 4.0.0.0.

  LOG: Post-policy reference: NLog, Version=4.0.0.0, Culture=neutral, PublicKeyToken=5120e14c03d0593c

  LOG: Binding succeeds. Returns assembly from d:\src\private\FusionLog\FusionLogRunner\bin\Debug\NLog.dll.

  LOG: Assembly is loaded in default load context.

The operation failed.

Now for something harder, but more interesting. A failed log file. Also for NLog.


*** Assembly Binder Log Entry  (25.02.2017 @ 14:46:28) ***




The operation failed.

Bind result: hr = 0x80131040. No description available.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll

Running under executable  d:\src\FusionLogTest\FusionLogRunner\bin\Debug\FusionLogRunner.exe



--- A detailed error log follows. 



=== Pre-bind state information ===

LOG: DisplayName = NLog, Version=4.0.0.0, Culture=neutral, PublicKeyToken=5120e14c03d0593c

(Fully-specified)

LOG: Appbase = file:///d:/src/FusionLog/FusionLogRunner/bin/Debug/

LOG: Initial PrivatePath = NULL

LOG: Dynamic Base = NULL

LOG: Cache Base = NULL

LOG: AppName = FusionLogRunner.exe

Calling assembly : FusionLogLib, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.

===

LOG: This bind starts in default load context.

LOG: Using application configuration file: d:\src\private\FusionLog\FusionLogRunner\bin\Debug\FusionLogRunner.exe.Config

LOG: Using host configuration file: 

LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.

LOG: Redirect found in application configuration file: 3.2.1.0 redirected to 4.0.0.0.

LOG: Post-policy reference: NLog, Version=4.0.0.0, Culture=neutral, PublicKeyToken=5120e14c03d0593c

LOG: GAC Lookup was unsuccessful.

LOG: Attempting download of new URL file:///d:/src/FusionLog/FusionLogRunner/bin/Debug/NLog.DLL.

LOG: Assembly download was successful. Attempting setup of file: d:\src\FusionLog\FusionLogRunner\bin\Debug\NLog.dll

LOG: Entering run-from-source setup phase.

LOG: Assembly Name is: NLog, Version=3.2.1.0, Culture=neutral, PublicKeyToken=5120e14c03d0593c

WRN: Comparing the assembly name resulted in the mismatch: Major Version

ERR: The assembly reference did not match the assembly definition found.

ERR: Run-from-source setup phase failed with hr = 0x80131040.

ERR: Failed to complete setup of assembly (hr = 0x80131040). Probing terminated.

How to fix “Could not load file or assembly”?

There can be many cases why the assembly fails to load, but 99% of them can be simplified to two solutions presented bellow:

Update the reference

If one of the projects in Your solution has a different version of the assembly this assembly may be copied with it down to the folder of the executable assembly often overwritten the version You would assume it should have. This is the reason why it is a good practice to have all references pointing to a single version of a given assembly. The easiest way to make sure that this is the case is by using Package Manager from Visual Studio (right click on the solution):

And then:

Then update, build, test, commit and push.
Things may not always be that as easy, because we can’t control references that external libraries, like NuGet packages, have. This is when the second solution comes in:

Add assembly binding redirect

Look at the succeded log file, and the entry about a found redirect. The runtime can be instructed to use another version of the assembly than the one that was called. This is called binding redirect and can be defined in the config file of the main assembly (adding this in the config file of a library won’t matter for the runtime). In case of a Windows application, it’s the app.config file, and in a case of an ASP app the config file structure (web.configs and down) is being searched.
An example:

<configuration>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="NLog" publicKeyToken="5120e14c03d0593c" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.0.0.0" newVersion="4.0.0.0" />
      </dependentAssembly>
    </assemblyBinding>
  </runtime>
</configuration>

This entry is saying that all assemblies matching all the rules:

named NLog
having a public key token with value 5120e14c03d0593c
culture equal to neutral
having version between 0.0.0.0 and 4.0.0.0

Should be redirected to a assemble with version 4.0.0.0.

Troubleshooting FusionLog

I don’t see the logs from my application

The application has to be restarted after enabling the log. When talking about IIS application the whole IIS process has to be restarted.

I don’t see the folder

You have to create the folder manually

The application runs slower

The logging adds some overhead, but not enough for it to be seen with a bare eye. Maybe You are doing a lot of dynamic assembly loading?

Something is eating up my disc space

Disable the log. When enabled it creates a lot of small files. The result is that they are taking up more disc space than they actual size.

The logs are still being created despite disabling the logging

Restart the application process.

Why is it called fusion log?

I suspect because fusion is the process of combining two atoms into one.

Choosing centralized logging and monitoring system

Sun, 12 Feb 2017 00:00:00 +0100

While I’m working on the next angle on how to speed up calculating similarities I started investigating how to get better telemetry from cookit. Getting telemetry is easy - making sense of it is the hard part. This also brought another pain point of current setup - logging and monitoring. Since cookit is my pet, nonprofit project it was time to do something.

TL;TR;

There is a comparison table at the end, and what I’ve choose.

Table of content:

The current state of logging and monitoring
Requirements
The showdown
Comparison table
What I’ve choose.

The current state of logging and monitoring

The current setup is based on:

NLog - used for almost all logging in the website and in the Windows Service (You can read more about cookit architecture here). Most errors are logged into files, the critical ones are emailed to me.
Hangfire dashboard - for checking job status and if it ended with a failure. If it crashed, only the main exception is displayed.
Remote Desktop - for troubleshooting, performance counters and all monitoring stuff (trust me I know how badly RDP and monitoring placed in one sentence sounds :( ).

Except for the last one the setup was more or less OK on the daily basis. But in more advanced scenarios it started lacking more and more:

email notifications are fine for being informed that something is wrong, but for the details, I have to RDP and see the files and this is tolerable only on a PC. Doing this on a tablet or a phone is wasting time. Another problem is that if something goes wrong I get thousands of emails.
no performance metrics except for the Hangfire dashboard.
since I am running cookit on crappy hardware RDP is slow.
any analytics or finding correlations had to be done manually. Doing it this way is a pain and a waste of my time.
no way to do a post mortem why the site was slow an hour ago. The only source are the NLog files.

Requirements

So what I wanted to achieve (in order of importance):

Visualizations. - Yes, this is the first place. Looking at tables is the worst way to get a glimpse of what is happening. A simple chart makes a huge difference.
Accessible everywhere. - This means o need to RDP or any app that has to be installed. It has to be web based.
Not running on my machine. - Every system needs some taking care of it. Checking the logs once in awhile, upgrading to a newer version and so on. This requires time and knowledge (knowledge can be translated into time needed to have it). For the current moment this is not the area where I want to invest my time, so SaaS it is.
performance counters of the machine. - Knowing the stats of the application without knowing the stats of the whole machine is close to pointless when things go haywire.
Centralized logging. - No more digging through multiple NLog files. It has to be in one place.
Custom events. - I want to add custom timings for each TPL Dataflow block that I am using in data processing workflows. Also seeing aggregated timings of Hangfire jobs would be nice and would show me where to invest time in optimizations.
Real-time. - when I get a report from Pingdom that the site is not responding I want to check what is happening right now on the machine. Hour old data, in this case, is no use.
Alerting - Having monitoring without alerting works only when you can have huge monitors displaying the dashboard in a room filled with people dedicated to monitoring one app. Even then it only works in movies. Alerting is a must have and a time saver.
Historic data. - The time between getting an alert and having time to look at it is almost always non-zero in my case. Having at least 8 hours (remember, sleep is a good thing?) of data is a must have.

The showdown Google Analytics vs. ELK +Graphite/Grafana vs. NewRelic vs. Retrace vs. Application Insights vs. Raygun vs. Datadog:

Google Analytics

Google Analytics is way more than SEO/SEM tool. With the support of custom client and server events, it can be turned into monitoring tool. Another plus for using it is that I have it already and are looking at it from time to time.

The good

good/decent visualizations
very good UI (fast and intuitive)
accessible everywhere
free (at my scale)
I already have it
has custom events
monitors browser performance
supports custom client and server events
has custom dashboards
good Android application

The bad

working with custom events is not that great
no collection of system performance counters (can be easily written, but still a minus)
no log aggregation
no alerting

ELK Stack + Graphite/Graphana

ELK stack with Graphite and Grafana is the market standard for monitoring and central logging. This means that it is very easy to find Docker images, help and almost everything what is needed. Viewing on mobile devices is possible, but far from being great, and this won’t change anytime soon judging from this Github issue.

Grafana panel:

Kibana panel:

The good

the market standard
great visualizations (both Grafana and Kibana)
great data crunching tools
there are some ELK SaaS providers.
great UI and visualization
web based
there are existing applications for server stats monitoring

The bad

mostly Linux based and since I am running on a Windows Server this meant getting another Linux machine. Setting every
most providers are not free
a lot of stuff to manage on my own
no out of the box monitoring

New Relic

New Relic is a power horse when it comes to features. It has almost everything, from APM (Application Performance Management) to log aggregation. It is a very interesting product since it is done in a way that will be readable to nontechnical people. It automatically sets up notifications for apdex index violations, the UI is orientated into showing the main KPI for performance (like Google Analytics), not tech details like for example Azure Application Insights (next)

The main dashboard:

I was quite impressed with how good New Relic inspects what went into each request. This is the Request monitoring page:

Another thing that is a nice feature, and very unique, is the Geo View. It shows how page speed differs for different geo locations. And as You can see I have a problem with Russia (I have no idea why):

Installation is the easiest of all reviewed. All that is needed is to install the NewRelic Agent on the server. Dependencies detection, metering, even browser performance tracking works out of the box.

One last thing to mention, and it may be subjective. I had the impression that after installing the NewRelic’s agent on the server it slowed down a bit. I could be due to the fact that Application Insights was also running on it, or that my machine is just slow as it is. If someone else also has this impression ping me.

To sum up - New Relic is a powerful beast with the looks that a manager can understand :)

The good

out of the box monitoring of .NET, Ruby, Node.js, PHP, Java, Python and Go applications
Supports web and non-web applications
Supports browser monitoring
fast and very good UI
Browser profiling works without the need to add a script (it is added automatically by the installed NewRelic agent)
very easy to install

The bad

only paid version (I get why they have only a paid version. This is a minus for me when choosing for this use case. Remember this is a personal review).
it consists of multiple products, and even when in trial mode, enabling some of them requires contacting the sales department.
every sub-product is separately paid.
at registration(demo) requires the phone number, company name, company size, and role.
uninstalling requires a machine reboot.
expensive
confusing pricing

Retrace (Stackify)

Retrace is a product from the same guys (and girls) that developed Prefix - a good local profiler. I must say I was impressed by the features and polish that this product has. The UI may not show it fully, but it is not that far away from the big players. In some features, it’s even way ahead.

Retrace panel:

Retrace APM+ panel (from a demo application they provide. I was not able to get it working on my project):

The good

registration initially requires only Name, Surname and email are required 👍. Unfortunately, after email verification they want the rest 👎
what is being collected by the agent is controlled from the website. So after installing the agent on the server, no additional work on the server is required.
nice attention to details - when installing the NLog adapter we are prompted for the key.
the best out of the box alert triggers from them all. The only system monitoring disk queue length.
machine performance alerts are correlated with requests to the application.
fast full-text search and data analytics for logs
option to mark an error as “fixed”
quite fast UI
very good email alerts. They look good and are very informative.
very good customer service :) (emails and the comments)

The bad

after verification, they want a whole lot of info
I was not able to get any APM+ statistics. Everything looked properly configured, but it did not work. ~~I've emailed the support, but no answer (granted it was on the weekend)~~ Support contacted me just after the weekend and is very helpful, but we were not able to get it working.
Dashboard page is in beta, so it doesn’t have that many features or great looking as Application Insights or New Relic.

Application Insights

Microsoft is investing heavily in its Azure cloud and it can be seen from the number of features they are rolling out every three months. They are investing more in the SaaS model and Application Insights fits quite good in this environment. Application Insights feels a lot more developer orientated platform than New Relic. It can bee easily seen that they thought more about query capabilities (AND, OR operators), inner request details, features than in how to get the visualizations super nice.

My customized main dashboard:

And the request details: As you can see all dependencies were detected neatly.

There is also one thing that strongly shows that Application Insights is targeted at IT specialists. By clicking Analyze in the top bar we can see the query behind every diagram, change it and use it for a custom report. The language used is very similar to F#. This means we have all the functional goodies like piping and support for data manipulation functions. As a person using F#, I must say “well done Microsoft”:)

The good

out of the box monitoring of sewer and browser (needs adding Google Analytics like script).
can be added without access to the code. This does not allow full capabilities (dependency tracking and performance monitoring is available), but most of them.
auto detected dependencies like HTTP requests and ADO(SQL) queries.
1 GB a month free.
has data limits.
has data sampling.
very good search build in Azure portal. It searches in products and instances of products. This is more important than one might think because the number of screens is massive. Even when counting only those in the Application Insights module.
customizable dashboard for every Azure offering (every panel can be moved to the main dashboard).
the top bar displays connected features. When you open the pricing window it displays a link to the data limit window.

The bad

detection of application map did not work that great with Oracle database (this may be a configuration issue, I haven’t had time to investigate)
UI can get sluggish sometimes.
UI sometimes refuses to open a window. A reload helps.

Raygun

I’ve decided to check Raygun mostly because of Scott Hanselman’s (it is even featured on the website) and Troy Hunt’s recommendations. Raygun, similar to New Relic, hides multiple services under its name. In this case, it’s Pulse and Crash Report. Pulse is available for: Android, iOS, maxOS, JavaScript, WordPress, Xamarin.Android and Xamarin.iOS. No .NET here so I will not be exploring this path. Crash report supports 27 options with most languages and platform covered(Ruby, Node, PHP, .NET, Go, ColdFusion and mobile). After playing with it a bit I can say that Raygun is not a tool I was looking for because it is focused on logs and crash reports, not on monitoring. That said it does it very well and has its place as a tool. How it works is that it will notify You when an error occurs and gives the possibility to mark it as fixed, but not deployed to prod. This will halt notifications of this type of errors. A simple feature, but a quite valuable one then managing production health and prioritizing bug fixes.

Raygun panel:

The good

the logo reminds me of an old game Earthworm Jim, oh the memories…:)
very carefully designed UI.
easy to setup (NuGet package + web config changes)
checking an error as fixed works good
the filtering won’t let You filter on non-existing values. This prevents from misspelling the browser type for example. Small, but useful feature.

The bad

at registration(demo) requires a phone number, company name, company size, and role.
it is only a log aggregating mechanism
the filtering has very little fields to work with

Datadog

I’ve heard earlier about Datadog, but never checked it before. The first impression was very good, but soon everything started going downhill (see the The bad section). It became clear quite fast that Datadog is targeted to administrators and people that need to have a big picture of servers and services that they have. This is a higher view than the application view I am looking for.

The good

it can integrate with a huge number of systems (the only tool that could monitor Solr and Docker containers)
when I finally managed to find a dashboard it looked really good.
very customizable dashboard. Reminds me strongly of Grafana
it has infrastructure map
it has a free (with limited features) version up to 5 hosts.

The bad

agent installation is strange. It is done by running the command in the command window and after it nothing happens. The stats started appearing after few minutes (~10), so I was already googling if anyone had a similar problem.
agent after installation does the bare minimum. Its functionality can be extended by adding integrations on the website. This is the good part. What is not so nice is that adding a new feature requires changes the agent config, and other services, done manually on the server. This means a more configuration and no clean uninstall :( Retrace showed how to do it better.
the configuration file (which You have to edit manually) is tab sensitive (??!!)
no log aggregation
to add an integration You have to: save the config file (edited in the agent), enable the monitoring and restart the agent.
it took me way longer than I would wish for to get the SQL and IIS monitoring to work

Comparison table

Service	Google Analytics	ELK + Graphite/Graphana	NewRelic	Retrace	Application Insights	Raygun	Data dog
Logging
Centralized logging	N	Y	Y	Y	Y	Y	N
Log querying	N	Y	S	Y	Y	N*9	N
Custom log reports	Y	Y	S	Y	Y	N	N
Browser error collection	N	CbI*4	Y	N	Y	N	N
APM
Server side performance	CbI	Y	Y	Y	Y	N	Y
Browser side performance	Y	CbI *4	Y	N	Y	N	N
Environment performance	CbI	3rd party tools *5	Y	Y	Y	N	N
Custom metrics	CbI	Y	Y	Y	Y*7	N	Y *10
Alerts	N	Y	Y	Y	Y	N	Y
Real time view	Y	Y	Y	Y	Y	N	Y
Custom perf. reports	Y	Y	N	N	S	N	Y
Request dependencies details	N	N	Y	N	Y	N	N
Making life easier
Mobile access	Y *13	3rd party*6	Y *13	N	Works *2	Y	S
OAuth	Y	N	N	N	N	Y	N
Features
Application map	N	N	Y	N	Y	N	N*11
Price	Free with limists	60$*14	150$*8	300$	Free*3	588$	Free*12

Legend:

CbI - Can be implemented
S - Sort of. Not fully supported, or with very few features.
*2 - Azure portal is usable on mobile, but not a mobile first experience ;)
*3 - Application Insights is free up to 1 GB per month. There is also a data cap and data sampling option, so staying in the limit is realistic.
*4 - You have to implement the browser side and the proxy
*5 - There are 3rd party tool available, although quality varies.
*6 - there is An Android app Graphitoid for watching Graphite (did not try it). Kibana seems not to work on mobile
*7 - Application Insights has three types of events: event (something happened), metric (something took x amount of time) and dependency (if auto detection didn’t see this one. Can also log time)
*8 - New Relic’s pricing is not that straight forward. It is based on the type of a machine instance and hours it will run. This is the cheapest option I could find.
*9 - it supports basic queries but doesn’t allow filtering on error text, so for me, it is a NO.
*10 - if You log this metric as a performance counter Datadog can read it.
*11 - there is only an infrastructure map.
*12 - the free version doesn’t include alerts.
*13 - dedicated app
*14 - Kibana and Graphite is free, bit since I don’t want to install it on my server I have to fing a SaaS(hosted) offering. The cheapest I could find was Digital Ocean. Also, logz.io have a free offering up to 1GB~~, but no [pricing](http://logz.io/pricing/) (?)~~ the pricing widget appears on Edge and after disabling all add blockers and similar extensions.

What I’ve choose?

Remember I am looking for a tool that will be used in my side project. This means I will be spending my own money, and I don’t like to spend too much of it. The dislike for overspending is one of the main reasons I am running on crapp (it costs me ~320$ a year). So price will be important.

This said I will admit I have been blown away but what Application Insights offers and the pricing. I like that it is targeted to developers, still being very accessible. The pricing model is the best fitting for me since I can easily control if I want to pay and how much. There are still some things that would make me happier:

Login with Google. This is the most popular OAuth account on the web and would make my life a bit easier. I get why this is not the option, but still.
Faster UI. I don’t get the SPA idea in the portal. It is great when moving in scope of one module, but I would not mind a reload when going from the dashboard to AI
More stable UI. Some windows open full screen, some open as a blade (internal Azure name). Sometimes they don’t open.

The second place goes to New Relic. It is expensive and not exactly what I am looking for, but if I had to report performance to nontechnical people this would be the service I choose. It is easy, nice looking (this is more important than You think) and yet manages to deliver just enough information to get a glimpse what is happening.

The third place goes to Retrace from Stackify. It has better alerts than the top two, but no single request analytics and that was the reason why it took the third place. Still, it is a very good tool. Another plus is that it has the option to mute an error similar to Raygun.

In the next entry - Application Insights configuration and architecture.

Understanding OutOfMemoryException

Mon, 02 Jan 2017 00:00:00 +0100

In my ongoing series on calculating similarities one angle always seemed worth trying, and was pointed out many times on Reddit - use C++ and matrix manipulations. Similarity calculation fits very nicely into matrix representation, and there are algorithms targeting sparse matrix manipulation. So why did I delay it for so long? Because I had other angles I wanted to try and, from the looks of it required significant changes in the existing code base. But since last optimizations didn’t bring the time cuts I’ve expected, the time has come. Brace yourself.

Use matrixes

After some refactoring, I’ve managed to create a place where I could swap the existing implementation for the new one.

It took ~30 minutes of thinking, going back and fourth, but in the end, I’ve maneged to fit in without adding changes to the external code only for this implementation. So think, think, think and then code. Not the other way around. It is faster that way.

The function signature is this:

public static Dictionary<int, List<RecipeSimilarityLight>> calculateSimilarityMapForAll(
        Dictionary<int, IngredientWeightsVector> recipeIngredientVector
        , ILogger logger)
        {

where:

Dictionary<int, IngredientWeightsVector> vectorMap is a dictionary where the key is RecipeId and IngredientWeightsVector is a IDictionary containing the owned ingredients and weights.

I first wanted to try the most brute force approach possible - using float[][] without any fancy algorithms. Just to have a reference point. So here it is:

public static Dictionary<int, List<RecipeSimilarityLight>> calculateSimilarityMapForAll(
    Dictionary<int, IngredientWeightsVector> vectorMap
    , NLog.Logger logger)
{
    //similarity calculations
    var matrix = CreateRecipeMatrix(vectorMap);
    var transposeMatrix = Transpose(matrix);
    var multiplyed = Multiply(matrix,transposeMatrix);
    var vectorsLength = ComputeVectorsLength(vectorMap);
    var resultMatrix = Normalize(matrix, vectorsLength);

    //finding the most similar recipes
    var dict = initSimilarityMap(allRecipes);
    int index = 0;
    for (int i = 0; i < resultMatrix.Length; i++)
    {
        if (index % 500 == 0)
            logger.Info("Calculated: " + index + " from: " + allRecipes.Count);

        var recipeId = allRecipes[i];
        for (int j = 0; j < resultMatrix[i].Length; j++)
        {
            var secondRecipeId = allRecipes[j];
            var similarity = resultMatrix[i][j];
            addSimilarityIfGoodEnough(dict[recipeId], recipeId, secondRecipeId, similarity);
        }
    }
    return
        dict
            .ToDictionary(a => a.Key, a => a.Value.ToList());
}

After clicking run I saw this:

Unhandled Exception: OutOfMemoryException.

and soon after, this:

So without terminating the application I fired up Task Manager and saw this:

So what is going on here? I have 16 Gigs of RAM and I get OutOfMemoryException from a process holding less than one?

Since there are many ways to look at one problem I fired up something more advanced - Process Explorer from Microsoft.

Process Explorer is a part of SysInternals Suite and it is a MUST HAVE for anyone using Windows environment. It requires no installation, just copy and paste.

Process Explorer showed the same and something completely different:

One number is the same (there is a slight difference due to my computer crashing because of low memory), but the other number is a bit, to say it politely, different. And it is huge - 35 gigabytes. So what are those two categories and why is Task Manager showing one, and not the other?

To fully understand why did I get OutOfMemoryException while still having much RAM memory free we will have to understand, at least a bit, memory allocation in Windows and .NET Framework.

Windows, memory allocation and .NET Framework

So how I ended up with ~800MB Working Set, ~45 times larger Private Byes an OutOfMemoryException and Your system is low on memory message from Windows? This has to do with the way .NET Framework behaves.

Windows has a few used memory measurement, but I will concentrate on two of them:

Private Bytes also called the Private Set
Working Set

What is the difference?

Before I continue one note: There is a lot more to Windows and .NET memory management. I’ve taken only the parts needed for this problem. Go and read the linked articles for more in depth knowledge.

Working Set

A simple explanation:

Working set is the RAM memory that the process is currently using. Touching memory fragment for the first time moves it from Private Bytes to Working Set[source]

Precise one: This is the memory that the process can is currently using. Here are all variables and mapped files.

Private Bytes

To put it simple:

Private Bytes is the memory that the process might need.

A more precise: This is the amount of memory that the OS thinks the process will need in the near future. To be even more precise it is the amount of space that has been allocated in the swap file to hold the contents of the private memory in the event that it is swapped out [source]. As mentioned by the cited article this value may, and very often is, overestimated by the OS.

So why is the OS thinking I will need 35 gigabytes of memory? It must be the .NET Framework.

.NET Framework memory allocation

When running managed code .NET Framework is taking care of memory allocation and collection. It is allocating memory in bigger chunks from the OS, so that when creating a variable, so allocating memory for it, we are not communicating with the underlying system, but only with the runtime. Why .NET is doing it? For several reasons:

To keep track of the allocated objects so it can garbage collect them.
For performance. Allocating a 20 MB chunk with one call will faster than doing it with 2000 100KB calls.
Memory fragmentation.

The last one needs a it more clarification: This is the situation when where is no continuous block of free memory of requested size, but the sum of free memory is bigger than requested. Because one ASCI art says more than 1000 words here it is:

Our memory:
|oo_o_ooo__o_o_o|
where:
- o - allocated
- _ - free

In this situation, although we have 6 slots of free memory we can’t allocate a 3 slot object. How is .NET different from the OS in this subject? .NET can change the physical address of the object in the compaction phase of the garbage collection as long as it is not pinned using fixed.

So .NET Framework is over allocating, and then Windows is also over allocating and this ends up in the system running out of memory and crashing my application? This means that the framework AND the operating system is broken?

It is best not to assume that step back, have a break and look at the code once again in few minutes.

The mystery

I finally figured it out and it was so obvious that I felt stupid. Can You guess why did I run out of memory? To make it easier:

the framework is not broken
Windows is not broken (at least in this case)
read the first and the second post.

There are no awards, but I will try to highlight those who get it.

Dividing a bit in two for performance

Tue, 20 Dec 2016 00:00:00 +0100

This post is an analysis of a very interesting optimization proposed by Nicholas Frechette in the comments under the previous post and t0rakka on Reddit. They proposed to use one of the oldest tricks in performance cookbook - divide and conquer. Well, it did not turn out as I expected.

Recap

The last implementation of GetNonZeroIndexes looked like this:

        public IList<int> GetNonZeroIndexes()
        {
            List<int> ret= new List<int>();
            for (var i = 0; i < _array.Length; i++)
            {
                var a = _array[i];
                if (a != 0UL)
                {
                    if ((a & 1UL) != 0UL) ret.Add(i * 64 + 0);
                    if ((a & 2UL) != 0UL) ret.Add(i * 64 + 1);
                    if ((a & 4UL) != 0UL) ret.Add(i * 64 + 2);
                    if ((a & 8UL) != 0UL) ret.Add(i * 64 + 3);
                     .
                    .
                    .
                    .
                    .
                    if ((a & 1152921504606846976UL) != 0UL) ret.Add(i * 64 + 60);
                    if ((a & 2305843009213693952UL) != 0UL) ret.Add(i * 64 + 61);
                    if ((a & 4611686018427387904UL) != 0UL) ret.Add(i * 64 + 62);
                    if ((a & 9223372036854775808UL) != 0UL) ret.Add(i * 64 + 63);
                }
            }
            return ret;
        }

How can this get better?

Divide and conquer

Theory

The above approach is a brute force approach. Why? I am iterating over the long and checking for every bit if it is set. This is wasteful since most of them are not set (remember those are sparse arrays). The better approach would be to test multiple bits at once. How? Bit masks - as always :)

Using bit masks and the AND operation I could check the first 32 bits (from 64 that ulong has):

0000000011111111 <- masc
0001100110001001 <- value

0000000010001001 <- after AND

If any bit in this range is set then the result will be greater than zero. Simple as that.

Implementation

I could replace GetNonZeroIndexes with this code:

public IList<int> GetNonZeroIndexes()
{
    var ret = new List<int>();
    for (var i = 0; i < _array.Length; i++)
    {
        var a = _array[i];
        if (a != 0UL) //is any bit set?
        {
            if ((a & BitVectorMaskConsts._0_32) > 0) //is any of the 32 least significant bits set?
            {
                if ((a & BitVectorMaskConsts._0_16) > 0) //is any of the 16 least significant bits set?
                {
                    if ((a & BitVectorMaskConsts._0_8) > 0) //is any of the 8 least significant bits set?
                    {
                        if ((a & BitVectorMaskConsts._0_4) > 0) //is any of the 4 least significant bits set?
                        {
                            if ((a & 1UL) != 0UL) ret.Add(i*64 + 0);
                            if ((a & 2UL) != 0UL) ret.Add(i*64 + 1);
                            if ((a & 4UL) != 0UL) ret.Add(i*64 + 2);
                            if ((a & 8UL) != 0UL) ret.Add(i*64 + 3);
                        }
                        if ((a & BitVectorMaskConsts._4_8) > 0){
                        .
                        .
                        .
                        .

                    }
                }
            }
        }
    }
    return ret;
}

Complexity

What divide and conquer is doing is simulating a binary tree consisting of 64 nodes.

Since my tree is fully balanced (equal number of nodes on the left and right) its max depth is equal to log n. With n equal to a number of leafs in the tree. I am doing a brute force iteration over the last 4 elements so my number of leafs is 64/4 = 16. This means that I’ll be doing from 5 to 9 checks in the worst case. Looks promising.

Performance results

How did it affect performance? The last best sample run took 297 seconds. With divide and conquer it takes 283 seconds. This translates to:

(283 / 2199) * 182184 ~ 6,5 hours (starting from 34 hours, and 6,8 in the best run)

So I am doing even an order of magnitude fewer operations than before and all I get is 15 seconds? Maybe if I go to leaf size of 2 it will be better? I will be doing less so it should be faster? Even more, let’s plot how execution time is dependent on leaf size

Leaf size vs. time

Below is a table comparing leaf size to execution time (they are an average of 3 runs).

Size of the chunk	Execution time	St.dev
64	299 seconds	2.3
32	277 seconds	3.5
16	296 seconds	3.5
8	293 seconds	2.3
4	283 seconds	6.5
2	297 seconds	4.3

Making sense of it all

Why did my execution time not go down as expected? Couple reasons:

Things changed

I’ve started from 968 seconds for the test run, and most of this time was spend in Similarity. Now if I look at the profiler:

Similarity is still using the most CPU, but it is not the GetNonZeroIndexes that is the hogger. I’m just slowly getting closer and closer to the minimum execution time.

Why leaf size of 2 is slower than leaf size of 4?

Execution in current processors in not linear. They execute instructions in advance hoping that they will guess the flow. When they are right it is very fast, when it is a miss performance suffers greatly. It is called branch prediction.

This situation leads to the fact that sorting and then processing the array may be faster then just processing, or calculating the minimum of two numbers is faster using bit operations then using if.

My guess is that two and four leaf implementation suffers mostly from that.

Making bits faster

Mon, 19 Dec 2016 00:00:00 +0100

This post was inspired by a discussion on Reddit that followed my previous post

In this post, I will cover a suggestion by BelowAverageITGuy that cut down the total execution time by almost one hour.

Recap

The last implementation of GetNonZeroIndexes looked like this:

public IList<int> GetNonZeroIndexes()
{
    List<int> ret= new List<int>();
    for (var i = 0; i < _array.Length; i++)
    {
        if (_array[i] != 0L)// this one check saves me 64 ones
        {
            if (IsBitSet(_array[i], 0)) ret.Add(i * 64 + 0);
            if (IsBitSet(_array[i], 1)) ret.Add(i * 64 + 1);
            if (IsBitSet(_array[i], 2)) ret.Add(i * 64 + 2);
            if (IsBitSet(_array[i], 3)) ret.Add(i * 64 + 3);
            .
            .
            .
            //You get the idea
            .
            .
            .
            if (IsBitSet(_array[i], 61)) ret.Add(i * 64 + 61);
            if (IsBitSet(_array[i], 62)) ret.Add(i * 64 + 62);
            if (IsBitSet(_array[i], 63)) ret.Add(i * 64 + 63);
        }
    }
    return ret;
}

With IsBitSet looking like this:

[MethodImpl(MethodImplOptions.AggressiveInlining)]
static bool IsBitSet(long b, int pos)
{
    return (b & (1L << pos)) != 0;
}

It is using bit operations that are very fast, so what can be done better?

Bit mask precalculation

His suggestion was to get rid of bit shifting and have the mask ready. This way I will get rid of IsBitSet and end up with GetNonZeroIndexes looking like this:

        public IList<int> GetNonZeroIndexes()
        {
            List<int> ret= new List<int>();
            for (var i = 0; i < _array.Length; i++)
            {
                var a = _array[i];
                if (a != 0UL)
                {
                    if ((a & 1UL) != 0UL) ret.Add(i * 64 + 0);
                    if ((a & 2UL) != 0UL) ret.Add(i * 64 + 1);
                    if ((a & 4UL) != 0UL) ret.Add(i * 64 + 2);
                    if ((a & 8UL) != 0UL) ret.Add(i * 64 + 3);
                     .
                    .
                    .
                    .
                    .
                    if ((a & 1152921504606846976UL) != 0UL) ret.Add(i * 64 + 60);
                    if ((a & 2305843009213693952UL) != 0UL) ret.Add(i * 64 + 61);
                    if ((a & 4611686018427387904UL) != 0UL) ret.Add(i * 64 + 62);
                    if ((a & 9223372036854775808UL) != 0UL) ret.Add(i * 64 + 63);
                }
            }
            return ret;
        }

Excel

I generate this code in Excel and after generating all the if statemenent the program started to crash. i wonder if You can spot the error:

Power of 2	Value
49	140 737 488 355 328
50	281 474 976 710 656
51	562 949 953 421 312
52	1 125 899 906 842 620
53	2 251 799 813 685 250

See it?
I’ve hit Excel (and Google docs for that matter) precision and in the 52th power of 2, it was rounded in a strange way. So instead of getting 1 125 899 906 842 624 I’ve got 1 125 899 906 842 620 :| There goes two hours lost on debugging the applications.

Results

How did it impact performance? It allowed the sample execution time to go down to 308.22 seconds. For all recipes I get:

(297 / 2199) * 182184 ~ 6,8 hours (starting from 34 hours)

This shaved off almost 1 hour(!) from the best time. It also shows that even things as fast as bit shifts can be made faster.

Using bit masks for high-performance calculations

Tue, 13 Dec 2016 00:00:00 +0100

Last time I’ve shown how I’ve gone from 34 hours to 11. This time we go faster. To go faster I have to do less.

The current implementation of Similarity iterates over one vector and checks if that ingredient exists in the second one. Since those vectors are sparse the chance of a miss is big. This means that I am losing computational power on iterating and calling TryGetValue.

How to iterate only over the mutually owned ones and do it fast?

Bit masks

The fastest and least memory consuming way is to use an int or a long and set their bits individually. There is one problem with this. I need a mask for 1800 values, so my number would have to be 1800 bit… But before I go further why not use an array of bools? The memory overhead would be minimal (~1MB), but the cost of iterating would be significant. Don’t trust me? Keep on reading :)

.NET’s BitArray

Luckily .NET has BitArray class and it implements the AND operation, so let’s give it a try:

public float Similarity(IngredientWeightsVector otherVector)
{
    var andMask = _ingredientMask.And(otherVector.IngredientMask);//find only the commonly found ingredients 
    var nonZeros = GetNonZeroIndexes(andMask); // get the indexes of the ingredients
    float num = nonZeros.Sum(t => otherVector.IngredientWeights[t]*IngredientWeights[t]);
    var denom = Length()*otherVector.Length();
    return num/denom;
}

private List<int> GetNonZeroIndexes(BitArray ba)
{
    List<int> ret = new List<int>(20);
    int index = 0;
    foreach (bool b in ba)
    {
        if(b)
            ret.Add(index);
        index++;
    }
    return ret;
}

Lets walk through what is happening here.
I calculate the AND between the masks of both vectors. This gives me the mask of ingredients present in both recipes. Then, since I need the weights for those ingredients, I iterate over the mask and find indexes (GetNonZeroIndexes takes care of it). With them, I can get the weights from both recipes vectors. Having those weights I can finally calculate the dot product.

The code is ready, so let’s fire it up!

Sadly I won’t give you the numbers because I’ve decided to terminate the calculation when it passed the point when it became clear that it will be way slower than the first implementation. So let’s fire up the profiler and see what is happening:

As I saw earlier when optimizing this problem - iteration is not something free :(
Maybe it’s foreach fault and a simple for will be OK? Let’s check:

Get is the function called on the indexer. So is this a dead end? Before taking this decision lets look at the code of Get in the BitArray:

public bool Get(int index)
{
    if (index < 0 || index >= Length)
    {
        throw new ArgumentOutOfRangeException(nameof(index), index, SR.ArgumentOutOfRange_Index);
    }
    Contract.EndContractBlock();

    return (m_array[index / 32] & (1 << (index % 32))) != 0;
}

What can I improve there?

remove the ifs at the start, because I like my performance unchecked ;)
BitArray is using array of int as storage and I think using a long be a bit faster

Custom BitArray implementation

Lets try with a custom implementation:

using System;
using System.Collections;
using System.Collections.Generic;
using System.Runtime.CompilerServices;

namespace Cookit.Utils.Lib.CSharp.RecipeSimilaritiesCalculator
{
    public class MyBitArray
    {
        private readonly long[] _array;
        public int Length;

        public MyBitArray(int size)
        {
            Length = size;
            var numberOfLongs = (int) Math.Ceiling(((float) size)/64);
            _array = new long[numberOfLongs];
        }

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        private void CalcIndex(int index, out int longIndex, out long masc)
        {
            longIndex = GetLongIndex(index);
            var rest = index - longIndex*64;
            masc = GetMask(rest, true);
        }

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        private int GetLongIndex(int index)
        {
            return index/64;
        }

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        internal static long GetMask(int bitnum, bool val)
        {
            if (val)
                return (1L << bitnum);
            return ~(1L << bitnum);
        }

        public void Set(int index, bool value)
        {
            var longIndex = 0;
            long longMasc = 0;
            CalcIndex(index, out longIndex, out longMasc);
            var currentValue = _array[longIndex];
            currentValue = currentValue | longMasc;
            _array[longIndex] = currentValue;
        }

        public BitVectorMask And(BitVectorMask other)
        {
            var ret = new BitVectorMask(other.Length);
            for (var i = 0; i < _array.Length; i++)
            {
                ret._array[i] = _array[i] & other._array[i];
            }
            return ret;
        }

        public IList<int> GetNonZeroIndexes()
        {
            List<int> ret= new List<int>();
            for (var i = 0; i < _array.Length; i++)
            {
                for (int j = 0; j < 64; j++)
                {
                    if (IsBitSet(_array[i], j))
                        ret.Add(i * 64 + j);
                }
            }
            return ret;
        }

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        static bool IsBitSet(long b, int pos)
        {
            return (b & (1L << pos)) != 0;
        }
    }
}

Test run for the sample took 2509 sec, so the full run will take:

(2509 / 2199) * 182184 ~ 57,7 hours (starting from 34 hours)

So I am getting slower :

Why was it so slow? Hopefully, dotTrace will tell the truth:

If we look at the profiler result GetNonZeroIndexes is the most expensive function, but there is almost nothing in it to blame. The time is hidden in native functions such as iterating, multiplication and soo on, and I am doing a lot of them here. How to simplify?

Loop unwinding

The concept is: Why to waste time on looping when I can unwind the code. Will it make that of a difference? Let’s see:

public IList<int> GetNonZeroIndexes()
{
    List<int> ret= new List<int>();
    for (var i = 0; i < _array.Length; i++)
    {
        if (_array[i] != 0L)// this one check saves me 64 ones
        {
            if (IsBitSet(_array[i], 0)) ret.Add(i * 64 + 0);
            if (IsBitSet(_array[i], 1)) ret.Add(i * 64 + 1);
            if (IsBitSet(_array[i], 2)) ret.Add(i * 64 + 2);
            if (IsBitSet(_array[i], 3)) ret.Add(i * 64 + 3);
             .
            .
            //You get the idea
            .
            .
            if (IsBitSet(_array[i], 61)) ret.Add(i * 64 + 61);
            if (IsBitSet(_array[i], 62)) ret.Add(i * 64 + 62);
            if (IsBitSet(_array[i], 63)) ret.Add(i * 64 + 63);
        }
    }
    return ret;
}

This brings down the sample time to 395 seconds and the total time:

(395 / 2199) * 182184 ~ 9 hours (starting from 34 hours)

For the first time in this post it is faster than the best time from the previous post (484 seconds).

But it still can go faster. A reminder here is how Similarity looks like:

public float Similarity(IngredientWeightsVector otherVector)
{
    var andMask = _ingredientMask.And(otherVector._ingredientMask);
    float num = 0;
    var nonZeros = andMask.GetNonZeroIndexes();
    for (var i = 0; i < nonZeros.Count; i++)
    {
       num += otherVector.IngredientWeights[nonZeros[i]] *IngredientWeights[nonZeros[i]];
    }
    var denom = Length()*otherVector.Length();
    return num/denom;
}

What else can be optimized?

Exit early

If the andMask doesn’t contain any ones I am still doing some calculations, allocations and so on. Granted, not much but still. A simple check in the BitVectorMask:

public bool IsZero()
{
    for (var i = 0; i < _array.Length; i++)
    {
        if (_array[i] != 0L)
            return false;
    }
    return true;
}

and a if in the Similarity function:

public float Similarity(IngredientWeightsVector otherVector)
{
    var andMask = _ingredientMask.And(otherVector._ingredientMask);
    if (andMask.IsZero())//this was added
        return 0;
    float num = 0;
    var nonZeros = andMask.GetNonZeroIndexes();
    for (var i = 0; i < nonZeros.Count; i++)
    {
       num += otherVector.IngredientWeights[nonZeros[i]] *IngredientWeights[nonZeros[i]];
    }
    var denom = Length()*otherVector.Length();
    return num/denom;
}

With those modifications the sample run finished in 334 seconds. This gives:

(334 / 2199) * 182184 ~ 7,6 hours (starting from 34 hours)

Can it still be improved significantly or am I looking at marginal gains here? This problem can still astound you, but this will be unveiled next week.

Update

This post sprung a very interesting conversation on Reddit with 104 comments as of writing this update.

Independent code in performance optimizations

Mon, 12 Dec 2016 00:00:00 +0100

This will be a fast errata to the previous one. This time I will expand the oldest performance mantra:

The fastest code is the one that doesn’t execute. Second to that is the one that executes once

Last time I’ve forgot to mention one very important optimization. It was one of two steps that allowed me to go from 1530 to 484 seconds in the sample run.

Let’s look once more at the code:

public float Similarity(IDictionary<int,float> otherVector)
{
    var num = this
        .Sum(ingredient =>
        {
            float value = 0;
            if (otherVector.TryGetValue(ingredient.Key, out value))
                return ingredient.Value*value;
            return 0;
        });

    var denom = Lengh()*otherVector.Length();
    return num/denom;
}

and Length looking like this:

public float Length(Vector a){    
    return (float)Math.Sqrt(a.Sum(value => value*value));
}

Similarity will be called for every pair of recipes and that means a lot:

(182184*182184)/2 = 16 595 504 928 times

But there is one fragment that doesn’t change depending on input parameters, so it means i can calculate it less often.
Can You spot it?

It’s Length.

Changing it’s code to this:

public float Length()
{
    if (_wasIngredientWeightsChanged)
    {
        _len = (float) Math.Sqrt(_ingredientWeightsInternal.Values.Sum(value => value*value));
        _wasIngredientWeightsChanged = false;
    }
    return _len;
}

Allowed me to go from 968 seconds to 745 seconds for the sample. Scaling it to all recipes gives:

(745 / 2199) * 182184 ~ 17 hours (starting from 34 hours)

Conslusion

“Can I make it faster” is not the only question to ask when optimizing. Sometimes asking “does it have to execute this often?” is also a very valid question.

How to calculate 17 billion similarities

Mon, 05 Dec 2016 00:00:00 +0100

The previous post described the methodology I’ve used to calculate similarities between recipes in cookit. If you haven’t read it I’ll give it 4 minutes because it will make understanding this post easier. Go one, I’ll wait.

It ended on a happy note and everything seemed to be downhill from there on. It was until I tried to run it. It took long. Very long. How long? I don’t know because I’ve canceled it after about one hour. Going with a famous quote (probably from Einstein, but there are some ambiguities in this subject)

Doing the same thing over and over again and expecting different results

I’ve decided to, once again, use math to assess how long the calculation will take.

Reminder - how similarities are calculated

In the previous post I’ve decided to calculate similarity by calculating the dot product between recipe ingredient vectors. In C# it will look more or less like this:

public float Similarity(Vector a, Vector b)
{
    float accumulator=0;
    for (int i = 0; i < a.Length; i++)
    {
        accumulator += b[i]*a[i];
    }
    var denom = Length(a)*Length(b); //convertion to absolute 
    return accumulator/denom;         //convertion to absolute    
}    

where Length looks like this:

public float Length(Vector a){    
    return (float)Math.Sqrt(a.Sum(value => value*value));
}

Complexity

To give you some scale. Cookit currently has:

182 184 recipes
2936 ingredients

As a consequence of the way I model similarities, I will need to create for each recipe a 2936 dimensional array representing ingredients. There are two things I should estimate: memory and number of calculations.
First the memory:

182184(number of recipes) * 2936 = 534 892 224 floats 
534 892 224 * 4(the size of float) ~ 2.14 Gig counting only space needed for floats

And if I will want to calculate the similarities for all recipes I will have to do:

((182184 * 2936)^2) /2 = 143 054 845 647 833 100 floating point multiplications

Is it much? Yes. Is it a lot for modern processors? IT Hare had a good article about the cost of each operation in the CPU. One note before we go further. Those calculations will be more for fun than actual time estimates. Even as IT Hate points out, trying to calculate the actual execution time in modern CPUs is hard enough to be pointless. But here we go:

143 054 845 647 833 100 * 5 (number of cycles for floating point operations) / 2 130 000 000 (this is what 2,13 GHz translates to - 2,13 billion operations per second)
This gives us ~ 335809496 seconds ~ 93280 hours ~ 3886 days  

Once again it is more for fun than anything else, but it appears that, citing Mark Watney

I will have to science the sh** out of it [youtube]

Assessing total time

To get a more real assessment how long will it take I’ve decided to do a test run on a small subset of recipes. Recipes are selected by selecting websites they are from, so my subset ended up being an unround 2199 recipes. Then I’ve made 3 test runs (I’ve learned the hard way that one test run doesn’t mean anything) and in average of 1530 seconds.

Now how to go from time of the subset to the time needed for all recipes? I can’t use it to assess the total execution time because it consists of two parts:

calculating ingredient vector for all recipes.
finding similar recipes for only 2199 one of them. The similarity is found by calculating one vector by all the others. So in this case I’ve done

182184*2199 = 400 622 616 vector multiplications

The first part is constant regardless how many similarities I will have to calculate.
The second part is directly proportional to the number of recipes for which I will calculate similarities.

The last missing piece for assessing the time of calculation for all recipes is the proportion of those two operations. So let’s fire up the profiler (dotTrace by choice) in sampling mode:

Well, bummer to put it lightly. Almost all time went into calculating the dot product (named here Similarity). From this I know that I can calculate the total time as a linear function from number of recipes without making to big of an error:

(1530 / 2199) * 182184 ~ 34 hours

where:
- 1530 - number of seconds it took
- 2199 - number of ingredients
- 182184 - number of recipes 

So the next step will be:

Optimising

Less means faster

The fastest code is the code that never runs

Following this mantra I removed the ingredients that are not used in any recipe.

This reduced the length of the vector to 1709 and the sample time be a whopping 562 seconds to 968 seconds. With some simple math:

(968 / 2199) * 182184 ~ 22,3 hours (starting from 34 hours)

This is a 34% improvement. Nice :)

So case closed? No. I believe I can do better. To have an idea let’s have a look with a profiler:

There did all the time go? Let’s have a look at another view:

This shows that most of the time went into native code meaning in my case multiplying floats and iterating over the array. This means that this idea is a dead end. Let’s change the angle and use domain knowledge to optimize a bit more.

Using domain knowledge for optimization

I know that vectors are very sparse. In average one recipe has ~150 non-zero values in an almost 2.2k long array. This means I am wasting time multiplying zero. Let’s remove them.

Use a dictionary

Instead of using an array I will use a dictionary. The idea behind this take is this:

it will allow me to store only those ingredients that the recipe has. This way the vector will shrink by an order of magnitude and as a side effect I will also shrink memory usage (although not that much since dictionary in .NET is a memory heavy structure)
since I shrunk the vector I don’t need to enumerate over all the ingredients, but only over those from one of the vectors

So let’s put it into code:

public float Similarity(IDictionary<int,float> otherVector)
{
    var num = this
        .Sum(ingredient =>
        {
            float value = 0;
            if (otherVector.TryGetValue(ingredient.Key, out value))
                return ingredient.Value*value;
            return 0;
        });

    var denom = Length()*otherVector.Length();
    return num/denom;
}

With this change the calculation took 484 seconds. Scaling it to the full data set I get:

(484 / 2199) * 182184 ~ 11 hours (starting from 34 hours)

This means a 67% improvement over the starting value. So case closed? No. But this will be the topic for the next post since this one is getting to long.

How I calculate similarities in cookit?

Mon, 28 Nov 2016 00:00:00 +0100

Warning this post contains some math. Better still, it shows how to use it to solve real-life problems.

This post describes how I calculate similarity between recipes in my pet project cookit.pl. For those not familiar with it, cookit is a search engine for recipes. It crawls websites extracting recipes, then parses them and tries to create a precise ingredient list replete with amounts and units.

By the time of writing it had:

182 184 recipes
2936 ingredients

This scale may not seem huge, but trust me - It’s enough to bring a slew of problems to light. And that cookit runs on a crappy server, partly by choice, can make things all the more complicated.

Before turning to those problems, lets first answer this question:

How to calculate similarities?

The first question to ask is: How do you calculate similarity between two articles, recipes, objects in general ? Let’s use math! If object can be expressed as a series of ordered numbers (a vector) then there are many ways to express how two such vectors are similar to each other.

So how can a recipe be expressed as a vector? My answer was: If they have similar ingredients they are similar This means I have to create for each recipe, a vector of ingredients with 0 representing the lack of an ingredient and 1 its presence.

Similarity between recipes can now be expressed as the number of common ingredients (do an AND operation and count all the ones). So, case closed, right? Well, not exactly. This design has some problems.

Building a similarities vector

Let’s consider some cases where the initial idea will be too much of a simplification.

Taking similarity into account

Let’s take two recipes, one containing sea salt and the other salt. In a boolean representation, the similarity between them would be equal to zero. But they are actually similar, and can often be used interchangeably. So they are not the same but, they are similar. Now how to represent that?

I can take the advantage of the fact that all ingredients in cookit are organized in a graph. Where the child is more specific than the parent. For example, in this case, I have:

Spices -> solid spices -> mineral spices -> salt -> sea salt

We can use this knowledge to work and put all the parents and children into the vector as owned ingredients. The problem is, I then lose the significance of the actual ingredient used (meaning that sea salt for this recipe is ideal, and salt is just OK). The best way would be to use weights on how accurate the ingredient is for this recipe. So my vector will look like this:

	Spices	Solid spices	Mineral spices	Salt	Sea salt
Recipe with `salt`	(1y)y	(1y)y	(1*y)	1	(1*x)
Recipe with `salt`(numbers)	0.512	0.64	0.8	1	0.9
Recipe with `sea salt`	((1y)y)*y	((1y)y)*y	(1y)y	1*y	1
Recipe with `sea salt`(numbers)	0.4096	0.512	0.64	0.8	1

where:

x is in the range of 0-1 and is the penalty for being further away for children
y is in the range of 0-1 and is the penalty for being further away for parents

Adding importance

Let’s take scrambled eggs made from these ingredients:

eggs
salt
pepper

And now a steak made from these:

meat
salt
pepper

The current model would have us believe that those dishes are, at roghly in 66% similar. This is because every ingredient is given the same importance. The question now becomes, how do I calculate an ingredient’s importance? Easily, using the same idea Lucene did - the inverted index:

Sum of recipes having ingredient x / Sum of all ingredients in all recipes

Because salt and pepper are very popular ingredients, their weight will be very small. Because Meat and eggs are less popular, they will have a higher weight. This approach has the neat feature that it is normalized in zero to one range.

So, for now, the model looks good.

Calculating the vector

Let’s transfer all this knowledge into code:

float[] vector = new float[NUMBER_OF_ALL_INGREDIENTS]
 
foreach(var ingredient in recipe.Ingredients){
    SetIngredientWeight(ingredient,vector,1);
}

private void SetIngredientWeight(Ingredient ing,float[] vector,float weight){
    vector[ing.Id] = weight//set the weight for the current ingredient
    foreach(var parent in ing.Parents)
        SetIngredientWeight(parent,vector,weight*0.8) //go recursivly over each parent and add them with smaller weight
    foreach(var child in ing.Children) //do the same for children
        SetIngredientWeight(child,vector,weight*0.9)
}

What is happening:

I iterate over all the ingredients in the recipe and set the weight to 1. I then go recursively up and down the ingredient tree and set the weights for parents and children. The further away from the original ingredient, the less significant the ingredient is. So the weight is lowered (this is accomplished care by the weight*0.9 part in SetIngredientWeight)

This gives me a vector representation of every recipe. The “only” thing left is to figure a way to compare two vectors.

Calculating similarity

Ok, I have recipes represented as a vector of floats. Now, how does one go about converting two of those vectors into a single digit, from zero to one, representing how similar they are.

Luckily, mathematicians figured this long time ago:) There are many algorithms that can be helpful, but the most common, is to calculate the dot product. Hence it is there we shall start.

Let’s assume that our vectors are two-dimensional (this is like saying that every dish can be made from a combination of only two ingredients). They can then be drawn like this (image taken from Wikipedia):

So what you’re looking at is one vector being projected onto another (the cosine part takes care of it). This gives its length relative to the other vector. However, relative values are nice but useless. Why? Because comparing the similarity of A and B to that of A and C would be to comparing the length of A projected onto B and A projected onto C. But the divisors are different, thus the comparison makes no sense. One way to deal with that is to normalize the value to one range (the most common being from zero to one). This is simple. Just divide the shorter vector, or projection, by the length of the longer projection.
As ComicStrip says:

Lets putting all of it into C#:

public float Similarity(Vector a, Vector b){    
    float accumulator=0;
    for (int i = 0; i < a.Length; i++)
    {
        accumulator += b[i]*a[i];
    }
    var denom = Length(a)*Length(b); //convertion to absolute 
    return accumulator/denom;         //convertion to absolute    
}    

where Length looks like this:

public float Length(Vector a){    
    return (float)Math.Sqrt(a.Sum(value => value*value));
}

So all well then? Far from it. But this will be the topic of the next post. Stay tune, rss, follow, or just don’t close to browser window ;)

Problems with AsParallel

Mon, 21 Nov 2016 00:00:00 +0100

This post is covering a subset of what I am talking in my talk How I stopped worrying and learned to love parallel processing (currently only in polish).

This will cover on how, in terms of performance, AsParallel can kick you in a place where it hurts a lot, simultaneously being a blessing in terms of… performance. How is that? Let’s look at some

History

AsParallel was introduced as an extension to LINQ with TPL in .NET 4.0. In theory, it’s God’s sent. The promise was that it will:

parallelize the LINQ query.
take care of all thread management and synchronization.
adjust the number of Tasks automatically.
not require any additional code changes except for .AsParallel()

And in the vast majority of cases, this promise was kept! For example look at this code:

When it is awesome

void Main()
{
    //create a list
    const int _numberCount = 1*1000*1000;
    var list = new List<int>(_numberCount);
    for (var i = 0; i < _numberCount; i++)
    {
        list.Add(i);
    }    
    //calculate in one thread
    MeasureTime(() => list
                        .Where(IsPrime)
                        .Count()
                );

    //use AsParallel
    MeasureTime(() => list
                        .AsParallel()
                        .Where(IsPrime)
                        .Count()
                );
}
//very bad way to calculate a prime. Not for production usage ;) 
internal static bool IsPrime(int number)
{
    if (number == 1) return false;
    if (number == 2) return true;

    var boundary = (int) Math.Floor(Math.Sqrt(number));

    for (var i = 2; i <= boundary; ++i)
    {
        if (number%i == 0) return false;
    }

    return true;
}

private static void MeasureTime(Func<int> a)
{
    var sw = Stopwatch.StartNew();
    a();
    sw.Stop();

    Console.WriteLine("Took: " + sw.ElapsedMilliseconds);
}

This is a trivial example (a more real life example was published recently by Ayende here, here, here and here), but stay with me. In this case, the difference between single threaded and parallel execution is as follows:

Single threaded: 570
AsParallel: 138

This is more than 4-times speed up with only 1 line and no additional code changes. Looks like all the promises were kept.

But sometimes things don’t go that good.

AsParallel on a to small set

Let’s change that code, and set the list size to 100. Then the results will look like that:

Single threaded: 4
AsParallel: 15

And yes. Those numbers are tiny. But if you have it in every web request or a small that’s frequently called function it cumulates and affects overall performance. When talking about small functions they have a habit of being used in other functions, thus creating bigger blocks. Those blocks tend to be … parallelized. So let’s see what happens then.

Nested AsParallel

It is more or less the same code as above, but now we have:

CalcInSingleAsParallel that calculates the primes using AsParallel (same as above)’
CalcInTwoNestedAsParallel splits a list into chunks and returns in parallel calls CalcInSingleAsParallel

One thing to note about CalcInTwoNestedAsParallel is that it splits the list into chunks when being invoked, but returns an IEnumerable. This trick defers the calculation of the prime until Count is being called. This way MeasureTime only measures the prime calculating part.

void Main()
{
    //create the list
    const int _numberCount = 1*1000*1000;
    var list = new List<int>(_numberCount);
    for (var i = 0; i < _numberCount; i++)
    {
        list.Add(i);
    }
    // split the list into 10 fragments and return an IEnumerable<IEnumerable<int>>
    var tenFragments = CalcInTwoNestedAsParallel(list, 10);
    MeasureTime(
                () => tenFragments.Sum(a => a.Count())
    );
    
    // split the list into 100 fragments and return an IEnumerable<IEnumerable<int>>
    var hundretFragments = CalcInTwoNestedAsParallel(list, 100);
    //calculate the prime number
    MeasureTime(
                () => hundretFragments.Sum(a => a.Count())
    );
    
    // split the list into 1000 fragments and return an IEnumerable<IEnumerable<int>>
    var thousandFragments = CalcInTwoNestedAsParallel(list, 1000);
    //calculate the prime number
    MeasureTime(
                () => thousandFragments.Sum(a => a.Count())
    );
}
private static IEnumerable<IEnumerable<int>> CalcInTwoNestedAsParallel(List<int> numbers, int chunkNumber)
{
    var numbersChunks = ToChunks(numbers, chunkNumber);
    return numbersChunks
        .AsParallel()
        .Select(CalcInSingleAsParallel)
        ;
}
private static IEnumerable<int> CalcInSingleAsParallel(List<int> numbers)
{
    return numbers
        .AsParallel()
        .Where(IsPrime)
        ;
}

private static List<List<int>> ToChunks(List<int> source, int nSize = 30)
{
    var ret = new List<List<int>>();
    for (var i = 0; i < source.Count; i += nSize)
    {
        var tmp = source.GetRange(i, Math.Min(nSize, source.Count - i));
        ret.Add(tmp);
    }
    return ret;
}
internal static bool IsPrime(int number)
{
    if (number == 1) return false;
    if (number == 2) return true;

    var boundary = (int) Math.Floor(Math.Sqrt(number));

    for (var i = 2; i <= boundary; ++i)
    {
        if (number%i == 0) return false;
    }

    return true;
}
private static void MeasureTime(Func<int> a)
{
    var sw = Stopwatch.StartNew();
    a();
    sw.Stop();
    Console.WriteLine("Took: " + sw.ElapsedMilliseconds);
}

The results look like this:

For 10 chunks containing 100 000 elements it took: 1737
For 100 chunks containing 10 000 elements it took: 361
For 1000 chunks containing 1 000 elements it took: 228

We have a ~7.6 times difference between the best and the worst run, and ~12.5 times difference when compared to a single AsParallel! It gets even worse when we nest it into another AsParallel (while maintaining the same total amount of calculations being done):

For 10*10 chunks containing 10 000 elements it took: 3061
For 100*100 chunks containing 100 elements it took: 571
For 1000*1000 chunks containing 1 elements it took: 284

Take a look again at the last line: For 1000*1000 chunks containing 1 elements it took: 284 It was the fastest in this run although having very tiny (one element) lists.

The difference between the best and the worst run is this case is 11 times. When compared to the best run (a single AsParallel) it is more than 22 times slower! Why is that?

The profiler

This is the dotTrace result for all of the above:

No nesting, single AsParallel:

One nesting (so two AsParallel), 10 chunks:

Two nestings (so three AsParallel), 10*10 chunks:

So what is happening in those applications?

The first thread is the main thread of the application. The long delay before spawning more threads is the start-up of the application and list creation. Next threads start to show up. This is because of the call to AsParallel. It triggers TPL’s TaskScheduler which tries to create just the right amount of threads to minimize context switching between them and, at the same time, parallelize as much as possible by assigning tasks to thread and if needed creating more threads.

So why do we have such huge difference in execution times? Then we compare the images, the answer what is wrong becomes obvious. The cost is hidden in switching and managing threads. TPL’s task scheduler is creating new threads and assigning tasks to them because it thinks adding more will help (a single operation is short). At the same time, the operating system is switching between threads, trying to give each a slice of processors time thus reducing the time real computation is being done. Those two managers are interfering each other leading to almost no work being done and in the end a 22 times increase in time needed for the work to finish.

This shows that micro-managing is not a good thing. In programming or in the real world ;)

Conclusion

AsParallel gives mind blowing performance gains with little to none effort. But it can’t be treated as a silver bullet. When you decide to use it:

profile the whole process.
AsParallel is dependent on the data size. So have performance tests with wide range of size of the processed data
when used in a web application, test it under load. IIS likes to have control over threads and you are messing with it.

Use with caution

Debugging high memory usage. Part 2 - .NET Memory Profiler

Fri, 30 Sep 2016 00:00:00 +0200

Diagnosing high memory usage can be tricky, here is the second part of how I found what was hogging to much memory in our system. In the previous post I’ve wrote how to create a memory dump and how many possibilities of catching just the right moment for it ProcDump has. When trying to analyze memory leaks, or high memory usage (not necessary meaning a leak) we have a few ways to approach it:

Attach a debugger

There are many problems with this approach, to name a few:

it has to be done on the machine where the memory outrage has happened (probably a test server). And there Visual Studio shouldn’t be installed.
when debugging an IIS managed process it is being stopped. While the process is stopped it won’t respond to IIS ping requests (they can be turn off, but it is a good practice to have them on). This in effect will make IIS believe that that the process is hanging and restart it. And there goes our debugging session :(
debugging a multithreaded application handling multiple incoming requests will lead to breakpoints being triggered by multiple requests, not just the one being debugged
step by step debugging doesn’t give a view of the memory allocated

To sum up, this is not a good way to approach the problem.

Attach an online profiler

Products like Redgate ANTS Profiler, JetBrains dotMemory can attach to a process and show current GC generation sizes, the paste of new memory being allocated, and compare memory between two points in time (snapshots). While those features are great when profiling and optimizing, they are not that useful when trying to diagnose what has allocated the memory.

Offline memory analyzers

To clarify, by offline I mean programs that can read a memory dump ( the previous post covers one of the methods how to create a memory dump). This criteria leaves us with not that many programs to choose from. Online profilers mentioned above can’t read a memory dump. From my knowledge at this point we can choose from:

The first one is the all powerful big boys profiler with text commands and text based gui. It has the power and the speed that none other profiler has, but

Another, more important in this cases, thing lacking is the ability to show dependencies between objects and memory they allocated. So let’s look at what .NET Memory profiler can do.

.NET Memory Profiler

.NET Memory Profiler can have a tendency to crash after few minutes, sometimes even in less then one. As strange as it seams restarting the computer removes the problem. This happened to me 3 times.

First let’s load the dump file with File > Import memory dump.... This opens a nice import window with a few options and data to fill:

Include instance addresses - in our case not important. I’ m not planing on reading the dump in search of individual objects.
Collect instance data - this gives the ability to view values of fields and objects. Very nice feature, but at the first run I usually try to get a high-level look at the objects in memory so this is not needed for now.

After pushing Start another window will appear asking for mscordacwks.dll and sos.dll. You need to take those files from the machine on which the dump was made. .NET Memory Profiler tries to give hints where to find them, but the exact framework version can be slightly different (4.5 instead of 4.6.1 etc).

After some time (depending from the dump size) the main window will appear:

So let’s look what we have here. While this screen alone can give the answer to someone knowing the application, let’s go through it in investigative mode. What we see immediately is that the main memory holder is StatefulPersitanceContext from NHibernate. With 28 instances it is holding almost 2 Gigs of memory. So lets double click on it and go into details. And we see this:

The window on the left shows the current instances and info about them. One instance is responsible for most used memory. So let’s drill into it. Double click and here we go.

Did You notice that the graph on the right changed? It is showing dependency graphs for the currently selected instances. What can be seen is that Session is keeping hold of Document entity, which has a few collections on its own. The next object referring to this session is on the right side of the graph:

So AutoCloseDocumentService in CloseSettledDocuments is holding to this large session, and the session holds Document entity. So let’s get back to the first screen and see just how many Document entities there are:

Yup. Over 301 thousand root entities in one NHibernate session. Just to be sure let’s look at the CloseSettledDocuments function:

    public void CloseSettledDocuments()
    {
        var settledDocumentsToBeClosed = GetApprovedDocuments()
        ...

        foreach (var document in settledDocumentsToBeClosed)
            CloseDocument(document);
    }

and a peek into GetApprovedDocuments:

    public IList<Document> GetApprovedDocuments(){
        return _session.QueryOver<Document>().ToList();
    }

So we listed 301 thousand documents and processed each one independently. Classic n+1 problem that escalated because of large amount of data.

Debugging high memory usage. Part 1 - ProcDump

Wed, 21 Sep 2016 00:00:00 +0200

I’m taking a short break from Hangfire series, but I will get back to it.

This time - Where did my memory go ? Or to be more exact: Why is this using so much memory?

The story starts with one IIS application pool using around 6 Gigabytes of memory on one of our test environments. It was several times above the values that we expected it to use, so we decided to investigate.

Without much thinking we fired up Visual Studio installed on the test server, and attached to the process. Since the application was build in Debug mode we had all the pdb files in the website folder.

Do I have your attention now? The above paragraph is of curse a joke and a bunch of anti patterns. Don’t do any of them!

Now that I have your attentions let’s get back to what really happened. Since it was a remote sever handling multiple enviroments the only sane step to take is to make a memory dump of the process, kill it and analyze it on a developers machine. Since we always have Sysinternals Suite in c:\tools (highly recommend having it, because when You need it the most, there is no time for download). Sysinternals Suite contains a simple console application for making process dump called procdump.exe.

ProcDump is a very powerful tool and it’s capabilities go far beyond what I will show now. To just give a glimpse flags that control then the dump will be triggered:

c and cl- create a dump when CPU threshold is above or below given limit

e - create a dump when unhandled exception is thrown

h - create a dump when processes window is not responsive

m and ml - create a dump when CPU threshold is above or below given limit

p and pl - create a dump when a given performance counter is above or below a given limit

t - create a dump when the process terminates

absence of this flags means trigger the dump now For a better insight of what ProcDump can do fire it up without any arguments and it will display a very good help.

In this case we wanted to get the dump right now, so the command line looked like this:

procdump.exe 25944 c:\temp\ -ma

To decompose:

25944 is the process id. We could pass a process name, but since it is a website hosted by IIS its process is named w3wp.exe.

Process id (PID in short) can be obtained be many tools such as ProcessExplorer which is also a part of SysInternals Suite. But the default Task Manager available on any Windows machine also has it. Just right click on the Name column, and check PID in the menu that will appear.

c:\temp is the folder where the dump will be written to
ma flag says that we want a full dump with all of processes memory written to the file.

If you get this error:

Error opening w3wp.exe (25944):
Error 0x00000005 (5): Access is denied.

That is because it has to be run with administrator privileges like any debugger. The normal output should look like this:

ProcDump v8.0 - Writes process dump files
Copyright (C) 2009-2016 Mark Russinovich
Sysinternals - www.sysinternals.com
With contributions from Andrew Richards

[12:48:59] Dump 1 initiated: c:\temp\w3wp.exe_160917_124859.dmp
[12:49:03] Dump 1 writing: Estimated dump file size is 5612 MB.
[12:49:59] Dump 1 complete: 5614 MB written in 55.6 seconds
[12:50:00] Dump count reached.

So we have the dump. What next? This is the topic for the next post :)

Don't do it now! Part 6. Hangfire details - recurring jobs and cron expressions

Wed, 07 Sep 2016 00:00:00 +0200

Parts 3, 4, and 5 covered the BackgroundJob class responsible for enqueuing single jobs (fire and forget). This post will cover RecurringJob class exposing API for recurring jobs (as the name suggests).

Recurring job

Before we go into the API, let’s take a look what is a recurring job in Hangfire. Recurring job is a timer that enqueues a job at specific time intervals defined with a cron expression. What is important is, that it does not execute the job. Only enqueues an ordinary Hangfire job. This implementation is very elegant, but it also means that if the queue is full the job will have to wait for its turn. So there is no guarantee about the time it will actually execute.

Cron expressions

I’ve mentioned cron expression few times, but what exactly are they? Cron expressions are a way to express time occurrence, like “every Monday at 8”, or “second Thursday of the month at 5:30”. How do they look? A few examples:

every Monday at 8 can be expressed by 0 0 8 ? * MON *
second Thursday of the month at 5:30 can be expressed by 0 30 5 ? 1/1 THU#2 *
second Thursday of every third month at 5 can be expressed by 0 30 5 ? 1/3 THU#2 *

Just looking at the examples above a pattern emerges:

[seconds] [minutes] [hours] [day of month] [month] [day of week] [year]

Most of the fields accept special characters:

* - all values
? - no specific value. So what is the difference between * and ? ? The first one selects all values and the second one says “I don’t care about it”. Like with the 0 0 8 ? * MON * example. We put ? into day of month field because which day of the month it will be is not important.
- - for ranges. For example: MON-FRI for selecting workweek days.
, - naming multiple values. For example: MON,FRI for selecting Monday and Friday only
/ - for adding intervals. It is used like x/y which means: “start at x and do it every y times”. For example: 0/10 in seconds field will result in values: 0,10,20,30,40,50. One thing to look out for is that cron uses human notation, so with days, months etc. we start counting from 1, not 0, like in example 2 and 3 above.
L - last. It is allowed in two fields: day of month and day of week. This comes in handy when talking about the last day of February and not having to worry about leap years. It gets even better, and more confusing, when used for example like this: L-1 in the day of month field. Why is it powerful and confusing? Because it doesn’t mean “the last day and day before the last day of the month” ( - is for expressing ranges), but it means day before the last day of the month. So strange, but powerful since it gives the possibility to enumerate on days starting from the end of the month/week.
W - nearest weekday. How is it different from naming MON-FRI? It returns the nearest working day from a given date. So:
- a cron: 0 0 12 4W 9 ? * in the year 2016 will return the 5 of September 2016 12:00 since 4 of September 2016 is Sunday, so Monday is the nearest.
- a cron: 0 0 12 3W 9 ? * in the year 2016 will return the 2 of September 2016 12:00 since 3 of September 2016 is Saturday, so Friday is the nearest.
- a cron: 0 0 12 1W 10 ? * in the year 2016 will return the 3 of October 2016 12:00 despite the fact that it is Saturday. Why? Because it returns the nearest workday in a given month
# - the nth value. Think of it as / but without the iteration. Only allowed in day of week field For example:
- a cron 6#2 used in day of week means second Friday of the week. Why Friday? Because in this case days of week are numbered from 0 (Saturday)

If this looks complicated there is a cron expression builder online and Hangfire has a helper class Hangfire.Cron. Inside Hangfire uses NCrontab (nuget, github).

Ok, lets get back to Hangfire and RecurringJob class. It has three functions (excluding the overrides):

AddOrUpdate

As the name suggests it enables to create or update a recurring job. Without further ado let’s look at the overrides, because there are a few of them:

    
    static void AddOrUpdate(Expression<Action> methodCall, Func<string> cronExpression, TimeZoneInfo timeZone = null, string queue = "default");
    static void AddOrUpdate(Expression<Action> methodCall, string cronExpression, TimeZoneInfo timeZone = null, string queue = "default");
    
    static void AddOrUpdate<T>(Expression<Action<T>> methodCall, Func<string> cronExpression, TimeZoneInfo timeZone = null, string queue = "default");
    static void AddOrUpdate<T>(Expression<Action<T>> methodCall, string cronExpression, TimeZoneInfo timeZone = null, string queue = "default");
    
    static void AddOrUpdate(string recurringJobId, Expression<Action> methodCall, Func<string> cronExpression, TimeZoneInfo timeZone = null, string queue = "default");
    static void AddOrUpdate(string recurringJobId, Expression<Action> methodCall, string cronExpression, TimeZoneInfo timeZone = null, string queue = "default");
    
    static void AddOrUpdate<T>(string recurringJobId, Expression<Action<T>> methodCall, Func<string> cronExpression, TimeZoneInfo timeZone = null, string queue = "default");
    static void AddOrUpdate<T>(string recurringJobId, Expression<Action<T>> methodCall, string cronExpression, TimeZoneInfo timeZone = null, string queue = "default");

    static void AddOrUpdate(string recurringJobId, Expression<Func<Task>> methodCall, Func<string> cronExpression, TimeZoneInfo timeZone = null, string queue = "default");
    static void AddOrUpdate(string recurringJobId, Expression<Func<Task>> methodCall, string cronExpression, TimeZoneInfo timeZone = null, string queue = "default");

    static void AddOrUpdate<T>(string recurringJobId, Expression<Func<T, Task>> methodCall, Func<string> cronExpression, TimeZoneInfo timeZone = null, string queue = "default");
    static void AddOrUpdate<T>(string recurringJobId, Expression<Func<T, Task>> methodCall, string cronExpression, TimeZoneInfo timeZone = null, string queue = "default");

The total count is twelve functions, so what exactly differentiates then?

generic vs. nongeneric
requiring or not job Id. When using the overload without id, one will be generated as a concatenation of object type name and method called.

From my personal experience specifying the id is a better practice. It makes code refactoring easier because we don’t have to worry that we will lose the ability to alter the job because of refactored name.

cron expression passed as a string or as a function returning a string (string vs. Func<string>). I personally don’t see the use case for a function returning a string cron expression, but maybe it will be useful to someone.
enabling scheduling synchronous or asynchronous jobs (Expression<Action> vs. Expression<Func<Task>> or Expression<Action<T>> vs. Expression<Func<T,Task>> for generic overrides)

They all have:

TimeZoneInfo - because knowing the timezone is important when talking about hours
string queue - as the name suggests jobs can be scheduled onto different queues, but I will leave it for now since it will be covered in future posts.

Trigger

With function signature looking like:

    static void Trigger(string recurringJobId);

There is no doubt what it does - triggers the job. This of curse does not influence next scheduled triggering since they are scheduled in absolute time, not relative from the last execution.

RemoveIfExists

    static void RemoveIfExists(string recurringJobId);

Not much to say. It removes a scheduled job. Again, the job has to have an id. And it will remove only the trigger. Any enqueued jobs will execute.

Corner cases

What happens if I schedule the job to start in the past?

The job will be triggered when the next interval happens. Past jobs won’t be queued.

What happens if during the time the job should be triggered there was no processing server?

It depends:

if the job was enqueued at least once and during downtime it should be triggered, then it will be enqueued.
if there was never any triggering, it will be triggered next time in the future according to cron expression.

This logic is in TryScheduleJob function in RecurringJobScheduler.

What is the minimal precision for triggering a job?

One minute. It is achieved by a busy with a one-second sleep wait in EveryMinuteThrottler.

Don't do it now! Part 5. Hangfire details - job continuation with ContinueWith

Tue, 30 Aug 2016 00:00:00 +0200

Previous post covered almost all functions in BackgroundJob class except for ContinueWith functions family. So here we go :)

The fact that it has the same name as a System.Threading.Tasks.Task function is not without a coincidence, or at least I hope so. This method allows chaining jobs where one will be enqueued when the previous finishes. To repeat - the job won’t be executed, but enqueued. So it will go at the end of the queue. So lets look at the function and overrides signature:

static string ContinueWith(string parentId, Expression<Action> methodCall);
static string ContinueWith(string parentId, Expression<Action> methodCall, JobContinuationOptions options);
static string ContinueWith(string parentId, Expression<Func<Task>> methodCall, JobContinuationOptions options = JobContinuationOptions.OnlyOnSucceededState);    
static string ContinueWith<T>(string parentId, Expression<Action<T>> methodCall);
static string ContinueWith<T>(string parentId, Expression<Action<T>> methodCall, JobContinuationOptions options);
static string ContinueWith<T>(string parentId, Expression<Func<T, Task>> methodCall, JobContinuationOptions options = JobContinuationOptions.OnlyOnSucceededState);

The API enables to fire multiple methods when one finishes, but not doing the opposite. This scenario was recently covered by Batches, but it is a topic for a separate post, and they are available only in the paid version. So let’s have a look at the function and their overrides:

So let’s look at what differentiates the overrides:

generic vs. nongeneric
synchronous vs. asynchronous. This time, they are not exactly equal because methods differ in the fact that there are two versions for the synchronous override. One with JobContinuationOptions and one without. The asynchronous only has one with default initialization. This was probably done to keep backward compatibility (reflection, assembly binding), and since async support was implemented after ContinueWith.

We looked at the differences, let’s look at the similarities:

parentId - remember the unique job id returned by all the functions from enqueue part of the API? This is one of the cases when it comes in handy.
JobContinuationOptions - this is an enum and has two options:
- OnAnyFinishedState - the default one, meaning the job will execute regardless of whatever parent finishes with success, or throws an error.
- OnlyOnSucceededState - the job will fire only when parent succeeded (didn’t throw any exception, or didn’t timeout)
the return value is of curse a unique job id, so we can chain many jobs.

Continuation can introduce some edge cases, here are those came into my mind:

Will continuations be executed once more if we enqueue the parent job that ended successfully?

The short answer is no. Continuation jobs will be enqueued only if they are in AwaitingState (special state for continuation jobs in Hangfire state machine).

What happens to continuations scheduled to execute on success when the parent fails and after retry succeeded?

To put it simply: is it possible for the parent job to finish and not enqueue continuations? No. They are done in one transaction.

What if we continue on a job that already executed?

Continuations will be enqueued immediately.

Why should I use ContinueWith if I can enqueue continuation job at the end of parent job?

To keep the code clean. Hangfire is just a way to execute functions, and in most cases, there is no need for it know, is it running from Hangfire or from a web request. It is just a matter of keeping code clean.

Can a parent job pass parameters to continuations?

No. If that’s the case I would recommend TPL Dataflow or some agent system like Akka.NET or Orleans. Another option is to enqueue the child job at the end of parent job, but this can get messy.

Don't do it now! Part 4. Hangfire details - dashboard, retries and job cancellation

Mon, 22 Aug 2016 00:00:00 +0200

This part will cover few small topics:

dashboard
retries
more technical part of the Hangfire.BackgroundJob class API
job cancellation

Dashboard

Let’s start with the administrative dashboard because it gives a good background for the rest of the post. It greets us with more or less this view: This is the main view of what the server is doing, and how well (are the jobs failing) The more interesting part is the next view: There’s a bit more detail about jobs in all states. They are self explanatory, maybe except the awaiting, but I will cover this in the next post. We can go as deep as the state of the specific job, which will look like this:

And this brings us to the main motives of this post:

auto retries
the ability to requeue a job
the ability to delete a job
job cancellation

Auto retry

This one is easy. Hangfire will auto retry every job that failed (timeouted or thrown an exception) configurable amount of times (10 by default). Each retry is an equivalent to normal enqueuing, so it lands at the end of the queue.

Manual requeue

This can be done in several ways:

on the job page
on the jobs page (allows requeuing multiple jobs).
manually using Requeue from Hangfire.BackgroundJob :

    static bool Requeue(string jobId);
    static bool Requeue(string jobId,string fromState);

Remember the unique job id returned by schedule methods from previous post? This is one of the places where it becomes useful. The overload has an additional parameter, fromState which is a fail switch. Job will be deleted only if it is in this exact state.

Delete a job

A similar story to requeuing. It can be done from the UI or with the API:

    static bool Delete(string jobId);
    static bool Delete(string jobId, string fromState);

No surprises here. The API is very similar as Requeue. This would seem to end the topic of Delete, but one very important question should pop into Your mind:

Just how does Hangfire delete a job?

This question becomes even more interesting when:

we look at IJobCancellationToken in the documentation, which means cancellation is supported.
we see that Hangfire is using System.Threading.CancellationToken.
we take into account that Delate may be called from a different machine than currently executing the job.

Job cancellation

Cancellation can be triggered by two events (and both of the handle it differently):

1. Server shut down

When BackgroundJobServer(github), which is responsible for job execution, is being shut down, it stops processing new messages and triggers Cancel on its CancellationTokenSource. This token is used by the Worker(github) class to create the ServerJobCancellationToken (github) instance that will be injected into the method if it has a parameter of type IJobCancellationToken. Calling ThrowIfCancellationRequested on it, checks the token and throws OperationCanceledException. This exception is recognised as finishing due to issued cancellation. This way the server can gently close currently processing jobs.

2. Job deletetion (job cancellation)

Delete method uses the IBackgroundJobStateChanger (github), implemented by BackgroundJobStateChanger(github), to change the state of the job to Deleted and write it to storage. On this its responsibility finishes.

The rest of the cancellation logic is done in ThrowIfCancellationRequested in ServerJobCancellationToken(github)

	public void ThrowIfCancellationRequested()
	{
	    _shutdownToken.ThrowIfCancellationRequested();
	
	    if (IsJobAborted())
	    {
	        throw new JobAbortedException();
	    }
	}

The first line is checking the CancellationToken mentioned in server shut down case. The second gets the job state from the database and checks if there were any state changes indicating whether it should be cancelled (like changing the state by calling Delete). If yes, JobAbortedException(github) exception is being thrown. This exception is handled in a very similar way as OperationCanceledException (from which it inherits) and also will be recognised as a indication of finishing due to issued cancellation.

Don't do it now! Part 3. Hangfire details - jobs

Wed, 17 Aug 2016 00:00:00 +0200

This part will focus on the basic scheduling API of Hangfire. The easiest way to create a fire and forget job is by using the classHangfire.BackgroundJob and its minimalistic (and this is a complement) API of static functions:

Enqueue

As the name suggests it is the method for enqueuing jobs. It comes in different overrides:

static string Enqueue(Expression<Func<Task>> methodCall);
static string Enqueue(Expression<Action> methodCall);
static string Enqueue<T>(Expression<Func<T, Task>> methodCall);
static string Enqueue<T>(Expression<Action<T>> methodCall);

Two of them are non generic implementations of the generic version, so it comes down to whatever we want to:

schedule an async function Func<T, Task>
schedule a synchronous function Action<T>

What will be important further on is that all this methods return a string which is a unique identifier of the job. It will come in handy in the next posts.

Schedule

As the name suggests - it allows to enqueue a job, but delay its execution by some time period. The overrides are:

static string Schedule(Expression<Func<Task>> methodCall, DateTimeOffset enqueueAt);
static string Schedule(Expression<Func<Task>> methodCall, TimeSpan delay);
static string Schedule(Expression<Action> methodCall, DateTimeOffset enqueueAt);
static string Schedule(Expression<Action> methodCall, TimeSpan delay);
static string Schedule<T>(Expression<Func<T, Task>> methodCall, DateTimeOffset enqueueAt);
static string Schedule<T>(Expression<Func<T, Task>> methodCall, TimeSpan delay);
static string Schedule<T>(Expression<Action<T>> methodCall, TimeSpan delay);
static string Schedule<T>(Expression<Action<T>> methodCall, DateTimeOffset enqueueAt);

A few overrides, but it boils down to:

generic or nongeneric version
asynchronous vs. synchronous (Func<T, Task> vs. Action<T>)
delay it by a period from now, or execute it at specific time in future (TimeSpan delay vs. DateTimeOffset)

Similar to the Enqueue this functions also returns a string identifier of a job.

This two function are enough to start to worry about corner cases when using SqlServerStorage (probably the most common usage), so here we go:

Corner cases

Does Hangfire using SQLServer storage requires an opened transaction?

No Hangfire will create its own transaction scope and manage it. Code handling it an be found in Hangfire.SqlServer.SqlServerStorage.CreateTransaction.

If there is an opened transaction will Hangfire use it?

Yes Hangfire will enlist to an opened transaction, and use it. This means that the job will be enqueued only if rest of the process succeeded and we commit the transaction.

Does Hangfie using SQLServer storage support distributed transactions?

Yes. Again the code can be found in Hangfire.SqlServer.SqlServerStorage.CreateTransaction.

Can I rely on object state?

No. Only things being serialized are:

object type
called method
passed parameters

This means that we can’t rely on any state that the object had at the time of scheduling its function because it won’t be recreated. This is why I wrote that Hangfire enables to schedule functions. Because in C# functions aren’t first class citizens, but they are tied to a object instance you have to be more careful. To give an example of how a “Hello world” job looks in storage:

{  
   "Type":"System.Console, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089",
   "Method":"WriteLine",
   "ParameterTypes":"[\"System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\"]",
   "Arguments":"[\"\\\"Hello, world!\\\"\"]"
}

Based on this info Hangfire will recreate the object and execute the function. This also brings us to a small performance tip to keep in mind: keep your parameters small, and be sure to know how they will be serialized

Don't do it now! Part 2. Background tasks, job queuing and scheduling with Hangfire

Mon, 08 Aug 2016 00:00:00 +0200

In the previous post I’ve written about why I think the ability to schedule tasks for later execution is a fundamental technical feature, but also a must have from business’ point of view. We are passed the whys, so let’s get to the hows. The answer is simple - Hangfire. I’ve written about it here, here and here, so yeah, you guessed it, I like it. Hangfire is an amazing library. It has shown it’s value in my pet project (cookit.pl) and in a huge ERP system that we are building at work, where we replaced Quartz.NET with it and never looked back.

Why do I like it so much?

1. Scheduling

It enables:

to fire(enqueue) and forget a job:

	BackgroundJob.Enqueue(() => Console.WriteLine("Simple!"));`

to schedule a one time job using a cron expression

	BackgroundJob.Schedule(() => Console.WriteLine("Reliable!"), TimeSpan.FromDays(7));

and to schedule a recurring job also with a cron expression

	RecurringJob.AddOrUpdate(() => Console.WriteLine("Transparent!"), Cron.Daily)

A side note: Cron expressions allow to express almost any time span, and using them is a standard in most task scheduling cases (Quartz.NET, Unix, TeamCity to name a few). There is a good Wikipedia page explaining the syntax. Think of them as regular expressions for time.

I personally fell in love with the simplicity of this API, and especially with the fact that it takes a function to execute, not an object. Although this examples are simple lambdas Hangfire allows to schedule executing methods on almost any objects, but this is a topic for another post. Probably the next one:)

2. Persistence

The key point in procrastinating tasks is to not do them now, but to have certainty that they will be executed. That is why persisting jobs in a database is a key feature. And Hangfire has a hand of stores to persist in:

SQL Server (nuget package)
PostgreSql (nuget package)
Redis (nuget package, but only in paid pro version)

And configuring any of them is as simple as scheduling a job:

JobStorage.Current = new SqlServerStorage("HangfireConnectionString")

3. Job execution

Persisting a job in a database gives the possibility for another process to execute it. And Hangfire does just that. Of course the process executing the job has to have all the assemblies needed to execute the code, but this is just another argument for not having everything jammed into one project. Hangfire can be hosted by a:

console application
Windows service
IIS website

And configuring it is as easy as scheduling a job:

GlobalConfiguration.Configuration.UseSqlServerStorage("HangfireConnectionString");
using (var server = new BackgroundJobServer())
{
    Console.WriteLine("Hangfire Server started. Press any key to exit...");
    Console.ReadKey();
}

4. Monitoring

For me the ability to have a graphic interface to see current jobs, queues, errors and processing servers was a must-have when choosing a library for a simple reason - I needed it and didn’t want to write it myself. And Hangfire has just that:

The dashboard is fully served be Hangfire (there is no need to add any CSS or JavaScript files) and configuring it is a one liner extension on OWIN IAppBuilder:

	app.UseHangfireDashboard();

This was a glimpse of what this library can do. It’s not always bells and whistles, but I’ve never regretted choosing it can’t imagine working without the ability to schedule tasks. In the next post - a better look into jobs.

Don't do it now.

Sun, 24 Jul 2016 00:00:00 +0200

This is a post about the importance of not doing stuff right here and right now. About doing stuff later, delaying them and in a perfect world making someone else do them. But I will still stick to IT, don’t expect any life tips or guidance.

A mindset I see constantly in system design is doing the right thing right now, and even more stuff after that. Like sending an email, doing a REST call, performing some CPU intense calculations, calling an external resource. We are doing those things where they should be from the logical point of view. So, if we are given a task like registering an user that sound more or less like that:

"Verify the required fields, check if a user with such email doesn't exist if everything is OK save it to the database, and send an confirmation email."

We will:

verify required fields
check for a duplicate
save it to the database
send the confirmation email

And for me this is insane and even more - wrong. Wrong as not implementing what the requirement told. For me it should be:

verify required fields
check for a duplicate
save it to the database
schedule sending the confirmation email

Did you spot the difference? It is in the last step. My point is that we should schedule sending the email and not send it. Why schedule not do it now? There are many reasons, but to name a few:

in a web application it makes the request faster (scheduling is faster then sending an email)
processing of jobs can be done on different server.
it is easy to scale job processing servers
it is more fail resilient (easy to retry)
it allows to batch operations
it is easier to monitor

But for me there is also a bigger point to make here. I believe it is more tha n a technical detail, or a performance enhancer. I think that only the second version is implementing the requirement properly. This is because failure to send the email now should not block the registration. It’s a technical problem, not a business one and it should be dealt with without the user knowing. Imagine getting an error from registering saying “SMTP port not responding”. Most of us will say, or I hope so at least, “This is obvious”. But most of us spend our days implementing more complicated processes, where it is not that obvious, and we don’t dare to ask:

"Does it need to be done now? It is an integral part of the process? Do you need it in real time?"

Why? Maybe we feel it implies that the system isn’t fast enough or we can’t code it to work fast enough. Those may be, but my bet is on that we assume it is an integral part and we don’t ask:

should the whole process fail if this part does?
can this action fail for any business reason?
how well should this scale? Long monolithic processes don’t scale.

If you hear “Yes” to any of this questions then do it later.

Keep in mind that latency between scheduling a job and picking it up for execution shouldn’t be greater than 5 second for most cases, because then you are still in “real time” frame of most use cases.

The cost of garbage collection. Part 2 - rectification.

Sat, 16 Jul 2016 00:00:00 +0200

This is a rectification for the previous post about the cost of garbage collection. If You didn’t read it give it a try and check if You can spot the bug/mistake.

Like Konrad pointed out in his comment not all objects were in generation 0 as I assumed. This is partly connected to the fact that .NET, seeing rapid need for memory in Setup, will try to collect some of them, calling garbage collection, but I make the matters worse by calling GC.Collect in the end. So making sure that objects created during Setup will be in generation 2 (in my defence it was left over after a bit different take on this problem).

This is the proper code (it also was committed to the github repo):

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Runtime.CompilerServices;
using NBench;

namespace Tests
{
    public class GCCollection
    {
        private readonly string[] _messages =
        {
            "Gen 0 ticks:       ",
            "Gen 0+1 ticks:     ",
            "Gen 0+1+2 ticks:   "
        };

        private readonly int _objectNumber = 20*1000;

        [PerfSetup]
        public void Setup(BenchmarkContext context)
        {
            Console.WriteLine("Objects generated.");
            GenerateObjects(_objectNumber*2);
            GC.Collect(2, GCCollectionMode.Forced, true, true);
        }

        [PerfBenchmark(Description = "Gen 2 collection with nothing", NumberOfIterations = 1, RunMode = RunMode.Iterations, TestMode = TestMode.Test)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen0, MustBe.ExactlyEqualTo, 3d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen1, MustBe.ExactlyEqualTo, 2d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen2, MustBe.ExactlyEqualTo, 1d)]
        public void Gen2CollectionWithNothing()
        {
            GC.Collect(2, GCCollectionMode.Forced, true,true);

            var sw = Stopwatch.StartNew();

            RunGCAndCheck(0, sw);/* Run collect for gen 0 - collect everything.*/

            RunGCAndCheck(1, sw);/* Nothing should be here.*/

            RunGCAndCheck(2, sw);/* Nothing should be here.*/
        }

        [PerfBenchmark(Description = "Gen 0 collection", NumberOfIterations = 1, RunMode = RunMode.Iterations,TestMode = TestMode.Test)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen0, MustBe.ExactlyEqualTo, 1d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen1, MustBe.ExactlyEqualTo, 0.0d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen2, MustBe.ExactlyEqualTo, 0.0d)]
        public void Gen0Collection()
        {
            var list = GenerateObjects(_objectNumber);
            list = null;

            var sw = Stopwatch.StartNew();

            RunGCAndCheck(0, sw);/* Run collect for gen 0 - collect everything.*/
        }


        [PerfBenchmark(Description = "Gen 1 collection", NumberOfIterations = 1, RunMode = RunMode.Iterations,TestMode = TestMode.Test)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen0, MustBe.ExactlyEqualTo, 2.0d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen1, MustBe.ExactlyEqualTo, 1.0d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen2, MustBe.ExactlyEqualTo, 0.0d)]
        public void Gen1Collection()
        {
            var list = GenerateObjects(_objectNumber);
            var sw = Stopwatch.StartNew();
            RunGCAndCheck(0, sw);/* Can't collect anything. Move it to gen 1.*/

            list = null;
            RunGCAndCheck(1, sw);/* Run collect for gen 0 - it is empty. Run gen 1 collection - collect everything.*/
        }

        [PerfBenchmark(Description = "Gen 2 collection", NumberOfIterations = 1, RunMode = RunMode.Iterations,TestMode = TestMode.Test)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen0, MustBe.ExactlyEqualTo, 3.0d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen1, MustBe.ExactlyEqualTo, 2.0d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen2, MustBe.ExactlyEqualTo, 1.0d)]
        public void Gen2Collection()
        {
            var list = GenerateObjects(_objectNumber);
            var sw = Stopwatch.StartNew();

            RunGCAndCheck(0, sw);/* Can't collect anything. Move it to gen 1.*/

            RunGCAndCheck(1, sw);/* Run collect for gen 0 - it is empty. Run gen 1 collection - collect the list elements.*/

            list = null;
            RunGCAndCheck(2, sw);/* Run collect for generation 0 - it is empty. Run collect for generation 1 - it is empty. Run collect for generation 2 - collect everything.*/
        }

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        private void RunGCAndCheck(int generationNumber, Stopwatch sw)
        {
            sw.Restart();
            GC.Collect(generationNumber, GCCollectionMode.Forced, true,true);
            Console.WriteLine(_messages[generationNumber] + sw.ElapsedTicks);
        }

        private List<MyGCTestClass> GenerateObjects(long count)
        {
            var ret = new List<MyGCTestClass>();
            for (var i = 0; i < count; i++)
            {
                ret.Add(new MyGCTestClass(Guid.NewGuid().ToString()));
            }
            return ret;
        }

        private class MyGCTestClass
        {
            private readonly string Text;

            public MyGCTestClass(string text)
            {
                Text = text;
            }

            public MyGCTestClass(MyGCTestClass source)
            {
                Text = source.Text;
            }
        }
    }
}

What has changed:

This code generates ‘only’ 20 thousand objects because it is the maximal number that does not trigger garbage collection at my machine.
Setup generates 40 thousand objects to make sure .NET will preallocate enough memory for the process so when tests are run it won’t need to ask the OS for additional memory. It also triggers GC.Collect(2) after it to make sure that all objects will be collected and the test will have a clean run.
GenerateObjects is called in every test to make sure I measure any GC.Collect that may affect in which generation objects are in.

And the results (I spared the NBench info, so just the numbers).

Gen0Collection

Gen 0 ticks: 6167

Gen1Collection

Gen 0 ticks: 6153 Gen 0+1 ticks: 5059

Gen2Collection

Gen 0 ticks: 4963 Gen 0+1 ticks: 4772 Gen 0+1+2 ticks: 2955

Gen2CollectionWithNothing

Gen 0 ticks: 171 Gen 0+1 ticks: 140 Gen 0+1+2 ticks: 1365

Final thoughts:

GC.Collect(1) takes a bit less time (keep in mind we are talking in ticks, not even miliseconds!) than GC.Collect(0). For me it was not intuitive, and strange since calling GC.Collect(1) also calls GC.Collect(0).
And finally to answer Michał’s question. I think generation 0 and generation 1 algorithms are a bit different, and definitely generation 2 uses different, more complicated algorithm.

The cost of garbage collection

Tue, 12 Jul 2016 00:00:00 +0200

Update There is a error in the first version of this code, and the tests are in fact always operating in a situation that almost all object are in generation 2. The answer is in the next post

In the previous post I’ve promised to write how to have differentiate the number of workers in Hangfie, but in the comments Michał asked one interesting question - “Is generation 1 collection more expensive then generation 0 collection?”. Going further how does generation 2 collection fit into it?

I always believed that running GC.Collect is more expensive the deeper we go. So GC.Collect(0) is the least expensive, and GC.Collect(2) is most expensive. This opinion was reinforced by articles I remember reading and general opinion that .NET is optimized for fast young object collection.

But after a moment of thought I came to conclusion that the same/very similar algorithm could be used for every garbage collection generation. I started a conversation with my team at work, and after a while they came to more or less the same conclusion:

We don’t know. And to make matters worse there seem to be articles stating opposite opinions in this topic.

The only thing left was to check.
To check I needed couple of things:

1. Large number of object that can be collected.

I’ve assumed that increasing the number of objects will influence garbage collection time - this is quite logical assumption. One thing to have in mind is that those have to be objects (instances of classes) not structs. Why? Because structs and classes are collected in different ways. Classes are collected by counting references during GC.Collect, and structs, since you can’t have a reference to it, are collected when they loose scope, or during collection of object that contains it. More on this can be read for example here So my class looks like this:

private class MyGCTestClass
{
    private readonly string Text;

    public MyGCTestClass(string text)
    {
        Text = text;
    }

    public MyGCTestClass(MyGCTestClass source)
    {
        Text = source.Text;
    }
}

And generating code looks like this:

private List<MyGCTestClass> GenerateObjects(long count)
{
    var ret = new List<MyGCTestClass>();
    for (var i = 0; i < count; i++)
    {
        ret.Add(new MyGCTestClass(Guid.NewGuid().ToString()));
    }
    return ret;
}

2. A way to trigger garbage collection

Since .NET has two garbage collectors (client and server) and I wanted to measure time impact so using the blocking one (client) was the only logical choice. A quick entry in app/web.config:

<configuration>
    <runtime>
        <gcServer enabled="false" />
    </runtime>
</configuration>

Also GC.Collect has several overloads. The four parameter overload enables to:

set the collection as blocking
force garbage collection to actually execute
force compacting (So not only marking objects for deletion, but also moving the rest together to minimize memory fragmentation ) The call will look like this:

GC.Collect(generationNumber, GCCollectionMode.Forced, true, true);

3. A way to check how many times collection was executed

Because .NET runtime can trigger garbage collection on its own I had to be sure that the only garbage collection was that triggered by me.
This can be done by using GC.RegisterForFullGCNotification, but performance benchmarks like NBench or BenchmarkDotNet have it nicely wrapped. I’ve decided to go with NBench and it’s GcTotalAssertion attribute.

So my test class looked like this:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Runtime.CompilerServices;
using NBench;

namespace Tests
{
    public class GCCollection
    {
        private readonly string[] _messages =
        {
            "Gen 0 ticks:       ",
            "Gen 0+1 ticks:     ",
            "Gen 0+1+2 ticks:   "
        };

        private List<MyGCTestClass> _list;

        [PerfSetup]
        public void Setup(BenchmarkContext context)
        {
            _list = GenerateObjects(20*1000*1000);
            Console.WriteLine("Objects generated.");
            GC.Collect(2, GCCollectionMode.Forced, true,true);/*To be sure everything that can will be collected.*/
        }

        [PerfBenchmark(Description = "Gen 0 collection", NumberOfIterations = 1, RunMode = RunMode.Iterations,TestMode = TestMode.Test)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen0, MustBe.ExactlyEqualTo, 1d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen1, MustBe.ExactlyEqualTo, 0.0d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen2, MustBe.ExactlyEqualTo, 0.0d)]
        public void Gen0Collection()
        {
            _list = null;

            var sw = Stopwatch.StartNew();

            RunGCAndCheck(0, sw);/* Run collect for gen 0 - collect everything.*/
        }


        [PerfBenchmark(Description = "Gen 1 collection", NumberOfIterations = 1, RunMode = RunMode.Iterations,TestMode = TestMode.Test)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen0, MustBe.ExactlyEqualTo, 2.0d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen1, MustBe.ExactlyEqualTo, 1.0d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen2, MustBe.ExactlyEqualTo, 0.0d)]
        public void Gen1Collection()
        {
            var sw = Stopwatch.StartNew();
            RunGCAndCheck(0, sw);/* Can't collect anything. Move it to gen 1.*/

            _list = null;
            RunGCAndCheck(1, sw);/* Run collect for gen 0 - it is empty. Run gen 1 collection - collect everything.*/
        }

        [PerfBenchmark(Description = "Gen 2 collection", NumberOfIterations = 1, RunMode = RunMode.Iterations,TestMode = TestMode.Test)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen0, MustBe.ExactlyEqualTo, 3.0d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen1, MustBe.ExactlyEqualTo, 2.0d)]
        [GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen2, MustBe.ExactlyEqualTo, 1.0d)]
        public void Gen2Collection()
        {
            var sw = Stopwatch.StartNew();

            RunGCAndCheck(0, sw);/* Can't collect anything. Move it to gen 1.*/

            RunGCAndCheck(1, sw);/* Run collect for gen 0 - it is empty. Run gen 1 collection - collect the list elements.*/

            _list = null;
            RunGCAndCheck(2, sw);/* Run collect for generation 0 - it is empty. Run collect for generation 1 - it is empty. Run collect for generation 2 - collect everything.*/
        }

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        private void RunGCAndCheck(int generationNumber, Stopwatch sw)
        {
            sw.Restart();
            GC.Collect(generationNumber, GCCollectionMode.Forced, true,true);
            Console.WriteLine(_messages[generationNumber] + sw.ElapsedTicks);
        }

        private List<MyGCTestClass> GenerateObjects(long count)
        {
            var ret = new List<MyGCTestClass>();
            for (var i = 0; i < count; i++)
            {
                ret.Add(new MyGCTestClass(Guid.NewGuid().ToString()));
            }
            return ret;
        }

        private class MyGCTestClass
        {
            private readonly string Text;

            public MyGCTestClass(string text)
            {
                Text = text;
            }

            public MyGCTestClass(MyGCTestClass source)
            {
                Text = source.Text;
            }
        }
    }
}

Some things to point out:

RunGCAndCheck has [MethodImpl(MethodImplOptions.AggressiveInlining)] to inline the method and thus minimize the chance garbage collection will be called after it exits.
Setup will be called before every test because of the PerfSetup attribute.
Running GC.Collect(n) will cause garbage collection on all generation up to and with n-th generation.

Running the test in NBench

First thing to note is that NBench requires it’s own runner (it is avilable on nuget). The second is that it runs every test twice as a warm up to minimize for such things like time spend by .NET allocating memory, and JITing the code.

Gen0Collection (objects collected in generation 0 collection):

------------ STARTING Tests.GCCollection+Gen0Collection ----------
Objects generated.
Gen 0 ticks:       21257
Objects generated.
Gen 0 ticks:       42541
--------------- BEGIN WARMUP ---------------
Elapsed: 00:00:00.0171519
TotalCollections [Gen0] - collections: 1,00 ,collections: /s 58,30 , ns / collections: 17 151 979,67
TotalCollections [Gen1] - collections: 0,00 ,collections: /s 0,00 , ns / collections: 17 151 979,67
TotalCollections [Gen2] - collections: 0,00 ,collections: /s 0,00 , ns / collections: 17 151 979,67
--------------- END WARMUP ---------------

Objects generated.
Gen 0 ticks:       47949
--------------- BEGIN RUN ---------------
Elapsed: 00:00:00.0192217
TotalCollections [Gen0] - collections: 1,00 ,collections: /s 52,02 , ns / collections: 19 221 705,61
TotalCollections [Gen1] - collections: 0,00 ,collections: /s 0,00 , ns / collections: 19 221 705,61
TotalCollections [Gen2] - collections: 0,00 ,collections: /s 0,00 , ns / collections: 19 221 705,61
--------------- END RUN ---------------

--------------- RESULTS: Tests.GCCollection+Gen0Collection ---------------
Gen 0 collection
--------------- DATA ---------------
TotalCollections [Gen0]: Max: 1,00 collections, Average: 1,00 collections, Min: 1,00 collections, StdDev: 0,00 collections
TotalCollections [Gen0]: Max / s: 52,02 collections, Average / s: 52,02 collections, Min / s: 52,02 collections, StdDev / s: 0,00 collections

TotalCollections [Gen1]: Max: 0,00 collections, Average: 0,00 collections, Min: 0,00 collections, StdDev: 0,00 collections
TotalCollections [Gen1]: Max / s: 0,00 collections, Average / s: 0,00 collections, Min / s: 0,00 collections, StdDev / s: 0,00 collections

TotalCollections [Gen2]: Max: 0,00 collections, Average: 0,00 collections, Min: 0,00 collections, StdDev: 0,00 collections
TotalCollections [Gen2]: Max / s: 0,00 collections, Average / s: 0,00 collections, Min / s: 0,00 collections, StdDev / s: 0,00 collections

--------------- ASSERTIONS ---------------
[PASS] Expected TotalCollections [Gen0] to must be exactly 1,00 collections; actual value was 1,00 collections.
[PASS] Expected TotalCollections [Gen1] to must be exactly 0,00 collections; actual value was 0,00 collections.
[PASS] Expected TotalCollections [Gen2] to must be exactly 0,00 collections; actual value was 0,00 collections.

------------ FINISHED Tests.GCCollection+Gen0Collection ----------

The test itself is the part after END WARMUP. And according do this run generation 0 collection collecting all the objects lasts 47949 ticks.

Gen1Collection (objects collected in generation 1 collection)

------------ STARTING Tests.GCCollection+Gen1Collection ----------
Objects generated.
Gen 0 ticks:       45994
Gen 0+1 ticks:     546
Objects generated.
Gen 0 ticks:       45196
Gen 0+1 ticks:     522
--------------- BEGIN WARMUP ---------------
Elapsed: 00:00:00.0194638
TotalCollections [Gen0] - collections: 2,00 ,collections: /s 102,75 , ns / collections: 9 731 939,08
TotalCollections [Gen1] - collections: 1,00 ,collections: /s 51,38 , ns / collections: 19 463 878,17
TotalCollections [Gen2] - collections: 0,00 ,collections: /s 0,00 , ns / collections: 19 463 878,17
--------------- END WARMUP ---------------

Objects generated.
Gen 0 ticks:       47584
Gen 0+1 ticks:     472
--------------- BEGIN RUN ---------------
Elapsed: 00:00:00.0206490
TotalCollections [Gen0] - collections: 2,00 ,collections: /s 96,86 , ns / collections: 10 324 530,97
TotalCollections [Gen1] - collections: 1,00 ,collections: /s 48,43 , ns / collections: 20 649 061,95
TotalCollections [Gen2] - collections: 0,00 ,collections: /s 0,00 , ns / collections: 20 649 061,95
--------------- END RUN ---------------

--------------- RESULTS: Tests.GCCollection+Gen1Collection ---------------
Gen 1 collection
--------------- DATA ---------------
TotalCollections [Gen0]: Max: 2,00 collections, Average: 2,00 collections, Min: 2,00 collections, StdDev: 0,00 collections
TotalCollections [Gen0]: Max / s: 96,86 collections, Average / s: 96,86 collections, Min / s: 96,86 collections, StdDev / s: 0,00 collections

TotalCollections [Gen1]: Max: 1,00 collections, Average: 1,00 collections, Min: 1,00 collections, StdDev: 0,00 collections
TotalCollections [Gen1]: Max / s: 48,43 collections, Average / s: 48,43 collections, Min / s: 48,43 collections, StdDev / s: 0,00 collections

TotalCollections [Gen2]: Max: 0,00 collections, Average: 0,00 collections, Min: 0,00 collections, StdDev: 0,00 collections
TotalCollections [Gen2]: Max / s: 0,00 collections, Average / s: 0,00 collections, Min / s: 0,00 collections, StdDev / s: 0,00 collections

--------------- ASSERTIONS ---------------
[PASS] Expected TotalCollections [Gen0] to must be exactly 2,00 collections; actual value was 2,00 collections.
[PASS] Expected TotalCollections [Gen1] to must be exactly 1,00 collections; actual value was 1,00 collections.
[PASS] Expected TotalCollections [Gen2] to must be exactly 0,00 collections; actual value was 0,00 collections.

------------ FINISHED Tests.GCCollection+Gen1Collection ----------

The difference between 47584 ticks for generation 0 and 472 ticks for generation 1 collection is big (more than 100 times), but keep reading because the best is yet to come.

Gen2Collection (objects collected in generation 2 collection)

------------ STARTING Tests.GCCollection+Gen2Collection ----------
Objects generated.
Gen 0 ticks:       46034
Gen 0+1 ticks:     559
Gen 0+1+2 ticks:   1461791
Objects generated.
Gen 0 ticks:       37438
Gen 0+1 ticks:     451
Gen 0+1+2 ticks:   793937
--------------- BEGIN WARMUP ---------------
Elapsed: 00:00:00.3300270
TotalCollections [Gen0] - collections: 3,00 ,collections: /s 9,09 , ns / collections: 110 009 021,88
TotalCollections [Gen1] - collections: 2,00 ,collections: /s 6,06 , ns / collections: 165 013 532,82
TotalCollections [Gen2] - collections: 1,00 ,collections: /s 3,03 , ns / collections: 330 027 065,65
--------------- END WARMUP ---------------

Objects generated.
Gen 0 ticks:       36631
Gen 0+1 ticks:     524
Gen 0+1+2 ticks:   1358065
--------------- BEGIN RUN ---------------
Elapsed: 00:00:00.5528463
TotalCollections [Gen0] - collections: 3,00 ,collections: /s 5,43 , ns / collections: 184 282 119,04
TotalCollections [Gen1] - collections: 2,00 ,collections: /s 3,62 , ns / collections: 276 423 178,56
TotalCollections [Gen2] - collections: 1,00 ,collections: /s 1,81 , ns / collections: 552 846 357,12
--------------- END RUN ---------------

--------------- RESULTS: Tests.GCCollection+Gen2Collection ---------------
Gen 2 collection
--------------- DATA ---------------
TotalCollections [Gen0]: Max: 3,00 collections, Average: 3,00 collections, Min: 3,00 collections, StdDev: 0,00 collections
TotalCollections [Gen0]: Max / s: 5,43 collections, Average / s: 5,43 collections, Min / s: 5,43 collections, StdDev / s: 0,00 collections

TotalCollections [Gen1]: Max: 2,00 collections, Average: 2,00 collections, Min: 2,00 collections, StdDev: 0,00 collections
TotalCollections [Gen1]: Max / s: 3,62 collections, Average / s: 3,62 collections, Min / s: 3,62 collections, StdDev / s: 0,00 collections

TotalCollections [Gen2]: Max: 1,00 collections, Average: 1,00 collections, Min: 1,00 collections, StdDev: 0,00 collections
TotalCollections [Gen2]: Max / s: 1,81 collections, Average / s: 1,81 collections, Min / s: 1,81 collections, StdDev / s: 0,00 collections

--------------- ASSERTIONS ---------------
[PASS] Expected TotalCollections [Gen0] to must be exactly 3,00 collections; actual value was 3,00 collections.
[PASS] Expected TotalCollections [Gen1] to must be exactly 2,00 collections; actual value was 2,00 collections.
[PASS] Expected TotalCollections [Gen2] to must be exactly 1,00 collections; actual value was 1,00 collections.

------------ FINISHED Tests.GCCollection+Gen2Collection ----------

The final results

Gen 0 ticks:       36631
Gen 0+1 ticks:     524
Gen 0+1+2 ticks:   1358065

Yes, the difference between generation 2 and 1 is almost 2600 times!

To be sure below are averages from 5 runs:

Gen 0 ticks:       38364
Gen 0+1 ticks:     424
Gen 0+1+2 ticks:   1403400.6

So generation 1 is surprisingly fast, and generation 2 collection is relatively slow (remember those are ticks! Not milliseconds!). But case closed?

Not exactly.
Take a look at this test:

[PerfBenchmark(Description = "Gen 2 collection with nothing", NumberOfIterations = 1, RunMode = RunMode.Iterations, TestMode = TestMode.Test)]
[GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen0, MustBe.ExactlyEqualTo, 3d)]
[GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen1, MustBe.ExactlyEqualTo, 2d)]
[GcTotalAssertion(GcMetric.TotalCollections, GcGeneration.Gen2, MustBe.ExactlyEqualTo, 1d)]
public void Gen2CollectionWithNothing()
{
    _list = null;

    var sw = Stopwatch.StartNew();

    RunGCAndCheck(0, sw);/* Run collect for gen 0 - collect everything.*/

    RunGCAndCheck(1, sw);/* Nothing should be here.*/

    RunGCAndCheck(2, sw);/* Nothing should be here.*/
}

It should collect everything in generation 0 collection, the rest should be close to 0, but here are the results:

Gen 0 ticks:       32872
Gen 0+1 ticks:     358
Gen 0+1+2 ticks:   1297617

So yeah. Basically no difference from previous test. So this post should be named “The cost of GC - a failed try”. I don’t know what to make of it. I have some ideas what to change and what to check to be more sure that those numbers are valid.

If anyone has any suggestions I’ve put this code in a repo on github, any pull requests are welcomed.

GC can kill You. Practical GC performance counters in .NET

Mon, 30 May 2016 00:00:00 +0200

One of the steps in cookit is calculating similar recipes. This is what you can see on the left on the recipe page like this

For the sake of clarity and manageability it’s scheduled as separate Hangfire jobs. Because cookit is running 5 workers, so similarities are calculated for 5 websites concurrently.
The process uses cosine similarity, so it allocates a huge list at start and calculates similarities. A very CPU heavy operation.

So some time after triggering all recipes recalculation I saw this in Hangfire console.

The process can take a while, but on average it lasts couple of minutes. First thing was to check was if the server was doing anything. So fire up Task Manager and here it is:

So, something is happening. To better know what, I’ve fired up Process Explorer(run it as administrator!) and in the Properties > .NET Performance tab I saw this.

A side note.
Process Explorer is an element of Sysinternals Suite. This is the must have for any developer using Windows. They don’t need installation and weight less than 16MB. I always place them under C:\tools

I really like this tab, because it shows all the main performance counters for .NET programs. And in this case it also worked. Spending 43% (in the peeks) in GC is not a good thing.

Just to be sure that high GC isn’t caused by another issue, like exceptions let’s look at .NET CLR Exceptions tab:

More than I would want, but:

those are all exceptions thrown during the lifetime of the process and earlier it has done some extraction jobs and they can cause exceptions.
those are also first level exceptions. So also the caught ones. Some libraries use exceptions as way to control logic, and this can be seen on this tab. It can be a major performance hit.
under heavy load Hangfrie can throw some sql timeout exceptions.

Process Explorer only shows snapshots. To see how it looks over time fire up Performance Monitor (perfmon.exe - it is installed on every Windows) and add .NET CLR Memory % Time in GC performance counter (using the green plus).

Well it looks really bad. To say the least.
This only gives a very high level view. To have a better look let’s add more performance counters:

.NET CLR Memory # Gen 0 Collections (how many times Gen 0, Gen 1 and Gen 2 collection was called. It is a sum of all Gen collections count not only Gen 0)
.NET CLR Memory \ # Gen 1 Collections (Sum of Gen 1 and Gen 2 collections called.)
.NET CLR Memory \ # Gen 2 Collections
.NET CLR Memory \ % Time in GC

and we see this:

So a lot of Gen 0 and Gen 1 collections, plus a little bit of Gen 2.
This is showing that .NET is under memory pressure and it’s trying to free memory. Because it’s not running a lot of Gen 2 collections means that enough memory is being freed during Gen 0 and 1.
To be sure lets see how big Gen 0 and 1 heaps are (how much memory is allocated on them), and how they behave over time. Lets have a look at those counters:

.NET CLR Memory \ # Gen 0 heap size
.NET CLR Memory \ # Gen 1 heap size
.NET CLR Memory \ # Gen 2 heap size

This almost what I expect how the process would use memory:

It allocates a huge list at start (the Gen 0)
It loops through every recipe, calculates similarity, and stores it in a sorted list. The list is limited to five top similarities. So a lot of objects are being moved into and from the list.

So were is the almost?
Most allocated object should have a short life span, so they should be collected cheaply in Gen 0. But because Gen 1 is this big it means that when GC is triggered it can’t collect them, so they are being moved to Gen 1. Then Gen 1 collection is being triggered, and previously moved objects are being collected. And Gen 1 collection is far more costly. In summary GC is being triggered too often.

To show my point where is the Performance Monitor with only one worker.

You may be wondering where is the Gen 1 line?
It is still there. It’s those few pixels just above the 0 line is the Gen 1 heap size. And this is the same scale as plot above.
To make a point I’ve added time in GC.
A very boring plot. Just as it should be.

But this only shows the problem. Possible solutions are:

Get a better machine with more RAM. This is out of the question, as I already explain why cookit runs on crap
Have one worker. This is also not a good solutions because some jobs are I/O (network) bound. And having one worker is in general not that good (anyone remembers 1 core processors?)

So how to have differentiate the number of workers in Hangfie?
This will be in the next post :)

System.Console is slow

Wed, 25 May 2016 00:00:00 +0200

This post is written more as a warning

Anyone who made any HackerRank problems considering performance has seen this phrase in the assignment: “watch out for slow IO”.
We are used to think about files, databases and such as potentially slow IO, but the Console?
Yes, and you will be amazed how much.

Couple words about the setup.
I am using NLog with file target (for normal logging) and mail target (for total failure, and aggregated reports).
When debugging or profiling I run the process as a console application, so I have a NLog console target. It outputs some basic info and debug messages summing up to exactly 329KB of text.

So here are the numbers of running the application with console target:

And here are the numbers with console target commented:

28 seconds for outputting 329 KB of text.
To be fair, if you output it in one big chunk it will be faster.

To sum it up:

don’t assume something is fast. Measure it!
control your output. If this was a program running normally as a console application with Console.WriteLine thrown all over the code base the change wouldn’t take less than 1 minute.
NLog is a good abstraction over where your logs go.

Local optimizations don't add up

Fri, 20 May 2016 00:00:00 +0200

One of the main processes in cookit is dealing with extracting recipe information from raw html. I know it isn’t the most elegant solution but it is the only universal one.

But to the point.

Every web page goes through a process involving html parsing, stemming, parsing, and n-gram token matching. Then it’s saved to Sql Server and after transformation to Solr. So a lot of string manipulation, math calculations and from time to time mostly 0-gen GC.

In the most pessimistic case this process has to be re-run for every cached website for every page. And to give you a sense of how much data this is, here is the properties window for the folder containing the cached html files:

And since I run on crapp this process can take “a while”. And by that I mean days. It used to be more than a week but after a lot of work put into optimizations it takes now from 3 to 4 days for ~1000 websites.

I do optimization sessions from time to time, or when it feels necessary to. And when is that? There are some performance tests for some of the most critical parts, but the whole process is to complex to have a full blown performance test coverage. But this isn’t an issue since all background jobs are run in the background with Hangfire I get total run times out of the box and even better with a great dashboard:

If those were parsing jobs being shown I would know that I have a big problem.

This will be a story of one of those “from time to time” sessions.

So lets begin and fire up Excel.

↑ This is probably the reaction of most people reading the above, but it is one of the most crucial steps in any optimization work.

Why?

it gives an easy place where I note the run times and current tasks under tests.
it can easily calculate means and any average function.
since those runs usually last for a couple of minutes each in the meantime I do some reading, catching up with RSS feeds and so on. So when the run ends notes in excel let me easily switch context to the task currently under tests.

To be as close to the real run I fire it as a console application and go through the whole process. From reading the files, extraction, to saving to Sql Server and exporting to Solr. This is important because optimizations can speed up one fragment, but slow down the process as a whole.

This is what I saw in the base run :

98 sec
96 sec
130 sec

Now at this moment I could treat the third run as just a glitch, but I decided to do some more runs. And I saw this:

70 sec
69 sec
108 sec
106 sec
109 sec

This meant one thing - I have no baseline. The results had such a huge dispersion that they didn’t mean nothing.

My Excel showed this:

Where:

avg is of curse the most common average - the arithmetic mean. And as anyone will tell you, it doesn’t mean anything without the second.
st dev is the standard deviation which shows how precise the average is. And in my case it is telling me it isn’t.

So I fired up the profiler at hand, witch was dotTrace and because I wanted a view what was going on I used the Timeline mode. I had to run it several times to catch the long run, but finally I’ve got it. And when I did here is what I saw:

The main problem with this is that it is not showing a CPU bound process. But from Process Explorer I knew that all cores were hard at work. The second thing is that it shows a lot of threads doing almost nothing.

From the two above it was clear what to search for in the code.

This is the culprit:

return ingredients
    .AsParallel()
    .Select(a =>
        {
            float relevance = 1;
            if (_ingredientRelevance == null)
                _ingredientRelevance = _getIngredientRelevance.Do();
            if (_ingredientRelevance.ContainsKey(a.Id))
                relevance = _ingredientRelevance[a.Id];
            return new IngredientWithRelevanceAndLevel(a.Name, relevance, a.Level);
        }).ToList();

Because the profiler is showing only my code that means that the rest of the processing power was used to switch threads.

After removing the AsParallel my excel looked like this:

A lot nicer and a bit more predictable. So how did this change got through? It was a local optimization. It worked in a separated test, but when used in the whole process became a problem.

The Excel can be downloaded form here, and a Google Spreadsheet

Neo4j stored procedures for Windows

Mon, 09 May 2016 00:00:00 +0200

In the previous post I’ve written about new features in Neo4j. One of the new game changing functions were stored procedures. But, as I experienced, getting them to run on a Windows / .NET environment wasn’t that easy, and I was seeing “There is no procedure with the name …” more often then I wished for. So here is a short how to. Hope to save you some googling.

JDK
1. Download and install JDK 8.x from here. I’ve used 8u92 version.
2. Set environment variable for JAVA_HOME pointing to JDK install location (in my case C:\Program Files\Java\jdk1.8.0_92). This can be done with some PowerShell:
```
[Environment]::SetEnvironmentVariable("JAVA_HOME","C:\Program Files\Java\jdk1.8.0_92","Machine")
 
```
The last parameter sets it as a machine level variable (visible for all users).

Install Maven

Download Maven from the official site
Unzip it to a directory. In my case C:\Program Files\Maven

Add environment variables for M2_HOME and MAVEN_HOME.

[Environment]::SetEnvironmentVariable("M2_HOME","C:\Program Files\Maven","Machine")

 [Environment]::SetEnvironmentVariable("MAVEN_HOME","C:\Program Files\Maven","Machine")

Update PATH environment.


$path=[Environment]::GetEnvironmentVariable("Path","Machine")

if($path -notcontains "%M2_HOME%\bin"){

       $newPath=$path+";"+ "%M2_HOME%\bin"

       [Environment]::SetEnvironmentVariable("Path",$newPath,"Machine")

}

Check. Run mvn -version in a new cmd.

Neo4j
1. If you haven’t, download the new Neo4j version (stored procedures are available in version 3.0 and up) from here.
2. Add NEO4J_HOME environment path for Neo4j folder(in my case C:\Program Files\Neo4j CE 3.0.0).
```
[Environment]::SetEnvironmentVariable("NEO4J_HOME","C:\Program Files\Neo4j CE 3.0.0","Machine")
 
```
3. Clone neo4j-apoc-procedures. Again some PowerShell - should be executed with admin privileges:
```
git clone http://github.com/jexp/neo4j-apoc-procedures

 cd neo4j-apoc-procedures

 mvn clean install

 copy target/apoc-1.0.0-SNAPSHOT.jar $Env:NEO4J_HOME/plugins/ 
 
```
4. Edit Neo4j config.
  
  Open %userprofile%\AppData\Roaming\Neo4j Community Edition and add this entry (change the path if yours was different then in point 3.1):
  
  dbms.directories.plugins=c:/Program\ Files/Neo4j\ CE\ 3.0.0/plugins
5. Restart Neo4j (if it was running)
6. Check if it is OK and enter for example call apoc.help('search') in Neo4j cmd. And this should appear:

Graph Connect Europe 2016

Fri, 06 May 2016 00:00:00 +0200

Last week I had the opportunity to attend Graph Connect Europe. Many great sessions, but one thing topped them all - Neo4j 3.0 is out!

And as with previous major release (it introduced Cypher) there are many bug fixes, tweaks, speed improvements, but here are my personal favorites:

Stored procedures…. Yeah I know I’ve lost you there. But those aren’t yours father’s stored procedures. Neo4j now enables to call any JVM custom function from Cypher. To start with, they published a repo with 99 procedures called apoc (and if ‘apoc’ sounds familiar, yes it’s a Matrix reference). It doesn’t make any sense to describe all of them since they are all quite well described in the GitHub repo, but just to point out the more awesome ones:
- Meta Graph - to be exact apoc.meta.graph. This procedure analyzes the whole graph and creates a meta-graph showing what node types have what relations with each other. For me a great one for exploring an unknown Neo4j database, or just to check if we didn’t screw up any relations.
  
  To have a look how it looks lets fire the :play movie graph in Neo4j cmd. It displays a widget with couple steps. The code generating the database is on the second screen. After executing the graph should look like this:
  
  As You can see in the top left corner it has nodes labeled Movie and Person and relations named ACTED_IN, DIRECTED, DIRECTED,FOLLOWS,PRODUCED,REVIEWED, WROTE.
  So lets see meta graph:
  
  This graph is a XXXXS size, but with 171 nodes and 253 relations it is hard to gasp it’s structure. The meta graph makes it extremely simple and easy.
- Loading data from RDBMS - It enables to connect to any database supporting JDBC connector.
  What is even more is the syntax: CALL apoc.load.jdbc('jdbc:derby:derbyDB','PERSON') YIELD row CREATE (:Person {name:row.name})
  for loading the whole table, or
  CALL apoc.load.jdbc('jdbc:derby:derbyDB','SELECT * FROM PERSON WHERE AGE > 18')
  for executing any SQL statement.
Bolt. A binary protocol for communicating with Neo4j. Because not long ago Ayende with Raven also announced they are moving to a binary protocol. For me it is a good signal. It means that those databases are maturing and HTTP overhead,which for millisecond operations can be bigger than time spend actually executing the query, is starting to be an issue. This would be a problem, because binary protocols are harder to implement drivers if not for the next announcement:
Official drivers. For Java, JavaScript, Python and .NET :). The syntax looks simple, but clear:

using (var driver = GraphDatabase.Driver("bolt://localhost"))
using (var session = driver.Session())
{
   var result = session.Run("MATCH (u:User) RETURN u.name");
}

No node limit. This may seem like a big thing (and probably was from implementation perspective), but previously the limit was 34 bilion nodes. So it wasn’t a real limit for most cases. But good to know:)
GUI changes. When You click on any node in the graph options will appear:

So what do these buttons do?
- x - removes the node from view (not from the database)
- lock - locks this node into position, so it won’t move when the graph is expanded.
- arrows - it shows all relations and nodes connected to this node. So from the above we get:
And if we expand Joel Silver we get this:

And so on and so on. You get the point. Why is it cool? Because it allows to actualy walk the graph and explore it. Imagine debugging Minority Report style (only without the gloves, the screen, and being Tom Cruse)

It doesn’t mean that those are all the features. If you got hooked go check Neo4j 3.0 blogpost on the official blog.

How is cookit build

Wed, 27 Apr 2016 00:00:00 +0200

The last time I described the hardware cookit is running on. Commonly referred as crap.

Now it’s time for some info what exactly is running on this fine pieace of hardware.

So, what is responsible for what?

The website - besides the obvious of serving the cookit.pl, it also hosts the admin panel and Hangfire dashboard. This part is almost read-only. Even the admin panel - all it does is schedule a Hangfire job and that’s it.
That idea stuck with me since reading a blog post about how reddit is architectured. And by the way it is a must read blog for anyone interested in how the big guys design their systems.
Windows service In two words: Hangfire service.

Why the separation and not going the easy way of hosting Hangfire in the IIS process? Couple of reasons:
- Scaling - Building this way gives an easy way for horizontally (more machines with services) or vertically (more powerful machine hosting more workers)
- Autonomous - I can easily deploy one without the effect on the other.
- Very easy to debug - This may sound strange since debugging a Windows Service isn’t the nicest thing. But since all the Hangfire service is doing is executing a function in a class, the execution can be moved to a console application. It is supported by Hangfire and it just works. Simple as that.
- Resistent - The main process it hosts is crawling and parsing. And it used to crash a LOT. And I don’t mean throwing an exception and ending. Since it is highly parallel (hundreds of jobs) it used to create couple gigabyte log files, use so much processor that everything became unresponsive or just tried to allocate insane amount of memory and going down with a out of memory exception or stack overflow. And since the last ones can’t be caught after 5 such exceptions IIS stopped the whole pool, so the site was also down.
- Ease of deployment - Few lines of PowerShell and that is it:
  
  Stop-Service $serviceName //replace the files Start-Service $serviceName
Full text search(Solr) - Searching (the no-brainer) and providing all the data for displaying the search results page.
SQL Server - the single point of truth. Except one case:
Ingredients graph - while it may not seam as a equal building block as the above, it is. It fills a small problem where SQL databases suck. Modeling relations. I am talking a bit about this problem in my graph talk, but to show what I mean:

This is how sql displays subgraph of my ingredient graph:

This view if great if You want to see data as numbers and values. For relations I prefer this (the same sub graph):

Meet yEd - a graph editing tool able of editing most graph files formats (mostly text files).

And while this data is being loaded into sql, this file is the point of truth.

The importance of running on crap

Sat, 09 Apr 2016 00:00:00 +0200

As a pet project, I’m running cookit.pl - a search engine for recipes, ingredients and food in general.

I will be writing some more about the software side and all the problems I’ve encountered, but because StackExchange published it’s hardware setup I also felt in need to share what is my hardware.

Let’s start with…

Application services:

An IIS 8 hosted website,
SOLR hosted on Tomcat as the search engine,
SQL Server as the main source of truth ,
a windows service for doing all the work (the website is almost in read only mode).

Supporting applications:

TeamCity to run builds, test, health checks, and deploy.
Search everything, because Windows Search doesn’t work to good when searching through almost 3 million images and 600 000 text files.
FastStone Photo Resizer for batch image optimization, cropping and rescaling.

Some numbers

550 GB of images,
584,834 dumped parse result files,
~ 110 000+ recipes,
~ 3000 ingredients,
~ 150 ms main page, and ~250 ms search results response time (this varies on what is going on on the server)

And all of this runs on, there is no better way of putting it, on (with added RAM).

To be more specific on a i3-3250 CPU, 8 GB of RAM, and 1 TB hard drive (with write speed sometimes between 3-4 MB/sec).

Why don’t I upgrade to a better server, or even better go to the cloud?

There are many reasons, but here are some of them:

I shouldn’t need a better server. The http load that cookit is taking isn’t that big, so this machine should have plenty of computing power spare. This wasn’t true in the past, but now it uses ~15% CPU and ~1GB of RAM.
It is auto testing my code. Any performance problems will be visible much sooner. Basically if in the morning my mail isn’t flooded with error logs, or any of my monitoring services didn’t send me an auto alert, in most cases it means things are fine.
Fast is callable. It isn’t true in general, but it is easier to scale a fast application, then a slow one. Running on this machine gives me a lot of space to go up without any need for major code changes.
It is hard. Running on crap is much harder than on an 8 core 32GB machine with SSD/nvme drives. It makes is easier to run into strange problems, and that means I have to learn more. And very often go outside of my comfort zone. That is a good thing.
Finally it turns out crap if cheep. And since I’m not making any money from it, spending little is nice.

Why dragons need graphs - demo files

Mon, 24 Aug 2015 00:00:00 +0200

This describes the content of a zip package containing demo files from my talk Why dragons need graphs. There are two folders containing a create script and some query scripts showing basic, and more advanced syntax of Neo4j Cypher.

Dragons

Demoes basic syntax of Cypher.

00_Create.txt - Graph creation script
01_SelectAllNodes.txt - Displays all nodes. Neo4j by default shows all edges, so it will show the whole graph.
02_SelectAllNodesConnectedToMonster.txt - Demonstrates how to filter nodes type.
03_SelectAllNodesConnectedTo.txt - Adds edge type filtering.
04_MultipleSelect.txt - Shows how to select multiple nodes by multiple filters.
05_SelectStrongestMonster.txt - Using two way relations (from left to right, and right to left), multiple edge and node filtering selects the strongest monster.
06_ShortestPath.txt - Uses Neo4j shortestPath and shortestPaths to select shortest paths between nodes.

Text recognition

A very simplified example how graphs can be used in text recognition tasks. It gives the answer to the sentence:

Sushi restaurants in New York that my friends like

All select scripts are additive (they extend the previous one)

00_Create.txt - Graph creation script
01_SelectMe.txt - Selects node responsible for ‘me’
02_SelectFriendsFoMe.txt - Selects my friends
03_SelectSushi.txt - Selects ‘sushi’ food type
04_SelectSushiRestaurants.txt - Selects sushi restaurants
05_SelectRestaurantsThatMyFriendsLike.txt - Selects sushi restaurants that my friends like.
06_SelectRestaurantsInNewYork.txt - Narrows the restaurants to those in New York.
selectAll.txt - Selects all edges.

Update And the video:

IndexOutOfRange

MongoDB vs RavenDB

TLTR

Table of content:

Why slapping on MongoDB isn’t the right idea?

If not MongoDB than what?

MongoDB vs. RavenDB - Fast comparison

Docker image size

Free on-premise tier

What is missing from the free version when compared to the paid one?

Paid vs. free comparison.

What features RavenDB limits in the free version?

What MongoDB limits in the free version?

Cloud offerings

Backups

Cloud offering Conclusion

Clustering model

Clustering goals

MongoDB vs. RavenDB clustering

MongoDB

MongoDB replication

MongoDB horizontal scaling

How does it work from the client’s perspective?

RavenDB clustering

RavenDB replication

RavenDB horizontal scaling

Clustering conclusion

Multi-document transactions

What are multi-document transactions?

MongoDB vs. RavenDB

Query Language

MongoDB querying language

RavenDB query language

Direct comparison

Query language conclusion

Indexing

Why is indexing vital in document databases?

RavenDB vs. MongoDB indexes

MongoDB indexes

RavenDB indexes

Auto indexes

Indexing comparison

Summary

MongoDB

RavenDB

So, what will it be?

Tips, tricks, and good practices for Data-Driven Testing. Part 2.

Before we begin

1. Reading data files

1.1 Use SetName

1.2 Wrap file parsing in a try/catch

1.3 Have a file reader class

2 Comparing

2.1 Don’t use ExpectedResult for complex objects.

2.2 Use a diff tool for comparing

3. Advanced testsing scenarios

3.1 One data set, multiple results

3.2 Multi step testing

4 General

4.1 Copy localon all files in Visual Studio/MSBuild

The problem

The Solution (?)

Why I have doubts about this solution?

Conclusion

Tips, tricks, and good practices for Data-Driven Testing. Part 1.

Before we begin

Keep data in files.

Tips on folder structure in Data-Driven Tests:

1. Have one Data folder in the test project.

2. Second level folders - the name of the dataset

3. Third level folders represent the data cases

4. Don’t be afraid to add additional grouping folders

File tips

1. Use proper file types.

JSON tips

CSV/TSV tips

2. How to handle large files in data-driven tests

2.1. Use Git LFS (Git Large File Storage)

2.2. Use Git submodules

This is not the end.

1.1 Use `SetName`

1. Have one `Data` folder in the test project.

Slap a `State` flag field with two states: `Draft` and `Confirmed` to the object.

Add a boolean field `IsDraft`.