Apr 022017
 
Share...Tweet about this on TwitterShare on FacebookShare on Google+Share on StumbleUponShare on LinkedInPin on PinterestShare on TumblrShare on RedditDigg this

If we were to implement a solution that uses queues, the first thing that comes to our mind is to use one of the already baked systems that support queuing. There is really a proliferation of open-and-closed source software such as:

There are however other kind of queues, and those are directly attached to the database. On the project I am currently working, we are extensively using the Microsoft SQL Server Service Broker, which is the Microsoft SQL Server bound implementation. Oracle has it’s own implementation the Oracle Database Advanced Queueing.

As the topic of this post is MongoDB, no MongoDB doesn’t have the queuing system bound to the database, at least not in the same way SQL Server or Oracle have. This post is about how to achieve a similar thing by using the plain MongoDB implementation.
We can simulate the queuing system by using the Capped Collections.

What is queueing after all…

If you are not fully sure what is a Queue and what is the message queueing all about, check some posts such as: Wikipedia Queue, or What is message queueing

What is a capped collection?

Definition directly taken from the MongoDB web site:

Capped collections are fixed-size collections that support high-throughput operations that insert and retrieve documents based on insertion order. Capped collections work in a way similar to circular buffers: once a collection fills its allocated space, it makes room for new documents by overwriting the oldest documents in the collection.

Graphically we can represent a capped collection as follows:

Therefore, based on the above definition Capped Collections are a special kind of collections with two attributes which makes them different from “normal” collections: fixed-size and circular collection.
What does this mean:

  • Fixed-Size refers to the fact that there is a predefined (configurable) limit of the maximum number of items this table sill support.
  • Circular: refers to the fact that once the maximum amount is reached, the oldest of the items get deleted to make room to the new one.

One of the interesting properties of capped collections is that the collection itself preserves the order in which the items get inserted. This is very important aspect, especially if this kind of table get used for a logging-type of problems where the order of entries should be preserved. On the other hand, it had some limitation such as: we cannot remove a document from a capped collection, and updates on the documents won’t work if an update or a replacement operation changes the document size.

Capped collections can be used for several purposes such as:

  • Logging (i.e. latest activity performed on the web site, etc…)
  • Caching: preserving the latest items, etc…
  • Act as a Queue. Capped collection might be also used to act as a queue where the first-in-first-out logic applies.
  • etc…

How to create a capped collection?

In the same way as when creating normal MongoDB collections but with specifying the options parameter as follows by using the MongoDB Shell:

db.createCollection("LogCollection", 
    {   
        capped:true, 
        size:10000, 
        max:1000
    }) ; 

There are 14 optional parameters. Three of the most common are:

  • capped: true (by default is false) sets the type of the collection as a capped
  • size: Sets the maximum size in bytes for this particular collection.
  • max: Specifies the maximum number of documents allowed for the given collection

Consuming a capped collection

MongoDB offers the option to watch a capped collection for changes using a tailable cursor.

The code below “tails” a capped collection and outputs documents to the console as they are added. The method also handles the possibility of a dead cursor by tracking the field insertDate. New documents are added with increasing values of insertDate.

private static void TailCollection(IMongoCollection collection)
{
    // Set lastInsertDate to the smallest value possible
    BsonValue lastInsertDate = BsonMinKey.Value;
    
    var options = new FindOptions 
    { 
        // Our cursor is a tailable cursor and informs the server to await
        CursorType = CursorType.TailableAwait
    };
    
    // Initially, we don't have a filter. An empty BsonDocument matches everything.
    BsonDocument filter = new BsonDocument();
    
    // NOTE: This loops forever. It would be prudent to provide some form of 
    // an escape condition based on your needs; e.g. the user presses a key.
    while (true)
    {
        // Start the cursor and wait for the initial response
        using (var cursor = collection.FindSync(filter, options))
        {
            foreach (var document in cursor.ToEnumerable())
            {
                // Set the last value we saw 
                lastInsertDate = document["insertDate"];
                
                // Write the document to the console.
                Console.WriteLine(document.ToString());
            }
        }

        // The tailable cursor died so loop through and restart it
        // Now, we want documents that are strictly greater than the last value we saw
        filter = new BsonDocument("$gt", new BsonDocument("insertDate", lastInsertDate));
    }
}
    Share...Tweet about this on TwitterShare on FacebookShare on Google+Share on StumbleUponShare on LinkedInPin on PinterestShare on TumblrShare on RedditDigg this

    I'm a Software Developer and Solution Architect interested in Software Development, Object-Oriented Design and Software Architecture all this especially bound to the Microsoft.NET platform.Feel free to contact me or know more in the about section

    Leave a Reply

    holthaus_sofia@mailxu.com gremo.rpo@mailxu.com