Programming4us
         
 
 
Windows

Windows Azure : Understanding Queue Operations

11/25/2010 4:15:32 PM
Like the rest of the Windows Azure storage services, the Windows Azure queue service has a REST API. The API uses similar concepts and conventions as the other Windows Azure storage APIs , so a lot of this will be familiar to you.

All queue operations are through HTTP on a URI. A typical queue URI is of the form http://<account>.queue.core.windows.net/<queuename>. For example, if you created a queue called testqueue in my storage account, it would show up at http://sriramk.queue.core.windows.net/testqueue.


Note:Unlike blobs (which can be made publicly viewable), queues always need authentication. Therefore, none of the queue URIs in this chapter will work when accessed through your browser. However, you’ll learn the details of how to get to queue messages later in this chapter. Authentication is through the same shared key authorization method as used by the rest of the Windows Azure storage services.


1. Creating a Queue

Let’s start this review of queue operations with the most fundamental of operations: creating a queue. To create a queue, send an HTTP PUT signed using your storage account credentials to http://<account>.queue.core.windows.net/<queuename>.

The following HTTP traffic capture shows the request for the creation of a new queue called testq1:

PUT /testq1?timeout=30 HTTP/1.1
x-ms-date: Fri, 26 Jun 2009 09:59:17 GMT
Authorization: SharedKey sriramk:SAkRrcobpRn0ishVIh0+4F/gLfJuFqSFLT28hxNp0lQ=
Host: sriramk.queue.core.windows.net
Content-Length: 0
Connection: Keep-Alive

You’ll see that this is the same format you used for dealing with blobs, except that it is directed to a different HTTP endpoint. The timeout parameter is optional, and specifies the amount of time you want to wait before the server should time out the creation operation and return an error back to you. Queue creation is nearly instantaneous, and you generally don’t have to worry about specifying this with a high value. The official Microsoft storage client library also takes care of generating a good default for timeout values for most operations.

If the queue was created successfully, you’ll get back an HTTP 201. Following is an HTTP traffic capture of a successful response:

HTTP/1.1 201 Created
Server: Queue Service Version 1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 15750e32-e158-4287-bf8d-297611cb2238
Date: Fri, 26 Jun 2009 09:58:01 GMT
Content-Length: 0

This queue can now be manipulated by sending HTTP messages to http://sriramk.core.queue.windows.net/testq1.

Using the Microsoft storage client library to create queues is very similar. Earlier in this book, you learned how to obtain storage account credentials and how to set up and use this library, so let’s skip over that now. Example 9-1 shows the code to create a queue.

Example 1. Creating a queue
            CloudStorageAccount cloudStorageAccount =
CloudStorageAccount.Parse(ConfigurationSettings.AppSettings
["DataConnectionString"]);
CloudQueueClient cloudQueueClient =
cloudStorageAccount.CreateCloudQueueClient();
CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("testq1");
cloudQueue.CreateIfNotExist();


The code checks whether an existing queue with the name testq1 exists. If it doesn’t exist, the code creates it. Later, you’ll see how you can add messages and manipulate this queue.

It is not directly obvious where you place the code to create queues. Do you try to create queues dynamically through code, or do you try to create them statically upfront? If you’re used to the MSMQ/SQL Service Broker model, you’re generally comfortable with doing your setup upfront and distinct from your application code.

In general, there are two approaches you can follow:

  • The simple approach is to separate all your “creation/setup” code from your application code. Run it separately as part of your install/setup process. This code should check whether the queue exists, and create one if it doesn’t. This code doesn’t need to run in the cloud—it could be as simple as a script that runs as part of your build process.

  • If you are running a heavyweight service, and you’re using queues to manage heavy load, you might want to think about using many queues, and changing the number of queues dynamically at runtime. In this case, your queue “creation/deletion” code should form a part of the code that manages load.

2. Using Queue Metadata

The container entities in both the Windows Azure blob storage service and queue storage service can have metadata attached to them. These are simple name/value pairs that can be used to describe additional data about the queue itself.

As of this writing, this metadata feature hadn’t been widely used by applications. Applications that do incorporate this use metadata to denote additional data about the queue—what it should be used for, data about the items it contains, and so on. For example, an application may use it to describe the priority of the work items the queue handles. For simple applications, you can use a naming convention to describe a lot of this data in the queue’s name. But since names can’t be changed, the data is static. Using metadata gives you a chance to change data regarding how and why your application should use that queue. For example, you can dynamically change the priority of the queue’s work items.

Working with metadata is easy, in terms of both the REST API and the Microsoft storage library. To set/add a name/value pair, do an HTTP PUT to the queue’s URI, followed by a ?comp=metadata, and add a special header in the form x-ms-meta-name:value.

Following is a sample HTTP request that sets a name/value pair with the name foo and the value bar:

PUT http://sriramk.queue.core.windows.net/testq1?comp=metadata HTTP/1.1

x-ms-date: Fri, 26 Jun 2009 01:47:14 GMT
x-ms-meta-foo: bar
Authorization: SharedKey sriramk:u6PSIebDltGW5xHpO77epRpiUhcsTkWMvcM4GTmfqqA=

The bold lines show the changes to the URI (adding a ?comp=metadata) and the header setting the name/value pair.

Warning:

If you plan to use metadata from outside Microsoft’s data centers, test thoroughly in the code environment which talks to Windows Azure storage. Issues have been noted with proxies not sending all headers across, or mangling headers across the way, resulting in difficult-to-debug errors.


The equivalent code for this in the storage client library is simple as well. The code in Example 1 shows how to add the same foo=bar metadata using the storage library. Note that this code assumes you’ve already created a queue called testq1.

Example 1. Adding queue metadata
  CloudStorageAccount cloudStorageAccount =
CloudStorageAccount.Parse(ConfigurationSettings.AppSettings
["DataConnectionString"]);
CloudQueueClient cloudQueueClient =
cloudStorageAccount.CreateCloudQueueClient();
CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("testq1");
cloudQueue.Metadata.Add("foo", "bar");
cloudQueue.SetMetadata();


This code does a read on all of the queue’s metadata, and then adds your metadata to it. Why do that? Setting queue metadata is a SET/GET call where you replace all of the queue’s metadata. Using this read-response pattern ensures that you don’t delete any headers already set.

This also tells you how to delete all the metadata associated with a queue—just do a PUT to ?comp=metadata with no metadata headers. That will clear out all the metadata associated with the queue.

The previous code snippet also shows you the storage client API to read a queue’s metadata. What does that look like in the underlying REST API? As with all good REST APIs, once you’ve figured out how to create a resource, reading that resource is very easy.

To read a queue’s metadata, just issue an HTTP GET to that queue’s URI, followed by ?comp=metadata. You should get back a series of x-ms-<name>=<value> pairs in the HTTP headers, which contain the names and values of your metadata. The following HTTP capture shows a sample request:

GET /testq1?comp=metadata&timeout=30 HTTP/1.1
x-ms-date: Fri, 26 Jun 2009 17:52:30 GMT
Authorization: SharedKey sriramk:dCPWLE8dUiiaRKHFeziN5c9XzVIjxaX5s3QVSMtonYQ=
Host: sriramk.queue.core.windows.net

HTTP/1.1 200 OK
Server: Queue Service Version 1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 24091f05-6e62-46a8-9481-dedec525f594
x-ms-approximate-messages-count: 0
x-ms-foo:bar
Date: Fri, 26 Jun 2009 17:50:46 GMT
Content-Length: 0

3. Counting Queue Messages

If you looked at the headers in the previous code, you saw a header fly by with the name x-ms-approximate-messages-count. As the name suggests, that header contains the approximate number of messages in the queue at any time. This is a special header automatically generated by the system.

This is useful for estimating the load on a given queue. You can use it to split work among queues and workers—assign each queue to a worker node, and then use the approximate message count to balance the load. Note the word approximate in the header. The value of messages is lazily refreshed, and should be used as a guideline, rather than as a hard indicator of messages in the queue.

To access this property through the storage client library, use CloudQueue.ApproximateMessageCount.

4. Listing Queues

The operation to list queues returns a list of queues in your storage account. It does basically what you would expect, but has additional options to filter the result of queues you want, and a continuation marker system to deal with storage accounts that have hundreds or thousands of queues.

To get the list of queues in an account, you send an authorized HTTP GET to http://<accountname>.queue.core.windows.net/?comp=list. Doing so returns an XML response body containing the list of queues in your account. The following HTTP trace shows a sample request response. (The XML response has been formatted a bit for readability.)

GET /?comp=list&maxresults=50&timeout=30 HTTP/1.1
x-ms-date: Fri, 26 Jun 2009 18:10:23 GMT
Authorization: SharedKey sriramk:AYYja2g3eKp4getsu62Jr8JEfxDjhqVu89mKCvXjd7A=
Host: sriramk.queue.core.windows.net
Connection: Keep-Alive


HTTP/1.1 200 OK
Content-Type: application/xml
Server: Queue Service Version 1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: ee48a862-52e6-4444-b1d5-3c29f0d4871a
Date: Fri, 26 Jun 2009 18:09:14 GMT
Content-Length: 292

<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults AccountName="http://sriramk.queue.core.windows.net/">
<MaxResults>50</MaxResults>
<Queues>
<Queue>
<QueueName>testq1</QueueName>
<Url>http://sriramk.queue.core.windows.net/testq1</Url>
</Queue>
</Queues><NextMarker />
</EnumerationResults>

The code to do that in the storage client library is as follows:
CloudStorageAccount cloudStorageAccount =
CloudStorageAccount.Parse(ConfigurationSettings.AppSettings
["DataConnectionString"]);
CloudQueueClient cloudQueueClient =
cloudStorageAccount.CreateCloudQueueClient();

foreach (CloudQueue q in cloudQueueClient.ListQueues())
{
Console.WriteLine(q.Name);
}

One additional option when listing queues is limiting the results to only those that match a prefix. This is useful when building a service that creates and tears downs lots of queues—you can create a namespace of queues, and list only a certain subset of that namespace using this prefix search mechanism.

To search using a prefix, specify the prefix parameter in the URI (for example, http://<accountname>.queue.core.windows.net/?comp=list&prefix=<prefix-term>).

The storage client library to do that is simple, and is a minor variant of the code to list all queues:

CloudStorageAccount cloudStorageAccount =
CloudStorageAccount.Parse(ConfigurationSettings.AppSettings
["DataConnectionString"]);
CloudQueueClient cloudQueueClient =
cloudStorageAccount.CreateCloudQueueClient();

foreach (CloudQueue q in cloudQueueClient.ListQueues("test"))
{
Console.WriteLine(q.Name);
}

If you look at the XML returned for any of these queries, you’ll see a NextMarker element. This is a continuation mechanism. By default, the server returns 5,000 queues. (Note that you can modify this using the maxresults option in the URI.) If you have more than 5,000 queues, the next marker contains an indicator that you must send back to the server using the marker URI parameter to tell the server where to resume. When using the Microsoft storage client library, this is done automatically under the covers.

5. Deleting Queues

The delete operation deletes a queue from your storage account. When a queue is deleted, it and its messages are instantly inaccessible. The storage system internally marks it for deletion and garbage-collects it at leisure.


Note:

Queue deletion is not an instantaneous operation, so you must check whether the queue’s deletion has completed before trying to create another queue of the same name.


To delete a queue, send an HTTP DELETE to the queue’s URI. If successful, the storage service will send down an HTTP 2xx code. Following is an HTTP capture of both the request and response parts of a delete operation:

DELETE /testq1?timeout=30 HTTP/1.1
x-ms-date: Thu, 02 Jul 2009 11:09:33 GMT
Authorization: SharedKey sriramk:YBAGIWCXRG+d6688SXPQ2wOR/gzKL7xzux8Lhh7mfiw=
Host: sriramk.queue.core.windows.net
Content-Length: 0


HTTP/1.1 204 No Content
Content-Length: 0
Server: Queue Service Version 1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: a7a7bf21-5f61-4e4a-a508-4700d19f08ab
Date: Thu, 02 Jul 2009 11:07:35 GMT

Example 2 shows the counterpart in the storage client library.
Example 2. Deleting a queue
CloudStorageAccount cloudStorageAccount =
CloudStorageAccount.Parse(ConfigurationSettings.AppSettings
["DataConnectionString"]);
CloudQueueClient cloudQueueClient =
cloudStorageAccount.CreateCloudQueueClient();
CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("testq1");
cloudQueue.Clear();
cloudQueue.Delete();


When you delete a queue, its messages are deleted as well. If workers are trying to remove messages of that queue, they’ll immediately start seeing errors. In short, don’t delete a queue unless you’re really sure that you don’t need the messages in it anymore. It might be safer to leave empty queues around with no messages in them. Since there’s no upper limit on the number of queues you may have, there is no harm in having old queues lying around (appropriately named and ignored by your code, of course!) and cleaning them up later.

Other -----------------
- Windows Azure Queue Overview
- Tuning Windows 7’s Performance : Optimizing Virtual Memory
- Tuning Windows 7’s Performance : Optimizing the Hard Disk
- Tuning Windows 7’s Performance : Optimizing Applications
- Tuning Windows 7’s Performance : Optimizing Startup
- Tuning Windows 7’s Performance : Monitoring Performance
- Windows Vista - File Encryption : Workings of BitLocker Drive Encryption
- Windows Vista - File Encryption : Encryption File System
- Windows 7 : Customizing the Taskbar for Easier Program and Document Launching
- Windows 7 : Customizing the Start Menu for Easier Program and Document Launching
- Windows Azure Storage : REST API (part 2) - Storage Client APIs
- Windows Azure Storage : REST API (part 1)
- Windows 7 : Customizing Your Notebook’s Power and Sleep Buttons
- Windows 7 : Customizing the Start Menu’s Power Button
- Windows 7 : Turning Off Your Windows 7 Computer from Anywhere
- Windows 7 : Setting Up One-Click Restarts and Shutdowns
- Windows 7 : Useful Windows 7 Logon Strategies
- Windows 7 : Customizing Startups with the Advanced Options Menu
- Windows 7 : Customizing Startups Using the Boot Configuration Data
- Windows Azure : Queue Service Architecture
 
 
Most View
- Windows Server 2008 R2 : Installing Windows SharePoint Services (part 1)
- Recovering from a Disaster in an Exchange Server 2010 Environment: Identifying the Extent of the Problem (part 1)
- Windows 7 : Understand Internet Explorer’s Advanced Security Options
- SharePoint 2007 : See What Lists and Document Libraries Are in a Site (part 2)
- Windows Server 2008: Understanding and Deploying BranchCache (part 3)
- iPhone 3D Programming : Vertices and Touch Points - Creating a Wireframe Viewer (part 3)
- Windows Phone 7 : Pinning a Website to Start
- SQL Injection Attacks and Defense : Executing Operating System Commands
- Windows Azure: Building a Secure Backup System (part 6) - Uploading Efficiently Using Blocks
- .NET Debugging : Introduction to the Tools - .NET 2.0—Redistributable & .NET 2.0—SDK
Top 10
- Implementing Edge Services for an Exchange Server 2007 Environment : Utilizing the Basic Sender and Recipient Connection Filters (part 3) - Configuring Recipient Filtering
- Implementing Edge Services for an Exchange Server 2007 Environment : Utilizing the Basic Sender and Recipient Connection Filters (part 2)
- Implementing Edge Services for an Exchange Server 2007 Environment : Utilizing the Basic Sender and Recipient Connection Filters (part 1)
- Implementing Edge Services for an Exchange Server 2007 Environment : Installing and Configuring the Edge Transport Server Components
- What's New in SharePoint 2013 (part 7) - BCS
- What's New in SharePoint 2013 (part 6) - SEARCH
- What's New in SharePoint 2013 (part 6) - WEB CONTENT MANAGEMENT
- What's New in SharePoint 2013 (part 5) - ENTERPRISE CONTENT MANAGEMENT
- What's New in SharePoint 2013 (part 4) - WORKFLOWS
- What's New in SharePoint 2013 (part 3) - REMOTE EVENTS