A Real-World Sample
Let me introduce a scenario that will help illustrate some of these concepts. Fabrikam, an ISV, would like to build a hosted digital asset management Web application. This application enables end users to store digital images in the cloud. Users must be able to preview the pictures submitted, tag them, and annotate them with location information. The application must allow users to use their own desktop applications to access these pictures directly from the storage. Figure 6 represents a high-level architectural view of the services exposed by Fabrikam. As shown in the diagram, Fabrikam exposes a Web application and Web service to the end users for managing their digital assets. These services store and retrieve digital assets from the asset store implemented on top of Windows Azure blob services. Fabrikam implements a background processor to execute tasks such as tagging, thumbnail generation, and so on.
Figure 6 Fabrikam Hosted in Windows Azure
Let's see how to design, develop, and deploy such an application to the Azure Services Platform. The discussions that follow demonstrate core concepts and features available in Windows Azure. You can download the code and documentation to see how the entire scenario is implemented and hosted in Windows Azure.
Application Design and Development
The first step is to understand the applications scenarios, select appropriate patterns, best practices, and programming models. During this process you will identify the type of the application and workloads targeting Windows Azure. In addition, you will figure out the inter-application messaging and data storage patterns. Here I'll assume that Fabrikam is building an interactive Web application with scale-out needs and a background processer to generate metadata from digital assets. The above scenario needs Azure Web and Worker roles, respectively. In addition, Fabrikam needs Blob storage to store digital assets and metadata, service interfaces to expose assets, and messaging patterns for communication.
To facilitate development, the Windows Azure SDK comes with a development environment that simulates Windows Azure on the local machine, thereby enabling offline development and debugging. The SDK supports Web and Worker roles, storage, Visual Studio templates, tools for packaging, viewing and managing role instances, and a Visual Studio add-in that enables the F5 experience. You can download the SDK from MS Download center.
To begin, use the Windows Azure Tools for Microsoft Visual Studio to create a new project using the Cloud Service project templates. You can add necessary user interface, application logic, and input processing to the Web application. Alternatively, you could start from an existing ASP.NET Web application. Because the application will execute under partial trust, avoid calling APIs that violate this restriction. For details about this restriction, please, see the
Windows Azure SDK Trust Policy Reference.
Next, import Microsoft.ServiceHosting.ServiceRuntime assembly and use the respective APIs to enable logging, reading local environment configuration, and reading application configuration sections from the service definition file, as shown below.
// Writing log information
RoleManager.WriteLog ("Information", "message to show in the log file");
// Getting service configuration information
RoleManager.GetConfigurationSetting("MyConfigSection");
// Getting local environment; code snippet below shows getting a pointer
//to a local file
ILocalResource resource = RoleManager.GetLocalResource( "ComputeLocalStorage")
By the way, it is a programming best practice to replace the above logging and resource access code with reusable application blocks (Logging and Configuration blocks) such as Microsoft Enterprise Library application blocks. You can implement specific providers for Windows Azure logging and configuration management and plug them into the runtime. These best practices decouple applications from platform-specific APIs and help increased reusability between on-premise and cloud environment.
Storage needs for your application will need to be rewritten to take advantage of the Azure storage patterns supported. I will address this shortly.
Next, add a Worker role project into your project and then open WorkRole.cs. This file includes a class called WorkerRole that derives from the RoleEntryPoint class. This class provides methods to manage the initialization, starting, and stopping of a Worker role service, as well as for monitoring the health of the service.
In this step you normally override two methods, as shown in Figure 7.
Figure 7 WorkerRole
public class WorkerRole : RoleEntryPoint { public override void Start() { RoleManager.WriteToLog("Information", "Generating metadata from digital assets"); while (true) { RoleManager.WriteToLog("Information", "background task is executing thumbnail extraction"); Thread.Sleep (1000); } } public override RoleStatus GetHealthStatus() { return RoleStatus.Healthy; } } Now run the application in the local development fabric by hitting F5. You will see logs written in the Web and Worker Roles console instances.
Now let's configure a blob store and access it from the Web role.
First you need to set up the blob storage account information.You'll need to specify the following parameters:
AccountName: Specify the name of your Windows Azure account.
AccountSharedKey: Specify the key used to authenticate a request made against Windows Azure storage. To authenticate a request, you must sign the request with the key for the account that is making the request.
BlobStorageEndpoint: Specifies the base URI of the blob storage service.
ContainerName: Specifies the name of the blob container used to store images for this application.
For example, take a look at the following configuration settings:
The REST API for Blob Storage exposes two resources: containers and blobs. A container is a set of blobs; every blob must belong to a container. Blobs are written using PUT operations as defined by the REST API. Using these APIs you can create a hierarchical namespace to organize assets such as fabrikamImagegallery/images and fabrikamImagegallery/documents, where fabrikamImagegallery is the name of the container and images refers to the blob name. It should be noted that each blob could be constructed from blocks of a maximum of 4MB in size for efficient management.
Working with the REST API is rather cumbersome and hence the SDK ships with a sample library that abstracts the REST calls and exposes well known access patterns through the BlobContainer and StorageAccountInfo classes.
Figure 8 shows how to work with Blob storage.
Figure 8 Using Blob Storage
BlobStorage blobStorage = BlobStorage.Create(StorageAccountInfo.GetDefaultBlobStorageAccountFromConfiguration()); BlobContainer newContainer = blobStorage.GetBlobContainer("ContainerName"); newContainer.CreateContainer(null, ContainerAccessControl.Public); this.container.ListBlobs(String.Empty, false); BlobProperties properties = new BlobProperties(string.Format(CultureInfo.InvariantCulture, "image_{0}", "idenitifier1")); NameValueCollection metadata = new NameValueCollection(); metadata["Id"] = id; metadata["Filename"] = "filename"; metadata["Tags"] = "Image"; properties.Metadata = metadata; this.container.CreateBlob(properties, imageBlob, true); Now, let's create a Queue storage service to send message from a Web role to a Worker role. The Windows Azure Queue service provides reliable, persistent messaging within and between services. The REST API for the Queue service exposes two resources: queues and messages. It is possible to create an unlimited number of queues each identified using a unique name. The maximum size of the message is restricted to 8KB. You are responsible for deleting the message after reading; otherwise the message will remain in the queue for the specified time.
Figure 9 shows how to access Queue storage and messages in a Worker Role.
Figure 9 The Queue Storage Operations
public class WorkerRole : RoleEntryPoint { public override void Start() { Uri baseUri = new Uri(RoleManager.GetConfigurationSetting("QueueEndpoint")); string accountName = RoleManager.GetConfigurationSetting("AccountName"); string accountKey = RoleManager.GetConfigurationSetting("AccountSharedKey"); StorageAccountInfo account = new StorageAccountInfo( baseUri, null, accountName, accountKey); QueueStorage service = QueueStorage.Create(account); MessageQueue queue = service.GetQueue("fabrikamimageprocessingqueue"); while (true) { Thread.Sleep(10000); if (queue.DoesQueueExist()) { Message msg = queue.GetMessage(); if (msg != null) { queue.DeleteMessage(msg); } } } } …. } Now you are ready to try out this application in the local development fabric by hitting F5. You will see Web Roles receiving inputs and sending messages to Worker Roles through Queues. By now you have completed all the necessary development activities needed and are ready to publish the application to Azure.
Deploying the Service
Deploying services to Windows Azure is relatively simple. However, you first need to decide what information to include in the service definition file (application configuration information) versus service configuration file (environment requirements), how to package the application, whether you can leverage existing tools to create and upload packages, and how to store your packages in the cloud storage and later deploy to Azure.
Here's how I will address these requirements for the sample application.
Fabrikam ISV has developer, deployment, and operational roles to manage its cloud applications. Developers create a service definition file that contains information about the application. The service definition file (.csdef) defines the roles available to a service, specifies the service endpoints, and establishes configuration parameters for the service. The service definition settings cannot be changed after a service is deployed. This file becomes part of the Service package file during the packaging process.
Developers also create a service configuration file (specifying the number of instances needed and Azure account information) for local debugging and Azure staging deployment. The service configuration file (.cscfg) specifies values for the configuration settings for one or more role instances within the running service. Operational staff can dynamically modify these service configuration settings without redeploying the service. The service configuration file is not packaged with the service, but is uploaded to the Windows Azure Fabric as a separate file.
The deployment staff updates the services configuration file before production deployment with Azure account-specific information and operational requirements (number of production instances needed).
Developers or the deployment staff create Azure deployment packages using Visual Studio tools or the CSPack tool that comes with the Azure SDK.
It is possible to store deployment packages in the Azure blob storage and retrieve while deploying to the Azure cloud platform
The service package file (.cspkg) is a zipped file that contains role definitions and all role related files (binaries, images and config) packaged into one artifact. You could generate these packages by using Visual Studio or using the CSPack tool. When you use Visual Studio to run on the development fabric the package created will have a .csx extension. However when you use the "Publish" feature, it creates a .cspkg file, which is a zipped and encrypted version of the .csx file. This encrypted and zipped version with .cspkg extension is uploaded to the cloud.
Now you can upload your service configuration file (.cscfg) and package file (.cspkg) to Windows Azure. For this, first you need to obtain Windows Azure compute and storage accounts from the Azure Services Developer Portal. After obtaining these accounts you can deploy the package to the cloud though the Azure developer portal. Once files are uploaded, you'll be provided with an internal staging URL that you can use to test your service in the Windows Azure Fabric. When you move into production you will get the production URL.
Administration
Administration functions cover the management, governance, and operational aspects of the applications deployed in a cloud environment. Leveraging the cloud platforms for application hosting assumes that the services deployed are in a controlled environment with appropriate service-level management, resource management, service provisioning, security/trust models, and monitoring. You should be familiar with the key functional capabilities needed for service management. They include:
- Developing services with operations-friendly implementation best practices by including proper instrumentation.
- Monitoring the health and availability of cloud applications and services.
- Collecting metrics and reporting on service usage, performance, and billing.
- Enabling automated provisioning of services and updating service configurations.
In Windows Azure, agents monitor each VM instance for failure conditions and collect metrics regarding failures, performance measures, and usage metrics.
Azure Fabric automates the provisioning and management aspect with limited human input. People and process are error prone. This automation helps avoid human errors. In Windows Azure, applications run in a partial-trust sandbox, requests are load balanced, and failure conditions are automatically managed.
As a developer you should be familiar with the operational behaviors of your service. These behaviors are documented as part of service definition and configuration models and communicated to the cloud platform to facilitate automation.
In addition, follow these best practices:
Adding logging information in the code
RoleManager.WriteLog ("Information", "message to show in the log file");
- Overiding GetHeatlthStatus method
public override RoleStatus GetHealthStatus()
{
// return the health status of worker role.
return RoleStatus.Healthy;
}
- Getting Log files for analyzing application states
You should use the "configure" option and "Copy Log" option in the Azure portal to copy the log messages to the blob storage.
Once the copy is completed you could access the logs from blob storage using the programming model described earlier.
In the future Windows Azure will expose management interfaces to the access and control management and operational activities. This helps on-premises service management tools to monitor and manage cloud applications.
Conclusion
Understanding the cloud computing architecture models, finding relevant patterns, deciding on the programming model, and enabling automated management are critical to the success of cloud computing evolution. Windows Azure is designed with developers in mind by enabling developers to quickly and easily create, deploy, manage, and distribute Web applications and services.
As a developer, you should accumulate a core set of patterns that will help you develop cloud applications most effectively. In addition, you should remember to capture and share recurring solutions to problems and other emerging patterns such that our solutions and products can provide better out-of-the-box support for core cloud computing patterns.
I would like to thank Steve Marx, Jason Hogg, David Hill, Fred Chong, Eugenio Pace, and Nataraja Koduru for their valuable feedback on this article.
Joshy Joseph is a principal architect with Microsoft Services Managed Solutions Group. His primary skills and expertise include distributed computing, grid computing, and Web services. He is the author of the book
Grid Computing (Prentice Hall, 2004) and a prolific inventor, with more than 35 patents on file. In addition, he has written numerous technical articles on distributed computing and business process development. Joshy can be reached at
jojoseph@microsoft.com.