Where can I store my Data on Windows Azure?

So, You’re building a cloud app on Windows Azure are you? Where are you going to store your data? This is a question that comes down to architecture, scale and options. Considering I don’t know your architectural decisions, or how scalable your particular application, I guess that leaves me with only one topic to talk about, What are your options for storing data on Windows Azure?

Windows Azure is an open cloud platform which enables you to quickly build and deploy your applications using your existing skillset. This is reflected in the data options which are available for use on Windows Azure which are also hosted on Windows Azure.

Relational Databases on Windows Azure

Some Applications (or Developers) can’t live without a trusty relational database. We’ve got that.

MySQL

This option surprises many developers, but Windows Azure does have MySQL databases available for use.

MySQL is provided by a partner ClearDB who maintains multi-master and multi-master with multi-replica MySQL configurations. ClearDB combines advanced replication techniques, advanced clustering technology and a web service interface to provide MySQL Databases for your consumption.

If you create a Windows Azure Web Site using the Create with Database or From Gallery option in the Windows Azure Management Portal a Mercury sized database is being provisioned on your behalf.

MySQL Sizing Options

Pricing available on the Windows Azure Store on the ClearDB Website.

Package Name

Database Size

Max. Connection limit

I/O Performance

Mercury 20MB

4

Low
Venus 1GB

15

Moderate
Saturn 5GB

30

Moderate
Jupiter 10GB

40

High

MySQL on a Linux Virtual Machine

In addition to the ability to leverage a third-party service for using MySQL on Windows Azure, you can also leverage the new Virtual Machines feature. Follow the steps for setting up MySQL on an OpenSUSE Virtual Machine on Windows Azure.

Pricing for Linux Virtual Machines are available on the Windows Azure Pricing page.

SQL Database

Microsoft has created a cloud equivalent of SQL Server known as Windows Azure SQL Database (or SQL Database for short). SQL Database uses the same TDS Protocol as SQL Server which means all standard libraries or drivers [like the SQL Server Driver for PHP] work against SQL Database for Connectivity and Querying.

SQL Database Sizing Options

Pricing available on Windows Azure Pricing page under Data Management.

Database Size
100 MB
1 GB
10 GB
150 GB

As you can see above, SQL Database offers sizing tiers which acts as a starting point for your database size, additional database sizing between the tiers is offered at a discounted database rate as described in the SQL Database section of the Windows Azure Pricing page.

You may have also noticed that the maximum database size is 150GB, once you reach a database of this size it becomes necessary to shard your data using a technical known as SQL Federation.

SQL Server on a Windows Virtual Machine

If you need a large database (Scaled Vertically) or rely on a feature that is not currently present in SQL Database, it is possible to install SQL Server in a Windows Virtual Machine. Depending on the version of SQL Server you would like to use there are two great tutorials: Provisioning a SQL Server Virtual Machine on Windows Azure [which installs SQL Server 2012 from an image in the gallery] or Creating and Uploading a Virtual Hard Disk that Contains the Windows Server Operating System [which explains how to create a reusable operating system image, you’ll need to install SQL Server before running sysprep the VHD]

Unstructured Data (NoSQL) on Windows Azure

Unstructured Storage is becoming all the rage, which is understandable due to its ability to massively scale.

Windows Azure Table Storage

Windows Azure Table Storage is one of the three abstractions of Windows Azure Storage. Table Storage is an Entity-Attribute-Value model data store with a 100TB per account limit. Each subscription has 5 storage accounts by default, and more storage accounts can be added by calling customer support.

An entity in Windows Azure Table Storage is comprised of 3 required properties partition key, row key & timestamp and up to 252 additional properties. Properties can be one of the following data types Int32, Int64, String, Guid, DateTime, Double, Boolean or Byte Array.

MongoDB on Windows Azure

There are a few different ways that MongoDB is supported on Windows Azure which include in a Cloud Service, on a Virtual Machine or as a Service.

MongoDB in a Cloud Service

The creators of MongoDB, 10gen, have created a MongoDB installer which will install a MongoDB replica set in a Windows Azure Cloud Service. To do this they leverage a worker role in which their installer leverages a startup script to install the MongoDB Server. It also utilizes Local Storage as a cache before committing new transactions to Blob Storage.

To use the MongoDB in a Cloud Service configure and Deploy the MongoDB Installer.

MongoDB in a Virtual Machine

Obviously MongoDB can be installed on a Virtual Machine and can be exposed to the internet. There are two tutorials on installing a standalone instance of MongoDB on Windows Azure: Install MongoDB on a virtual machine running CentOS Linux in Windows Azure and Install MongoDB on a virtual machine running Windows Server 2008 R2 in Windows Azure.

If you would like to set up a highly available MongoDB replica set on Windows Azure, follow the instructions laid out in MongoDB on Windows Azure VM – Linux Tutorial or MongoDB on Windows Azure VM – Windows Installer,

MongoDB as a Service

Another option for MongoDB on Windows Azure is MongoDB as a Service provided by MongoLab. This service is [at time of writing] in preview and provides 500MB of storage for FREE. The MongoLab Preview is currently only available in the East US datacenter.

Preview Warning

We are pleased to be able to offer free databases running on Windows Azure. Be aware that both Windows Azure and MongoLab’s presence on Windows Azure are still in preview mode, and that these databases are not appropriate for production use. For example, we may need to take them down for maintenance from time to time without advance warning.

image

CouchDB on Windows Azure

Apache CouchDB is a JSON document database exposed over HTTP.

CouchDB comes with a suite of features, such as on-the-fly document transformation and real-time change notifications, that makes web app development a breeze. It even comes with an easy to use web administration console. You guessed it, served up directly out of CouchDB! We care a lot about distributed scaling. CouchDB is highly available and partition tolerant, but is also eventually consistent. And we care a lot about your data. CouchDB has a fault-tolerant storage engine that puts the safety of your data first.

BigCouch by Cloudant

Cloudant has expanded the base implementation of CouchDB in a new incarnation they are calling BigCouch. BigCouch as a Service is offered in the West US datacenter in a cluster Cloudant refers to Lagoon.

The Cloudant Data Layer collects, stores, analyzes and distributes application data across a global network of secure, high-performance data centers, delivering low-latency, non-stop data access to users no matter where they’re located.

BigCouch is chalked full of goodies including secondary indexes, lucene-based full text search, as well as built in replication and synchronization.

Cassandra on Windows Azure

Looking to build the next Netflix or Twitter? Run an elastic, decentralized and fault-tolerant Cassandra cluster on Windows Azure. Learn how to harness the power of Cassandra while a Node.js application in the tutorial Running Cassandra with Linux on Windows Azure and Accessing it from Node.js.

Choose your own Adventure

Have a favorite data store of your own? Why not try running it on Windows Azure? If you do, leave a comment below with a link off to a blog post explaining how you got it running.

Stay Cloudy my Friends…

Windows Azure SDK for PHP – ‘PartitionKey’ property value must satisfy is_string.

When building out an application there are many factors you need to consider, these considerations boil down to functional and non-functional requirements that you as a developer or architect are set out to achieve. In many circumstances, if not all, we are accepting data as an input from a user or other medium, transforming or normalizing that data into something that will help fulfill other requirements of an application.

In order to fulfill scalability goals it may be necessary to find alternative means to traditional storage, something that was designed with scalability in mind. One such service is Windows Azure Table Storage which is defined as:

[Windows Azure Storage] Tables offer NoSQL capabilities for applications that require storage of large amounts of unstructured data. Tables are an ISO 27001 certified managed service which can auto scale to meet massive volume of up to 100 terabytes and throughput and accessible from virtually anywhere via REST and managed API’s.

The topic of Windows Azure Storage Abstractions and their scalability targets is some what out of scope of this post, however, keep in mind that it is important to understand the Windows Azure Storage Architecture and more specifically How to get the most out of Windows Azure [Storage] Tables.

Integer values for PartitionKey or RowKey with PHP

If you’ve done your research on designing for scale with Windows Azure Storage and you find that Integer values are what you need for your specific data model, keep in mind that PartitionKey and RowKey values are of type string.

While this doesn’t seem like a foreign concept you may find yourself in a situation where you have an error while attempting to use integers as strings while interacting with the Windows Azure SDK for PHP. The error you will receive is “PartitionKey [or RowKey] property value must satisfy is_string”. In my particular scenario I was loading my tables by iterating through an array of key/value pairs.

My array looked like this:


In which you may expect the key to be treated as a string, which isn’t the case. To understand the reasoning behind this, we’ll go right to the source.

The key can either be an integer or a string. The value can be of any type.

Additionally the following key casts will occur:

  • Strings containing valid integers will be cast to the integer type. E.g. the key "8" will actually be stored under 8. On the other hand "08" will not be cast, as it isn’t a valid decimal integer.
  • Floats are also cast to integers, which means that the fractional part will be truncated. E.g. the key 8.7 will actually be stored under 8.
  • Bools are cast to integers, too, i.e. the key true will actually be stored under 1 and the key false under 0.
  • Null will be cast to the empty string, i.e. the key null will actually be stored under "".
  • Arrays and objects can not be used as keys. Doing so will result in a warning: Illegal offset type.

According to the documentation, you’ll notice that “strings containing valid integers will be cast to integer type”, in order to ensure that the now converted integer is provided to the service proxy as a string, we need to call strval() on the key when providing it as a PartitionKey or RowKey. To provide an example of this consider the following:


Where addEntity() constructs an entity and inserts it into Windows Azure Table Storage.

Conclusion

In this post I clarified a potential issue that may arise when iterating over an array with key/value pairs to push entities into Windows Azure Table Storage.

Stay Cloudy My Friends…

Update Released: Windows Azure Storage Plugin for WordPress

Microsoft Open Technologies, Inc. announced today that they have released an update to the existing WordPress Plug-in, Windows Azure Storage for WordPress.

If you aren’t familiar with the tool, essentially it allows you to sign up for Windows Azure, create a storage account and save your blog post assets like images, videos and documents in a durable, CDN enabled [enabling CDN in Windows Azure Storage] storage system with up to 100TB of storage!

Why Update the WordPress Plug-in?

The major reason for the update was to bring the tool inline with the most recent release of the Windows Azure SDK for PHP [Windows Azure SDK for PHP on GitHub]. For this reason alone, it is a great example of a real world application which reaches thousands of users [Current Downloads as of this post: 2635].

Need Installation Instructions? Read Windows Azure Storage Plug-in for WordPress Updated by Brian Swan.
Need a Blog? Sign up for a Free Windows Azure Account, then Setup a new WordPress Blog on Windows Azure Web Sites using the OSS Gallery.

If you use WordPress as your blog engine or as a CMS, the Windows Azure Storage for WordPress Plug-in is definitely worth a look.

How does the Plug-in Work?

More interested in the Architecture of the Plugin? Read Windows Azure Storage Plug-in for WordPress by Doug Mahugh

The plug-in uses the WordPress Filters to inject new storage functionality such as a tab for accessing existing items in the storage container, a mechanism for saving content to Windows Azure Storage, as well as providing XML-RPC support for saving attachments from third party blog tools (like Windows Live Writer).

Under the covers the tool saves your media files as Block Blobs, it utilizes the BlobRestProxy exposed by the Windows Azure SDK for PHP to commit the files in individual blocks (max. 4mb in size) before committing the entire blob. The benefit to using this technique is if any one (or multiple) block doesn’t succeed in uploading, the maximum amount of data that needs to be resent to Blob Storage is reduced (which in turn saves in bandwidth usage).

All thumbnails which are auto-generated by WordPress are uploaded in the same fashion, then the metadata (Container Name, Blob Name, Thumbnail urls) are all saved to the database using add_post_meta.

Can I Contribute?

If you want to add additional features, or view the Source code for the Windows Azure Storage for WordPress Plugin it is available from the WordPress Plug-ins SVN Repository under a New BSD License.

More into Git? [Me too…] Instructions for Bidirectional operations between SVN and GIT

Once you’ve downloaded the source, you may want to consider checking out WebMatrix which has support for PHP syntax highlighting and code completion, as well as WordPress specific code completion which will be helpful when you’re extending the plug-in.

Happy Blogging!

Windows Azure Storage Emulator Account Name and Key

It’s almost funny to say, but I’m working on a legacy Windows Azure Application in which the need of the Windows Azure Storage Account Name and Account Key need to be given instead of a Storage Account Connection String.

What’s the Difference? You need the Name & Key for a Connection String

While this is true in some scenarios, there is one distinct difference especially when it comes down to running in the local Storage Emulator. The following are considered Connection Strings:

<Setting name="DataConnectionString" value="usedevelopmentstorage=true" />

or

<appSettings>
     <add key="DataConnectionString" value="DefaultEndpointsProtocol=https;AccountName=myuniqueaccountname;AccountKey=superlongsecurestoragekey" />
</appSettings>

The second example of a connection string could be easily parsed to extract the AccountName and AccountKey, unlike the first example which is absolutely impossible to provide any valuable information. If you’re using the original Microsoft Sample code for Windows Azure Storage Session State Management which doesn’t use the convention of the connection string, you’d be sadly out of luck. This is what you’ll find in the Web.Config file:

<appSettings>
     <add key="AccountName" value="myuniqueaccountname"/>
     <add key="AccountKey" value="superlongsecurestoragekey"/>
</appSettings>

Oh No, We’re doomed right?

Not at all. Microsoft has provided the storage credentials for the local Storage Emulator on MSDN. Here’s a quick place to copy the default name and key:

AccountName

devstoreaccount1

AccountKey

Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

Happy Clouding!

Setting up the Development Storage Service

If you are setting up your Development Environment for Windows Azure and want to avoid installing SQL Server Express. I didn’t want to use SQL Server Express because I’ve already installed SQL Server 2008 R2 CTP so I can interact with my SQL Azure Database in the Cloud.

From the command line, Navigate to the Windows Azure SDK directory.

cd "C:\Program Files\Windows Azure SDK\v1.0\bin\devstore"

Then you will Launch the Development Storage Initialization Tool [DSInit.exe]. You will need to set the SqlInstance [/sqlinstance:] that you wish to create the tables on, and you will need to force [/forceCreate] the tool to create the tables.

DSInit.exe /sqlinstance:<YourDatabaseName>/forceCreate

This will launch the Initialization Tool.

Development Storage Initialization
If you receive the error message below you it is most likely because the instance name was improperly set. Ensure to be use only the InstanceName do not include periods or slashes.

Development Storage Initialization Error

After your Development Storage is set up make sure you  go into your Development Fabric and Start the Development Storage Service.

Happy Coding!