cloudcamp nyc

i am organizing my notes from cloudcamp nyc 2009. the whole thing was done the unconference style, and i bounced around different groups for a few hours.

taxonomy

it seems that folks were struggling to come up with the taxonomy of the cloud. i see it is a continuum from bare OS instances (amazon’s EC2) to increasingly specialized services (google app engine, S3, salesforce, and even facebook).

bare OS instances are easy – nothing there is new, and what is changing is the elasticity of the capacity – you can spin up and wind down instances much faster (and recently announced amazon’s auto scaling makes it even easier). the only catch there is that you have to design for scalability yourself. but you have all the power and the flexibility, and there is no lock-in.

the latter is when you give up some of the flexibility for scalability out of the box – the design and implementation are taken care of for you – google app engine and facebook just scale. there is a lock-in there, but you might not care. somewhat unrelated, but pmarca had some things to say about it a while ago.

automation

when dealing with OS instances, these developments make it obvious that if you have not embraced automation yet, it is time to do so. the lone admin doing things by hand does not cut it anymore; even barebones shell scripts and basic packaging might not be enough.

the sad thing is that i do not see an easy, thorough, cross-platform management tool that does everything out of the box. cfengine was the closest thing back in the day, but i want something that embraces source control and declarative configuration; something that is transparent and easy to use; something that is cross-platform. chef and puppet look promising.

so there are a lot of new vendors coming out and touting their products for managing the cloud.

monitoring

same thing with monitoring – sscope was a killer app almost a decade ago, but i have not seen a compelling polished replacement (until, possibly, hyperic) – something that could scale down and up, something that can be either agentless for small easy deployments and agent-based to be able to scale beyond that. amazon’s cloud watch is a step in the right direction, but it is just the beginning.

i hope that cloud computing developments will give this industry a push it needs to clean up its act and finally come up with comprehensive convenient set of tools that is sorely overdue.

bottom line is that with clouds there will be more stuff to manage, and the demands for the management tools will become stricter.

i want commodization of these tools – monitoring, deployment, etc. ideally they will be independent of cloud provider and i want them to be simple and transparent and composable and hackable.

usage scenarios

it is almost a no-brainer these days for startups writing green-field apps to rely on amazon for their capacity. it is great, since they are not as concerned with SLAs and lock-in is not that big of a deal.

amazon truly has kick-started the industry that lurked in the shadows for so many years with all the major players trying and failing to launch a successful pay-as-you-go cloud computing service. i guess their secret was extracting a service from their own successful projects, instead of trying to come up with something new for others.

as for enterprises though, i do not see them embracing the cloud platform as it is now, however there are some scenarios possible right now, and i am sure many more will emerge:

• a single batch that needs to be processed (amazon now has hadoop services ; also consider load testing from the cloud, some sort of one-off processing, or anything else that is a spike in demand could be good)

• overflow capacity for something like flash traffic during holidays for retailers – extend your private infrastructure, or certain pieces of it into the cloud and wind down when you are done

• POC, development – it is hard to test on the same hardware as prod, so roll your stuff out on the cloud and then for prod/uat deployment put it back inside

• one-off isolated apps that can deal with current state of SLAs and lack of private clouds, or even pieces of existing apps

there is a chance for internal infrastructure folks to learn something from the cloud experience. in most large companies servers and storage have been increasingly provided as a service and billed back to the projects, but provisioning is still not elastic, and utilization is often the usual low numbers. granted, more and more are using vmware to virtualize (at least dev/uat) to increase utilization, but it is a far cry from the experience of the likes of amazon.

the ideas and some of the tech from the public clouds could and should be looked at by the internal infrastructure people, and not just the servers/storage, but also those that provide middleware services – messaging and monitoring, for instance.

clouds in your own data centers? some of the vendors seem to head this direction – give us the servers and the datacenters and we will create cloud infrastructure for you, taking care of management, monitoring, apis, provisioning, etc.

i wonder how the chargebacks in organizations will change between departments – since instead of a capital investment upfront with depreciation we can convert it to operating expense with pay for use billing – would that allow us to win anything? of course, for small companies it is great – no upfront costs, no hardware, etc.

ms azure is somewhat an outlier here – first of all, it lacks tinkerability (yes, it is a term) – the barrier for entry is too high, but at the same time it is not a vertical app, but a full-blown platform with all the development tools in place. the problem is that it is too opaque, and i would imagine it will be a while before the whole kitchen sink has proper api, billing, provisioning, etc working properly – as usual the idea is fantastic, but the implementation, knowing the track record, is something to be wary of. since windows is notoriously hard to automate, perhaps azure will finally sidestep this problem and offer that as a service, but i cannot see how it can completely do it, unless it provides something as easy and transparent as ssh+shell scripts and declarative versioned management for that.

coming up

development in several directions – granularity of service, privacy of the cloud; standards – one must talk about a certain place in the continuum – standards for infrastructure level monitoring/provisioning are perhaps possible, but once we run into more vertical services, standards make little sense, since you are buying into the provider.

there will be consolidation, since, as failures of ibm and sun showed and success of amazon shows, it is a low-margin business, and amazon knows it first-hand.