Foggy Software

Or, the Animal kingdom in the cloud Something about IT seems to attract references to the animal kingdom. It might be caused by lively imagination or the unkempt nature of practitioners in the field (I'm thinking about myself). For example, the Agile movement seems to like the story of " the Chicken and the Pig ". Just as an indication as to how prevalent this meme became, I've been asked (actually, usually I do the asking) - who are the chickens and the pigs in a meeting. Only those few unaware of the meme took offense. As most powerful  memes , this one helps in communicating briefly a much more complex set of ideas, hence it's power. "You promised Cats…." you might say about now, "and all you've talked about are pigs. And when did this blog become veterinarian focused?" Bare with me for a bit... Ok. Cats. A more recent meme I've been hearing about is "Cat vs Cattle", or more frequently known as "Pet...

Or, the one and the many The inspiration for this post was a discussion with Q&A folks about how Crowbar should behave when failures are encountered while configuring  the storage subsystem on a node. Well, that and a binge of reading and listening to folks talking about Lean Startups and the importance of solving real customer issues. The Q&A engineer was adamant, that on a server with 24 drives, Crowbar should be just ignoring a single failed drive, and just use the other 23. For the use case he was trying to solve, this might make sense. He had limited resources (only a handful of servers), and needed to quickly turn up a cluster. The fact that Crowbar flagged the server with the bad disk as having a problem, and refused to use it was nothing but annoyance to him. Crowbar was designed to enable DevOps operations at very large scale. In a recent customer install (more about it in another post, i hope), the customer purchased 5 racks of servers, rather than 5 ...

Or, you control your bits Traditional storage solutions gravitated towards some central bank of disks - SAN, NAS, Fiber Channel, take your pick, they share a few traits that are not very democratic: They cost lots, and large parts of the cost is Intellectual Property embedded in the solution (i.e. the markup on the underlying hardware is huge) The OEM makes lots of trade-off decisions for you - e.g. ratio of controllers to disks, replication policies and lots of others (most OEM's graciously expose some options that the user can control, but those are just a tiny fraction) They typically require 'forklift updates' - if you use up your capacity, call the forklift to install the next increment, which typically requires a forklift worth of equipment On the plus side, in general those type of systems provide you with reliable, performant storage solution (based on the $$$ you spend, you get more of either qualities). But, in the world of large scale deployments ba...

Or, some less than obvious reasons why refactoring is "A Good Thing" At a meetup tonight, someone challenged me to explain what's really good about Openstack. This was in the context of Openstack-Boston /  Chef-Boston discussion about Openstack and the effort around Community deployment cookbooks, and an approach that uses Pull From Source (which I'll post about in a later date). While I could have spent lots of time describing the CI testing infrastructure and the great work done by Monty and his team, frankly that's not unique to Openstack. It's an enabler for lots of other things. To me, one of the primary sources of excellence in Openstack is the courage to refactor. Not too long ago, there were only 2 services - Nova for Compute, and Swift for Object storage. In Grizzly, through large efforts, there are dedicated services with clear focus and dedicated team passionate about the technology area each services. One of the first refactors was Key...

  Or, why swift rocks Swift is the Object Storage component of Openstack. Object Storage is a new paradigm for storage. It’s not files (as in a filesystem) or blocks (as in SCSI variants), but rather Objects. Objects are immutable; once written, their contents can’t be changed, only replaced. Why would you want that? Think facebook, or tumblr or flickr – you’re not likely to ever update the content of an image… and the benefits that Swift brings are worth the loss of this capability. To clear some confusion, Swift is SOFTWARE. It is not a piece of hardware or an appliance, like a NAS box. It is not a Service (Like the S3 offering from Amazon, or CloudFiles from Rackspace).  Swift is software, and free open-source software at that. The cost to deploy it is driven by the choices of hardware (physical or virtual) and operational choices made. But what considerations drive those decisions, and what are the tradeoffs? A coarse understanding of what magic Swift performs and how,...

Or, my impressions from the OpenStack Folsom Summit OpenStack proved again that it's a community effort. The number of developers in design half and overall participants in the Conference part was amazing. Most design sessions (I focused on Quantum and a few Nova sessions) were standing room only. literally. The representation of big name vendors was palpable, though it was nice to see that big names didn't necessarily get undue influence on the conversation. (For my part, the fun started on the way there , on an RV for 1800 miles...but that's a story told somewhere else ) My major takeaway is that Openstack is open for business, in multiple ways. The first is that Openstack is production ready. This was manifested even prior to the summit, with dueling announcements from HP and Rackspace about deploying the Essex release into their respective production public clouds. A second way in which business friendliness is achieved is by opportunities More supporting e...

Or, Running Lisp Probabilistically, backwards. It started at a talk I attended in previous post . Yes, there's been lots of talk about analytics and cloud, even made famous in popular media .... But the "whats" and the "hows" are in constant flux. The curiosity triggered by the subtitle, lead to a weekend filled with AI and to this post. AI is a big space. I'm focusing on one small part, Machine Learning. And specifically, translation and categorization, in exploring how The cloud supports AI, and AI supports the cloud... What does the could have to do with machine translation? The early attempts at machine translation (circa 1950's) went down the path of Natural Language Processing. They failed for various reasons. current approaches  (do the quizzes!!), as in Google Translate, play a matching game. Use the cloud to collect ridiculous amounts of sample translations, e.g a newspaper which publishes in more than language, restaurant menus an...

Or, if you have the bits, but not the information.... I'm sitting at an event  which combines a common thread of thoughts that has been floating in my head - Hadoop deals with lots of data... but how do I get to the Information contained there. ThaDa... Big Data and machine learning are made for each other. I've just learned about the Mahout  - the tool to make information out of data, by using machine learning! Definitely something to look at. Update: I've spent some time reading and digesting some of the AI topics, with the results in this followup  post

Or ... some thoughts about organizing OpenStack Meetups On February 1st I've had to joy (and hassle) to coordinate another Boston OpenStack meetu p. This time, using Harvard facilities, which are quite different than our previous venue - suffice to say that Harvard is very different than the Lexington Historical Society (you don't need to know where the vacuum is). But I digress. If you were in the room, you know all about the value of a community, so skip ahead to the "closing notes" for links to preso's, future events, acknowledgements and such. What I find exciting about open source projects is the sense of shared mission and destiny. An open source project fails or succeeds to a large extent based on how well it builds a community.  To be successful a project needs to create a dedicated community of users, developers, vendors and service providers. Users have real problems to solve. Real use cases, real businesses, real money. Devel...

Or, what does it really mean to be highly available in the cloud Good IT practices try to maximize SLA conformance, especially around availability. Lessons learned from a disk failure in the Exchange server leading to mail outages and the inevitable fire drills have been deeply embedded into minds. REDUNDANCY EVERYWHERE. power supplies, network connections, disks - if you can put 2 of them suckers in there, you do. Just to keep that machine running. That machine should never fail. The web has mitigated things somewhat. Rather than a relying on hardware redundancy (where you don't use half your equipment), deployment strategies have evolved. A large pool of web servers can sustain SLA's with some servers failing by utilizing load-balancers to only direct traffic to live web servers. This scheme brings with it worries about session state availability and other share information (e.g database) but nonetheless its progress. Since hardware is now allowed to fail, software...

or,dissipating   the fog in the cloud Update (Feb 3,2012):  read the meetup retrospective here OpenStack  is creating lots of buzz in the IT industry. Its promise is to revolutionize the cloud market in the same way that Linux revolutionized the operating system market. Part of the reason that Linux was successful is the community of devotees that saw the potential and ushered the code base to where it is today. But communities don't always just happen, they're made with sweat and pizza. My group has taken on the goal of making OpenStack users in the Boston area a vibrant community. The promise of OpenStack and quick pace of activity has spurred a worldwide  wave of excitement, with local groups spanning the globe - from Argentina  to Japan, Australia to British Columbia. I half stumbled into organizing the  Boston meetup group . Being an engineer, it gave me a new appreciation to what it takes to make an event happen. From gett...

or, what's this DevOps thing everybody is talking about Agile in software development is an attempt to be more real - rather than attempt to predict the future, the methodology is all about taking stabs and correcting with a fast feedback cycle. Start producing real value quickly, and keep chasing the value fanatically. In a recent Openstack meetup discussing Swift  an attendee asked - so swift has all these parameters, do you have good guidance on how to set them ?  A traditional approach would involve enormous efforts attempting to predict workloads, methodically simulating them in a lab environment, and laboriously attempting to tune parameter after parameter to find the optimum setting for those workloads. Can you predict what would happen? You will end up deploying optimized the system based on the lab results driven by the simulated workloads.  By the second week in production you realize that the actual workloads are substantially different t...

or... what does actually matter in an agile process? The holly grail of any software outfit is: deliver high value, high quality results; consistently and predictably. Talk about a loaded sentence ! but those are the requirements to make a software team  valuable to the business (or is the business). Breaking it down:: Team - A single developer doesn't need process.... And finding a unified process that works a huge conglamorate corp is not likely. A process then is meant for a TEAM of some reasonable size. High value - if you deliver the wrong thing at the right time.... well, I've been at too many (now deceased) startups to recognize that value is key.  High quality - need I say more? If you want to deliver 2.0, but your stuck bug-fixing 1.0 into existence you will have 2 customers - the one that is unhappy with 1.0 and the one waiting for the features in 2.0. Neither would keep the revenue coming in. Results - activities by themselves, with no re...

Or, here's my pet mini-cloud I first used VMWare workstation as a development tool somewhere around 20003. It was a cost effective way to test the software I was working on against quirks of different operating systems. On one linux desktop I could have all my development tools, including a few Windows95 installations. So, now you're asking - "what does that have to do with  clouds?" Well, no much yet. For my current project ( Crowbar ) I found myself needing a setup that includes: Multiple OS flavors  Win XP for access to corporate resources ubuntu 10.10 and 11.04 - main testing targets Redhat  and Centos - for some dev work and testing Varying numbers of machines, pretty much depending on the day For basic development 3 machines - development, admin node and a target node For swift and nova (components of openstack ) related development - at least 5, sometimes a few more Weird-ish network configurations Network segments to simulate public, pr...

Or.. .why do you care about DevOps (and why you want devOps ++) If you've ever written (or tested) a piece of software, you've heard this all before - "but... it works in my setup". That endless back and forth of finger pointing, until way too often, it ends up being some silly environment setting that causes the application or service to miss-behave. DevOps is there to help. No more manual deployment steps to deploy an app. No longer 13 page project plans requiring coordination of at least 5 geo-distributed teams (half of which are in a timezone where the sun has long stopped shining). The DevOps motto is simple - "if it hurts and its dangerous, do it often". The industry has accepted this in the Continuous Integration case - builds that integrate disparate development streams are painful. Assumptions made in isolation prove to be wrong, and whole hell breaks loose when you try to put different, independently developed pieces together. This is a pai...

Recently I’ve been finding myself at the same awkward moment repeatedly; a social gathering of one sort or another, introductions, platitudes and the inevitable 2 questions:  “what do you do?” and “oh, software, what kind of software?” I have been experimenting with different explanations and examples, but judging by the faces looking at me I might as well just said I herd cats for a living. Using familiar examples seems to resonate, it goes something like this: A:  do you used Gmail or Yahoo? WEP:  “yes… doesn’t everyone? A: Why then you’re using ‘’the cloud’’! That is just an example of SaaS. Ok. That gets the ball rolling, a bit.  But WEP (wide eyed person) is somewhat left unconvinced about this whole cloud thing. Heck, web based email has been around long before Cisco started advertising cloud on TV. Setting up this blog triggered a thought – blogging is almost as common as ATM machines and Oprah. What if this could be the linchpin in explaining cloud to my m...

I have succumbed to the web completely - so here's a blog. Working on open source projects and interacting with the world seems to absolutely mandate that I have a blog.  Well - here it is. Some of the topics I'm involved with include, in no particular order: Open source cloud platforms, mostly OpenStack but poking at things SOA and Cloud - best buddies? Java, python ruby php and their ilk seem to be part of the territory... For the occasional excursion from excitement I try to go hiking camping and when the weather in the US NE is like today, riding. I'm still hoping to get (back) around to flying and fly my homebuilt Q200 (at that point in time, I'd probably need a pig-radar) ... that's it for today. I've got me some RAID's to configure!