Archive
Big News: #Dell and #Cloudera Partner on #Hadoop Solution
.
Big news!
Today, Dell (the company I work for) has announced a solution offering in conjunction with Cloudera, called the Dell | Cloudera Solution for Apache Hadoop! It’s a fully deployable Hadoop solution made up of Dell PowerEdge C hardware, Dell and Cloudera software, and Dell and Cloudera services – the complete package.
(Note that this is the second solution that my team has put out in the last two weeks – if you missed last week’s OpenStack announcement, you can get more info here.)
But on to the star of today’s show!
We’d often hear from our customers that they were experimenting with Hadoop on Dell hardware to analyze sales data, marketing information, etc in order to better serve their own customers. And pretty soon, we were being asked, “What’s Dell’s point of view? What recommendations do you have for distributions, architecture, and implementation?”
Well, here’s the answer. The Dell | Cloudera Solution for Apache Hadoop.
What’s Under the Hood
- Hardware – similar to OpenStack, the reference architecture that this solution is built on is the innovative Dell PowerEdge C server line. Based out of Dell’s well known Data Center Solutions team (aka DCS), PowerEdge C server technology was born out of learnings from custom engagements with some of the biggest cloud and hyperscale environments in the world. What we discovered is that, despite custom requirements and specs, there was some commonality among the requests. Concepts like density, power efficiency, etc were at the top of the list when it came to these environments.
So we took what was common and built a line of servers that was ideal for cloud and hyperscale environments, which became the Dell PowerEdge C server family. And it is an ideal platform for our Hadoop solution, as all the tenets of scale apply when building out an optimal Hadoop instance.
- Software – Our partners, Cloudera, who are among the most established Hadoop vendors in the market, will be providing their distributions of the Hadoop software, as well as their CDH management tools, which are top notch. In addition, Dell will be providing our Crowbar software framework and a specific Cloudera barclamp as part of the solution. (As a refresher, we announced availability of Crowbar last week as part of our OpenStack announcement, which helps speed deployment CONSIDERABLY. In the case of the Dell Cloudera Solution for Apache Hadoop, it can speed bare metal deployment of a Hadoop cluster from days or weeks to less than one day!)
- Services – Dell and Cloudera are partnering to enable our customers to get going on a validated instance of Hadoop, and offer a variety of services from support of the entire stack of hardware and software, as well as training on the technologies that will enable our customers on Hadoop.
Now’s the time to get started on Hadoop, and the Dell | Cloudera Solution for Apache Hadoop can get you there.
Find out more about this new solution at www.Dell.com/Hadoop or get in touch with us at Hadoop@Dell.com.
More info:
- The Dell Cloudera solution website – http://www.Dell.com/Hadoop
- Barton George’s blog – http://bartongeorge.net/2011/08/04/introducing-the-dell-cloudera-solution-for-hadoop-harnessing-the-power-of-big-data/
- Rob Hirschfeld’s blog – http://www.RobHirschfeld.com/2011/08/04/dell-cloudera-hadoop
- My recent blog about Crowbar – https://jbgeorge.net/2011/08/01/the-dell-crowbar-software-framework-who-what-when-where-and-how/
- Email us for more info at Hadoop@Dell.com
Until next time,
JBG
@jbgeorge
The #Dell #Crowbar Software Framework: Who, What, When, Where, and How
.
It’s been about a week since Dell (the company I work for) officially announced the Dell OpenStack Cloud Solution, the market’s first HW / SW / Services solution for OpenStack – and the response has been great!
(Check out my last blog post about it here for more details.)
Crowbar from Dell
Since this wasn’t your average announcement with a big company like Dell actually contributing to the community, I received a number of questions, comments, and accolades around our contributed software framework – Crowbar.
So I thought I’d pass on some of the people, history, functionality, and background behind Dell’s Crowbar software.
History
Since Dell had been a committed partner of OpenStack’s since it’s announcement in July of 2010 (over a year ago), we were among the first to begin working with the OpenStack code as we began developing the Dell solution for it. Since it was very early, very raw code, there was a lot of installing / testing / coding / re-testing / blowing it away / reinstalling / testing / coding / re-testing / blowing it away… you get the idea.
And as any cloud technology goes, deploying OpenStack on bare metal took time, effort, and expertise.
It wasn’t long before our solutions development team decided we’d automate this process to make our efforts quicker. But lo and behold, we discovered that others in the community were dealing with the same thing. A few early users indicated that it was taking them a full day to deploy OpenStack from start to finish, and in most cases, it was taking multiple days.
So we decided build out a quick, automated method of deploying OpenStack on bare metal technology that the whole community could use.
And Crowbar was born.
Functionality
Extending the capabilities of the Chef configuration management framework from Opscode, our team began developing Crowbar with the goal of installing a multi-node OpenStack cloud on bare metal PowerEdge C servers in less than four hours. “Multi-node” was a key requirement as almost every use case we encountered required that.
We announced Crowbar’s existence and development in early 2011, with our intent to field test, and then release to the open source community. Over the course of time, we began testing it out in various instances and scenarios, as well as demo-ing it at various cloud events. Many of you may remember us showcasing our progress at SCaLE, CloudConnect, and the Apr OpenStack Design Summit (which Dell sponsored).
With time and experience, we came across a number of requirements for this type of a tool – automated BIOS configuration, RAID configuration, network discovery and set up, installation of the open source system monitoring tool Nagios, installation of the open source performance monitoring tool Ganglia, and more. So we built it all in.
And based on the extensive cloud cloud experience on our team, we knew that cloud development and management is an ever evolving exercise – as the Dell OpenStack lead architect and OpenStack community celeb Rob Hirschfeld would say, “Cloud is always ready, and never finished.” So Crowbar is designed in a way to manage change. Add in a new supported server to the mix, and Crowbar detects it and proceeds to add it to the OpenStack pool. When software needs to be updated, Crowbar can be leveraged to update once, and deploy to all.
Barclamps
Dell Crowbar featuring Barclamps!
And what about new components that come out? There are a number of new OpenStack capabilities coming out of various projects – how do those get rolled in?
Barclamps.
(Yes, we designed Crowbar with modularity in mind.)
Think of it this way: Crowbar is made up of two general functionality categories – the Crowbar core and Crowbar barclamps. The Crowbar core contains all the main functionality that allows Crowbar to perform tasks. Barclamps are modules that actually perform a function. So need to deploy OpenStack Nova (aka cloud compute)? Leverage the Nova barclamp. Need to deploy OpenStack Swift (aka cloud storage)? Leverage the Swift barclamp.
Need to deploy some software that doesn’t have a barclamp?
You can create your own barclamp!
And that’s the beauty of Crowbar – we created (and will continue to create) barclamps that work for our customers, but we’re encouraging / training our customers to develop their own barclamps, and we want the community at large to do the same.
What real customers are doing with the Crowbar and the Dell OpenStack Cloud Solution
In our press release, one of our OpenStack customers, DreamHost, spoke about their success with the Dell OpenStack Cloud Solution. Along with Dell server technology and services, the innovative DreamHost team really picked up on the Crowbar tool. In fact, as they are working on hosted options for their Ceph technology, the DreamHost team has begun developing a barclamp for Ceph!
In fact, here’s what the DreamHost GM of Emerging Technologies, Ben Cherian, had to say about Crowbar:
“If Dell had not developed Crowbar, we would have been forced to write a similar tool ourselves. It’s a fundamental goal that we have in the company: automate, automate, automate. That’s how we drive down prices.”
Ben also went on to say that since Dell had already developed a technology like Crowbar, we ended up saving them four to six months of development time.
Common Questions
- Is Crowbar for Dell products only?
The Crowbar software that we’ve developed at Dell will obviously work with Dell supported platforms. But there is no reason that this framework will not work for any other vendor technology – all that needs to happen is for that vendor / user / community member to create a barclamp for that technology.
- Is Crowbar for OpenStack only?
Obviously, our original use case is for OpenStack – it has been tested and validated for that model. However, again, it is a flexible software framework. We can see a number of use cases for OpenStack, and welcome the community to use it where it makes sense.
- What do I get with Crowbar as a part of the Dell OpenStack Cloud Solution vs open source Crowbar?
There are two main differences at this point:
First, open source Crowbar has everything that the Dell OpenStack Cloud Solution Crowbar has, except for the BIOS and RAID capabilities, where we’re trying to clear some legal hurdles before we open source. Once those are cleared, we expect to open source those capabilities.
Second, when you get Crowbar as a part of the Dell OpenStack Cloud Solution, you will get full support from Dell via Dell Services. Any questions, support, training you need on Crowbar, Dell delivers it.
Where do I get Crowbar?
OK, the question of the hour. How do you get Crowbar?
- You get Dell-supported Crowbar when you purchase the Dell OpenStack Cloud Solution.
- You get open source Crowbar from our github site – http://www.github.com/dellcloudedge.
You can head over to http://www.RobHirschfeld.com and see much more TECHNICAL content on the Dell OpenStack Cloud Solution and Crowbar. In addition to being our lead architect on OpenStack, Rob is actually one of the developers of the actual Crowbar software. (You can also follow him on Twitter at @zehicle.)
And as always, if you want to learn more about the Dell OpenStack Cloud Solution, you can head to http://www.Dell.com/OpenStack. Or if you’d like to reach out to us to learn more, drop me and the gang a line at OpenStack@Dell.com.
Until next time,
JOSEPH
@jbgeorge
Dell OpenStack Cloud Solution
The JBGeorge Tech Blog 