Archive
Living Large: The Challenge of Storing Video, Graphics, and other “LARGE Data”
.
UPDATE SEP 2, 2016: SUSE has released a brand new customer case study on 
“large data” featuring New York’s Orchard Park Police Department, focused on video storage. Read the full story here!
————–
This is a duplicate of a blog I authored for SUSE, originally published at the SUSE Blog Site.
Everyone’s talking about “big data” – I’m even hearing about “small data” – but those of you who deal in video, audio, and graphics are in the throes of a new challenge: large data.
Big Data vs Large Data
It’s 2016 – we’ve all heard about big data in some capacity – generally speaking, it is truckloads of data points from various sources, magnanimous in its volume (LOTS of individual pieces of data), its velocity (the speed at which that data is generated), and its variety (both structured and unstructured data types). Open source projects like Hadoop have been enabling a generation of analytics work on big data.
So what in the world am I referring to when I say “large data?”
For comparison, while “big data” is a significant number of individual data that is of “normal” size, I’m defining “large data” as an individual piece of data that is massive in its size. Large data, generally, is not required to have real-time or fast access, and is often unstructured in its form (ie. doesn’t conform to the parameters of relational databases).
Some examples of large data:
Why Traditional Storage Has Trouble with Large Data
So why not just throw this into our legacy storage appliances?
Traditional storage solutions (much of what is in most datacenters today) is great at handling standard data. Meaning, data that is:
- average / normal in individual data size
- structured and fits nicely in a relational database
- when totaled up, doesn’t exceed ~400TB or so in total space
Unfortunately, none of this works for large data. Large data, due to its size, can consume traditional storage appliances VERY rapidly. And, since traditional storage was developed when data was thought of in smaller terms (megabytes and gigabytes), large data on traditional storage can bring about performance / SLA impacts.
So when it comes to large data, traditional storage ends up being consumed too rapidly, forcing us to consider adding expensive traditional storage appliances to accommodate.
“Overall cost, performance concerns, complexity and inability to support innovation are the top four frustrations with current storage systems.” – SUSE, Software Defined Storage Research Findings (Aug 2016)
Object Storage Tames Large Data
Object storage, as a technology, is designed to handle storage of large, unstructured data from the ground up. And since it was built on scalable cloud principles, it can scale to terabytes, petabytes, exabytes, and theoretically beyond.
When you introduce open source to the equation of object storage software, the economics of the whole solution become even better. And since scale-out, open source object storage is essentially software running on commodity servers with local drives, the object storage should scale without issue – as more capacity is needed, you just add more servers.
When it comes to large data – data that is unstructured and individually large, such as video, audio, and graphics – SUSE Enterprise Storage provides the open, scalable, cost-effective, and performant storage experience you need.
SUSE Enterprise Storage – Using Object Storage to Tame Large Data
Here’s how:
- It is designed from the ground-up to tackle large data. The Ceph project, which is core of SUSE Enterprise Storage, is built on a foundation of RADOS (Reliable Autonomic Distributed Object Store), and leverages the CRUSH algorithm to scale data across whatever size cluster you have available, without performance hits.
- It provides a frequent and rapid innovation pace. It is 100% open source, which means you have the power of the Ceph community to drive innovation, like erasure coding. SUSE gives these advantages to their customers by providing a full updated release every six months, while other Ceph vendors give customers large data features only as part of a once-a-year release.
- It offers pricing that works for large data. Many object storage vendors, both commercial and open source, choose to charge you based on how much data you store. The price to store 50TB of large data is different than the price to store 100TB of large data, and there is a different price if you want to store 400TB of large data – even if all that data stays on one server! SUSE chooses to provide their customers “per node” pricing – you pay subscription only as servers are added. And when you use storage dense servers, like the HPE Apollo 4000 storage servers, you get tremendous value.
Why wait? It’s time to kick off a conversation with your SUSE rep on how SUSE Enterprise Storage can help you with your large data storage needs. You can also click here to learn more.
Until next time,
JOSEPH
Address Your Company’s Data Explosion with Storage That Scales
.
This is a duplicate of a blog I authored for SUSE, originally published at the SUSE Blog Site.
Experts predict that our world will generate 44 ZETTABYTES of digital data by 2020.
How about some context?
Now, you may think that these are all teenage selfies and funny cat videos – in actuality, much of it is legitimate data your company will need to stay competitive and to serve your customers.
The Data Explosion Happening in YOUR Industry
Some interesting factoids:
- An automated manufacturing facility can generate many terabytes of data in a single hour.
- In the airline industry, a commercial airplane can generate upwards of 40 TB of data per hour.
- Mining and drilling companies can gather multiple terabytes of data per minute in their day-to-day operations.
- In the retail world, a single store can collect many TB of customer data, financial data, and inventory data.
- Hospitals quickly generate terabytes of data on patient health, medical equipment data, and patient x-rays.
The list goes on and on. Service providers, telecommunications, digital media, law enforcement, energy companies, HPC research groups, governments, the financial world, and many other industries (including yours) are experiencing this data deluge now.
And with terabytes of data being generated by single products by the hour or by the minute, the next stop is coming up quick: PETABYTES OF DATA.
Status Quo Doesn’t Cut It
I know what you’re thinking: “What’s the problem? I‘ve had a storage solution in place for years. It should be fine.”
Not quite.
- You are going to need to deal with a LOT more data than you are storing today in order to maintain your competitive edge.
- The storage solutions you’ve been using for years have likely not been designed to handle this unfathomable amount of data.
- The costs of merely “adding more” of your current storage solutions to deal with this amount of data can be extremely expensive.
The good news is that there is a way to store data at this scale with better performance at a much better price point.
Open Source Scale Out Storage
Why is this route better?
- It was designed from the ground up for scale.
Much like how mobile devices changed the way we communicate / interact / take pictures / trade stock, scale out storage is different design for storage. Instead of all-in-one storage boxes, it uses a “distributed model” – farming out the storage to as many servers / hard drives as it has access to, making it very scalable and very performant. (Cloud environments leverage a very similar model for computing.) - It’s cost is primarily commodity servers with hard drives and software.
Traditional storage solutions are expensive to scale in capacity or performance. Instead of expensive engineered black boxes, we are looking at commodity servers and a bit of software that sits on each server – you then just add a “software + server” combo as you need to scale. - When you go open source, the software benefits get even better.
Much like other open source technologies, like Linux operating systems, open source scale out storage allows users to take advantage of rapid innovation from the developer communities, as well as cost benefits which are primarily support or services, as opposed to software license fees.
Ready. Set. Go.
At SUSE, we’ve put this together in an offering called SUSE Enterprise Storage, an intelligent software-defined storage management solution, powered by the open source Ceph project.
It delivers what we’ve talked about: open source scale out storage. It scales, it performs, and it’s open source – a great solution to manage all that data that’s coming your way, that will scale as your data needs grow.
And with SUSE behind you, you’ll get full services and support to any level you need.
OK, enough talk – it’s time for you to get started.
And here’s a great way to kick this off: Go get your FREE TRIAL of SUSE Enterprise Storage. Just click this link, and you’ll be directed to the site (note you’ll be prompted to do a quick registration.) It will give you quick access to the scale out storage tech we’ve talked about, and you can begin your transition over to the new evolution of storage technology.
Until next time,
JOSEPH
@jbgeorge
The Next Adventure Begins…
.
As I’ve updated my audiences on SoMe this week, I’ve taken on an executive role at SUSE, where I will lead the efforts for SUSE’s solutions strategy when it comes to cloud, storage, big data, and more.
The company has been doing a great job in these areas, and I’ve been fortunate to partner with them in various roles across OpenStack, Ceph, and more.
My goal is to start with the good work already deployed in market, and help take it to the next level to enable customers to address serious business challenges with these solutions, simply and effectively.
More to come as I get my feet set in the new role, but I am very excited to join SUSE, a company that understands the emerging technology space, it’s customers, and it’s communities – and is poised to do big things with it’s contributions to open source and it’s commercial offerings.
Let the adventure begin!
Until next time,
JBG
@jbgeorge
Michael Dell Comments on the “Data Economy”
This is a repost of my blog at http://dell.to/1euUcqk.
In this short interview with Inc., Michael Dell provides an overview of the company’s transformation into a leading player in the “data economy.”
As Michael notes, with the costs of collecting data decreasing, more companies in a growing number of industries are making better use of existing data sources, and gathering data from new sources.
And that’s where Dell has been enabling customers for years with solutions built with technologies like Hadoop and NoSql. Helping companies and organizations make better use of this data, and assisting them in using it to solve their challenges, are just a few of the ways Dell has changed the Big Data conversation, and built an entirely new enterprise business along the way.
As a member of the Technology CEO Council, Michael also recently joined other tech CEOs to discuss the data economy with policy makers. As an example of the potential of the data economy, he explained how Dell’s growing health information technology practice includes 7 billion medical images. These images are in an aggregated data set allowing researchers to mine them for patterns and predictive analytics.
“There’s lots that can be done with this data that was very, very siloed in the past,” Michael toldComputerworld, “We’re really just kind of scratching the surface.”
It’s certainly an exciting time to be at Dell – and the data revolution continues!
THIS JUST IN: VMware and Dell Partner to Enable Cloud Foundry via #Crowbar
.
And the goodness just keeps on coming!
A few weeks ago, Dell (they company I work for) unleashed the power of the Dell developed, open source Crowbar software framework as a part of the announcement of the Dell OpenStack Cloud Solution. It allowed users to deploy a full OpenStack IaaS cloud on bare metal PowerEdge C servers in less than two hours (vs multiple days if done manually), and allows for a continuous integration mechanism for the stood up cloud.
A week later, we announced the Dell | Cloudera Solution for Apache Hadoop, which also leverages the powerful Crowbar software to deploy a running Hadoop cluster on to bare metal PowerEdge C servers in less than a day, where it can take days or even weeks if deployed by other means.
So….
Infrastructure as a Service (IaaS)? Check.
Hadoop / Big Data? Check.
But what about Platform as a Service (PaaS)?
Big time check.
Today, VMware is announcing their development of a Cloud Foundry barclamp for Dell’s Crowbar software!
VMware’s Cloud Foundry is an open platform as a service (PaaS) project initiated by VMware designed to support multiple frameworks, multiple cloud providers, and multiple application services all on a cloud scale platform. It’s a project that is only a few months old, but one that has been getting outstanding interest from enterprises who want PaaS to be the new developer UI to their private cloud. And Cloud Foundry is already powering real solutions.
And now it can be deployed quickly, simply, and automated with Dell’s Crowbar software.
Crowbar, software that leverages OpsCode’s Chef configuration management tool, allows users to get up and running on powerful technologies like Cloud Foundry, but actually does much more. It actually does BIOS configuration, RAID configuration, network discovery, deploys Nagios and Ganglia, and more to enable an environment ideal for complex technologies. It is also aware of changes in its environment, and adjusts to them in an automated manner. (Learn more about Crowbar here.)
This is another win for open source in my book, and a real indicator of the impact open source is going to have on the next era of IT.
So who’s the next Crowbar barclamp rockstar?
You tell me.
You. Crowbar. Download. Build barclamp. Share.
I’d love to be telling your story here next. 🙂
More info:
- The Cloud Foundry Blog – http://blog.cloudfoundry.com
- Get open source Crowbar – www.github.com/DellCloudEdge
- Rob Hirschfeld’s Blog (Dell)
- Barton George’s Blog (Dell)
Until next time,
JBGeorge
@jbgeorge
Citrix Synergy 2011: Day 1
- Welcome to Citrix Synergy 2011!
.
Hello from San Fran!
Boy, what a day at Day 1 of the Citrix Synergy event. I’m kicking back at the Serrano Hotel after a long day of meeting, greeting, networking, and learning at they event. Made a lot of new friends today, and got to touch base with a lot of old acquaintences.
I had a lot of high hopes for the conference this week. And the keynote got it off to a rocking start.
I was particularly interested in the OpenStack announcement, but there was a ton of great announcements and demos today. Also had some great meetings with the press today since Dell was a part of the announcements.
Here are some of the highlights I found interesting…
- First off, the OpenStack news – Citrix announced “Project Olympus” which would include a Citrix distribution of OpenStack and a cloud-optimized version of XenServer. There’s an early access program that customers can engage in to work with Citrix, Dell, and Rackspace to get things kicked off there. Check out more at my blog on it here.
- Google did a pretty cool thing by doing a demo with the Citrix crew on stage, and then proceeded to give away about 100 Chromebooks to keynote attendees. Unfortunately, I was not one of them.
- Citrix CEO Mark Templeton did a great job of pitching a nice connected story of BYO-ThreePC – Personal Cloud, Private Cloud, and Public Cloud.
- Citrix showcased a couple of new new NetScaler products – Cloud Gateway to helps in orchestrating web and windows apps, and Cloud Bridge to connect cloud data, directory, web, and apps between public and private clouds.
- Great demos on the new and improved Citrix Receiver. Lots of nice UI updates, drag and drop, etc. And features are translated into the administration side of things as well. It was also cool to see Android apps running in Windows via Citrix Receiver. Things are becoming seamless….
Some great quotes:
Live from Citrix Synergy 2011...
- Citrix’s mantra = “Whatever, whenever, wherever” – I like it. (Also heard a lot of “any, any, any”.)
- “Don’t encrypt the laptop – encrypts the data.”
- Consumerization is a major force and will force changes in IT
OK, well that’s all I’ve got in me for tonight – there was a ton more – feel free to comment about items you found interesting.
I’ll be back tomorrow with another update.
Until next time,
JOSEPH
@jbgeorge
One Giant Leap for Cloud: Citrix, Dell, and Rackspace Step Up with OpenStack
Last July, when the world learned about OpenStack for the first time, it was clear that it needed a group of partners to share the vision for OpenStack’s potential – open, standards based, and a platform for cloud innovation.
And there were partners those who stepped up.
Dell (my employer), Citrix, and others joined Rackspace and NASA, and committed to what they foresaw as a force in cloud. This was a critical juncture in OpenStack’s evolution – industry heavyweights had to be visionary at this stage, while OpenStack was still developing as a technology and as an initiative.
(I’m proud to say that Dell was the only hardware solutions vendor who committed to the OpenStack initiative right from the beginning.)
Fast forward to almost a year later, and OpenStack’s pioneer partners are once again stepping up to help drive OpenStack as a technology platform and further the OpenStack community.
Today, Citrix is announcing Project Olympus, a new cloud infrastructure product based on OpenStack, which will include a certified version of OpenStack as well as a cloud optimized version of XenServer.
And to help drive this, Citrix is announcing the launch of an Early Access Program, with support from Dell and Rackspace and a host of other partners in the OpenStack community, allowing customers to get – you guessed it – early access to Olympus.
(Read the entire announcement here – http://www.citrix.com/English/NE/news/news.asp?newsID=2311980)
At Dell, “Open, Capable, Affordable” is our mantra, and we view certified distributions of open source code as an important part of adoption. It provides customers with peace of mind knowing that a company like Citrix is behind them as they themselves step up to OpenStack. And Citrix is a company we all know and respect, and one that many of us regularly depend on as a software provider – Xen, XenServer, XenDesktop, NetScaler, and on and on.
With this announcement, it’s important that we not gloss over what we’re seeing happening here in the OpenStack community.
It’s not that these key vendors are just supporting the OpenStack movement…
They’re participating in it.
And, yes – it’s a big deal.
This isn’t an announcement of some new consortium based on OpenStack – this is an announcement about key technology leaders doing what they do best to advance an initiative we believe in. All of the companies mentioned have teams dedicated to developing OpenStack, contributing in technical conversations, learning about how customers can benefit from it, and driving the business of the open source cloud platform.
- Citrix is focusing its strengths and core competencies to help enable customers in the software stack via this new distribution and cloud optimized XenServer.
- Rackspace has launched an entire business unit to OpenStack installation, training, and support by way of Rackspace Cloud Builders.
- Dell’s OpenStack team (of which I’m a part) has been leading the way in bare metal deployment of multi-node OpenStack clouds with Crowbar, and an operational model to base full OpenStack solutions on. (You may have seen our live demos at Cloud Connect, SXSW, and the OpenStack Design Summit.)
And that’s the difference. No one is watching from the bench – these guys have been in since the beginning and are living it daily.
We, as the Openstack community, believe we need an open alternative, believe in OpenStack, and believe it is going to change how we develop, build, and run the cloud. And each of us brings our core competencies to the table to help mature the technology, enabling this community to grow and thrive.
And this is a great time for YOU to get involved in OpenStack as well. Check it out at www.openstack.org and get involved.
If you’re learning more about Project Olympus, check out the Olympus webpage at http://www.citrix.com/olympus.
If you’re interested in learning more about Dell and OpenStack visit www.Dell.com/OpenStack or email us at OpenStack@Dell.com.
Kudos to Citrix on Project Olympus, as well as to all of us in the OpenStack community, as we continue to drive this initiative forward.
I’m actually here in San Francisco at the Citrix Synergy conference this week, and I hope to be blogging on the happenings going on. If you’re here as well, I’d be interesting in hearing your thoughts on OpenStack, the announcement, and how you can participate as well. Contact me via Twitter and let’s chat. I’m @jbgeorge – try not to be distracted by the incredibly handsome profile picture. 🙂
Until next time,
JOSEPH
@jbgeorge
Learn more:
- Today’s Citrix announcement
- The Citrix Olympus website
- Barton George – Dell’s Cloud Evangelist on today’s news
- Rob Hirschfeld – Dell Cloud Architect on today’s news
- The OpenStack website
- Dell’s OpenStack website (with link’s to Dell’s OpenStack whitepaper – “Bootstrapping OpenStack Clouds”)
The “Open” Trend in Cloud…
"Yes We're Open!"
C’mon – all the cool kids are doing it…
Going “open” that is.
How many announcements have we seen in the last few months having something to do with open technology when it comes to the cloud space? With a great mix of business benefits AND customer / end user benefits, going open is certainly the latest trend.
I think this is a great and overdue market direction – standards are needed in this space, and I’m hopeful that this level of vendor response will help get us there. Additionally, its important that we understand that open technology, while enabling users, should also benefit vendors, so that they can continue to invest in and support open products.
Here’s a sampling of all the fantastic-ness.
—-
OpenStack
OpenStack
This one is near and dear to my heart since I’m the OpenStack business lead at Dell. But my view is that OpenStack really kicked off this season’s “open” trend. Announced last July, it’s an open source cloud platform that has gained serious momentum with over 60 vendors in the community and quite a few developers world wide. With the latest release of their code base (codenamed Cactus), and their recent Design Summit (at the Hyatt Regency in Santa Clara – where else?), service providers and large enterprises are taking a serious look at OpenStack as a viable cloud solution. (Learn more at www.OpenStack.org. Also shameless plug for www.Dell.com/OpenStack.)
—-
OpenCompute
Facebook helped launch this initiative as they strove to build a low cost, yet highly efficient computing infrastructure. Partnering with other industry stalwarts like Dell and Rackspace, Facebook opened up the specs on the efficient servers that make up their environment, in an effort to encourage other companies to build energy efficient infrastructures. (Learn more at www.opencompute.org.)
—
VMware Cloud Foundry (Open PaaS)
With its aquisition of the SpringSource Java development framework, and of RabbitMQ, the cloud messaging technology, VMware makes a bold move into the cloud space by unveling Cloud Foundry as an open source “Platform as a Service”. Focused on Java application developers , Cloud Foundry supports Ruby on Rails, Sinatra, and Rails, and allows developers a quick and easy way to get development platforms up and available.
Another plus – my good buddy Dave McCrory (@mccrory) is helping drive this initiative at VMware. 🙂
(Learn more at www.cloudfoundry.com)
—
openshift.redhat.com/app
OpenShift
Speaking of open PaaS, Red Hat announced its OpenShift Platform as a Service today. Their focus is to produce exportable workloads that can be used via private cloud or public cloud, such as Amazon. OpenShift will support Java, PHP, Python, and Ruby. Ironically, the open source is not yet open source, but will be shortly.
(Learn more at openshift.redhat.com/app.)
—
OpenFlow
How can I continue without mentioning OpenFlow? As many experts agree, the network will be one of the most critical components to cloud success. Created to help drive innovation in networking, a number of vendors are investigating ways to add OpenFlow as a feature to their networking portfolio. Dell, Nicira, and others are some interesting names in this space. I’m hopeful we’ll hear some interesting news about commercial networking providers and OpenFlow as Interop kicks off next week.
(Learn more at www.openflow.org.)
—-
So what do you think should be next here? Do you feel open sourcing cloud technologies advance us as an industry?
Drop me a line or leave a comment – would love to hear your thoughts.
Until next time.
JOSEPH
@jbgeorge
The JBGeorge Tech Blog 