Archive
HPC: Enabling Fraud Detection, Keeping Drivers Safe, Helping Cure Disease, and So Much More
.
This is a duplicate of a blog I authored for SUSE, originally published at the SUSE Blog Site.
Ask kids what they want to be when they grow up, and you will often hear aspirations like:
- “I want to be an astronaut, and go to outer space!”
- “I want to be a policeman / policewoman, and keep people safe!”
- “I want to be a doctor – I could find a cure for cancer!”
- “I want to build cars / planes / rocket ships / buildings / etc – the best ones in the world!”
- “I want to be an engineer in a high performance computing department!”
OK, that last one rarely comes up.
Actually, I’ve NEVER heard it come up.
But here’s the irony of it all…
That last item is often a major enabler for all the other items.
Surprised? A number of people are.
For many, “high performance computing” – or “HPC” for short – often has a reputation for being a largely academic engineering model, reserved for university PhDs seeking to prove out futuristic theories.
The reality is the high performance computing has influenced a number of things we take for granted on a daily basis, across a number of industries, from healthcare to finance to energy and more.
HPC has been an engineering staple in a number of industries for many years, and has enabled a number of the innovations we all enjoy on a daily basis. And it’s a critical function of how we will function as a society going forward.
Here are a few examples:
- Do you enjoy driving a car or other vehicle? Thank an HPC department at the auto manufacturer. There’s a good chance that an HPC effort was behind modeling the safety hazards that you may encounter as a driver. Unexpected road obstacles, component failure, wear and tear of parts after thousands of miles, and even human driver behavior and error.
- Have you fueled your car with gas recently? Thank an HPC department at the energy / oil & gas company. While there is innovation around electric vehicles, many of us still use gasoline to ensure we can get around. Exploring for, and finding, oil can be an arduous, time-consuming, expensive effort. With HPC modeling, energy companies can find pockets of resources sooner, limiting the amount of exploratory drilling, and get fuel for your car to you more efficiently.
- Have you been treated for illnesses with medical innovation? Thank an HPC department at the pharmaceutical firm and associated research hospitals. Progress in the treatment of health ailments can trace innovation to HPC teams working in partnership with health care professionals. In fact, much of the research done today to cure some of the world’s diseases and genomics are done on HPC clusters.
- Have you ever been proactively contacted by your bank on suspected fraud on your account? Thank an HPC department at the financial institution. Often times, banking companies will use HPC cluster to run millions of “monte carlo simulations” to model out financial fraud and intrusive hacks. The amount of models required and the depth at which they analyze requires a significant amount of processing power, requiring a stronger-than-normal computing structure.
And the list goes on and on.
- Security enablement
- Banking risk analysis
- Space exploration
- Aircraft safety
- Forecasting natural disasters
- So much more…
If it’s a tough problem to solve, more likely than not, HPC is involved.
And that’s what’s so galvanizing about HPC. It is a computing model that enables us to do so many of the things we take for granted today, but is also on the forefront of new innovation coming from multiple industries in the future.
HPC is also a place where emerging technologies get early adoption, mainly because experts in HPC require new tech to get even deeper into their trade. Open source is a major staple of this group of users, especially with deep adoption of Linux.
You also see early adoption of open source cloud tech like OpenStack (to help institutions share compute power and storage to collaborate) and of open source distributed storage tech like Ceph (to connect highly performant file systems to colder, back up storage, often holding “large data.”) I anticipate we will see this space be among the first to broadly adopt tech in Internet-of-Things (IoT), blockchain, and more.
HPC has been important enough that the governments around the world have funded multiple initiatives to drive more innovation using the model. Here are a few examples:
- The National Strategic Computing Initiative
- The Cancer Moonshot Program
- The Exascale Computing Project
- The STEM Program for Students (Science, Technology, Engineering, Math)
This week, there is a large gathering of HPC experts in Salt Lake City (SuperComputing16) for engineers and researchers to meet / discuss / collaborate on enabling HPC more. From implementing tech like cloud and distributed storage, to best practices in modeling and infrastructure, and driving progress more in medicine, energy, finance, security, and manufacturing, this should be a stellar week of some of the best minds around. (SUSE is out here as well – David has a great blog here on everything that we’re doing at the event.)
High performance computing: take a second look – it may be much more than you originally thought.
And maybe it can help revolutionize YOUR industry.
Until next time,
JOSEPH
@jbgeorge
Highlights from OpenStack Summit Barcelona
.
This is a duplicate of a blog I authored for SUSE, originally published at the SUSE Blog Site.
What a great week at the OpenStack Summit this past week in Barcelona! Fantastic keynotes, great sessions, and excellent hallway conversations. It was great to meet a number of new Stackers as well as rekindle old friendships from back when OpenStack kicked off in 2010.
A few items of note from my perspective:
OpenStack Foundation Board of Directors Meeting
As I mentioned in my last blog, it is the right of every OpenStack member to attend / listen in on each board meeting that the OpenStack Foundation Board of Directors holds. I made sure to head out on Monday and attend most of the day. There was a packed agenda so here a few highlights:
- Interesting discussion around the User Committee project that board member Edgar Magana is working toward, with discussion on its composition, whether members should be elected, and if bylaw changes are warranted. It was a deep topic that required further time, so the topic was deferred to a later discussion with work to be done to map out the details. This is an important endeavor for the community in my opinion – I will be keeping an eye on how this progresses.
- A number of strong presentations by prospective gold members were delivered as they made their cases to be added to that tier. I was especially happy to see a number of Chinese companies presenting and making their case. China is a fantastic growth opportunity for the OpenStack projecct, and it was encouraging to see players in that market discuss all they are doing for OpenStack in the region. Ultimately, we saw City Network, Deutsche Telekom, 99Cloud and China Mobile all get voted in as Gold members.
- Lauren Sell (VP of Marketing for the Foundation) spoke on a visionary model to where her team is investigating how our community can engage with other projects in terms of user events and conferences. Kubernetes, Ceph, and other projects were named as examples. This is a great indicator of how we’ve evolved, as it highlights that often multiple projects are needed to address actual business challenges. A strong indicator of maturity for the community.
Two Major SUSE OpenStack Announcements
SUSE advancements in enterprise ready OpenStack made its way to the Summit in a big way this week.
- SUSE OpenStack Cloud 7: While we are very proud to be one of the first vendors to provide an enterprise-grade Newton based OpenStack distribution, this release also offers features like new Container-as-a-Service capabilities and non-disruptive upgrade capabilities.
Wait, non-disruptive upgrade? As in, no downtime? And no service interruptions? That’s right – disruption to service is a big no-no in the enterprise IT world, and now SUSE OpenStack Cloud 7 provides you the direct ability to stay live during OpenStack upgrade. - Even more reason to become a COA. All the buzz around the Foundation’s “Certified OpenStack Administrator” exam got even better this week when SUSE announced that the exam would now feature the SUSE platform as an option.
And BIG bonus win – if you pass the COA using the SUSE platform, you will be granted
- the Foundation’s COA certification
- SUSE Certified Administrator in OpenStack Cloud certificatio
That’s two certifications with one exam. (Be sure to specify the SUSE platform when taking the exam to take advantage of this option.)
There’s much more to these critical announcements so take a deeper look into them with these blogs by Pete Chadwick and Mark Smith. Worth a read.
Further Enabling the Enterprise
As you know, enterprise adoption of OpenStack is a major passion of mine – I’ve captured a couple more signs I saw this week of OpenStack continuing to head in the right direction.
- Progress in Security. On stage this week, OpenStack was awarded the CII, the Core Infrastructure Initiative Best Practices badge. The CII is a project out of the Linux Foundation project that validates open source projects, specific for security, quality and stability. By winning this award, OpenStack is now validated by a trusted third party and is 100% compliant. Security FTW!
- Workload-based Sample Configs. This stable of assets has been building for some time, but OpenStack.org now boasts a number of reference architectures addressing some of the most critical workloads. From web apps to HPC to video processing and more, there are great resources on how to get optimize OpenStack for these workloads. (Being a big data fan, I was particularly happy with the big data resources here.)
I’d be interested in hearing what you saw as highlights as well – feel free to leave your thoughts in the comments section.
OK, time to get home, get rested – and do this all over again in 6 months in Boston.
(If you missed the event this time, OpenStack.org has you covered – start here to check out a number of videos from the event, and go from there.)
Until next time,
JOSEPH
@jbgeorge
This Week: OpenStack Summit Barcelona!
This is a duplicate of a blog I authored for SUSE, originally published at the SUSE Blog Site.
In a few days, Stackers will congregate in beautiful Barcelona, Spain to kick off the bi-annual OpenStack Summit and User Conference, the 14th of its kind.
On the heels of the recent OpenStack Newton launch, we will see a wide variety of people, backgrounds, industries, and skill sets represented, all focused on learning about, sharing best practices on, and working on the future of OpenStack.
There are many great sessions, workshops, and evening events happening at the summit this coming week, but three in particular that I want to highlight.
OpenStack Board of Directors Meeting
Did you know that, since OpenStack is an open community, the OpenStack Foundation board meeting is open for members to attend and listen in to the discussion? It’s great for members to have this level of access , so take advantage of the openness built into the OpenStack community, and take a listen.
While there are some portions of the meeting that will be a closed session (rightly so), most of the meeting you’ll hear about progress in specific initiatives, comments on new members to the community, and hear back on future directions.
It’s a great experience that more of our members need to participate in, so I highly recommend it to members. You can check out the planned agenda and WebEx details here.
OpenStack Ops Meetup
I mentioned the OpenStack operator community in my last blog (“Renewing Focus on Bringing OpenStack to the Masses”), and how I feel strongly about championing the cause of the operators out there.
While many of us are focused on code design, quality, new projects, etc, the operators are tasked with implementing Openstack. This involves the day-to-day effort of running OpenStack clouds, which include readying IT environments for OpenStack deployments, first hand implementation of the project, the ongoing maintenance and upgrade aspects of the cluster, and being driven by a specific business goal they will be measured by.
At this Summit, the Operators will be hosting an Ops Meetup to get into the meat of OpenStack Ops. Now this stands to be an intense, down-in-the weeds discussion – not for the faint of heart! 
So if you are among the many tasked with getting OpenStack operational in your environment, head on over and get to know your peers in this space, swap stories of what works well, share best practices, and make connections you can stay in touch with for years to come.
Learn more about the Ops Meetup here.
Certified OpenStack Administrator (COA)
Are you aware that you can now be CERTIFIED as an OpenStack Admin?
The COA is an exam you can take to prove your ability to solve problems using both command line and graphical interfaces OpenStack, demonstrating that you have mastered a number of skills required to operate the solution.
At OpenStack Summit, there are a few COA activities occurring that you should be aware of:
- COA 101. Anne Bertucio and Heidi Bretz of the OpenStack Foundation will be hosting a 30min beginner-level session on the topic of COA, touching on the why / what / how relating to the COA exam. (More info here.)
- COA booth. The Foundation Lounge at the Summit will feature an area dedicated to learning more about the COA. A variety of OpenStack community volunteers will be pitching in to answer questions, find trainings, and even sign up for the COA. I plan on helping out on Wednesday right after the morning keynotes, so stop by and let’s chat COA.
- COA exams. If you’re ready now to take the exam, head on over to https://www.openstack.org/coa/ and get the details on when you can take the exam. The world needs Certified OpenStack Admins!
(PS – If you need help with some prep, SUSE’s happy to help there – click here to get details on COA training from SUSE.)
I’m looking forward to a great week of re-connecting with a number of you I haven’t seen in some time, so if you’re out at Barcelona, look me up – I’d love to hear what OpenStack project you’re working on, or learn about how you are implementing OpenStack, or where I can help with places OpenStack can further help you in your business objectives.
See you in Barcelona!
JOSEPH
@jbgeorge
Renewing Focus on Bringing OpenStack to the Masses
.
This is a duplicate of a blog I authored for SUSE, originally published at the SUSE Blog Site.
Happy Newton Release Week!
This 14th release of OpenStack is one we’ve all been anticipating, with its focus on scalability, resiliency, and overall user experience – important aspects that really matter to enterprise IT organizations, and that help with broader adoption of OpenStack with those users.
(More on that later.)
Six years of Community, Collaboration, and Growth
I was fortunate enough to have been at the very first OpenStack “meeting of the minds” at the Omni hotel in Austin, back in 2010, when just the IDEA of OpenStack was in preliminary discussions. Just a room full of regular everyday people, across numerous industries, who saw the need for a radical new open source cloud platform, and had a passion to make something happen.
And over the years, we’ve seen progress: the creation of the OpenStack Foundation, thousands of contributors to the project, scores of companies throwing their support and resources behind OpenStack, the first steps beyond North America to Europe and Asia (especially with all the OpenStack excitement in India, China, and Japan), numerous customers adopting our revolutionary cloud project, and on and on.
Power to the People
But this project is also about our people.
And we, as a community of individuals, have grown and evolved over the years as well – as have the developers, customers and end users we hope to serve with the project. What started out with a few visionary souls has now blossomed into a community of 62,000+ members from 629 supporting companies across 186 countries.
As a member of the community, I’ve seen positive growth in my own life since then as well. I’ve been fortunate to have been part of some great companies – like Dell, HPE, and now SUSE. And I’ve been able to help enterprise customers solve real problems with impactful open source solutions in big data, storage, HPC, and of course, cloud with OpenStack.
And there might have been one other minor area of self-transformation since those days as well, as the graphic illustrates…
Clearly we – the OpenStack project and the OpenStack people – are evolving for the better.
So Where Do We Go From Here?
In 2013, I was able to serve the community by being a Director on the OpenStack Foundation Board. Granted, things were still fairly new – new project ideas emerging, new companies wanting to sponsor, developers being added by the day, etc – but there was a personal focus I wanted to drive among our community.
“Bringing OpenStack to the masses.”
And today, my hope for the community remains the same.
While we celebrate all the progress we have made, here are some of my thoughts on what we, as a community, should continue to focus on to bring OpenStack to the masses.
Adoption with the Enterprise by Speaking their Language
Take a look at the most recent OpenStack User Survey. It provides a great snapshot into the areas that users need help.
- “OpenStack is great to recommend, however there’s a fair amount of complexity that needs to be tackled if one wishes to use it.”
- “OpenStack lacks far too many core components for anything other than very specialized deployments.”
- “Technology is good, but no synergies between the sub-projects.”
2016 data suggests that enterprise customers are looking for these sorts of issues to be addressed, as well as security and management practices keeping pace with new features. And, with all the very visible security breaches in recent months, the enterprise is looking for open source projects to put more emphasis on security. In fact, many of the customers I engage with love the idea of OpenStack, but still need help with these fundamental requirements they have to deal with.
Designing / Positioning OpenStack to Address Business Challenges
Have you ever wondered why so many tall office buildings have revolving doors? Isn’t a regular door simpler, less complex, and easier to use? Why do we see so many revolving doors, when access can be achieved so much simpler with other means?
Because revolving doors don’t exist to solve access problems – they solve a heated-air loss problem.
When a door in a tall building is opened, cold air from outside forces it’s way inside when doors are open, pushing warmer / lighter air up – which is then lost through vents at the top of the building. Revolving doors limit that loss incredibly by sealing off portions of outside access when rotating.
Think of our project in the same way.
- What business challenges can be addressed today / near-term by implementing an OpenStack-based solution?
- Beyond the customer set looking to build a hosted cloud to resell, what further applications can OpenStack be applied to?
- How can OpenStack provide an industry-specific competitive advantage to the financial sector? To healthcare? To HPC? To the energy sector? How about retail or media?
Address the Cultural IT Changes that Need to Happen
I recently read a piece where Jonathan Bryce spoke to the “cultural IT changes that need to occur” – and I love that line of thinking.
Jonathan specifically said “What you do with Windows and Linux administrators is the bigger challenge for a lot of companies. Once you prove the technology, you need policies and training to push people.”
That is spot on.
What we are all working on with OpenStack will fundamentally shift how IT organizations will operate in the future. Let’s take the extra step, and provide guidance to our audiences on how they can evolve and adapt to the coming changes with training, tools, community support, and collaboration. A good example of this is the Certified OpenStack Administrator certification being offered by the Foundation, and training for the COA offered by the OpenStack partner ecosystem.
Further Champion the OpenStack Operator
Operators are on the front lines of implementing OpenStack, and “making things work.” There is no truer test of the validity of our OpenStack efforts than when more and more operators can easily and simply deploy / run / maintain OpenStack instances.
I am encouraged by the increased focus on documentation, connecting developers and operators more, and the growth of a community of operators sharing stories on what works best in practical implementation of OpenStack. And we will see this grow even more at the Operator Summit in Barcelona at OpenStack Summit (details here).
We are making progress here but there’s so much more we can do to better enable a key part of our OpenStack family – the operators.
The Future is Bright
Since we’re celebrating the Newton release, the quote from Isaac Newton on looking ahead is seems fitting…
“To myself I am only a child playing on the beach, while vast oceans of truth lie undiscovered before me.”
When it comes to where OpenStack is heading, I’m greatly optimistic. As it has always been, it will not be easy. But we are making a difference.
And with continued and increased focus on enterprise adoption, addressing business challenges, aiding in the cultural IT change, and an increased focus on the operator, we can go to the next level.
We can bring OpenStack to the masses.
See you in Barcelona in a few weeks.
Until next time,
JOSEPH
@jbgeorge
Living Large: The Challenge of Storing Video, Graphics, and other “LARGE Data”
.
UPDATE SEP 2, 2016: SUSE has released a brand new customer case study on 
“large data” featuring New York’s Orchard Park Police Department, focused on video storage. Read the full story here!
————–
This is a duplicate of a blog I authored for SUSE, originally published at the SUSE Blog Site.
Everyone’s talking about “big data” – I’m even hearing about “small data” – but those of you who deal in video, audio, and graphics are in the throes of a new challenge: large data.
Big Data vs Large Data
It’s 2016 – we’ve all heard about big data in some capacity – generally speaking, it is truckloads of data points from various sources, magnanimous in its volume (LOTS of individual pieces of data), its velocity (the speed at which that data is generated), and its variety (both structured and unstructured data types). Open source projects like Hadoop have been enabling a generation of analytics work on big data.
So what in the world am I referring to when I say “large data?”
For comparison, while “big data” is a significant number of individual data that is of “normal” size, I’m defining “large data” as an individual piece of data that is massive in its size. Large data, generally, is not required to have real-time or fast access, and is often unstructured in its form (ie. doesn’t conform to the parameters of relational databases).
Some examples of large data:
Why Traditional Storage Has Trouble with Large Data
So why not just throw this into our legacy storage appliances?
Traditional storage solutions (much of what is in most datacenters today) is great at handling standard data. Meaning, data that is:
- average / normal in individual data size
- structured and fits nicely in a relational database
- when totaled up, doesn’t exceed ~400TB or so in total space
Unfortunately, none of this works for large data. Large data, due to its size, can consume traditional storage appliances VERY rapidly. And, since traditional storage was developed when data was thought of in smaller terms (megabytes and gigabytes), large data on traditional storage can bring about performance / SLA impacts.
So when it comes to large data, traditional storage ends up being consumed too rapidly, forcing us to consider adding expensive traditional storage appliances to accommodate.
“Overall cost, performance concerns, complexity and inability to support innovation are the top four frustrations with current storage systems.” – SUSE, Software Defined Storage Research Findings (Aug 2016)
Object Storage Tames Large Data
Object storage, as a technology, is designed to handle storage of large, unstructured data from the ground up. And since it was built on scalable cloud principles, it can scale to terabytes, petabytes, exabytes, and theoretically beyond.
When you introduce open source to the equation of object storage software, the economics of the whole solution become even better. And since scale-out, open source object storage is essentially software running on commodity servers with local drives, the object storage should scale without issue – as more capacity is needed, you just add more servers.
When it comes to large data – data that is unstructured and individually large, such as video, audio, and graphics – SUSE Enterprise Storage provides the open, scalable, cost-effective, and performant storage experience you need.
SUSE Enterprise Storage – Using Object Storage to Tame Large Data
Here’s how:
- It is designed from the ground-up to tackle large data. The Ceph project, which is core of SUSE Enterprise Storage, is built on a foundation of RADOS (Reliable Autonomic Distributed Object Store), and leverages the CRUSH algorithm to scale data across whatever size cluster you have available, without performance hits.
- It provides a frequent and rapid innovation pace. It is 100% open source, which means you have the power of the Ceph community to drive innovation, like erasure coding. SUSE gives these advantages to their customers by providing a full updated release every six months, while other Ceph vendors give customers large data features only as part of a once-a-year release.
- It offers pricing that works for large data. Many object storage vendors, both commercial and open source, choose to charge you based on how much data you store. The price to store 50TB of large data is different than the price to store 100TB of large data, and there is a different price if you want to store 400TB of large data – even if all that data stays on one server! SUSE chooses to provide their customers “per node” pricing – you pay subscription only as servers are added. And when you use storage dense servers, like the HPE Apollo 4000 storage servers, you get tremendous value.
Why wait? It’s time to kick off a conversation with your SUSE rep on how SUSE Enterprise Storage can help you with your large data storage needs. You can also click here to learn more.
Until next time,
JOSEPH
Address Your Company’s Data Explosion with Storage That Scales
.
This is a duplicate of a blog I authored for SUSE, originally published at the SUSE Blog Site.
Experts predict that our world will generate 44 ZETTABYTES of digital data by 2020.
How about some context?
Now, you may think that these are all teenage selfies and funny cat videos – in actuality, much of it is legitimate data your company will need to stay competitive and to serve your customers.
The Data Explosion Happening in YOUR Industry
Some interesting factoids:
- An automated manufacturing facility can generate many terabytes of data in a single hour.
- In the airline industry, a commercial airplane can generate upwards of 40 TB of data per hour.
- Mining and drilling companies can gather multiple terabytes of data per minute in their day-to-day operations.
- In the retail world, a single store can collect many TB of customer data, financial data, and inventory data.
- Hospitals quickly generate terabytes of data on patient health, medical equipment data, and patient x-rays.
The list goes on and on. Service providers, telecommunications, digital media, law enforcement, energy companies, HPC research groups, governments, the financial world, and many other industries (including yours) are experiencing this data deluge now.
And with terabytes of data being generated by single products by the hour or by the minute, the next stop is coming up quick: PETABYTES OF DATA.
Status Quo Doesn’t Cut It
I know what you’re thinking: “What’s the problem? I‘ve had a storage solution in place for years. It should be fine.”
Not quite.
- You are going to need to deal with a LOT more data than you are storing today in order to maintain your competitive edge.
- The storage solutions you’ve been using for years have likely not been designed to handle this unfathomable amount of data.
- The costs of merely “adding more” of your current storage solutions to deal with this amount of data can be extremely expensive.
The good news is that there is a way to store data at this scale with better performance at a much better price point.
Open Source Scale Out Storage
Why is this route better?
- It was designed from the ground up for scale.
Much like how mobile devices changed the way we communicate / interact / take pictures / trade stock, scale out storage is different design for storage. Instead of all-in-one storage boxes, it uses a “distributed model” – farming out the storage to as many servers / hard drives as it has access to, making it very scalable and very performant. (Cloud environments leverage a very similar model for computing.) - It’s cost is primarily commodity servers with hard drives and software.
Traditional storage solutions are expensive to scale in capacity or performance. Instead of expensive engineered black boxes, we are looking at commodity servers and a bit of software that sits on each server – you then just add a “software + server” combo as you need to scale. - When you go open source, the software benefits get even better.
Much like other open source technologies, like Linux operating systems, open source scale out storage allows users to take advantage of rapid innovation from the developer communities, as well as cost benefits which are primarily support or services, as opposed to software license fees.
Ready. Set. Go.
At SUSE, we’ve put this together in an offering called SUSE Enterprise Storage, an intelligent software-defined storage management solution, powered by the open source Ceph project.
It delivers what we’ve talked about: open source scale out storage. It scales, it performs, and it’s open source – a great solution to manage all that data that’s coming your way, that will scale as your data needs grow.
And with SUSE behind you, you’ll get full services and support to any level you need.
OK, enough talk – it’s time for you to get started.
And here’s a great way to kick this off: Go get your FREE TRIAL of SUSE Enterprise Storage. Just click this link, and you’ll be directed to the site (note you’ll be prompted to do a quick registration.) It will give you quick access to the scale out storage tech we’ve talked about, and you can begin your transition over to the new evolution of storage technology.
Until next time,
JOSEPH
@jbgeorge
The Next Adventure Begins…
.
As I’ve updated my audiences on SoMe this week, I’ve taken on an executive role at SUSE, where I will lead the efforts for SUSE’s solutions strategy when it comes to cloud, storage, big data, and more.
The company has been doing a great job in these areas, and I’ve been fortunate to partner with them in various roles across OpenStack, Ceph, and more.
My goal is to start with the good work already deployed in market, and help take it to the next level to enable customers to address serious business challenges with these solutions, simply and effectively.
More to come as I get my feet set in the new role, but I am very excited to join SUSE, a company that understands the emerging technology space, it’s customers, and it’s communities – and is poised to do big things with it’s contributions to open source and it’s commercial offerings.
Let the adventure begin!
Until next time,
JBG
@jbgeorge
Tech in Real Life: Content Delivery Networks, Big Data Servers and Object Storage
.
This is a duplicate of a blog I authored for HP, originally published at hp.nu/Lg3KF.
In a joint blog authored with theCube’s John Furrier and Scality’s Leo Leung, we pointed out some of the unique characteristics of data that make it act and look like a vector.
At that time, I promised we’d delve into specific customer uses for data and emerging data technologies – so let’s begin with our friends in the telecommunications and media industries, specifically around the topic of content distribution.
But let’s start at a familiar point for many of us…
If you’re like most people, when it comes to TV, movies, and video content, you’re an avid (sometimes binge-watching) fan of video streaming and video on-demand. More and more people are opting to view content via streaming technologies. In fact, a growing number of broadcast shows are viewed on mobile and streaming devices, as are a number of live events, such as this year’s NCAA basketball tournament via streaming devices.
These are fascinating data points to ponder, but think about what goes on behind them.
How does all this video content get stored, managed, and streamed?
Suffice it to say, telecom and media companies around the world are addressing this exact challenge with content delivery networks (CDN). There are a variety of interesting technologies out there to help develop CDNs, and one interesting new technology to enable this is object storage, especially when it comes to petabytes of data.
Here’s how object storage helps when it comes to streaming content.
- With streaming content comes a LOT of data. Managing and moving that data is a key area to address, and object storage handles it well. It allows telecom and media companies to effectively manage many petabytes of content with ease – many IT options lack that ability to scale. Features in object storage like replication and erasure coding allow users to break large volumes of data into bite size chunks, and disperse it over several different server nodes, and often times, several different geographic locations. As data is needed, it is rapidly re-compiled and distributed as needed.
- Raise your hand if you absolutely love to wait for your video content to load. (Silence.) The fact is, no one likes to see the status bar slowly creeping along, while you’re waiting for zombies, your futbol club, or the next big singing sensation to show up on the screen. Because object storage technologies are able to support super high bandwidth and millions of HTTP requests per minute, any customer looking to distribute media is able to allow their customers access to content with superior performance metrics. It has a lot to do with the network, but also with the software managing the data behind the network, and object storage fits the bill.
These are just two of the considerations, and there are many others, but object storage becomes an interesting technology to consider if you’re looking to get content or media online, especially if you are in the telecom or media space.
Want a real life example? Check out how our customer RTL II, a European based television station, addressed their video streaming challenge with object storage. It’s all detaile here in this case study – “RTL II shifts video archive into hyperscale with HP and Scality.” Using HP ProLiant SL4540 big data servers and object storage software from HP partner Scality, RTL II was able to boost their video transfer speeds by 10x!
Webinar this week! If this is a space you could use more education on, Scality and HP will be hosting a couple of webinars this week, specifically around object storage and content delivery networks. If you’re looking for more on this, be sure to join us – here are the details:
Session 1 (Time-friendly for European and APJ audiences)
- Who: HP’s big data strategist, Sanjeet Singh, and Scality VP, Leo Leung
- Date: Wed, Apr 8, 2015
- Time: 3pm Central Europe Summer / 8am Central US
- Registration Link
Session 2 (Time-friendly for North American audiences)
- Who: HP Director, Joseph George, and Scality VP, Leo Leung
- Date: Wed, Apr 8, 2015
- Time: 10am Pacific US / 12 noon Central US
- Registration Link
And as always, for any questions at all, you can always send us an email at BigDataEcosystem@hp.com or visit us at www.hp.com/go/ProLiant/BigDataServer.
And now off to relax and watch some TV – via streaming video of course!
Until next time,
JOSEPH
@jbgeorge
Recognizing the Layers of Critical Insight That Data Offers
This is a joint blog I did with John Furrier of SiliconAngle / theCube and Leo Leung from Scality, originally published at http://bit.ly/1E6nQuR
Data is an interesting concept.
During a recent CrowdChat a number of us started talking about server based storage, big data, etc., and the topic quickly developed into a forum on data and its inherent qualities. The discussion led us to realize that data actually has a number of attributes that clearly define it – similar to how a vector has both a direction and magnitude.
Several of the attributes we uncovered as we delved into this notion of data as a vector include:
- Data Gravity: This was a concept developed by my friend, Dave McCrory, a few years ago, and it is a burgeoning area of study today. The idea is that as data is accumulated, additional services and applications are attracted to this data – similar to how a planet’s gravitational pull attracts objects to it. An example would be the number 10. If you the “years old” context is “attracted” to that original data point, it adds a certain meaning to it. If the “who” context is applied to a dog vs. a human being, it takes on additional meaning.
- Relative Location with Similar Data: You could argue that this is related to data gravity, but I see it as more of a poignant a point that bears calling out. At a Hadoop World conference many years ago, I heard Tim O’Reilly make the comment that our data is most meaningful when it’s around other data. A good example of this is medical data. Health information of a single individual (one person) may lead to some insights, but when placed together with data from a members of a family, co-workers on a job location, or the citizens of a town, you are able to draw meaningful conclusions. When grouped with other data, individual pieces of data take on more meaning.
- Time: This came up when someone posed the question “does anyone delete data anymore?” With the storage costs at scale becoming more and more affordable, we concluded that there is no longer an economic need to delete data (though there may be regulatory reasons to do so). Then came the question of determining what data was not valuable enough to keep, which led to the epiphany that data that might be viewed as not valuable today, may become significantly valuable tomorrow. Medical information is a good example here as well – capturing the data that certain individuals in the 1800’s were plagued with a specific medical condition may not seem meaningful at the time, until you’ve tracked data on specific descendants of his family being plagued by similar ills over the next few centuries. It is difficult to quantify the value of specific data at the time of its creation.
In discussing this with my colleagues, it became very clear how early we are in the evolution of data / big data / software defined storage. With so many angles yet to be discussed and discovered, the possibilities are endless.
This is why it is critical that you start your own journey to salvage the critical insights your data offers. It can help you drive efficiency in product development, it can help you better serve you constituents, and it can help you solve seemingly unsolvable problems. Technologies like object storage, cloud based storage, Hadoop, and more are allowing us to learn from our data in ways we couldn’t imagine 10 years ago.
And there’s a lot happening today – it’s not science fiction. In fact, we are seeing customers implement these technologies and make a turn for the better – figuring out how to treat more patients, enabling student researchers to share data across geographic boundaries, moving media companies to stream content across the web, and allowing financial institutions to detect fraud when it happens. Though the technologies may be considered “emerging,” the results are very, very real.
Over the next few months, we’ll discuss specific examples of how customers are making this work in their environments, tips on implementing these innovative technologies, some unique innovations that we’ve developed in both server hardware and open source software, and maybe even some best practices that we’ve developed after deploying so many of these big data solutions.
Stay tuned.
Until next time,
Joseph George – @jbgeorge
Director, HP Servers
Leo Leung – @lleung
VP, Scality
John Furrier – @furrier
Founder of SiliconANGLE Media
Cohost of @theCUBE
CEO of CrowdChat
The HP Big Data Reference Architecture: It’s Worth Taking a Closer Look…
This is a duplicate of the blog I’ve authored on the HP blog site at http://h30507.www3.hp.com/t5/Hyperscale-Computing-Blog/The-HP-Big-Data-Reference-Architecture-It-s-Worth-Taking-a/ba-p/179502#.VMfTrrHnb4Z
I recently posted a blog on the value that purpose-built products and solutions bring to the table, specifically around the HP ProLiant SL4540 and how it really steps up your game when it comes to big data, object storage, and other server based storage instances.
Last month, at the Discover event in Barcelona, we announced the revolutionary HP Big Data Reference Architecture – a major step forward in how we, as a community of users, do Hadoop and big data – and it is a stellar example of how purpose-built solutions can revolutionize how you accelerate IT technology, like big data. We’re proud that HP is leading the way in driving this new model of innovation, with the support and partnership of the leading voices in Hadoop today.
Here’s the quick version on what the HP Big Data Reference Architecture is all about:
Think about all the Hadoop clusters you’ve implemented in your environment – they could be pilot or production clusters, hosted by developer or business teams, and hosting a variety of applications. If you’re following standard Hadoop guidance, each instance is most likely a set of general purpose server nodes with local storage.
For example, your IT group may be running a 10 node Hadoop pilot on servers with local drives, your marketing team may have a 25 node Hadoop production cluster monitoring social media on similar servers with local drives, and perhaps similar for the web team tracking logs, the support team tracking customer cases, and sales projecting pipeline – each with their own set of compute + local storage instances.
There’s nothing wrong with that set up – It’s the standard configuration that most people use. And it works well.
However….
Just imagine if we made a few tweaks to that architecture.
- What if we replaced the good-enough general purpose nodes, and replaced them with purpose-built nodes?
- For compute, what if we used HP Moonshot, which is purpose-built for maximum compute density and price performance?
- For storage, what if we used HP ProLiant SL4540, which is purpose-built for dense storage capacity, able to get over 3PB of capacity in a single rack?
- What if we took all the individual silos of storage, and aggregated them into a single volume using the purpose-built SL4540? This way all the individual compute nodes would be pinging a single volume of storage.
- And what if we ensured we were using some of the newer high speed Ethernet networking to interconnect the nodes?
Well, we did.
And the results are astounding.
While there is a very apparent cost benefit and easier management, there is a surprising bump in performance in terms of read and write.
It was a surprise to us in the labs, but we have validated it in a variety of test cases. It works, and it’s a big deal.
And Hadoop industry leaders agree.
“Apache Hadoop is evolving and it is important that the user and developer communities are included in how the IT infrastructure landscape is changing. As the leader in driving innovation of the Hadoop platform across the industry, Cloudera is working with and across the technology industry to enable organizations to derive business value from all of their data. We continue to extend our partnership with HP to provide our customers with an array of platform options for their enterprise data hub deployments. Customers today can choose to run Cloudera on several HP solutions, including the ultra-dense HP Moonshot, purpose-built HP ProLiant SL4540, and work-horse HP Proliant DL servers. Together, Cloudera and HP are collaborating on enabling customers to run Cloudera on the HP Big Data architecture, which will provide even more choice to organizations and allow them the flexibility to deploy an enterprise data hub on both traditional and newer infrastructure solutions.” – Tim Stevens, VP Business and Corporate Development, Cloudera
“We are pleased to work closely with HP to enable our joint customers’ journey towards their data lake with the HP Big Data Architecture. Through joint engineering with HP and our work within the Apache Hadoop community, HP customers will be able to take advantage of the latest innovations from the Hadoop community and the additional infrastructure flexibility and optimization of the HP Big Data Architecture.” – Mitch Ferguson, VP Corporate Business Development, Hortonworks
And this is just a sample of what HP is doing to think about “what’s next” when it comes to your IT architecture, Hadoop, and broader big data. There’s more that we’re working on to make your IT run better, and to lead the communities to improved experience with data.
If you’re just now considering a Hadoop implementation or if you’re deep into your journey with Hadoop, you really need to check into this, so here’s what you can do:
- my pal, Greg Battas posted on the new architecture and goes technically deep into it, so give his blog a read to learn more about the details.
- Hortonworks has also weighed in with their own blog.
If you’d like to learn more, you can check out the new published reference architectures that follow this design featuring HP Moonshot and ProLiant SL4540:
- HP Big Data Reference Architecture: Cloudera Enterprise reference architecture implementation
- HP Big Data Reference Architecture: Hortonworks Data Platform reference architecture implementation
If you’re looking for even more information, reach out to your HP rep and mention the HP Big Data Reference Architecture. They can connect you with the right folks to have a deeper conversation on what’s new and innovative with HP, Hadoop, and big data. And, the fun is just getting started – stay tuned for more!
Until next time,
JOSEPH
The JBGeorge Tech Blog 