Archive
Is Your HPC System Fully Optimized? + Live Chat
.
This is a duplicate of a high-performance computing blog I authored for Cray, originally published at the Cray Blog Site.
Let’s talk operating systems…
Take a step back and look at the systems you have in place to run applications and workloads that require high core counts to get the job done. (You know, jobs like fraud detection, seismic analysis, cancer research, patient data analysis, weather prediction, and more — jobs that can really take your organization to the next level, trying to solve seriously tough challenges.)
You’re likely running a cutting-edge processor, looking to take advantage of the latest and greatest innovations in compute processing and floating-point calculations. The models you’re running are complex, pushing the bounds of math and science, so you want the best you can get when it comes to processor power.
You’re probably looking at a high-performance file system to ensure that your gigantic swaths of data can be accessed, processed, and stored so your outcomes tell the full story.
You’re most likely using a network interconnect that can really move so that your system can hum, connecting data and processing in a seamless manner, with limited performance hit.
We can go on and on, in terms of drive types, memory choices, and even cabling decisions. The more optimized your system can be for a high-performance workload, the faster and better your results will be.
But what about your software stack, starting with your operating system?
Live tweet chat
Join us for a live online chat at 10 a.m. Tuesday, June 19 — “Do you need an HPC-optimized OS?” — to learn about optimizing your HPC workloads with an improved OS. You can participate using a Twitter, LinkedIn or Facebook account and the hashtag #GetHPCOS.
Sunny Sundstrom and I will discuss what tools to use to customize the environment for your user base, whether there’s value in packaging things together, and what the underlying OS really means for you.
Bring your questions or offer your own expertise.
Moment of truth time
Are you optimizing your system at the operating environment level, like you are across the rest of your system? Are you squeezing every bit of performance benefit out of your system, even when it comes to the OS?
Think about it — if you’re searching for an additional area to drive a performance edge, the operating system could be just the place to look:
- There are methods to allocate jobs to run on specific nodes tuned for specific workloads — driven at the OS level.
- You can deploy a lighter-weight OS instance on your working compute node to avoid unnecessarily bringing down the performance of the job, while ensuring your service nodes can adequately run and manage your system.
- You can decrease jitter of the overall system by using the OS as your control point.
All of this at the OS level.
Take a deeper look at your overall system, including the software stack, and specifically your operating system and operating environment.
You may be pleasantly surprised to find that you can drive even more performance in a seemingly unexpected place.
Until next time,
JBG
@jbgeorge
Add our 6/19 tweet chat to your calendar for further discussion.
8 Letters to HPC Success – SOFTWARE
.
This is a duplicate of a high-performance computing blog I authored for Cray, originally published at the Cray Blog Site.
These days, more and more organizations are using high-performance computing systems to seek out answers to big questions. From detecting fraud in financial transactions within milliseconds, to scouring the universe for life, to working toward a cure for cancer, the applications working on these questions require an infrastructure that allows for numerous cores, large swaths of data, and rapid results.
These systems are carefully designed to optimize application performance at scale. Processors, memory, storage, networking — all carefully selected and implemented to drive optimal performance.
However, far too many vendors ignore the impact of another key area of the infrastructure stack … the SOFTWARE.
Software is a key component and a remarkable enabler for gleaning optimal performance from an application and a system. And that means getting that much more performance out of your system, getting you to insights and answers faster.
Along with fine-tuning your hardware infrastructure, imagine the impacts of an HPC-optimized software OS and operating environment. You could witness the impact of lower jitter, more focused intelligence built into to the various levels of the hardware system, and greater levels of job deployment flexibility.
Or think of the impact of a unified, cross-processor programming environment, designed to help your application developers get the most of their HPC applications — a programming interface that gives them broad control and flexibility when it comes to porting, compiling, debugging and optimizing the applications that get you results.
These software applications are the reality for today’s Cray customers.
From the Cray Linux® Environment to the Cray Programming Environment, from powerful systems management to amazing storage telemetry, and from emerging cloud capabilities to effective analytics packages, we enable our customers to take performance to the next level with our comprehensive software portfolio.
Here at Cray, we understand the power of the software stack and have been committed to building the best for several decades. In addition to our work with our ecosystem of software partners and open source communities, we remain big believers in rolling up our sleeves and getting into the code. Our software engineering teams work hand in hand with our system hardware teams to ensure Cray systems are optimized at all levels, and find ways to extend the reach of our partners and open-source projects.
And there’s no slowing down in sight.
It’s time you leveraged the power of HPC-optimized software to answer the big questions you have in your business.
In the coming weeks we’ll bring you a more in-depth look at Cray’s HPC-optimized software stack with blogs, videos, webinars and other helpful tools so that you can maximize the performance of your Cray systems and applications.
OpenStack, Now and Moving Ahead: Lessons from My Own Personal Transformation
.
This is a duplicate of a blog I authored for SUSE, originally published at the SUSE Blog Site.
It’s a transformative time for OpenStack.
And I know a thing or two about transformations.
Over the last two and a half years, I’ve managed to lose over 90 pounds.
(Yes, you read that right.)
It was a long and arduous effort, and it is a major personal accomplishment that I take a lot of pride in.
Lessons I’ve Learned
When you go through a major transformation like that, you learn a few things about the process, the journey, and about yourself.
With OpenStack on my mind these days – especially after being nominated for the OpenStack Foundation Board election – I can see correlations between my story and where we need to go with OpenStack.
While there are a number of lessons learned, I’d like to delve into three that are particularly pertinent for our open source cloud project.
1. Clarity on the Goal and the Motivation
It’s a very familiar story for many people. Over the years, I had gained a little bit of weight here and there as life events occurred – graduated college, first job, moved cities, etc. And I had always told myself (and others), “By the time I turned 40 years old, I will be back to my high school weight.”
The year I was to turn 40, I realized that I was running out of time to make good on my word!
And there it was – my goal and my motivation.
So let’s turn to OpenStack – what is our goal and motivation as a project?
According to wiki.OpenStack.org, the Openstack Mission is “to produce the ubiquitous Open Source Cloud Computing platform that will meet the needs of public and private clouds regardless of size, by being simple to implement and massively scalable. OpenStack is open source, openly designed, openly developed by an open community.”
That’s our goal and motivation
- meet the needs of public and private clouds
- no matter the size
- simple to deploy
- very scalable
- open across all parameters
While we exist in a time where it’s very easy to be distracted by every new, shiny item that comes along, we must remember our mission, our goal, our motivation – and stay true to what we set out to accomplish.
2. Staying Focused During the “Middle” of the Journey
When I was on the path to lose 90 pounds, it was very tempting to be satisfied during the middle part of the journey.
After losing 50 pounds, needless to say, I looked and felt dramatically better than I had been before. Oftentimes, I was congratulated – as if I had reached my destination.
But I had not reached my destination.
While I had made great progress – and there were very tangible results to demonstrate that – I had not yet fully achieved my goal. And looking back, I am happy that I was not content to stop halfway through. While I had a lot to be proud of at that point, there was much more to be done.
OpenStack has come a long way in its fourteen releases:
The phenomenal Newton release focused on scalability, interoperability, and resiliency – things that many potential customers and users have been waiting for.- The project has now been validated as 100% compliant by the Core Infrastructure Initiative (CII) as part of the Linux Foundation, a major milestone toward the security of OpenStack.
- Our community now offers the “Certified OpenStack Adminstrator” certification, a staple of datacenter software that much of the enterprise expects, further validating OpenStack for them.
We’ve come a long way. But there is more to go to achieve our ultimate goal. Remember our mission: open source cloud, public and private, across all size clouds, massively scalable, and simple to implement.
We are enabling an amazing number of users now, but there is more to do to achieve our goal. While we celebrate our current success, and as more and more customers are being successful with OpenStack in production, we need to keep our eyes on the prize we committed to.
3. Constantly Learning and Adapting
While losing 90 pounds was a major personal accomplishment, it could all have been in vain if I did not learn how to maintain the weight loss once it was achieved.
This meant learning what worked and what didn’t work, as well as adapting to achieve a permanent solution.
Case in point: a part of most weight loss plans is to get plenty of water daily, something I still do to this day. While providing numerous health advantages, it is also a big help with weight loss. However, I found that throughout the day, I would get consumed with daily activities and reach the end of the day without having reached my water requirement goal.
Through some experimentation with tactics – which included setting up reminders on my phone and keeping water with me at all times, among other ideas – I arrived at my personal solution: GET IT DONE EARLY.
I made it a point to get through my water goal at the beginning of the day, before my daily activities began. This way, if I did not remember to drink regularly throughout the day, it was of no consequence since I had already met my daily goal.
We live in a world where open source is getting ever more adopted by more people and open source newbies. From Linux to Hadoop to Ceph to Kubernetes, we are seeing more and more projects find success with a new breed of users. OpenStack’s role is not to shun these projects as isolationists, but rather understand how OpenStack adapts so that we get maximum attainment of our mission.
This also means that we understand how our project gets “translated” to the bevy of customers who have legitimate challenges to address that OpenStack can help with. It means that we help potential user wade through the cultural IT changes that will be required.
Learning where our market is taking us, as well as adapting to the changing technology landscape, remains crucial for the success of the project.
Room for Optimism
I am personally very optimistic about where OpenStack goes from here. We have come a long way, and have much to be proud of. But much remains to be done to achieve our goal, so we must be steadfast in our resolve and focus.
And it is a mission that we can certainly accomplish. I believe in our vision, our project, and our community.
And take it from me – reaching BIG milestones are very, very rewarding.
Until next time,
JOSEPH
@jbgeorge
HPC: Enabling Fraud Detection, Keeping Drivers Safe, Helping Cure Disease, and So Much More
.
This is a duplicate of a blog I authored for SUSE, originally published at the SUSE Blog Site.
Ask kids what they want to be when they grow up, and you will often hear aspirations like:
- “I want to be an astronaut, and go to outer space!”
- “I want to be a policeman / policewoman, and keep people safe!”
- “I want to be a doctor – I could find a cure for cancer!”
- “I want to build cars / planes / rocket ships / buildings / etc – the best ones in the world!”
- “I want to be an engineer in a high performance computing department!”
OK, that last one rarely comes up.
Actually, I’ve NEVER heard it come up.
But here’s the irony of it all…
That last item is often a major enabler for all the other items.
Surprised? A number of people are.
For many, “high performance computing” – or “HPC” for short – often has a reputation for being a largely academic engineering model, reserved for university PhDs seeking to prove out futuristic theories.
The reality is the high performance computing has influenced a number of things we take for granted on a daily basis, across a number of industries, from healthcare to finance to energy and more.
HPC has been an engineering staple in a number of industries for many years, and has enabled a number of the innovations we all enjoy on a daily basis. And it’s a critical function of how we will function as a society going forward.
Here are a few examples:
- Do you enjoy driving a car or other vehicle? Thank an HPC department at the auto manufacturer. There’s a good chance that an HPC effort was behind modeling the safety hazards that you may encounter as a driver. Unexpected road obstacles, component failure, wear and tear of parts after thousands of miles, and even human driver behavior and error.
- Have you fueled your car with gas recently? Thank an HPC department at the energy / oil & gas company. While there is innovation around electric vehicles, many of us still use gasoline to ensure we can get around. Exploring for, and finding, oil can be an arduous, time-consuming, expensive effort. With HPC modeling, energy companies can find pockets of resources sooner, limiting the amount of exploratory drilling, and get fuel for your car to you more efficiently.
- Have you been treated for illnesses with medical innovation? Thank an HPC department at the pharmaceutical firm and associated research hospitals. Progress in the treatment of health ailments can trace innovation to HPC teams working in partnership with health care professionals. In fact, much of the research done today to cure some of the world’s diseases and genomics are done on HPC clusters.
- Have you ever been proactively contacted by your bank on suspected fraud on your account? Thank an HPC department at the financial institution. Often times, banking companies will use HPC cluster to run millions of “monte carlo simulations” to model out financial fraud and intrusive hacks. The amount of models required and the depth at which they analyze requires a significant amount of processing power, requiring a stronger-than-normal computing structure.
And the list goes on and on.
- Security enablement
- Banking risk analysis
- Space exploration
- Aircraft safety
- Forecasting natural disasters
- So much more…
If it’s a tough problem to solve, more likely than not, HPC is involved.
And that’s what’s so galvanizing about HPC. It is a computing model that enables us to do so many of the things we take for granted today, but is also on the forefront of new innovation coming from multiple industries in the future.
HPC is also a place where emerging technologies get early adoption, mainly because experts in HPC require new tech to get even deeper into their trade. Open source is a major staple of this group of users, especially with deep adoption of Linux.
You also see early adoption of open source cloud tech like OpenStack (to help institutions share compute power and storage to collaborate) and of open source distributed storage tech like Ceph (to connect highly performant file systems to colder, back up storage, often holding “large data.”) I anticipate we will see this space be among the first to broadly adopt tech in Internet-of-Things (IoT), blockchain, and more.
HPC has been important enough that the governments around the world have funded multiple initiatives to drive more innovation using the model. Here are a few examples:
- The National Strategic Computing Initiative
- The Cancer Moonshot Program
- The Exascale Computing Project
- The STEM Program for Students (Science, Technology, Engineering, Math)
This week, there is a large gathering of HPC experts in Salt Lake City (SuperComputing16) for engineers and researchers to meet / discuss / collaborate on enabling HPC more. From implementing tech like cloud and distributed storage, to best practices in modeling and infrastructure, and driving progress more in medicine, energy, finance, security, and manufacturing, this should be a stellar week of some of the best minds around. (SUSE is out here as well – David has a great blog here on everything that we’re doing at the event.)
High performance computing: take a second look – it may be much more than you originally thought.
And maybe it can help revolutionize YOUR industry.
Until next time,
JOSEPH
@jbgeorge
The JBGeorge Tech Blog 