Anyone here deploy a Microsoft HPC Cluster?

I'm just about to deploy a 250 node MS HPC SP2 cluster on Win2k8R2 compute nodes. I successfully deployed a small scale version of the design with good success. Anyone here have experience with large build-out's? Any problems with database performance?

 

unless this is a new type of bacon I have no idea wtf you are talking about

 

that happened to me once. i got better though.

 

that will get you 24-36 months in the State Pen!

 

If Microsoft made bacon, it would taste like $hit, and you'd have a lot of trouble "downloading" it!

 

I guess I'll have to wait for the smart people to show up on Monday...

 

Holy cripes man, that's a huge deployment. Is this all one cluster or multiple clusters? How about some more details of your design?

I would imagine database performance issues to be of no difference to any other MS HA solution (given that it's shared nothing disk). Or, are you going to be running a scalable shared-database across the nodes?

Details, man, we need details!



Edit:

I haven't had much time to fool around with HA on 2K8 so I'm not completely knowledgeable of all the cluster changes that may have occurred since server 2003...


#2:

After five minutes looking at MS HPC.... which cluster type are you going to build? As a DBA, my preference for a cluster design would be resilient one with multiple head nodes and a separate/remote database server.

IMO and experience, mixing an SQL server with any other workload (esp. something as critical as a 'head-node') is almost always bad news. At some point one application begins stealing enough resources to affect the others, and it's usually the database server that's the culprit.

 

Dude, it's a holiday. Have some fun.

 

BACKGROUND, sorry so long but the other nerds here will appreciate it:

The design is a bit complicated, yet elegant.

The cluster nodes are actually virtual machines running on VMware ESXi 5. The design is 6 Win2k8R2 VM's and 6 CentOS 6.2 VM's, each with 4 vCPU's and 6GB RAM each.

The physical servers running ESXi have 24 cores and 72GB RAM each so the CPU's are over subscribed at a 2:1 ratio but the RAM allocation is 1:1.

The idea behind the design is to leverage the physical hardware that was natively deployed as a separate Linux HPC cluster and a separate Windows HPC cluster, but do to the irregular periodicity of use, at times, one grid was lit up to 100% while the other grid sat idle. Now both grids are always available with the ability to scale out to a much larger hardware pool in the virtualized environment than was available in the native environment. I've been running this configuration for about a year with great success, but now I'm exploring other grid schedulers for the Windows side. We use SunGridEngine on the Linux side without issues, and have been using it on Windows as well, but have run into several limitations with Windows.

That being said, I started playing with Windows HPC Server R2. The work thats performed on the grid is high fidelity modeling and simulation runs. Some studies take two to three weeks of CPU activity at 100% to complete. That's on a grid with 324 cores. My design puts the new vGrid (virtual grid) at 1124 cores. I hope to see dramatic improvements in productivity while handling jobs through a native Windows grid scheduling solution.

 

So you recommend a remote SQL database vs on hosted on the head node? I can scale up the hardware on the head node up to 24 cores and 72GB of RAM if needed.

The cluster type is type 5. High availability is handled at the virtualization layer so I didn't see the need for more then one head node, unless having two load balances the work?

I'm no DBA so any other advice you can provide is greatly appreciated!

 

Cool setup, I was expecting more hardware actually.

I'm still on the fence wrt virtualization (keep in mind all I deal with are database systems), but I can recognize it's advantages and do support it where it makes sense. We have a 3.x cluster and the scheduling subsystem isn't all that great. We're seeing some significant improvements in scheduling with the 4.x version, but it's taking forever to get our systems upgraded...

I still would recommend the separate database server for a design. I imagine that the database server might end up being the source system that is responsible for managing 'synchronization' between the nodes and infrastructure. The documentation on MS's site doesn't really give out a lot of details of what the database server is used for in an HPC setup.

 

Just use this...

Question: Will I go faster this year..
Answer: Not on your life...

 

i deployed a cluster to the toilet this morning.

 

Thanks for the info! The hardware in the virtualization pool currently is 48 ESXi 5 servers configured with 24 cores and 72GB of RAM each. This is phase one. Phase two is the buildout of two additional racks giving us a total of 144 physical servers with the same configuration. Phase three is to go to a dynamic design allowing users to configure their grid nodes as needed and deploy them to the private cloud. I hope to get other groups that do M&S work in my department to adopt the design and pool their resources into the private cloud. This design could potentially grow to 24, 48U racks of HPC compute for a total of 1152 physical servers, all for mixed Linux and Windows high performance computing!!

 

Smart people don't use Microsoft products.

What kind of disk are you using for the DB machines VM's? How complicated is the schema with regards to joins or unions amongst the tables along with size of tables and data?

A remote DB may suffer on wire speed depending on the result set, but bad storage decision on locals with regards to VM's may hurt you on performance. Also, if it's large data sets that change often you can hurt yourself with replication lag with writes depending on storage on the cluster.

-steve

 

Keep us posted, that's a very cool idea and design. I'm always interested in this type of stuff (yes, I read the full disclosure TPC benchmarks)....

Last bit of info....

I imagine that your licensing agreement with MS might be pretty liberal. I know ours is at the Enterprise Agreement level. If it's a virtual database server, I don't pay any license fee as we're licensed on the socket that VMHost provides via DataCenter licensing. If it's physical, I pay a per-socket fee like normal, just with a significant discount.

With that said, it's very easy to move database servers around these days. A competent DBA (heck, even a good IT person) should be able to do it without issue. It's my preference to build separate systems to start with and collapse back into a consolidated stack if the workload on the database server affords it. Heck you may even be able to utilize an existing low-usage database server to consolidate onto.

 

This sounds like the database server is part of the HPC cluster management for the infrastructure and not part of the end-user processing on the system. If that's the case, it's likely to be a significantly lower io consumer if it's only managing 'state' for the cluster.

The end-user processing seems to be highly dependent on cpu and not disk. I could be wrong though...

 

Smart people deploy the best products available for the requirements. But I'm in your camp, I'm not a MS fan. Unfortunately SGE doesn't play as well with Windows as I hoped. Simple parallell grid runs work great, but without the ability to do check-pointing, we're limited to the type of simulations we can run.

Storage for the static vGrid design is local storage to each ESXi host. Each host has two 1TB drives, each housing three VM's of each OS. our simulations are not I/O intensive, they are CPU intensive. Runs generate light network and drive I/O and output is in the .5 to 2.5 GB range. Network is two, 1Gbit per ESXi server for the grid and 6, 1Gbit per ESXi for the management cluster. vGrid nodes are disposable. The granular design of 6 of each OS allows for total OS failures in a VM without a significant loss of compute resources to the overall grid. It also complies to our current licensing model we have with VMware for their vSphere Enterprise Plus licenses.

The management VM cluster is backed by two NetApp FAS3240's with two 512GB SSD PAM modules and two 10Gbit ethernet uplinks. The storage is rated at 70,000 IOPS.

The architecture design of the vGrid has been proven with a years use. The changes to my design are only grid scheduling for Windows. After working with Oracle for several months, they most likely won't be updating Sun's grid engine code to work more seamlessly with Windows 2008R2, so I have to look for another solution.

We have a ELA with Microsoft that gives us all of their products with unlimited use.

 

Agreed. I've no religious bias with technology when it comes to making money for a company (which in turn puts money in my pocket). Whatever technology makes sense or a customer is comfortable with.

Even more reason to start scaled-out first and consolidate if it's worth it later.

 

Do you have a CSS contract?

-Knotcher
Microsoft