When you can get them juice...
We got the first 10 of our 30 additional render machines in Saturday. Dual Opteron 252s with 4G RAM and gigabit NICs for those who may be curious. Unfortunately, we couldn't really use them until today because Boxx forgot to give us the manuals for them. Oridnarily that would not be a problem, but since we had them pre-install and config Redhat 9 for us, that became a problem since the manual is where they tell you what their default root password is. Being impatient, I tried reinstalling the OS on one of the boxes, and found I didn't have the right SATA drivers or NIC drivers. Which kinda bollucksed that machine for now but such is life. It also irked me enough that I decided "To hell with it, I'll wait". For the future, we've ordered a SATA drive cloner that will make imaging these buggers easier, so once I have one machine that works, in theory they all will... 
What we have setup for our existing farm is a utility called SystemImager which rather handily enables you to take one system's disk drive and clone it across many machines. Perfect for a renderfarm like ours where every machine is the same. Now that we are adding a different machine type into the mix, we found we needed to build a new image specifically for the new machines. Once Monday rolled around and I could get ahold of someone at Boxx, I went ahead and setup one machine exactly the way we need it to be, and then set about generating an image of it for cloning, hoping to fix the damage I'd done to the second machine while simultaneously testing the new image. Alas, twas not to be. Apparently, the kernel that SystemImager uses to boot the target machine does not contain the proper SATA drivers so it could not properly partition the drive to be cloned. Bugger. I hunted the net for fixes, and found some, but none that contained the right driver for the SATA controllers in the systems we have. Of course, I could always build a custom mini-kernel with all the stuff I need in it, but I don't hate myself enough for that.
So, we have 9 new machines and one paper weight until the cloner comes in. On the plus side though, the new opteron 252 systems are fast. VERY fast. Betweeen the faster procs, faster bus, and faster NICs (Our older render boxxen are dual Athlon 2200MPs with 2G RAM and 100/t NICs), we're seeing about a 50% reduction in frame render times. Which has the same effect as if we'd added 20 of the older models. Nice. This will come in handy since we've currently got about 10:45 done for Weds. episode. But hey, we've still got at least a day and a half left before we have to be done. Nooo problem!
Another fun note for the tech geeks out there. We currently have our entire machine room protected by an APC Symmetra 16KVA UPS. When the entire farm is rendering, we average about 84% load. After adding 6 of these opteron based machines, we are 99%. Yikes. So, we had to run an extension cable into the machine room from my office (which is right outside the machine room) and plug in a SmartUPS 2200 to power the remaining 4 render machines. Reminds me of when I first started working for South Park. That was in Jan. of 1998. They were working on episode 111, Mecha Streisand, and were located in a loft above a small grocery store in Westwood. Rumor has it the place used to be a sperm bank, but I don't know that for a fact. At any rate, the place's power was certainly not designed to be used by a computer animation studio. For the first 3 months I worked there (until we moved to our first studio), a good portion of my day was spent hunting down which UPS was chirping and then hunting down the breaker that had tripped... Needless to say, things have gotten better.
In fact, it's almost time for Cash Grab Monday! Trey brings in one of those clear walled booths with a big fan in the floor. Everyone who wants to participate puts in a buck with their name on it (or they can nominate someone they want to see in the booth). Then Matt and Trey throw in some extra cash for spice (even a few C-notes). A bill is pulled at random from the pile, and the person who's name is on it get's to go into the booth. The cash is dumped in with them and the fan is turned on for 20 seconds. Whatever they can grab out of the air in that time they get to keep. It's a way to blow off some steam and get a little break before the real hard part begins, finishing the show. Personally, I'm not too enthused with this concept, but the crew seems to enjoy it. Someone actually walked away with over $160 last week. We'll see if anyone makes it big tonight. Later,
J^2