One of the really weird things about my job is that I am the person who specifies what new hardware we buy. Which can be fun, but it also amounts to an ass-load of responsibility. Shining example: When we first start doing South Park, we were using SGI workstations, whch were at the time the best, and only, option for doing professional 3D animation. They were fine for a couple years, but as PC technology finally reached the point were you could take them seriously as a professional workstation, it was a no brainer to switch to them, as the cost of purchasing and maintaining them was less then it cost to just keep up maintenance on the SGIs. We had hopes to switch to Linux, but the Linux version of Maya just wasn't a reality at that time, so we had to use Windows. Luckily, Windows 2000 was out by then, and it is by far the best version of that OS to date. So, all was good.
We ended up going with SGI's own PC machines at the time, called 330s. They were fine, dual Pentium III/866MHz with the VR3 (SGIs own rebranded version of nVidia's Quadro card), and 512 MB of RAM. Nothing compared to today's standards, but back in the fall of 2000, they kicked the snot outta the older IRIX SGI workstations. At the same time we switched platforms, we switched programs, finally moving from Alias' Power Animator 9 to Maya 2.5. We took a full 3 months to do it right, including developing a lot of MEL scripts to make working with Maya easier and faster, converting most of our charcters, props, backgrounds, etc. from Power Animator to Maya, and even developing a customized interface inside Maya specific to our workflow. Everything seemed to go very smoothly. Until our first episode of season 5, 5-something.
I remember it wasn't 501, since that aired later in 5th Season, but whatever the first episode in 5th season was, it was a rough one. The main problem wasn't that the workstations weren't fast enough, it was that they were too fast. The artists weren't optimizing like they had in the past since they didn't have to anymore. Unfortunately, while we did upgrade the artist's workstations, we did NOT upgrade the render farm, which was still using 16 4-processor SGI Origin 200s. So, suddenly, the render farm was rendering much slower then the artists could submit shots to it. That's a problem for us since the last thing we want is to have artists sitting around for 4-5 hours on a Weds. morning waiting to find out if their shot is approved. So, we had to re-address the issue of optimization and remind everyone to keep things as tight as they could. And things got better. For a while.
Two years later, the fall of 2002, we were finally able to upgrade the render farm. For less then the cost of one year of maintenance on the IRIX machines, we were able to replace our 64 IRIX processors with 60 Linux-based Athlon MPs that were able to render about 250% times faster. Not a bad deal, all things considered. Again, all was good. For a while... Fast forward to the beginning of 2004. Our Pentium III workstations are on their last legs. This fact is nailed home when one of the system drives fail. We call up SGI, and find out that the 330s are no longer supported. What they call End Of Lifed. So, if we want to get a new system drive, we can always buy one from them. For the paltry sum of $750. A quick search on pricewatch.com showed the EXACT same drive, a Seagate Barracuda 18G SCSI 160, averaging at $150. And a 40G IDE ATA-66 drive was about $80. So, after a brief pause to try and invent an experlative to properly express our displeasure (and failing utterly), we sent a PA out to buy a new IDE drive. And I began looking into the state of Workstations with the intent of finding a replacement.
One of the first decisions I made was to drop Windows like the whore that it is. Sure, it's been working alright, but the never-ending battle against spam, virii, malware, etc. had just gotten too old. It was time for a paradigm shift. So, that left two other real options. Linux, always a contender and getting better every year. And Mac OS X, which I hadn't played with until June 2003, when our Associate Producer, Jennifer Howell, decided she was done with the Powerbook she was using and wanted a new desktop machine. I inherited the Powerbook, and the first thing I did was look into getting Linux on it. After much searching and realizing it was gonna take more effort then I was really interested in applying, I decided to give OSX a shot. At the time, the thing was running OS 9.2, and sucked balls. Big time.
Anyone who's known me over the last 15 years knows how much I've hated Macs. From my first experience in high school, I've just hated how clunky they feel... Which is odd when you consider my first experience on a computer was on a Apple IIc. Any old-school user can tell you that the feel of the OS on an Apple IIc is actually much closer to OSX then OS9. I think that not having the ability to quickly and easily jump into the guts of the OS and see what the machine was doing was an instant turn-off from Macs for me. So, after decades of bitching about Macs, I decided to give this new OSX a shot. Everyone's been raving about how nice and pretty it is. And to my pleasant surprise, it was good. Very good. So good that 6 months later, I bought myself a new 12" powerbook. And have never regretted it. It is solid, fast, and caters to every level of user, from the first timer to the really old-school types. So, when it came time to decide on a new workstation, I decided OSX was ready to be a contender.
So, now that we have our two contenders, it was time to figure out the hardware. On the OSX side, the choice was rather simple. Just get the fasest one they got, stuff it with RAM, get the best GFX card they sell, and call it done. I specced a dual 2.5 GHz G5 with 2G of RAM, 160G of HD, and a nVidia 6800 Ultra. For Linux, I went with a dual Opteron 250 with 2G of RAM, 160G of HD, and a Quadro 1100. I did my best at trying to get 2 systems at a close enough price range, and with similiar enough specs to make it a fair test. I had to get 64-bit procs to make the machines as future compatible as possible, and besides they're just faster! Next was OS. Since the version of OSX that runs on the G5 is 64-bit, I had to get a 64-bit version of Linux to run on the Opteron system, so I could do as even a comparison as possible. Silly Me. Maya does not support ANY 64-bit version of Linux. It will run, but is NOT stable.
Now, as I've mentioned before, our main app is Maya. You could say that its our only app, really. So, if it won't run well, we're SOL. So, I had to put Redhat 9 on the Opteron machine, which is sad on more ways then I can mention. Anyways, the test went was follows: We put each machine in front of an animator for 3 shows, then we had them swap so they each got some experience on both platforms. The idea was to try and get a balanced evaluation of the two to make a final decision. And it was interesting. The linux box was faster. Much much faster. But it had serious stability issues. And sound was iffy at best. Now, I know that many other studios have been successful with running Maya on Linux, but one thing to keep in mind that South Park's technology department consists of myself and my assistant, Keith Nesson. That's it. So, we don't really have the time to make a custom version of Linux. Our creedo here has always been K.I.S.S., Keep It Simple Stupid! So, whatever is going to be the cleanest, most streamlined solution is going to be our first choice. So, OSX won. Kinda.
It wasn't really an even fight. Since Maya only supports a version of Linux that is now several years old, and is NOT 64-bit, that immediately put the Linux machine at a disadvantage. We had an rather aged version of Linux running on fairly recent hardware. Problems are bound to happen. Especially if you can not, or will not, customize the kernel to add the features needed to properly support the afore mentioned hardware. Add to that the fact that Maya's sound support under Linux is spotty at best, by their own Qual Charts, and you have a recipe for failure. And not having a reliable sound option is unacceptable for us since almost every shot we animate has someone talking, or is synchronized to sound in some way. So, even though the graphics were MUCH faster on the Linux machine (by almost a factor of 4), we had to go with the OSX machines, since the sound worked better. Mostly. There was one problem that was almost a show stopper.
Maya has a feature called Repeat on Hold. What it does is when you drag your mouse along the time line, you hear the sound play. With RoH activated, when you click and hold on the timeline, the sound for that frame loops. This feature was absolutely necessary for our lipsynch department to work efficiently, and was 100% broken in Maya 6.0.1. It just plain did nothing. So, we called Alias, and after much complaining and bitching, got them to sneak a fix into the next version of Maya, 6.5, which was about to be released. I will never cease regretting that decision. The first reason for regret is it meant we had to start using a new version of Maya in production without spending any actual time testing it. Any systems person with half a brain can tell you that was REAL dumb. But we had little choice. We had to get new systems, since the old workstations were dropping like flies, and parts to fix them with were getting scarce. So, we struggled on. It's been a rough run, but not entirely my fault.
Sure, if we'd had the time, we could have found that Maya 6.5 crashes like mad when you play back animation. That's a BAD thing. We could have found some way to get lipsynch to deal with the lack of a repeat on hold function for one run so we didn't have to go into a run with an untested version of Maya. We could have not upgraded at all and just run the machines we had into the ground for one last run and hope we made it through. But even then, this lastest run would have been brutal. 901 = lots of large crowds, 902 = a TON of crowd shots (1 million hippies? Yikes! The story boards actually said "The largest crowd shot ever seen on South Park"), 904 = 10 MILLION DEMONS? (I kid you not, that's what the script said!). No matter what system you work on, dealing with that kind of shot is rough. But we did it. Barely, but we still pulled it off. So, when it all comes to an end, the switch over was for the better. Even if Maya is less stable then it might have been on the older machines, the machines themselves are more stable. At the end of the day, that has to count for something, right?
Wow. I just realized how bad my verbal diarreah is. Yikes. Well, I hope this isn't too boring. I'll leave off for now. Later,
J^2