Jul 26, 2013 (07:07 AM EDT)
Brave Tales From The SysAdmin Trenches
Read the Original Article at InformationWeek
Today, July 26, is National SysAdmin Appreciation Day. So take a moment and thank the unsung heroes at your company who manage software deployments, keep networks safe and speedy, handle bizarre help desk tickets, and politely remind you -- for the tenth time -- that you must be connected to the VPN to use that app.
To honor these "overlooked network ninjas," OpenDNS, the vendor that sells the Umbrella Web Security service, launched the 2013 OpenDNS SysAdmin Awards. Self-selected candidates submitted stories for awards such as the "Hand-Me-Down Hacker," for creating your own solution because the budget wouldn't cover buying one; "Flying Solo," for solving an IT crisis alone; and "PEBCAK" (problem exists between chair and keyboard), for the weirdest IT support request and how it was resolved.
The theme of this year's SysAdmin awards program is "resourcefulness under fire amid tight IT budgets" -- an all-too-common state of affairs. The contest includes tales of genuine creativity and courage, with some comedy thrown in. Here's a summary of some of the winning award submissions.
Doing More With Less Award: Erik Sheldon, with KSBJ, a Christian music radio station in Texas. After being called upon by management to install Microsoft's Hyper-V without much working knowledge of the virtualization software and no training, Sheldon somehow made it all work. Here's his award entry:
Last year I was working on trying to put together a business case to take us from separate free ESXi VM servers to a VM Cluster for disaster recovery and high availability. Being a non-profit, any time we need to spend money it's a nightmare if it's not budgeted.
That being said, most of the purchases we have each year were not budgeted for because that's just how things go. Anyway, we went with a Hyper-V solution because in my old director's words, "It was the cheapest." Since Microsoft has amazing deals for non-profits, I get it, but we had never used the Hyper-V environment, and I was very comfortable in the VMWare world after using it for a few years at that point. Being in control of all I was (100+ older workstations that always have issues, 30+ servers at the time that were also old and had cranky issues but were moving to VMs to ease that pain, a network that was ever-evolving, a flat network to all of a sudden the need for 20+ VLANS per my director, a complete overhaul of our radio station's control rooms, boards and software), I was ready to snap if I had to handle one more thing.
Luckily, my director found some vendor who would give us a cheap deal on the setup, configure and training of the Hyper-V environment. "Cheap deal" meant that they wouldn't come every day, all day, until the project was done from start to finish. It meant they would start it, then come out when other clients didn't need them at other locations.
So fast-forward two weeks later. My boss is upset with how things are going with the project, and up and fires the consulting company without letting me know. When they don't show for a few days, I call them to get a timeframe and they tell me to speak with my boss, who then in turn tells me that he fired them "because they were too slow, and since I've been watching them, I should be able to take the project and run with it." He didn't care to find out that they were only about 30% done, that they hadn't done any training yet, and that all I saw while watching them was just trying to figure out what exactly I was looking at, since I had seen a SAN setup, a cluster setup or … oh yeah … a Hyper-V environment!
So, add the next week of working crazier hours than I normally do, an unhealthy amount of caffeine (quad-shot grande white mocha from Starbucks will put hair on anyone's chest), and every article I could find to help me, and I came away with a 90% completed Hyper-V cluster. The reason I say 90% is that over the next few months, I'd find little things here and there that I had to tweak, or that I wish I would have understood more to set it up better.
The experience was rough, but I know a ton more than I did, am able to work in Hyper-V or VMware, and have an amazing private cloud to slap on the resume. The best part of the whole thing was that my director at least listened to me on what hardware to purchase, even though we didn't have the funds budgeted for entirely. Over a year later, things are still working fantastically. The projects still come like that -- even more so now, though, since he and my new director have seen what I can do and want to get every last drop of sweat out of me.
PEBCAK Award: Eric Reynolds, with Hudson Printing, Salt Lake City, Utah. And now for an amusing tale of user error, Reynolds' award entry:
We had a colored-pencil department full of creative types. They loved their Macs with the color-corrected screens and crazy graphics programs that never would run right on a Windows machine.
Not long after I became the sysadmin, one user developed problems with her monitor. She arrived an hour before I did and would call and leave a message that her monitor was "acting funny." She said the color would be off, it would blink, and sometimes she complained of smoke. By the time I would get in, there was no problem. Color looked good, no problems, and she was working. After a week of this I decided to come in early to see what might be going on.
When I arrived, she was not there yet. So I sat down in an empty chair and waited for her. We spoke for a few moments when she arrived, and I told her to go about her normal routine. She flipped on her computer and monitor and there weren't any problems. I waited around for an hour but no problem ever showed itself. So I went back to my office. Before I got back I had a message waiting for me. So I ran back to her workspace, and sure enough, her monitor was going crazy. I asked what happened and she said, nothing. I asked her about every step she took after I walked away.
"All I did was water my plant!" she cried.
I looked and noticed a small potted plant on the shelf above her monitor. Under the shelf there was water damage where it would drip down after she watered it.
I solved the issue with nothing more than a gentle suggestion to keep all liquids away from all electronics, and by moving her plant.
Hand-Me-Down-Hacker Award: Ian Silber, with MIBAR, a New York-based managed service provider. Silber and his team were faced with the task of redoing a client's entire infrastructure after four feet of flood water wiped it out during Hurricane Sandy. His award entry:
I am Ian, and I work for a small- to medium-sized business consulting/MSP firm in the NY/NJ area. Picture the aftermath of Hurricane Sandy taking out an entire SMB company's infrastructure due to four feet of flooding. Your heroic systems administrator, Ian, was able to get a long-time client's mission-critical applications and data back up and running in 24 hours and all employees working at almost 100% capacity. I used three already-decommissioned servers to house the mission-critical applications and data as a temporary measure.
All data was restored using tape backup that was brought offsite, as well as offsite/online data backup.
After a period of two weeks or so, we brought in an entire new infrastructure utilizing server virtualization to replace all the temporary hardware. All in all, due to flooding, about 30 PCs, 10 physical servers, 4 network switches, 1 firewall, and LTO backup drives were replaced.
Flying Solo Award: Gary Hartwell, with PBC & Associates Consulting, Tampa, Fla.
Walking into an outdated server environment, Hartwell took matters into his own hands. From running help desk to handling database administration, he's a shining example of the do-it-all, always-available sys admin. Hartwell's award entry:
Not only am I flying solo as a sys admin, I am flying solo as the only IT support for the business. Mostly coming from large corporate environments, my users are used to the full array of IT services. When hired, I inherited four physical servers in two offices that were already either EOL (end of life) or over five years outdated. Shortly after, I relocated, consolidated, virtualized and expanded services to running more than 18 different VMs for such a small user base. I have solved outages from a cruise vacation to a family member visit. As the only IT person, I handle help desk, network engineer, SysAdmin, security and database administration. From convincing the management that a storage closet converted to a server room with six servers needs HVAC, to fixing the classic 'I cannot get email' Internet requests, I have been flying solo for three years.