TL;DR – Anybody that tells you qakbot is a simple little thing is grossly misinformed. It is a smart, mean, and ugly virus.
I get into work a few minutes early so I can check in with the Ops guys and see how their night went. It didn’t go well. There was no new developments in stopping the issue, and they now could see that a few of the servers have been infected as well. They are in the process of getting all but the absolutely critical servers shut down, and they’ve told everybody to shut down their computers as well. Except for a few computers the Ops and networking room that are clean, we’re in a state of “darkness.” I hear our director is going to talk to the county administrator and if some kind of press release is going to be made, they’ll want to put it on the web site and that’s where I’ll come in to be able to help a little.
That reminds me that I have to go check in to my desk. I’m in a small area away from the main area with the rest of the MIS folks, so I know I’ll be doing some walking today. I get to my desk, just as the phone rings on one of my fell web teammate’s phone I answer it, and they were looking for my colleague to get some info on what’s going on, and then realizes she’s out for the day. I give them the basics of what I know (we’re still down, don’t turn on your machine, we’ll make contact with all departments as the situation is updated) and finish the call. It dawns on me that it might be a busy day like this, so I forward the phones of both my web teammates to my phone, check in with my other colleague that is in the office today. They were supposed to be out for training, but their web connection couldn’t handle the video stream and fortunately we have an outside line that is unaffected by the virus. So they came in today to keep with the training. I walk over to the administrators office.
I get into the reception area and there’s already a group forming. The director is there, as well as two of our Ops guys, the administrator, and a couple of folks from the another department. They’re talking around a computer that is on and known to be infected. Our director is pointing out a couple of tell-tale signs that the system has been infected (folder names that are made out of random characters that sit in the user’s folder, Symantec Antivirus briefly failing upon load and then saying all is fine, weird activity in Internet Explorer). It is asked if we know precisely when and where we got the virus, and the director answers that it we don’t know precisely, but we have a pretty good idea.
It’s a tale as old as time really about boy meets girl. However in this case the boy is a typical computer user and the girl is a well dressed phishing scam e-mail that they received. The user was doing their typical routine of checking e-mail and a couple news sites and clicked the link in the e-mail that looked a little odd, but nothing too bad. They became distracted with another issue, and when they looked back at the screen, they noticed that there was an odd window on the page and the mouse started acting up. Almost immediately after that they noticed that a couple computers next to them immediately started acting up, and that something had happened.
With the entire network shut down, there are critical services that still have to be provided, and that’s what the department is here to ask about. We’re getting into triage type services now and the director assures them that their own private network is safe at this point, but they’ll have to identify what other critical services they need and they’ll make it happen. The administrator indicates that they are already working on a press release about the issue, and I tell her I can put it up on the public site whenever they’re ready. She says thank you and I coordinate with her assistant to make that happen.
I head back to my desk and realize how reliant we are upon computers and the network these days. I know that I’m a programmer and that most of this comes naturally, but at the same time when even some of the basic work functions rely upon it, you quickly become a little speechless at the idea. I decide for the time being that since I don’t have access to a computer, I can do some layout design on my current project on my whiteboard. This is a refreshing exercise because I can easily erase and redesign until I’m happy with things, and then transfer it to a piece of paper for my notes. My artwork is atrocious, but hey, it gets the job done.
I get the copy of the press release that was sent out and then head over to the Ops section to use one of their computers to access the website. I have to install a couple small pieces of software to get things done, but it takes longer than I had anticipated, but I’ve been tossing out a couple of ideas to the networking and Ops folks as I prepped things and was able to confirm a few thoughts I had on the issue. I get done with the press release and start to head back to my desk. The director, our tech support lead, and a couple of other folks are talking about the issue, and they’re meeting in a little bit with everybody to discuss things. The director looks at me briefly and asks me if I want to sit in on things. At first I’m not sure, since I don’t have any major leads, but I’m always willing to toss in whatever thoughts I have, so I decide to sit in.
The meeting has a decent amount of folks in it. There’s at least two Ops guys, two tech support guys, the administrator, the heads of all of our groups there. As the meeting begins, I become speechless with what I hear. One of the tech support guys has been the one to do the most research on the issue, tweaking around with a machine. What he’s found is that the virus likes to spread itself through network connections. By default, all our users have at least 2 network connections, so that can be tricky. Worse, it uses some Internet Explorer hacks to start getting more programs down. As he finds a registry hack or a folder that needs removing, the virus comes around right after you and removes your permissions from doing the same thing again. It already knows how to cripple Symantec Antivirus, even with the latest definitions. It is a wicked bugger that we currently don’t have a solid bead on beating. We don’t have the information we need about it in order to stop it. There are talks about reformatting the entire network. That involves roughly 1600 machines and since some of our servers are compromised, we can’t use our Altiris system to fix everybody over the network, since they may get infected, or may get it again by another machine. We could potentially setup some cluster networks offline to fix machines, but again we don’t know what the virus is doing, so reintroducing them to the network would be futile.
I consider myself a pretty well rounded geek. I focus on slingin’ code, but I try to keep my wits about me in regards to servers, networking, programs, and the like. This is one of the few times I’m quite literally speechless about how to even attack the issue. It’s rather humbling I do agree with one of the tech guys that we just need more time to figure this virus out, so we can plan an attack. Our director assures us that we ARE going to fix this issue, and we are all determined to do so. A couple of the networking guys are going to continue to pursue options with Symantec for a fix, and the rest of us are going to look into research and preserving the network. I leave the meeting a little disheartened, but determined.
I update a couple programmers on things, and talk to Networking about getting a Ubuntu disc to boot off of in the morning. Our network circuit is still open, so I can get out on the net that way to do some more research. It’s already a bit after 5, and we were invited to dinner with some friends, so I plan on hitting things hard tomorrow, don my “Googlian Monk” robes, and try to track down some more information. There’s gotta be something somewhere.