Let me tell you all how I once put pretty shut down the whole mobile phone network in Chile circa 2004-5 [1/21]

I used to work for a now-defunct company in the telco sector. Arguably, this company built the first pre-paid billing systems for GSM networks ever, and we had clients all over the world [2/21]

I worked in Rio; we served all Latin American clients—mostly subsidiaries of European mobile carriers. In Latin America, pre-paid phones were popular, and we were doing real-time authorization and billing for probably 30% of all mobile phone calls in places like Brazil [3/21]

Our system was the most complicated piece of C code I have ever seen in my life. Each carried ran a different version—with different patch levels and customizations. There was no plugin architecture; flexibility was based on #ifdef client_abc enable feature xyz [4/21]

I was sent to Santiago to install the new release of a system I had been writing. This was a virtual PABX system, something that sounds alien today. Basically, it used USSD messages to do lots of shit. USSD is this weird stateful SMS thing that deserves its own thread. [5/21]

There I was, on my first international trip in my life. It was also the first time I wore a suit outside a wedding. Here’s a picture I found on Flickr. [6/21]

As I get there, the lovely folk at the largest telco company in Chile show me a Sun workstation I am supposed to use. I don’t speak Spanish; they don’t speak English. We met in the middle in the mix of Portuguese and Spanish we are used to in South America, aka Portuñol [7/21]

The system I am supposed to install a new release is called wIQ. They give me username wiq, pwd wiq123 to log in the server. It works, and they leave me alone to do my work. I pop in a CD I had brought from Brazil and copy the binaries over to the server using trusty scp. [8/21]

I then kill the running process and wait for whatever version of SysV init that thing had to respawn it—I am pretty sure it was a DEC Alpha running Tru64 UNIX. The process comes back up, and start core dumping like there’s no tomorrow. [9/21]

If you don’t know what a core dump is, that’s when god kills your process, and it vomits all of its memory, register, stack pointers, and shit to disk for some future debugging nobody ever does [10/21]

One thing about these old school UNIX systems is that they often do IPC through some fucked up ways. In our system’’s case, it was a lot of shared files implementing a FIFO (think stone age Kafka) and shared memory (think you’re fucked) [11/21]

In fact, a lot of this system’s architecture is similar to what some in-memory databases such as Prevayler (https://en.wikipedia.org/wiki/Prevayler) built, except ours was from 1998—but they suck just as bad [12/21]

Anyway, the billing process writes to shared memory and that was crashing my new code. I was nervous, but still keeping my cool. Until I looked around and saw that the whole operations floor for the largest telco company in Chile were running around like headless chickens [13/21]

It turns out that for whatever weird coupling thing, the whole Chilean mobile network was down. And for some even stranger reason, nobody thought it could be the twenty-something Brazilian boy in the corner [14/21]

I knew I had minutes before people come to my desk. I eventually work out that the problem was that the fucking billing process cached some information. All I needed to do was to restart that process [15/21]

But I was running as user wiq, and that process was running as user opsc. I tried everything I could think of that would kill a random process, and because I am a horrible h4x0r I fail miserably [16/21]

At some point, just before I give up and call the operators, I randomly open another terminal just to keep my fingers busy while I am thinking about my excuse. It prompts for a user; I am about to write the credentials they gave me, user wiq, password wiq123 [17/21]

But for some reason, I type in user root. And for password, I type in root123. And I am in. [18/21]

I can’t believe my eyes, but I kill -9 the fuck out of that process and wait for it to respawn. And everything is back to normal. And people start running around with their hands on their heads. [19/21]

I can’t believe my eyes, but I kill -9 the fuck out of that process and wait for it to respawn. And everything is back to normal. And people start running around with their hands on their heads. [19/21]

And everyone was so happy they took me to Calle Suecia, where there was a Flintstone’s themed night club called Bedrock where some artists enacted scenes from El Chavo del Ocho with the audience (mostly Doña Florinda sucker slaps) [20/21]

And it’s been 15 years, and I never told this story publicly, although a hint of it has been part of my “fun facts about Phil” emails since forever. Fin. [21/21]