Ivory Coast: Not Just Soaps!

About six weeks ago there was talk about going to the Ivory Coast to do a site survey in anticipation of the project coming up there next month. I had volunteered because I thought it would be an interesting break from Ghana. Well you know what they say about volunteering. Now that we are close to wrapping up someone said ah, what about Ivory Coast! We were going to send someone... oh yes, you! So it looks like I am stopping over in Ivory Coast to inspect a shore office and a vessel before I wrap things up here and return to the US. There is one other person coming with me, and people who work here go there all the time for work so I don't feel like it's going to be unsafe. I am hoping to be home on or around Sept 6--in only 10 more days!

A site survey should be a lot of fun because it gives you the chance to inspect the existing site and identify the issues. Sort of a "what's wrong with this picture" but on a bigger scale. Plus I get to visit another country--this one a former French colony, so it should be quite different from Ghana. It has been a good project and I'm glad I did it, but I am also glad it will be wrapped up soon.

Singapore Sling

Well we are reaching the end of the road and just as Odysseus encounters new challenges even after he reaches the shores of Ithaca I have had new items pop up. We had to run back to Takoradi today to move a server to the DMZ and make a few otehr adjustments. We go back to Accra in the morning. Meanwhile there was a "oh by the way we need you to set up xyz for us too" and that will be a whole new mini-project in itself! Here's the fun part: the management team is in Houston (-5 hours) and the support team is in Singapore (+8 hours) and I am in the middle. So for example right now (9:00 PM in Ghana) it's 4:00 PM in Houston and 5:00 AM tomorrow in Singapore. Whatever directive I get from Houston can't be implemented by the Singapore team till the next day, and if I need clearance from management for something it has to wait until after 2:00 PM Ghana time, which is of course long after the Singapore guys went home!

No going-home date yet but we were hoping by the end of this week. Not sure yet how the scope change will affect that date. May know more Wednesday.

Meanwhile congrats to SS who had her first day today of HIGH SCHOOL! :)

Peep

No real updates; except to say I am still alive and working long hours. We are seeing the finish line ahead but there is still much to do. Looking forward to coming home. More later! :)

Back in Accra

I am back in Accra today. We flew Ghana Air and I was a little concerned about it but everything was fine. We are about to start the big wrap up; almost everything is online and accessible but we still have a couple of weeks worth of things to do. There is one satellite office that we are going to set up but the equipment hasn't come in yet. Some changes have been requested and this is something I'll be working on until we get the other equipment in. However we have been kicked out of the office room that we did have in Accra because the guys from the upcoming satellite office are here waiting for it and they took our old room over in our absense! So I am working from a guest reception area in the lobby. I am closer to the kitchen now but they haven't replenished the supply of McVitie's, so I'm not so tempted to snack.

Overland and Sea

Got back to Takoradi. The helicopter ride is always cool for the first 10 minutes. We lurch off the ground a few feet and hover, then glide away. After that they get up in the clouds and there is nothing to see, so I read for a bit. Odysseus makes it back to Ithica but is in disguise.

Because I had prepped the servers today was going to be a quick and easy day and I would have most everything done by 5:00. Ah but like Odysseus, the fates conspire to keep me from completing the journey. It wasn't anything as interesting as a cyclops or the wrath of Poseidon, though. Here is what happened: Plugged in the Overland tape drive and flipped it on. Waited for the screen to come up. Waited some more. Noticed the ominous flashing amber light in the shape of a Caution sign. Noooooo! Because it was bought in the US they wanted me to open a US support ticket. They open at 7:00 AM PACIFIC time, which is 3:00 PM here. Wait 5 hours. Call. Wait on hold 30 minutes. Register with support. Read the serial number. Repeat serial number five times. Verify serial number twice. Call back bacause line was dropped. Repeat three times. Lose Internet when power goes out. Phones go through Internet. Wait for Internet to come back up. Call again. Get trouble ticket issued. Was asked other then nothing on the screen and a flashing light, were there any other problems I was having with it? Get an engineer on phone. Drop call and repeat. I now have an open ticket with Overland. But where did my day go?

The Beat Goes On

My coworker left today for Takoradi, leaving me to fend for myself amongst the IT deprived. I am scheduled to go back tomorrow but it all kind of depends on corporate. We have been emailing back and forth all afternoon about a support issue. Meanwhile I am doing remote work on the office back in Takoradi. Wish there was something interesting to report but all I do is work and I doubt that would interest too many readers. So I apologize for the banality of this post. There was a bit of excitement as they changed out some satellite descramblers for the tv channels and most of twhe channels promptly went out. There were raised voices and some enthusiastic gesturing among the people trying to get it fixed, as everyone was convinced it was the other person's fault. Judging by the current quiet they got it resolved. Good thing too; at lunch and dinner the satellite channel plays B-grade movies. Lunch wouldn't be the same without watching Tom Selleck or Meg Ryan or Zac Efron... no David Hasselhoff yet, though. And no one seems to want to watch BBC or CNN. We did get a new shipment of food in (same caterer) and we had pork chops the other night. Assured they were fresh and free of trichinosis, I ate four. But the food has not been too great since.

No time for light reading; Odysseus is still conversing with the ghost of Tiresias and it may be tomorrow before I get to Scylla and Charibdis.

I have not been sleeping well but since the snorers are both gone maybe I will sleep better tonight. We got a third roommate the other day, a short, thick-necked, affable Aussie who warned us he sleeps nude. I don't look; I keep the curtains around my bunk closed! He is still here but at least he doesn't snore.

Progress is Good

Got some momentum now. Have 3 of 6 servers migrated and 8 of 26 user workstations. I will be hitting the workstations today and tomorrow. It's time consuming for a variety of reasons. One is that the first workstation I migrated deleted the existing profile which included everything in My Documents and Desktop and I hed to try to go back and undelete the deleted files. So from now on I back that up first and sometimes it's gigs. I know it's not a best practice to store things locally but frankly I am not here to break bad habits.

We are looking at trying to go back to Takoradi by Tuesday. Monday would be the earliest I could go back as there are no hielcopters coming out this weekend but I won't be ready that fast.

Didn't sleep much the past two days but last night someone gave me an Ambien and I crawled into bed and went back to reading Homer's Odyssey--something I read in airports and waiting on helicopters. The next thing I knew 10 hours had passed and the light was still on over the bed. So I guess the Ambien worked!

Side note about the Odyssey: Though I am not so familiar with the Greek classics I can say that for a hero Odysseus is oddly prone to emotional tantrums. Examples about but for one when Circe tells him he must go to see the spirits of the dead and meet with the ghost of Tiresias he describes throwing himself onto a bed and rolling around and crying until he tired of it. Sounds like something a two year old might do. It's an interesting visual and I have to think the movie version wouldn't be so faithful to the original text.

Grousing

Don't get me wrong. I'm glad to be here working. But let me vent about my day so far.

So far today I have hacked into an EMC SAN (easier than it sounds, it turns out); talked a manager through restarting servers via satellite phone, talked an engineer through configuring his laptop for DHCP via satphone and a translator; set up new user accounts, and sundry things related to AD. All while being constantly interrupted by people with side issues. You can be sitting in the server room surrounded by equipment and wires and working feverishly, banging away at the keyboard like Faulkner, avoiding eye contact with the person hovering over you like vulture, and then then they open their mouth, hesitate, then jump right in with the inevitable opener. "Hey can you look at something?" No, because I'm already looking at something! The only way I could get the DC up was to use a media install off a system state backup that took 12 hours to copy. Now it's up but it still acts like it has to check with HQ before it shows anything. Still waiting on DNS to magically appear.

The Filipinos made squid today at lunch and it stinks like the trash bin behind a Chinese restaurant. Rumor has it salmonella has broken out in the past couple of weeks. I am now eating rice and bread. I can do it for a couple of days but I may need someone to drop ship some pizzas.

Is Weds still Hump Day when you work 7 day weeks?

There are three of us remaining in the team (see earlier post) and we are all present. The two of them are now snug in their beds but I am up and back at work. Both of them snore terribly. It sounds like two lovestruck toads calling out to one another, and I cannot sleep through that.

Yesterday was a long day and today probably will be as well. The Internet connection that they had wasn't that well-kept a secret and when we arrived the server closet was criss-crossed like a Christmas Tree as people surreptitiously connected more and more ports to the connection. There were at least FIVE rogue wireless networks, all broadcasting DHCP on networks that we are trying to move off of, all undocumented.

About 1:00 AM I noticed a file copy off a server in Accra had aborted. A little checking and I found our servers were overheating and shutting down! The blade center was reporting temperatures of 55 C. No idea whether there really is an issue like the AC in the server closet went out or if the place was simply on fire. We have asked someone to look at it in the morning, which it is now, and I expect a report in a couple of hours. I may have to go all the way back to Accra to deal with this, which would take about 6-7 hours.

Rack & Stack Day

We arrived in Takoradi about 10 AM and drove over to the office. We were looking around as we drove through some market streets and looked at the bright colors and the people. I took a few pictures and will try to post as I can. We go to the office and went straight to work. They had a server room ready and an empty rack. With our shipment of about 80 boxes already here from Accra, we immediately set into putting equipment together and racking. I keep getting interrupted by office staff who haven't seen an IT guy before and have many questions and tasks...

The local manager is taking us out for dinner. Our flight to the vessel tomorrow is at 6:30 AM! Argh!

Takoradi

We are nearing a place where we are ready to go set up the other sites. One is a shore office in Takoradi and the other is to redo the vessel by migrating it to the corporate domain. The servers were packed and shipped out today to Takoradi and tomorrow we are flying out to collect them and set them up. I get all of one day (assuming the shipment arrives timely) and then we are off to the Kobyashi Maru. One member of the team went ahead and is there now getting the Internet set up. You can hear the cheering from here, 130 miles away. Well, not really. But I assume they are ecstatic.

Tomorrow begins a big push towards the finish. If you had a chart of our progress it would look rather flat up to this point because we had to get the infrastructure issues addressed first. But these next sites should go up pretty quickly and I am hoping by the end of the week we will have a lot to show for it. So this is the big bubble in the progress chart! None of us want to be in Takoradi or on the vessel very long so we will be pushing hard to get it done quickly. There won't be a lot of sleeping going on this week...