Saturday, December 31, 2005
What a difference a year makes!
I look at my blog from last December, and I had a life. I talk about it in my blog-my son, my church, my old colleagues. I got out socializing. I miss my walks to the mailbox when I would talk to or at least see people in the neighbourhood.
Now there is just work. I spend way too much time commuting. I just veg out when I get home in the evenings. I spend half the weekends just trying to recuperate from the week previous. A year ago, I hated being at home. Now, I hate working. I mean I love what I do, but I'd rather be at home full-time with the old man and the kids. I know I won't have any problem retiring.
The one thing that hasn't changed is that our pension plan and its surplus are still in limbo.
D.
Now there is just work. I spend way too much time commuting. I just veg out when I get home in the evenings. I spend half the weekends just trying to recuperate from the week previous. A year ago, I hated being at home. Now, I hate working. I mean I love what I do, but I'd rather be at home full-time with the old man and the kids. I know I won't have any problem retiring.
The one thing that hasn't changed is that our pension plan and its surplus are still in limbo.
D.
Thursday, December 29, 2005
... the bad ...
Last week, I had to reschedule a day off because of IBM. I found out the day before that they needed some changes in order to test. This wasn't a last minute thing in that they could have said something earlier. I knew I would have to do the work, but didn't think it would be required for another two weeks. I came in and shepherded the request through change management. This took the better part of a day because of the yearend freeze. It wasn't even a production change. I got two half days off before Christmas and New Year's so I thought it worked out OK.
I only worked two days this week because of the holidays and the half day off, but I spent part of it getting ready for some server setups IBM is doing next week. I prepared MQ configurations files for two new QA servers. Late this afternoon, I find out that a third configuration file is required. I had assumed that IBM was going to do it, but I was wrong. I had planned to leave early so I did. I'm off next week but promised to come in if necessary. The hardware is already a week late.
I don't know what the IBM project manager is doing to earn his money. These requirements for our resources just fall out of the sky. We're paying them a half million dollars for this behaviour. Meanwhile, our PM is working 12 hr days to keep everything on track. I asked to be assigned to this project so that I could make sure that IBM earned their money. Stupid, stupid, stupid!
D.
I only worked two days this week because of the holidays and the half day off, but I spent part of it getting ready for some server setups IBM is doing next week. I prepared MQ configurations files for two new QA servers. Late this afternoon, I find out that a third configuration file is required. I had assumed that IBM was going to do it, but I was wrong. I had planned to leave early so I did. I'm off next week but promised to come in if necessary. The hardware is already a week late.
I don't know what the IBM project manager is doing to earn his money. These requirements for our resources just fall out of the sky. We're paying them a half million dollars for this behaviour. Meanwhile, our PM is working 12 hr days to keep everything on track. I asked to be assigned to this project so that I could make sure that IBM earned their money. Stupid, stupid, stupid!
D.
Friday, December 23, 2005
Helping a drinking buddy
I went to borrow a cellphone off the Intel server team leader. I wanted to play with its email capabilities for oncall purposes. The phone was turned in by someone who resigned. I found out that he has three openings that he is trying to fill. I told him I knew someone competent, and he was interested. He's looking for a Novell/network administrator right now so I wasn't sure of the fit.
I fired off an email with caveats to a Windows sysadmin who's now looking for an IT job. I need someone to have lunch as noone I know seems to bother anymore. He has been dabbling outside the industry since he left CLA. He responded almost immediately with his resume. Of course, it makes him sound like the greatest thing since sliced bread (I peeked). I guess he didn't use the outplacement service because parts of it are dated.
Anyway, I passed it on and wished him luck. I really don't know where I stand with this team leader and whether he values my judgement. It's only been three months (almost!) since I joined the company.
D.
I fired off an email with caveats to a Windows sysadmin who's now looking for an IT job. I need someone to have lunch as noone I know seems to bother anymore. He has been dabbling outside the industry since he left CLA. He responded almost immediately with his resume. Of course, it makes him sound like the greatest thing since sliced bread (I peeked). I guess he didn't use the outplacement service because parts of it are dated.
Anyway, I passed it on and wished him luck. I really don't know where I stand with this team leader and whether he values my judgement. It's only been three months (almost!) since I joined the company.
D.
Saturday, December 17, 2005
What's wrong with this picture?
Once I got access to a production server (a couple of weeks ago), I started reviewing the MQ error logs. I see a TCP/IP communications error. This is appearing in all four queue managers used by the premier web application here.
I report the problem to the Service Desk. I don't know it at the time but external business partners also report the problem. The trouble ticket get assigned to the sysadmin for this application. He consults the mainframe MQ guy who says that the occasional TCP/IP error is normal and tolerable. In four years of supporting MQ on distributed platforms, I have never seen this error except during scheduled network maintenance or as a result of a real problem.
The network guy says that he needs to be involved at the time of the problem and that he doesn't know how to use the sniffer. The network has just been outsourced to Bell, and the rules of engagement have yet to be defined. In fact, there seems to be a question whether Bell is responsible for the internal switches and routers. If they don't, these devices are without support.
I do some more digging. The problem is occurring at 35 minutes after the hour, only during even numbered hours but not consistently. The problem reoccurs a week to the minute after the previous report. I try to engage someone (anyone!) to take ownership of the problem. The sysadmin is over his head and really shouldn't be the primary. The new IT problem manager has "bigger" fish to fry as does the network manager. These people keep referring me to other people who can't help with the problem.
The sysadmin does tell me that the problem has also manifested itself between the application server and the database server. I do a traceroute and there is only one hop between two. The problem must be on the device that they are both plugged into. I decide to bounce my theory off the security operations manager and my boss. I go to the former because the firewall is the default gateway of the application server where MQ also resides. My boss know a lot about the network. They both reply with the same info.
The next day, the security manager comes back to me with a suspect, but he wants to confirm with his staff. A day or so after that, his firewall guy comes to talk to me, and yes, it is a known problem. There is a multicast (IGMP?) on the core switch that is causing the connectivity break if I understood him correctly.
His manager gives me more background. This problem has been going on for over two years. When the problem was first diagnosed, the resolution was judged too painful to pursue. Actually, the firewall guy did rebuild his cluster as part of this. It was one of the first things he did after joining the company.
This is just unacceptable to me. Everything on the backbone is a victim of this problem. The high profile web application supporting 10,000 brokers and external business partners (who we white label for) is being impacted. I haven't received an answer from the network manager about the devices involved. He appears not to know. I think his VP is going to be in over her head as well with this one (she's more a call center/voice person). I hope I'm pleasantly surprised.
I asked the security manager what the next step was. I told him I was here to fix new problems not find old ones. He is going to escalate it to his manager. I sent an email to my boss asking him to do the same. I expect he'll try to escalate technically rather than politically because that's the way he is. We'll see if us newbies can make anything happen here.
D.
I report the problem to the Service Desk. I don't know it at the time but external business partners also report the problem. The trouble ticket get assigned to the sysadmin for this application. He consults the mainframe MQ guy who says that the occasional TCP/IP error is normal and tolerable. In four years of supporting MQ on distributed platforms, I have never seen this error except during scheduled network maintenance or as a result of a real problem.
The network guy says that he needs to be involved at the time of the problem and that he doesn't know how to use the sniffer. The network has just been outsourced to Bell, and the rules of engagement have yet to be defined. In fact, there seems to be a question whether Bell is responsible for the internal switches and routers. If they don't, these devices are without support.
I do some more digging. The problem is occurring at 35 minutes after the hour, only during even numbered hours but not consistently. The problem reoccurs a week to the minute after the previous report. I try to engage someone (anyone!) to take ownership of the problem. The sysadmin is over his head and really shouldn't be the primary. The new IT problem manager has "bigger" fish to fry as does the network manager. These people keep referring me to other people who can't help with the problem.
The sysadmin does tell me that the problem has also manifested itself between the application server and the database server. I do a traceroute and there is only one hop between two. The problem must be on the device that they are both plugged into. I decide to bounce my theory off the security operations manager and my boss. I go to the former because the firewall is the default gateway of the application server where MQ also resides. My boss know a lot about the network. They both reply with the same info.
The next day, the security manager comes back to me with a suspect, but he wants to confirm with his staff. A day or so after that, his firewall guy comes to talk to me, and yes, it is a known problem. There is a multicast (IGMP?) on the core switch that is causing the connectivity break if I understood him correctly.
His manager gives me more background. This problem has been going on for over two years. When the problem was first diagnosed, the resolution was judged too painful to pursue. Actually, the firewall guy did rebuild his cluster as part of this. It was one of the first things he did after joining the company.
This is just unacceptable to me. Everything on the backbone is a victim of this problem. The high profile web application supporting 10,000 brokers and external business partners (who we white label for) is being impacted. I haven't received an answer from the network manager about the devices involved. He appears not to know. I think his VP is going to be in over her head as well with this one (she's more a call center/voice person). I hope I'm pleasantly surprised.
I asked the security manager what the next step was. I told him I was here to fix new problems not find old ones. He is going to escalate it to his manager. I sent an email to my boss asking him to do the same. I expect he'll try to escalate technically rather than politically because that's the way he is. We'll see if us newbies can make anything happen here.
D.
Thursday, December 15, 2005
You can't get there from here
Today, I am officially oncall for distributed MQ.
Tuesday, I brought my laptop home to check my ssh and remote control access. It doesn't work. This is after lugging it around Yorkdale doing Xmas shopping. They don't have any lockers at the bus terminal anymore. I had assumed that because my Notes was working that the other stuff would be OK.
Wednesday, I schlep the laptop back to the office. I escalate the problem because I expect to be stuck at home the next day because of snow. The floor desktop technician is off, and his backup is busy. Another guy shows up at my desk in the afternoon, and installs some stuff that was originally requested mid October. We go to the lab to test. It doesn't work. We keep trying . Finally, the VPN expert shows up, and declares that it won't work with AGN dialer because of conflicts. I take the laptop home to give another try anyway. No joy.
No snow yet this morning so I went into the office. I let the tech know what was happening. I suggested that the Sympatico dialer be installed on my laptop. Mirabile dictu, this is an option. Unfortunately, all the software he has is for high speed access. He downloads a dialer from Bell. It's not supported but it works, at least in the lab. I still have to try it from home. We finally got the storm they were calling for so I may have to work from home tomorrow. He renamed the icon so it looks like the supported dialer. It was so nice to work with someone who knows what they're doing. I think he's normally a third level guy.
I also found out today that my Regina teammate got a job. She was the last one left from the old team without one. We said we would keep buying Super 7 tickets until we won or we both had jobs. She doesn't start until the new year.
D.
Tuesday, I brought my laptop home to check my ssh and remote control access. It doesn't work. This is after lugging it around Yorkdale doing Xmas shopping. They don't have any lockers at the bus terminal anymore. I had assumed that because my Notes was working that the other stuff would be OK.
Wednesday, I schlep the laptop back to the office. I escalate the problem because I expect to be stuck at home the next day because of snow. The floor desktop technician is off, and his backup is busy. Another guy shows up at my desk in the afternoon, and installs some stuff that was originally requested mid October. We go to the lab to test. It doesn't work. We keep trying . Finally, the VPN expert shows up, and declares that it won't work with AGN dialer because of conflicts. I take the laptop home to give another try anyway. No joy.
No snow yet this morning so I went into the office. I let the tech know what was happening. I suggested that the Sympatico dialer be installed on my laptop. Mirabile dictu, this is an option. Unfortunately, all the software he has is for high speed access. He downloads a dialer from Bell. It's not supported but it works, at least in the lab. I still have to try it from home. We finally got the storm they were calling for so I may have to work from home tomorrow. He renamed the icon so it looks like the supported dialer. It was so nice to work with someone who knows what they're doing. I think he's normally a third level guy.
I also found out today that my Regina teammate got a job. She was the last one left from the old team without one. We said we would keep buying Super 7 tickets until we won or we both had jobs. She doesn't start until the new year.
D.
Saturday, December 10, 2005
The last couple of weeks
The department Xmas party was held offsite after work last Friday. I couldn't find the place right away but I still got there early. I intro'd myself to one woman and talked to her about an application I expect to support. I hung out with the voice people (they've been friendly) until one of my teammates showed. I had a couple of beers. Finally, my boss arrived. I chatted with the Change Manager at the upper management table for a while.
I sat with my boss at dinner (food was good), along with an account manager and an ex-CLAer. We played hot potato for the centrepiece (a bit juvenile). There were a couple of karaoke performances and door prizes. My boss won the centrepiece and a Roots gym bag. The music started in earnest. My boss and I met with the server team leader to talk about my position, and how our two teams would work together. He said something about "why bother" and how his guys were certified. This is out of context so it sounds worse than he intended, but I think there is a kernel of truth here, and he is getting some pushback from his team. By the time we finished, I had had enough of the company for one day (I'd gotten up early to drive in) so I left. My boss thought that I went to the washroom. The whole party was like a throwback to a different era.
I took Tuesday off to do Xmas shopping, and all hell broke loose. My boss paged me even though I'm not oncall yet. A queue had filled up and was writing messages to the MQ dead letter queue. The main MQ guy was also off. I wasn't much help over the phone. It turned out that some syslog server had stopped functioning causing a filesystem (probably root) to fill on the MQ server. The server team didn't know the root password for this syslog server at the time.
This past Friday, we got another whack of snow. I had an all-day session with IBM so I had to go in; otherwise, I wouldn't have bothered. They are setting up a performance and capacity testbed for the company's premier web application. They are also making recommendations for changes to the app itself. Apparently, this is costing the company $500,000. My secret agenda was to ensure that we get our money's worth, but my actual piece of the action is very small. Aside from the odd outburst, the IBM people have been pretty good which has been a pleasant surprise. The same cannot be said for my company's people, some of whom seem to running their own agenda. At least two of them were obstructionists or not listening prolonging the session.
D.
I sat with my boss at dinner (food was good), along with an account manager and an ex-CLAer. We played hot potato for the centrepiece (a bit juvenile). There were a couple of karaoke performances and door prizes. My boss won the centrepiece and a Roots gym bag. The music started in earnest. My boss and I met with the server team leader to talk about my position, and how our two teams would work together. He said something about "why bother" and how his guys were certified. This is out of context so it sounds worse than he intended, but I think there is a kernel of truth here, and he is getting some pushback from his team. By the time we finished, I had had enough of the company for one day (I'd gotten up early to drive in) so I left. My boss thought that I went to the washroom. The whole party was like a throwback to a different era.
I took Tuesday off to do Xmas shopping, and all hell broke loose. My boss paged me even though I'm not oncall yet. A queue had filled up and was writing messages to the MQ dead letter queue. The main MQ guy was also off. I wasn't much help over the phone. It turned out that some syslog server had stopped functioning causing a filesystem (probably root) to fill on the MQ server. The server team didn't know the root password for this syslog server at the time.
This past Friday, we got another whack of snow. I had an all-day session with IBM so I had to go in; otherwise, I wouldn't have bothered. They are setting up a performance and capacity testbed for the company's premier web application. They are also making recommendations for changes to the app itself. Apparently, this is costing the company $500,000. My secret agenda was to ensure that we get our money's worth, but my actual piece of the action is very small. Aside from the odd outburst, the IBM people have been pretty good which has been a pleasant surprise. The same cannot be said for my company's people, some of whom seem to running their own agenda. At least two of them were obstructionists or not listening prolonging the session.
D.
