Saturday, November 18, 2006
Best practice bites
An MQ administrator should monitor the dead letter queue for undelivered messages. This is a sign of major system problems.
My pager started going off at the beginning of October. I tried to see what was on the dead letter queue, but the message was too big for the sample programs I normally use. Then the message expired. I let the usual suspects know there was a problem.
While I was on WMB training, I got exposure to a utility, rfhutil(c), that allows access to queues. This problem gave me the impetus to set it up on my laptop. Late one afternoon, I was able to capture one of the DLQ messages and the application developer. Turns out the data is too long to get to the backend and it's an application problem. This should be impossible for a well designed application. Someone hasn't done their testing properly. We only have a SIT team, a UAT team and a QA team, but this got by all of them. I ask for program correction, but it's a piece of the application that's been outsourced so there's no telling when this problem will be fixed.
While I'm investigating this problem, I see that messages are also hitting the mainframe DLQ. I know that my mainframe counterpart ignores this because of another design issue-messages are arriving at the backend without a valid userid (TAM and RACF aren't synchronized). I start sending out alerts when these problems occur, but no one seems to know what they're about or what to do about them, and they keep trying to involve me in the problem resolution.
Maybe I should increase the maximum message length and just let them flow through to the mainframe DLQ and not have to worry about it. One of my teammates is setting up a Tivoli alert for the support areas, but it's taken forever to get through the change management process and I'm not even sure it is working yet.
D.
My pager started going off at the beginning of October. I tried to see what was on the dead letter queue, but the message was too big for the sample programs I normally use. Then the message expired. I let the usual suspects know there was a problem.
While I was on WMB training, I got exposure to a utility, rfhutil(c), that allows access to queues. This problem gave me the impetus to set it up on my laptop. Late one afternoon, I was able to capture one of the DLQ messages and the application developer. Turns out the data is too long to get to the backend and it's an application problem. This should be impossible for a well designed application. Someone hasn't done their testing properly. We only have a SIT team, a UAT team and a QA team, but this got by all of them. I ask for program correction, but it's a piece of the application that's been outsourced so there's no telling when this problem will be fixed.
While I'm investigating this problem, I see that messages are also hitting the mainframe DLQ. I know that my mainframe counterpart ignores this because of another design issue-messages are arriving at the backend without a valid userid (TAM and RACF aren't synchronized). I start sending out alerts when these problems occur, but no one seems to know what they're about or what to do about them, and they keep trying to involve me in the problem resolution.
Maybe I should increase the maximum message length and just let them flow through to the mainframe DLQ and not have to worry about it. One of my teammates is setting up a Tivoli alert for the support areas, but it's taken forever to get through the change management process and I'm not even sure it is working yet.
D.
Saturday, November 11, 2006
All atwitter
My boss called me Monday morning on his way into the office because he'd an email titled Announcement on his cellphone and he wanted to know the content (just a new security operations manager replacing the ex-CLAer).
Two new AVPs in application development were announced in the week so the rumour mill was right. I met the one when I kicked him and the contractor next door out of a room I had reserved for a meeting. Getting possession of a room you've booked is a crap shoot thanks to the state of the Domino environment (severely backlevelled).
The current AVP in charge of application development has been shuffled aside. I guess she won't be bringing in samosas for the whole department any more. Because they initially only announced the reporting structure for two of her five managers, a lot of the developers were confused and concerned about being outsourced. I made a joke of it with one of them, but I don't think the other person found it very funny. It was all straightened out by the end of the week.
No changes have been announced for my unit yet.
D.
Two new AVPs in application development were announced in the week so the rumour mill was right. I met the one when I kicked him and the contractor next door out of a room I had reserved for a meeting. Getting possession of a room you've booked is a crap shoot thanks to the state of the Domino environment (severely backlevelled).
The current AVP in charge of application development has been shuffled aside. I guess she won't be bringing in samosas for the whole department any more. Because they initially only announced the reporting structure for two of her five managers, a lot of the developers were confused and concerned about being outsourced. I made a joke of it with one of them, but I don't think the other person found it very funny. It was all straightened out by the end of the week.
No changes have been announced for my unit yet.
D.
Saturday, November 04, 2006
The "calm" before the storm
Four people were let go earlier this week, all from the Service Delivery area which is headed by a former CLA person. My boss had just consulted the one on a process issue. He wanted me to close a ticket that had been opened by the Service Desk, and she supported my position that it was the responsibility of the Service Desk to close their own tickets. I've opened dozens of tickets with the Service Desk and never had to close any of them.
The final Gartner report on outsourcing has been received. The contractor next to me says that it will take a couple of weeks to digest it. My senior VP owns this task. Meanwhile, the grapevine says a reorg will be announced next week. When I mentioned this to the CIO's admin assistant, she made a quip but had a funny look on her face. That was all the confirmation I needed. According to one of my older teammates, people have put their lives on hold because of the uncertainty. It's like they're expecting everyone to go. As the contractor next door reminded me a while age, it's not an acquistion. Actually, she confirmed a rumour that my big boss's area was at the bottom of the outsourcing list.
I had breakfast with the CIO last week, me and about a dozen others. I was disappointed that it wasn't more interactive. She repeated her introduction from the Town Hall. Then she made a comment on the schism between new employees and old ones which seemed to squelch further discussion in that area. After that, one of the long-term employees dominated the conversation. It was funny, but the comments that he and a few others made were the same ones that I had heard at my old shop, e.g. upper management interference and lack of direction, new technology implementations dying on the vine (Active Directory), the large investment in the old infrastructure. Someone hasn't done their due diligence/risk mitigation though, and the company is running major systems on unsupported software. It's like driving a 10 year old car. Sooner or later, you going to have a major breakdown that you won't want to bother fixing/paying for. Towards the end, the CIO mentioned that she had never done an outsourcing where people lost their jobs. She also mentioned that poor performers would not be outsourced. This week, the fellow who brought up Active Directory submitted his resignation.
D.
The final Gartner report on outsourcing has been received. The contractor next to me says that it will take a couple of weeks to digest it. My senior VP owns this task. Meanwhile, the grapevine says a reorg will be announced next week. When I mentioned this to the CIO's admin assistant, she made a quip but had a funny look on her face. That was all the confirmation I needed. According to one of my older teammates, people have put their lives on hold because of the uncertainty. It's like they're expecting everyone to go. As the contractor next door reminded me a while age, it's not an acquistion. Actually, she confirmed a rumour that my big boss's area was at the bottom of the outsourcing list.
I had breakfast with the CIO last week, me and about a dozen others. I was disappointed that it wasn't more interactive. She repeated her introduction from the Town Hall. Then she made a comment on the schism between new employees and old ones which seemed to squelch further discussion in that area. After that, one of the long-term employees dominated the conversation. It was funny, but the comments that he and a few others made were the same ones that I had heard at my old shop, e.g. upper management interference and lack of direction, new technology implementations dying on the vine (Active Directory), the large investment in the old infrastructure. Someone hasn't done their due diligence/risk mitigation though, and the company is running major systems on unsupported software. It's like driving a 10 year old car. Sooner or later, you going to have a major breakdown that you won't want to bother fixing/paying for. Towards the end, the CIO mentioned that she had never done an outsourcing where people lost their jobs. She also mentioned that poor performers would not be outsourced. This week, the fellow who brought up Active Directory submitted his resignation.
D.
