Saturday, November 08, 2008
Murphy's law, big time
The eComm project wanted the highest availability so I had to move the connectivity to CGI to a different queue manager that recycled for backup in the early hours of the morning. It took a half dozen tries to schedule the outage because of various other conflicting changes. November 2nd was finally chosen, but I forgot about the time change. The change had to be made between 2 and 4 a.m. due to contractual obligations with a business partner. The MQ admin at CGI had to validate the time change before making the eComm change. I laid out a schedule of events listing both timezones, and the CGI guy confirmed.
He said that their mainframe would be coming down at 1 a.m. DST. I made my changes from home at that time figuring that the connection was down already. I should have known something was up when my test messages transmitted OK instead of staying on the transmit queue.
I then drove up to my church to use the office phone line because my cell phone has a very poor signal at home, and I needed to type and talk at the same time. It took me a couple of tries to get my computer to dial out. It said there was no dial tone, but I knew they were connecting to the Internet from the church. I finally figured out that it was just the pulse/tone setting. This was about 2 a.m. EDT now 1 a.m. EST. The CGI mainframe was supposed to be available in another half hour so I planned to catch up on my email. For some reason, more than 25 messages had arrived in my inbox since I left the office Friday afternoon.
After not hearing from CGI by 1:40 a.m., I gave their MQ admin a call. He informed me that the mainframe was still down and would be for another half hour. I couldn't believe it. I had to call my tester and tell him that CGI was about an hour behind schedule. I went back to my email and got caught up on some reading.
The CGI MQ admin called me back earlier than I expected at 2:30 a.m. Our MQ testing went off without a hitch. I let him go after getting the CGI off-hours support number. I called my tester to let him know he could start. Then he couldn't get the app to work properly. A dropdown wasn't, but I could see that messages were flowing to and from CGI OK. The tester said he would send me the screenshots of the problem. His mailbox was full so he used his private mailbox and asked me to forward it to the Service Desk.
I decided to call the Service Desk at 3 a.m. to report the problem. I woke the technician, and we were both pretty punchy. I tried to describe the problem without actually having seen it. The SD technician proceeded to tell me that it sounded like an application problem and that he didn't have an off-hours support number for eComm. I felt really guilty about calling him in the middle of the night. I had almost completely forgotten about the document. I told him I would forward it. He said he would create a ticket in the morning so I let him go back to bed. The tester called me back, and I told him that we wouldn't get any support that night. He said it wasn't like that on the project. I let him go to bed.
By this time, it's about 3:30 a.m., and I'm really punchy. I try to forward the document but my mailbox had reached its quota. I delete some larger documents and compact my mail database. No joy. I delete and compact again. Still no joy. For some reason, I decided to close my email and re-open it. It works! I can open the document and forward it now. I pack up my stuff and get home at 4:30 a.m.
When I get to the office on Monday, I learn that the problem was escalated to a high severity on Sunday. A code fix was developed and applied that evening. We had found an application defect. So much for not having off-hours support.
D.
He said that their mainframe would be coming down at 1 a.m. DST. I made my changes from home at that time figuring that the connection was down already. I should have known something was up when my test messages transmitted OK instead of staying on the transmit queue.
I then drove up to my church to use the office phone line because my cell phone has a very poor signal at home, and I needed to type and talk at the same time. It took me a couple of tries to get my computer to dial out. It said there was no dial tone, but I knew they were connecting to the Internet from the church. I finally figured out that it was just the pulse/tone setting. This was about 2 a.m. EDT now 1 a.m. EST. The CGI mainframe was supposed to be available in another half hour so I planned to catch up on my email. For some reason, more than 25 messages had arrived in my inbox since I left the office Friday afternoon.
After not hearing from CGI by 1:40 a.m., I gave their MQ admin a call. He informed me that the mainframe was still down and would be for another half hour. I couldn't believe it. I had to call my tester and tell him that CGI was about an hour behind schedule. I went back to my email and got caught up on some reading.
The CGI MQ admin called me back earlier than I expected at 2:30 a.m. Our MQ testing went off without a hitch. I let him go after getting the CGI off-hours support number. I called my tester to let him know he could start. Then he couldn't get the app to work properly. A dropdown wasn't, but I could see that messages were flowing to and from CGI OK. The tester said he would send me the screenshots of the problem. His mailbox was full so he used his private mailbox and asked me to forward it to the Service Desk.
I decided to call the Service Desk at 3 a.m. to report the problem. I woke the technician, and we were both pretty punchy. I tried to describe the problem without actually having seen it. The SD technician proceeded to tell me that it sounded like an application problem and that he didn't have an off-hours support number for eComm. I felt really guilty about calling him in the middle of the night. I had almost completely forgotten about the document. I told him I would forward it. He said he would create a ticket in the morning so I let him go back to bed. The tester called me back, and I told him that we wouldn't get any support that night. He said it wasn't like that on the project. I let him go to bed.
By this time, it's about 3:30 a.m., and I'm really punchy. I try to forward the document but my mailbox had reached its quota. I delete some larger documents and compact my mail database. No joy. I delete and compact again. Still no joy. For some reason, I decided to close my email and re-open it. It works! I can open the document and forward it now. I pack up my stuff and get home at 4:30 a.m.
When I get to the office on Monday, I learn that the problem was escalated to a high severity on Sunday. A code fix was developed and applied that evening. We had found an application defect. So much for not having off-hours support.
D.
Labels: Work
