While recently troubleshooting a T1 circuit issue for a MPLS connection I was getting the run around from my service provider. The classic phrase we have all heard. “Cleared on Test”, “Clear to the smart jack”, and “Testing clean on loop” as well as other variations are common phrases I hear. Telco is all to quick to dismiss my complaints and they always think they have no issues.
What if I am testing clean on loop? What if I am clearing on test? Who do we point the finger to then?
Well that is exactly what I had happen to me. Telco never saw the input and crc errors that I was getting on my serial interface. They could loop smart jack telco facing and test clean for all patterns. 0x1111 was giving me the most trouble. I could put my gear in hardware loop and send 0x1111 all day long just fine. But when I was connected to telco I would get about a 28% success rate. I knew it would be a cabling or telco issue. So I changed the patch cable between smart jack and CSU. After the 3rd patch cable the issue remained.
Telco stuck to their guns telling me to change the cable. Finnaly after getting telco and the various local exchange carriers on the line at the same time, with the backing of my hardware vendor I was able to find the issue. I had them loop customer facing from “test point” which happened to be on the telco side of the smart jack, about a 60 miles from my gear. Somewhere in this 60 miles was an issue. I was determined to find it.
I asked telco to loop the smart jack customer facing. They couldn’t do this. How convenient that they can loop telco facing to prove it isn’t their issue but cannot loop customer facing to prove the customer doesn’t have a problem. So I did the next best thing. I had them but a coupler on the end of my patch cable before it plugs into the smart jack. I then was able to test clean on all test patterns, 0x1111 included. This ruled out my patch cable and my gear.
The LEC who was onsite assisting this troubleshooting happened to have a smart jack. I asked that he install it so we could rule out the smart jack. I had the telco in the major city close to this office give me a loop customer facing from the test point. Sent my test patterns and tested clean!!! It was a telco issue, Imagine that. A bad smart jack. The kicker was the smart jack had been changed 4 days before hand in attempt to resolve the errors. Bad batch of smart jacks possibly.
So the next time the telco has you running around to prove it isn’t you. Stick to YOUR guns. Clear to smart jack doesn’t always rule them out!