While recently troubleshooting a T1 circuit issue for a MPLS connection I was getting the run around from my service provider. The classic phrase we have all heard. “Cleared on Test”, “Clear to the smart jack”, and “Testing clean on loop” as well as other variations are common phrases I hear. Telco is all to quick to dismiss my complaints and they always think they have no issues.

What if I am testing clean on loop? What if I am clearing on test? Who do we point the finger to then?

Well that is exactly what I had happen to me. Telco never saw the input and crc errors that I was getting on my serial interface. They could loop smart jack telco facing and test clean for all patterns. 0×1111 was giving me the most trouble. I could put my gear in hardware loop and send 0×1111 all day long just fine. But when I was connected to telco I would get about a 28% success rate. I knew it would be a cabling or telco issue. So I changed the patch cable between smart jack and CSU. After the 3rd patch cable the issue remained.

Telco stuck to their guns telling me to change the cable. Finnaly after getting telco and the various local exchange carriers on the line at the same time, with the backing of my hardware vendor I was able to find the issue. I had them loop customer facing from “test point” which happened to be on the telco side of the smart jack, about a 60 miles from my gear. Somewhere in this 60 miles was an issue. I was determined to find it.

I asked telco to loop the smart jack customer facing. They couldn’t do this. How convenient that they can loop telco facing to prove it isn’t their issue but cannot loop customer facing to prove the customer doesn’t have a problem. So I did the next best thing. I had them but a coupler on the end of my patch cable before it plugs into the smart jack. I then was able to test clean on all test patterns, 0×1111 included. This ruled out my patch cable and my gear.

The LEC who was onsite assisting this troubleshooting happened to have a smart jack. I asked that he install it so we could rule out the smart jack. I had the telco in the major city close to this office give me a loop customer facing from the test point. Sent my test patterns and tested clean!!! It was a telco issue, Imagine that. A bad smart jack. The kicker was the smart jack had been changed 4 days before hand in attempt to resolve the errors. Bad batch of smart jacks possibly.

So the next time the telco has you running around to prove it isn’t you. Stick to YOUR guns. Clear to smart jack doesn’t always rule them out!

Bookmark to

While troubleshooting a problems with circuits provided by AT&T I’ve had to deal with a number of people. If I end up building a simple tech support relationship after a few hours I sometimes ask where in the world the tech is. I also sometimes will ask if I don’t recognize as accent or if I think I do know the accent and want to verify it.

In some cases though, I ask out of frustration. Frustration caused by not being able to understand the person because English is not their first language. Today was one of those days. The support person was from Singapore. I was troubleshooting errors on a T1 circuit that was provided to a city in Arkansas. This troubleshooting was being performed in the middle of the day. The office had been running on the backup circuit so intrusively testing the main circuit was ok.

Now I work with vendors a lot. I realize that it is cheaper to provide around the clock support by using call centers in other parts of the world during prime times. By doing this you can have employees working ~8am to ~5pm local times. I am ok with this. However if it is 1pm Central Standard Time. Why am I speaking to singapore? Is there no one in the United States of America who can troubleshoot a T1 circuit?

ATT, why am I speaking to Mr. Gonzales in singapore? To add to the situation, we are talking over each other because of the lagged out voice call. It feels like we need to say “over” each time we complete a statement.

From what I can gather Tier1 is in India or Pakistan. and Tier2 is in Singapore. There is a tier3 but you can only talk to them if ATT manages the router. I know the implementation and designer guys are in the USA. New York actually I believe. So setting up the circuits I get to speak to people in the USA. For everything else I am having to deal with Singapore and India. For what I pay, I want someone in the USA during business hours of 8am-5pm Central.

Bookmark to