To take routing out of the picture, try changing a machine in Site A to a Class B subnet mask and a machine in Site B to a Class B subnet mask. This will put them both on the same Network ID. Then ping the machine in Site A from the machine in Site B and vice-versa. Change your Ping to use the /L switch as follows:

Note: Before changing the subnet masks, run the following pings to get a baseline performance indicator. Then make the changes.

Ping /l 64 PC_IN_SITE_A
Ping /l 640 PC_IN_SITE_A
Ping /l 6400 PC_IN_SITE_A

Repeat in the opposite direction. You should see about a 10% increase in time from 64 to 640. Then about double the time from 640 to 6400.

Note your findings. Did you see any changes in speed with these two machines. If so, there is a problem with your route tables or router configuration.

If it is not a route issue, then my next question is when the fibers were laid, were they certified by the installer for transmission quality? Do you have the correct modules in the switches for the fiber types? The modules for Multi-mode fiber are not strong enough to push light through the much smaller diameter Single-mode fiber.

No change in speed. Site A to B replys were <1ms. Site B to A <10ms. No changes when Subnet mask was changed or packet size was increased. Neither way would reply to 6400 size. Fiber test good per vendor but like I stated previously even when we bypass the fiber and go with our wireless backup link the slowness still exists.

When you replaced the fiber with wireless, this was on the short link. There would be no way for you to replace the fiber on the long 2 mile link unless you have a microwave system. This has to be where the problem lies. There are two fibers that handle a single data connection. One is for transmit and one is for receive.

When you ping from A to B, your time is <1ms. When you ping from B to A, your time is <10ms. This means that the Transmit fiber from B to A is your likely culprit.

A way to verify: Have someone stationed at both ends. Remove your Fibers from the Transceiver and swap the connections. Then rerun your Ping tests. The results should be exactly opposite.

This would verify that one of your fibers is degraded in quality and there is significant light loss. The ends are almost always the culprit. Replace the connectors.

As far as the 6400 size goes. This is approaching the size of a Jumbo Frame at 9000 bytes. Your network gear should be able to handle Jumbo Frames but Flow Control must be turned OFF on most switches. There are only a few Cisco switches that can handle Jumbo Frames AND Flow Control. Your HP switches can not even though their IOS allows it to be turned on when Jumbos are enabled.

Bump.

I will try switching the fiber pairs but I do not think that is the problem because of the same issue when we switch to the wireless link instead of using the fiber.

You diagram shows that you are running MM fiber from A to your switch/router and SM fiber from B to your switch/router. And since The sites are interconnected and routed via two fiber connections, it does not matter if you replace the MM with a wireless connection, you still have the other 2 mile SM connection to contend with. This is where I believe the problem lies. Two miles is ceartainly within the limits of the fiber capabilities but at that distance, a suspect quality fiber will degrade your performance substantially. I have had it happen to me at only 2,000 feet.

The fiber connections run in pairs at each end of the connection. One end transmits and the other receives. The other fiber in the pair runs in the other direction. By switching the pairs at both ends, you should see the ping performance also change directions if one of the two fibers is degraded in quality. Often, the fibers are installed in multipair trunks. If you have additional fibers run to your facillity, you can swap the line with another fiber to see if performance changes. If not, you will need to have an end to end tester check the line quality.

To Clarify: Site A -> Core 2 miles SM Fiber OR Wireless. Site B-> Core MM Fiber about 10 Feet ( same Room).

OK. This going to be a real needle in the haystack for you but here is a great utility.

Wireshark is a protocol analyzer that you can install onto a laptop and plug straignt into your switch. In the switch configuration, setup the port that you plug it into as a Monitoring Port.

Wireshark: Go deep.

It is a free utility with excellent support on the web site. Look for patterns. See if a device is storming the network with broadcasts. You can select which network adapter to listen on if you want to use it to troubleshoot wired or wireless networks.

Try to chenge the split speed... try to dive in 6:4... then check what happen`s