Before discussing the results, it should be mentioned that the excellent performance of Linux Ethernet is due to the drivers written by Don Becker of Scyld Computing. The Netperf and Netpipe results are what one would expect. Sharing the interface reduces the bandwidth available to an application. Perhaps the most interesting result is the Tyan 2762 does seem to keep pace with single interface performance until it reaches larger bloc sizes. More interestingly, the Tyan 2762 shows curious behavior when the two interfaces are communicating separately. In the two other cases, the two Netpipe jobs showed virtually identical performance to the single job/single interface test.
The MPICH test is a bit more telling. The tests were run for 1, 4 and 8 CPU cases. In the the 8 CPU case, an extra interface was employed to see if contention on one interface had any effect on the tests. In general the answer seem to be that two interfaces on a dual node are better than one. With exception of the EP (embarrassing parallel) test, all the other tests, have a high rate of communication and seem to benefit from the extra interface. Of note is the poor performance of IS (integer sort) in all cases. This test is a highly latency sensitive test and even on 8 CPUs with two interfaces can not reach the single CPU performance.
The tests indicate the dual SMP nodes with a single network connection generate contention on the interface and therefore lower performance. Using two separate interfaces seems to help the performance for some of the NAS suite tests reinforcing the claim that sharing a single interface can reduce performance for certain applications. There does not appear to be many clusters that have utilized two interfaces as separate communication channels as a mean to increase performance. Of course, the addition of an extra interface is also highly dependent on the applications communication behavior.