I have actually done a detailed evaluation on this and technically it is a no brainier. No discernible advantage in using Ethernet other than lack of availability of pcie switches. Latency is dramatically lower and SW stack is grossly simplified. Having said that for my upcoming storage proposal, I felt RapidIO was even a better choice since it scored over PCIE in every respect technically. Even lower latency, peer to peer support, a proper routable fabric, ability to use 100g silicon photonic of the 802.3bm ethernet standard. Currently I am working with 1 or 2 standards bodies to move this forward. If anyone wants a intro ppt , let me know and I can send you one.
PCI Express is getting quite a lot of traction, so it'll be interesting to see how it does with PLX's adapter-card approach. It's got the speed and low power. Infiniband may feel the squeeze. Interesting progress by PLX: Thanks for the update Rick.