First, it is important to remember that buffered IO are irrelevant when testing a client to server data transfer. The only thing we measure is a combination of the NFS buffering, server buffering, network layout and operating system asynchronous IO handling. So, the scaled CPU rates are here useless.

Second, IOzone will not be able to perform large buffer IO tests because of the simple fact that NFS will break IO into it's own block size. Pics should be seen whenever IOzone matches the NFS settings. The only test which may show a server limitation is (once again) to perform local IO tests.

IOPerf results

Local IO on server

Nothing much to say

fwrite ~ 30 MB/sec, fread ~ 36 MB/sec, this is a surprinsing factor of 2.5 better than the preceedingly tested MTI IO rates in fwrite mode. And only 0.7 times worst in fread.
However, Multi-thread Performance scales exactly with number of threads. Was not true with the MTI (10 threads got the IO rate down by a factor of ~ 2.5 only). This may be due to the RAID configuration itself (how many drives are usable).
If 10 threads are opened at a time, we now have the MTI showing a factor of 2 better performance than what we get here.

		Block IO						Character						Random
		fwrite			fread			putc			getc			seek
Machine/FS	KB	KB/sec Real	KB/sec CPU	%CPU	KB/sec Real	KB/sec CPU	%CPU	KB/sec Real	KB/sec CPU	%CPU	KB/sec Real	KB/sec CPU	%CPU	KB/sec Real	KB/sec CPU	%CPU
_ibmtest_	400000	27797.08	93896.71	29.60	35874.44	224719.10	15.96	9182.74	9765.62	94.03	8266.17	9205.98	89.79	1457.56	1457.56	100.00
1PV_ibmtest_test_	400000	33030.55	102040.82	32.37	36968.58	238095.24	15.53	9142.86	9768.01	93.60	8271.30	9214.47	89.76	1415.31	1415.31	100.00
NPV_ibmtest_test_	400000	2991.51	38876.38	7.69	3007.73	214530.06	1.41	961.54	9747.68	9.86	908.15	9137.90	9.94	238.41	1420.01	16.83

Client to Server

Results were obtained using a 8 kByte buffer, 5000 IO 1 pass. The test order in that table is :

IO test on individual partition (one at a time, no prefix)
One IO thread per volume, N volumes (1PV_)
Parralele IO, 10 threads per volume. One result-average shown. (NPV_)

For those tests, 8 kByte buffer, 50000 IO 1 pass. All nodes invintoried here started their process at the same time. The first line shows a gross-average over all available nodes. We performed 47 simultaneous IO tests.
Comparing to the base-line ( ~ 9 M/sec), the expected fwrite rate in a straight scaling scenario would be 9/47 i.e 215 kB/sec . The number we woud get in a network-saturated scenario is (100 BT connection) 12 MB/sec and 47 nodes i.e. 261 kB/sec.
Looking at the gross-average of 9 M/sec wth a single thread and 188 kB/sec (47 thread), it appears that the numbers scales well with the client-to-server base line number. The only conclusion to this would be that either the server is IO limited or that the NFS-daemon is the limitation.

Block IO Character Random

fwrite fread putc getc seek

Machine/FS KB KB/sec Real KB/sec CPU %CPU KB/sec Real KB/sec CPU %CPU KB/sec Real KB/sec CPU %CPU KB/sec Real KB/sec CPU %CPU KB/sec Real KB/sec CPU %CPU

<rcas6*> 400000 187.75 43215.41 0.43 166280.69 208008.25 66.07 115.65 6411.00 1.81 6752.18 7349.41 91.07 956.65 957.66 99.89

rcas6001 400000 173.47 43478.26 0.40 254777.07 253164.56 100.64 99.08 6443.30 1.54 6132.15 7432.18 82.51 966.89 976.56 99.01

rcas6002 400000 144.89 43196.54 0.34 367.70 88495.58 0.42 139.08 6360.31 2.19 1069.58 6850.49 15.61 986.43 986.43 100.00

rcas6005 400000 232.05 43196.54 0.54 542.42 120481.93 0.45 125.60 6433.97 1.95 7425.28 7430.80 99.93 986.43 986.43 100.00

rcas6006 400000 165.05 43907.79 0.38 253164.56 254777.07 99.37 98.59 6420.55 1.54 7426.66 7426.66 100.00 986.43 986.43 100.00

rcas6007 400000 155.92 42060.99 0.37 464.71 120120.12 0.39 152.75 6426.74 2.38 7417.02 7444.63 99.63 986.43 986.43 100.00

rcas6008 400000 207.08 42553.19 0.49 270.33 45454.55 0.59 121.95 6374.50 1.91 7441.86 7447.40 99.93 986.43 986.43 100.00

rcas6009 400000 240.43 43956.04 0.55 253164.56 253164.56 100.00 103.98 6419.52 1.62 7426.66 7426.66 100.00 986.43 986.43 100.00

rcas6010 400000 205.03 43383.95 0.47 359.80 148148.15 0.24 121.42 6413.34 1.89 7417.02 7419.77 99.96 976.56 976.56 100.00

rcas6011 400000 105.03 43010.75 0.24 253164.56 254777.07 99.37 140.94 6409.23 2.20 7419.77 7419.77 100.00 986.43 986.43 100.00

rcas6012 400000 183.43 42780.75 0.43 250000.00 250000.00 100.00 91.39 6398.98 1.43 7397.82 7422.53 99.67 976.56 966.89 101.00

rcas6013 400000 197.50 42643.92 0.46 251572.33 250000.00 100.63 102.25 6431.90 1.59 7425.28 7426.66 99.98 986.43 986.43 100.00

rcas6014 400000 142.20 42507.97 0.33 393.65 99502.49 0.40 141.92 6386.72 2.22 1012.63 7031.11 14.40 986.43 996.49 98.99

rcas6015 400000 251.96 43668.12 0.58 253164.56 254777.07 99.37 90.50 6436.04 1.41 7403.29 7415.65 99.83 996.49 986.43 101.02

rcas6016 400000 169.90 43431.05 0.39 245398.77 253164.56 96.93 132.40 6440.19 2.06 7425.28 7426.66 99.98 986.43 986.43 100.00

rcas6017 400000 138.88 42060.99 0.33 317.45 61443.93 0.52 147.73 6333.12 2.33 5056.25 7441.86 67.94 976.56 996.49 98.00

rcas6018 400000 149.53 44345.90 0.34 251572.33 251572.33 100.00 118.42 6426.74 1.84 7425.28 7428.04 99.96 966.89 976.56 99.01

rcas6019 400000 156.86 43956.04 0.36 253164.56 254777.07 99.37 126.79 6466.21 1.96 7425.28 7428.04 99.96 986.43 986.43 100.00

rcas6020 400000 123.05 43336.94 0.28 380.15 133333.33 0.29 160.39 6333.12 2.53 7423.90 7429.42 99.93 986.43 986.43 100.00

rcas6021 400000 248.81 43668.12 0.57 248447.20 251572.33 98.76 118.43 6407.18 1.85 7423.90 7426.66 99.96 986.43 986.43 100.00

rcas6022 400000 103.75 43383.95 0.24 239520.96 242424.24 98.80 140.82 6382.64 2.21 7408.78 7412.90 99.94 986.43 986.43 100.00

rcas6023 400000 212.75 43668.12 0.49 245398.77 250000.00 98.16 98.13 6443.30 1.52 7403.29 7437.71 99.54 626.00 626.00 100.00

rcas6024 400000 235.85 43811.61 0.54 251572.33 251572.33 100.00 109.13 6443.30 1.69 7443.25 7444.63 99.98 966.89 976.56 99.01

rcas6025 400000 176.30 43668.12 0.40 253164.56 251572.33 100.63 119.94 6379.59 1.88 7417.02 7415.65 100.02 976.56 966.89 101.00

rcas6026 400000 220.88 43010.75 0.51 254777.07 254777.07 100.00 90.99 6429.83 1.42 7425.28 7432.18 99.91 986.43 986.43 100.00

rcas6027 400000 162.74 43431.05 0.37 246913.58 248447.20 99.38 110.83 6400.00 1.73 6694.56 7425.28 90.16 966.89 966.89 100.00

rcas6028 400000 249.38 43196.54 0.58 250000.00 250000.00 100.00 109.51 6392.84 1.71 7406.04 7426.66 99.72 921.29 912.68 100.94

rcas6029 400000 244.04 43383.95 0.56 22637.24 251572.33 9.00 107.75 6430.87 1.68 7423.90 7426.66 99.96 986.43 986.43 100.00

rcas6030 400000 254.93 42417.82 0.60 533.40 194174.76 0.27 100.33 6425.70 1.56 7441.86 7446.02 99.94 986.43 986.43 100.00

rcas6031 400000 238.07 44004.40 0.54 253164.56 253164.56 100.00 87.00 6430.87 1.35 7423.90 7429.42 99.93 986.43 986.43 100.00

rcas6032 400000 129.35 42964.55 0.30 253164.56 253164.56 100.00 111.36 6405.12 1.74 7411.52 7411.52 100.00 986.43 986.43 100.00

rcas6033 400000 199.86 42964.55 0.47 493.47 122699.39 0.40 113.14 6418.49 1.76 6382.64 7429.42 85.91 976.56 986.43 99.00

rcas6034 400000 179.29 44004.40 0.41 254777.07 256410.26 99.36 91.56 6405.12 1.43 7425.28 7426.66 99.98 986.43 976.56 101.01

rcas6035 400000 254.98 42598.51 0.60 254777.07 254777.07 100.00 91.32 6404.10 1.43 7422.53 7423.90 99.98 976.56 976.56 100.00

rcas6036 400000 241.05 43811.61 0.55 238095.24 239520.96 99.40 85.00 6442.26 1.32 7425.28 7433.56 99.89 986.43 986.43 100.00

rcas6037 400000 144.15 43010.75 0.34 254777.07 253164.56 100.64 115.18 6449.53 1.79 7425.28 7433.56 99.89 986.43 986.43 100.00

rcas6038 400000 137.51 42872.45 0.32 253164.56 253164.56 100.00 127.39 6414.37 1.99 7426.66 7425.28 100.02 986.43 986.43 100.00

rcas6039 400000 188.97 43715.85 0.43 1622.72 195121.95 0.83 110.38 6428.80 1.72 7422.53 7428.04 99.93 545.57 548.63 99.44

rcas6040 400000 218.21 42149.63 0.52 247.37 53120.85 0.47 122.85 6310.14 1.95 1219.48 6029.54 20.22 904.22 948.12 95.37

rcas6041 400000 173.18 40899.80 0.42 284.85 60606.06 0.47 141.08 6336.13 2.23 737.15 6087.35 12.11 986.43 986.43 100.00

rcas6042 400000 155.50 43010.75 0.36 253164.56 253164.56 100.00 100.69 6417.46 1.57 7425.28 7428.04 99.96 996.49 986.43 101.02

rcas6043 400000 152.12 43859.65 0.35 370.16 170940.17 0.22 134.76 6393.86 2.11 7425.28 7433.56 99.89 996.49 986.43 101.02

rcas6044 400000 194.51 43383.95 0.45 250000.00 248447.20 100.62 117.54 6468.31 1.82 7425.28 7433.56 99.89 986.43 986.43 100.00

rcas6045 400000 149.66 42598.51 0.35 250000.00 251572.33 99.38 136.83 6449.53 2.12 7426.66 7425.28 100.02 986.43 986.43 100.00

rcas6046 400000 134.20 44150.11 0.30 254777.07 254777.07 100.00 116.99 6417.46 1.82 7426.66 7426.66 100.00 986.43 986.43 100.00

rcas6047 400000 272.75 43620.50 0.63 253164.56 254777.07 99.37 103.55 6405.12 1.62 7423.90 7425.28 99.98 986.43 986.43 100.00

rcas6048 400000 199.42 44150.11 0.45 253164.56 254777.07 99.37 99.94 6425.70 1.56 7198.13 7419.77 97.01 642.48 642.48 100.00

rcas6049 400000 209.84 42238.65 0.50 778.24 99750.62 0.78 107.85 6405.12 1.68 7220.22 7429.42 97.18 986.43 986.43 100.00

IOzone profiling

The profiling gives a gross performance view for several kByte buffer and several file size.
The Read/Write tests are unbuffered while FRead/FWrite tests are buffered.

Client to Server

The truncated distribution here are the result of the inhability of the client to push large file sizes ; the performance drops to about nothing. In addition, the unbuffered IO results are deceptive. They show a very large fwrite rate, larger than the ethernet capability. This is again due to a NFS side effect i.e. large blocks pushed through NFS will be broken in smaller buffer size (see rsize=32768 and wsize=32768 parameters) and what we measure is the NFS client performance and its buffering capabilities. Note that ioperf forces a sync/flush after each IO in write mode and isnot subject to this effect.

Local IO on server

There is a server-side intrinsic limitation when writting to a large file. Note that this axis (log2 of file size) is only the file size and has nothing to do with its internal structure. This limitation however seems to be a buffering effect since the unbuffered IO tests (bottom 2 FWrite and FRead) does not show that of a sharp fall (there is still something fishy for large file and large record size but that's expected).

Local IO on Server Client to Server (beware of NFS)