Cluster Disks
From BNL Physics Computing
Contents |
[edit]
Benchmarks
Bonnie++ is run like:
bonnie++ -u nobody -d /path/to/workspace
[edit]
coop018, 7 disk raid5, ext3
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
coop018 6592M 54132 98 129200 53 67341 26 56411 96 369428 76 501.1 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 2771 99 +++++ +++ +++++ +++ 3263 99 +++++ +++ 11030 94
coop018,6592M,54132,98,129200,53,67341,26,56411,96,369428,76,501.1,1,16,2771,99,+++++,+++,+++++,+++,3263,99,+++++,+++,11030,94
[edit]
coop018, 7 disk raid5, ext3, nfs mounted on coop017
Note: md0_raid5 at 5-10% when running, nfsd 20-50% when running. Average CPU usage ~5% sys.
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
coop017 6592M 36025 57 29304 8 15280 4 38848 65 46455 5 423.4 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 210 0 9452 10 186 0 213 0 10510 12 139 0
coop017,6592M,36025,57,29304,8,15280,4,38848,65,46455,5,423.4,1,16,210,0,9452,10,186,0,213,0,10510,12,139,0
[edit]
coop018, 7 disk raid5 + lvm, ext3
[edit]
coop017, one disk
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
coop017 6592M 51216 89 66507 20 27202 7 56245 93 69936 6 207.5 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
coop017,6592M,51216,89,66507,20,27202,7,56245,93,69936,6,207.5,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
[edit]
coop009, 6 disk, lvm, no raid (/local/mnt/00)
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
coop009 4G 41370 90 43146 12 22005 6 31155 70 48762 7 216.2 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 354 1 +++++ +++ 388 1 324 2 +++++ +++ 280 1
coop009,4G,41370,90,43146,12,22005,6,31155,70,48762,7,216.2,0,16,354,1,+++++,+++,388,1,324,2,+++++,+++,280,1
[edit]
coop, 2 disk raid0, scsi (/home)
[edit]
coop, 2 disk raid0, sata (/mnt/scratch/00)
Note: CPUs in use by others.
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
coop 4G 36858 77 74766 28 29810 9 28761 62 54615 8 59.1 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 2209 87 +++++ +++ +++++ +++ 1164 48 +++++ +++ 4764 50
coop,4G,36858,77,74766,28,29810,9,28761,62,54615,8,59.1,0,16,2209,87,+++++,+++,+++++,+++,1164,48,+++++,+++,4764,50
[edit]
Info dump
There are a number of different disk generations on the coop.
[edit]
Home directories
Served by main coop node, backed up by legato. Served by coop, NFS mounted on all nodes.
coop:~# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.01
Creation Time : Tue Apr 20 15:11:58 2004
Raid Level : raid0
Array Size : 141693056 (135.13 GiB 145.09 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Tue Aug 31 12:43:22 2004
State : clean, no-errors
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Chunk Size : 16K
Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
1 8 18 1 active sync /dev/sdb2
UUID : dbbb7ee1:a5a29b4c:d686cfa4:cb8a534f
Events : 0.23
coop:~# df -hl /home Filesystem Size Used Avail Use% Mounted on /dev/md0 134G 124G 3.3G 98% /home
[edit]
General Use Scratch
Scratch disk for general use. Not backed up. Served by coop, NFS mounted on all nodes.
coop:~# mdadm --detail /dev/md1
/dev/md1:
Version : 00.90.01
Creation Time : Tue Apr 20 18:50:12 2004
Raid Level : raid0
Array Size : 488391808 (465.77 GiB 500.11 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Tue Aug 31 12:43:22 2004
State : clean, no-errors
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Chunk Size : 16K
Number Major Minor RaidDevice State
0 22 1 0 active sync /dev/hdc1
1 22 65 1 active sync /dev/hdd1
UUID : 573a4f51:f264fd40:c7cf2bf2:0516f4c4
Events : 0.24
coop:~# df -h /mnt/scratch/00 Filesystem Size Used Avail Use% Mounted on /dev/md1 459G 404G 33G 93% /mnt/scratch/00
[edit]
MINOS Scratch
Disk bought by and dedicated for MINOS work. Not backed up. Served by coop009. NFS exported to all the cluster. This is a pure LVM with no RAID layer. It is built from /dev/sd[a-f]1.
coop009:root# df -hl /local/mnt/00
Filesystem Size Used Avail Use% Mounted on
/dev/minos_volume1/lv1
1.4T 1.3T 104G 93% /local/mnt/00
[edit]
Daya Bay Scratch
Under development.
