Bringing up dart
History
2014-Feb-08 Installing BigBlueButton
2013-May-03
Upgrade replacing 4 150GB SAS drives and 1TB WD Black with 7 WD Red 3TB drives. Installing RAID controller to replace one borrowed from AGI.
Hardware
- SuperMicro SC742 4U server case
- SuperMicro X9SRA motherboard
- Intel Xeon E5-2650 Sandy Bridge-EP 2.0GHz (2.8GHz Turbo Boost) LGA 2011 95W 8-Core Server Processor
- SUPERMICRO SNK-P0050AP4 Heatsink for Supermicro X9DR3-F Motherboard
- 32GB ECC registered RAM (4 x 8GB)
- LSI MegaRAID SATA/SAS 9260-8i RAID controller
Drives
- Corsair Neutron Series GTX CSSD-N240GBGTX-BK 2.5" 240GB SATA III Internal Solid State Drive (SSD)
- 7 Western Digital Red WD30EFRX 3TB IntelliPower SATA 6.0Gb/s 3.5" Internal Hard Drive -Bare Drive
Drive Config
LSI controller
storcli64 /c0 show storcli64 /c0/v0 delete storcli64 /c0 add vd type=raid6 Size=all name=RED drives=252:0,252:1,252:2,252:3,252:4,252:5,252:6 storcli64 /c0/v0 start init full storcli64 /c0/v0 show init
# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_root-lv_root
230546596 7322588 211512932 4% /
tmpfs 16424832 0 16424832 0% /dev/shm
/dev/md0 198325 135192 52894 72% /boot
/dev/mapper/vg_black-lv_home
945301884 138794936 758488312 16% /home
/dev/mapper/vg_var-lv_var
418102048 1076260 395787416 1% /var
none 16424832 36 16424796 1% /tmp
previously
I ended up doing the entire config using the CentOS installer, but left this information here because going through the steps myself helped me understand how LVM works. I created some virtual drives and played around with them in a VirtualBox installation of CentOS to work in a safe environment where I would not be able to break anything.
Hardware
sda 3Ware 435 (4 x Hitachi 150GB in RAID 5) sdb Neutron SSD 240 sdc Neutron SSD 240 sdd WDC Black 1000 sde Revo 60 sdf Revo 60
2 x Neutrons => partitioned for boot and md0 RAID 1 = operating system 2 x Revos => md1 RAID 0 = fast workspace 3ware RAID => /var and other fast data storage Black = home, bulky data, swap
Volume groups
vg_mirror vg_workspace vg_raid vg_bulk
Logical volumes
lv_root / lv_workspace /workspace lv_var /var lv_home /home lv_swap SWAP
Neutron drive: Build a broken mirror, add second drive later
[See http://www.gagme.com/greg/linux/raid-lvm.php Managing RAID and LVM with Linux] For install onto RAID see also some of the steps here Install on partitionable RAID1
Here are the commands that I used.
# Use the entire drive and put the partitions inside /dev/md_d0 mdadm --create --metadata=0.90 --level=mirror --raid-devices=2 /dev/md_d0 /dev/sda missing # put RAID1 into LVM2 pvcreate /dev/md0 vgcreate vg_mirror /dev/md0
# later, set up second drive as the mirror mdadm --add --verbose /dev/md0 /dev/sdb2 # the mirror should automatically be added to LVM since it's part of /dev/md0 cat /proc/mdstat
Revo drive: completely managed under LVM2
The Revo drive has its own RAID onboard. I set up the two halves as RAID 0 for performance at the cost of potential dataloss. The drive is visible to Linux as /dev/sde and /dev/sdf and also as a RAID at /dev/mapper/sil_bibibgajfbdj (a name only its mother could love.) I set it up under LVM2 thusly
[root@dart ~]# pvcreate -v /dev/mapper/sil_bibibgajfbdj
Set up physical volume for "/dev/mapper/sil_bibibgajfbdj" with 234458624 available sectors Zeroing start of device /dev/mapper/sil_bibibgajfbdj Writing physical volume data to disk "/dev/mapper/sil_bibibgajfbdj" Physical volume "/dev/mapper/sil_bibibgajfbdj" successfully created
[root@dart ~]# vgcreate --verbose vg_workspace /dev/mapper/sil_bibibgajfbdj
Wiping cache of LVM-capable devices Adding physical volume '/dev/mapper/sil_bibibgajfbdj' to volume group 'vg_workspace' Archiving volume group "vg_workspace" metadata (seqno 0). Creating volume group backup "/etc/lvm/backup/vg_workspace" (seqno 1). Volume group "vg_workspace" successfully created
[root@dart ~]# lvcreate --verbose --extents 100%FREE -n lv_workspace vg_workspace
Setting logging type to disk Finding volume group "vg_workspace" Archiving volume group "vg_workspace" metadata (seqno 1). Creating logical volume lv_workspace Creating volume group backup "/etc/lvm/backup/vg_workspace" (seqno 2). Found volume group "vg_workspace" activation/volume_list configuration setting not defined: Checking only host tags for vg_workspace/lv_workspace Creating vg_workspace-lv_workspace Loading vg_workspace-lv_workspace table (253:5) Resuming vg_workspace-lv_workspace (253:5) Clearing start of logical volume "lv_workspace" Creating volume group backup "/etc/lvm/backup/vg_workspace" (seqno 2). Logical volume "lv_workspace" created
[root@dart ~]# mkfs.ext4 /dev/vg_workspace/lv_workspace
/dev/vg_workspace/lv_workspace will be mounted at /workspace
3Ware RAID5: completely managed under LVM2
pvcreate /dev/sd? vgcreate vg_raid /dev/sd? lvcreate -l ????? vg_raid -n lv_var mkfs.ext4 /dev/vg_raid/lv_var
After installing OS, /dev/vg_raid/lvm0 will be mounted at /var
Black drive: completely managed under LVM2
Might also want some SWAP on here?
parted set up one LVM partitions mark partition type as 8e = LVM pvcreate /dev/sdd vgcreate vg_bulk /dev/sdd lvcreate -l ????? vg_bulk -n lv_home lvcreate -l ????? vg_bulk -n lv_swap mkfs.ext4 /dev/vg_bulk/lv_home mkswap /dev/vg_bulk/lv_swap
Will be mounted as /home and swap
Commands to view what we've done
This is how things actually look right now..
[root@dart bwilson]# pvdisplay
--- Physical volume ---
PV Name /dev/sdd1
VG Name vg_black
PV Size 931.51 GiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 238466
Free PE 0
Allocated PE 238466
PV UUID 2ZWwrU-Eyp9-erVR-DaSZ-ld3T-g6LK-8SBJcn
--- Physical volume ---
PV Name /dev/disk/by-id/dm-name-sil_bibibgajfbdj
VG Name vg_workspace
PV Size 111.80 GiB / not usable 1.75 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 28620
Free PE 0
Allocated PE 28620
PV UUID F86o1d-tmhZ-4KVW-Ke06-b9iE-7HGW-EjhLrF
--- Physical volume ---
PV Name /dev/md1
VG Name vg_root
PV Size 223.37 GiB / not usable 1.87 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 57183
Free PE 0
Allocated PE 57183
PV UUID drVc1G-694L-6GSl-cTNl-uINo-zbEc-ZBpILd
--- Physical volume ---
PV Name /dev/sda1
VG Name vg_var
PV Size 405.09 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 103703
Free PE 0
Allocated PE 103703
PV UUID nMZCtR-oZkS-2Ctu-9HwB-Vt5r-7iPD-vf5v7H
[root@dart bwilson]# vgdisplay
--- Volume group --- VG Name vg_black System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 1 Act PV 1 VG Size 931.51 GiB PE Size 4.00 MiB Total PE 238466 Alloc PE / Size 238466 / 931.51 GiB Free PE / Size 0 / 0 VG UUID fYgPgc-cXaW-6ryU-nU1v-5m3F-Rj0r-Afe9p6 --- Volume group --- VG Name vg_workspace System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 111.80 GiB PE Size 4.00 MiB Total PE 28620 Alloc PE / Size 28620 / 111.80 GiB Free PE / Size 0 / 0 VG UUID 9OkF0O-E5f7-bfvH-qAfb-E7Wh-cOGS-EhVydH --- Volume group --- VG Name vg_root System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 223.37 GiB PE Size 4.00 MiB Total PE 57183 Alloc PE / Size 57183 / 223.37 GiB Free PE / Size 0 / 0 VG UUID 2Q5ooC-B0ZW-BcxA-jufX-80zh-hkQh-qYNm2G --- Volume group --- VG Name vg_var System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 405.09 GiB PE Size 4.00 MiB Total PE 103703 Alloc PE / Size 103703 / 405.09 GiB Free PE / Size 0 / 0 VG UUID mIgZdQ-4tzM-T2KP-S2f7-QA14-iwp9-cTwyly
[root@dart bwilson]# lvdisplay
--- Logical volume ---
LV Path /dev/vg_black/lv_swap
LV Name lv_swap
VG Name vg_black
LV UUID 3ZtM9o-a5Z8-iAs9-Sm1K-O1zU-jfLM-iKX7pj
LV Write Access read/write
LV Creation host, time dart.alseageo.com, 2012-12-08 02:26:11 -0800
LV Status available
# open 1
LV Size 15.62 GiB
Current LE 4000
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1
--- Logical volume ---
LV Path /dev/vg_black/lv_home
LV Name lv_home
VG Name vg_black
LV UUID rwvORk-YHTr-jmMR-lxey-Q0Ue-NwTO-mOcrUO
LV Write Access read/write
LV Creation host, time dart.alseageo.com, 2012-12-08 02:26:12 -0800
LV Status available
# open 1
LV Size 915.88 GiB
Current LE 234466
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2
--- Logical volume ---
LV Path /dev/vg_workspace/lv_workspace
LV Name lv_workspace
VG Name vg_workspace
LV UUID wUhofL-LtjP-p6Rr-wMj0-jKRF-X2fB-Bjyy9H
LV Write Access read/write
LV Creation host, time dart.alseageo.com, 2012-12-09 18:29:17 -0800
LV Status available
# open 0
LV Size 111.80 GiB
Current LE 28620
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:5
--- Logical volume ---
LV Path /dev/vg_root/lv_root
LV Name lv_root
VG Name vg_root
LV UUID 5cCyeM-yuTM-GsfE-M4Xu-em5h-6olD-sMucxV
LV Write Access read/write
LV Creation host, time dart.alseageo.com, 2012-12-08 02:26:00 -0800
LV Status available
# open 1
LV Size 223.37 GiB
Current LE 57183
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
--- Logical volume ---
LV Path /dev/vg_var/lv_var
LV Name lv_var
VG Name vg_var
LV UUID GZFOdu-D6uN-UCAR-Gt3R-Ddni-2dGd-ZNmhNh
LV Write Access read/write
LV Creation host, time dart.alseageo.com, 2012-12-08 02:25:17 -0800
LV Status available
# open 1
LV Size 405.09 GiB
Current LE 103703
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3
/etc/fstab
/dev/mapper/vg_root-lv_root / ext4 defaults,noatime,discard 1 1 UUID=488a0c3b-689e-4f2b-8688-d8eefdf0a7f8 /boot ext4 defaults,noatime,discard 1 2 /dev/mapper/vg_black-lv_home /home ext4 defaults,noatime 1 2 /dev/mapper/vg_var-lv_var /var ext4 defaults,noatime 1 2 /dev/mapper/vg_black-lv_swap swap swap defaults 0 0 /dev/mapper/vg_workspace-lv_workspace /workspace ext4 defaults,noatime,discard 1 2 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 # SSD optimizations none /tmp tmpfs defaults,noatime,mode=1777 0 0
CentOS install
Install server system from DVD
vi /etc/sysconfig/network-scripts/ifcfg-eth0 vi /etc/resolv.conf service network restart yum -y install emacs yum -y install openssh-clients yum -y install gcc unzip yum -y install hdparm yum -y install httpd yum update reboot
Basic software install
PostGIS
Install PostGIS 2.0 See PostGIS page
PHP 5
Build php so that you can use your own PostgreSQL build. See Ben Ramsey's page I had to add pgsql to Ben's configuration.
Using the standard php won't work because it pulls in postgresql 8.4 client library
This generates an SELINUX error though:
Cannot load /usr/lib64/httpd/modules/libphp5.so into server: libpq.so.5: failed to map segment from shared object: Permission denied
You fix this with
chcon --reference=/usr/lib64/httpd/modules/libphp5.so /usr/local/pgsql/lib/libpq.so.5.4
And then it works! The audience goes crazy!!! http://dart.geo-ceg.org/phpPgAdmin/
Install ArcGIS Server
I actually had a working copy of ArcGIS Server for a few months. As of Mar 2014 I am taking the system offline and moving it to California. Once I get it back onto the Internet, I will be putting Debian on as the primary operating system and running virtual machines if I decide I need something like ArcGIS Server + CentOS.
Mondo back up
2014-Mar-26
I had to add mondorescue.repo from here to /etc/yum.repos.d/ and then do 'yum install mondo'
This is how I build a bare metal restore boot disc that can get me going again after an SSD failure. How much space do I need? Well, 'df -h' shows this.
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_root-lv_root 220G 9.9G 199G 5% / tmpfs 16G 0 16G 0% /dev/shm /dev/md0 194M 142M 43M 77% /boot /dev/sda 14T 838G 13T 6% /var none 16G 8.2M 16G 1% /tmp
I am going to try doing a backup to DVD, hoping compression works else it will go to 2 discs. I want to exclude the bulk of /var since it's a 14 terabyte RAID!
The commands
export MRTMP="/var/tmp" export MRSCRATCH=$MRTMP mondoarchive -OVr -z -E "/var/home|/var/lib/pgsql|/var/lib/libvirt" -d /dev/sr0 -s 20G
What the options mean:
-OVr = Do a backup, Verify it, and output to a DVD device -E "/var/home|/var/lib/pgsql|/var/lib/libvirt" = Ignore these, they are big -d /dev/sr0 = Device to write backup to. -z = use ACLs, for SELINUX -s 20G = using 25 GB media
