Bringing up dart

History

2014-Feb-08 Installing BigBlueButton

2013-May-03

Upgrade replacing 4 150GB SAS drives and 1TB WD Black with 7 WD Red 3TB drives. Installing RAID controller to replace one borrowed from AGI.

Hardware

• SuperMicro SC742 4U server case
• SuperMicro X9SRA motherboard
• Intel Xeon E5-2650 Sandy Bridge-EP 2.0GHz (2.8GHz Turbo Boost) LGA 2011 95W 8-Core Server Processor
• SUPERMICRO SNK-P0050AP4 Heatsink for Supermicro X9DR3-F Motherboard
• 32GB ECC registered RAM (4 x 8GB)
• LSI MegaRAID SATA/SAS 9260-8i RAID controller

Drives

• Corsair Neutron Series GTX CSSD-N240GBGTX-BK 2.5" 240GB SATA III Internal Solid State Drive (SSD)
• 7 Western Digital Red WD30EFRX 3TB IntelliPower SATA 6.0Gb/s 3.5" Internal Hard Drive -Bare Drive

Drive Config

LSI controller

storcli64 /c0 show
storcli64 /c0/v0 delete
storcli64 /c0 add vd type=raid6 Size=all name=RED drives=252:0,252:1,252:2,252:3,252:4,252:5,252:6
storcli64 /c0/v0 start init full
storcli64 /c0/v0 show init

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_root-lv_root
                     230546596   7322588 211512932   4% /
tmpfs                 16424832         0  16424832   0% /dev/shm
/dev/md0                198325    135192     52894  72% /boot
/dev/mapper/vg_black-lv_home
                     945301884 138794936 758488312  16% /home
/dev/mapper/vg_var-lv_var
                     418102048   1076260 395787416   1% /var
none                  16424832        36  16424796   1% /tmp

previously

I ended up doing the entire config using the CentOS installer, but left this information here because going through the steps myself helped me understand how LVM works. I created some virtual drives and played around with them in a VirtualBox installation of CentOS to work in a safe environment where I would not be able to break anything.

Hardware

sda 3Ware 435 (4 x Hitachi 150GB in RAID 5)
sdb Neutron SSD 240
sdc Neutron SSD 240
sdd WDC Black 1000
sde Revo 60
sdf Revo 60

2 x Neutrons => partitioned for boot and md0 RAID 1 = operating system
2 x Revos => md1 RAID 0 = fast workspace
3ware RAID => /var and other fast data storage
Black = home, bulky data, swap

Volume groups

vg_mirror
vg_workspace
vg_raid
vg_bulk

Logical volumes

lv_root      /
lv_workspace /workspace
lv_var       /var
lv_home      /home
lv_swap      SWAP

Neutron drive: Build a broken mirror, add second drive later

[See http://www.gagme.com/greg/linux/raid-lvm.php Managing RAID and LVM with Linux] For install onto RAID see also some of the steps here Install on partitionable RAID1

Here are the commands that I used.

# Use the entire drive and put the partitions inside /dev/md_d0
mdadm --create --metadata=0.90 --level=mirror --raid-devices=2 /dev/md_d0 /dev/sda missing
# put RAID1 into LVM2
pvcreate /dev/md0
vgcreate vg_mirror /dev/md0

# later, set up second drive as the mirror
mdadm --add --verbose /dev/md0 /dev/sdb2
# the mirror should automatically be added to LVM since it's part of /dev/md0
cat /proc/mdstat

Revo drive: completely managed under LVM2

The Revo drive has its own RAID onboard. I set up the two halves as RAID 0 for performance at the cost of potential dataloss. The drive is visible to Linux as /dev/sde and /dev/sdf and also as a RAID at /dev/mapper/sil_bibibgajfbdj (a name only its mother could love.) I set it up under LVM2 thusly

[root@dart ~]# pvcreate -v /dev/mapper/sil_bibibgajfbdj

   Set up physical volume for "/dev/mapper/sil_bibibgajfbdj" with 234458624 available sectors
   Zeroing start of device /dev/mapper/sil_bibibgajfbdj
 Writing physical volume data to disk "/dev/mapper/sil_bibibgajfbdj"
 Physical volume "/dev/mapper/sil_bibibgajfbdj" successfully created

[root@dart ~]# vgcreate --verbose vg_workspace /dev/mapper/sil_bibibgajfbdj

   Wiping cache of LVM-capable devices
   Adding physical volume '/dev/mapper/sil_bibibgajfbdj' to volume group 'vg_workspace'
   Archiving volume group "vg_workspace" metadata (seqno 0).
   Creating volume group backup "/etc/lvm/backup/vg_workspace" (seqno 1).
 Volume group "vg_workspace" successfully created

[root@dart ~]# lvcreate --verbose --extents 100%FREE -n lv_workspace vg_workspace

   Setting logging type to disk
   Finding volume group "vg_workspace"
   Archiving volume group "vg_workspace" metadata (seqno 1).
   Creating logical volume lv_workspace
   Creating volume group backup "/etc/lvm/backup/vg_workspace" (seqno 2).
   Found volume group "vg_workspace"
   activation/volume_list configuration setting not defined: Checking only host tags for vg_workspace/lv_workspace
   Creating vg_workspace-lv_workspace
   Loading vg_workspace-lv_workspace table (253:5)
   Resuming vg_workspace-lv_workspace (253:5)
   Clearing start of logical volume "lv_workspace"
   Creating volume group backup "/etc/lvm/backup/vg_workspace" (seqno 2).
 Logical volume "lv_workspace" created

[root@dart ~]# mkfs.ext4 /dev/vg_workspace/lv_workspace

/dev/vg_workspace/lv_workspace will be mounted at /workspace

3Ware RAID5: completely managed under LVM2

pvcreate /dev/sd?
vgcreate vg_raid /dev/sd?
lvcreate -l ????? vg_raid -n lv_var
mkfs.ext4 /dev/vg_raid/lv_var

After installing OS, /dev/vg_raid/lvm0 will be mounted at /var

Black drive: completely managed under LVM2

Might also want some SWAP on here?

parted set up one LVM partitions
mark partition type as 8e = LVM
pvcreate /dev/sdd
vgcreate vg_bulk /dev/sdd
lvcreate -l ????? vg_bulk -n lv_home
lvcreate -l ????? vg_bulk -n lv_swap
mkfs.ext4 /dev/vg_bulk/lv_home
mkswap /dev/vg_bulk/lv_swap

Will be mounted as /home and swap

Commands to view what we've done

This is how things actually look right now..

[root@dart bwilson]# pvdisplay

 --- Physical volume ---
 PV Name               /dev/sdd1
 VG Name               vg_black
 PV Size               931.51 GiB / not usable 4.00 MiB
 Allocatable           yes (but full)
 PE Size               4.00 MiB
 Total PE              238466
 Free PE               0
 Allocated PE          238466
 PV UUID               2ZWwrU-Eyp9-erVR-DaSZ-ld3T-g6LK-8SBJcn

  --- Physical volume ---
 PV Name               /dev/disk/by-id/dm-name-sil_bibibgajfbdj
 VG Name               vg_workspace
 PV Size               111.80 GiB / not usable 1.75 MiB
 Allocatable           yes (but full)
 PE Size               4.00 MiB
 Total PE              28620
 Free PE               0
 Allocated PE          28620
 PV UUID               F86o1d-tmhZ-4KVW-Ke06-b9iE-7HGW-EjhLrF
    
 --- Physical volume ---
 PV Name               /dev/md1
 VG Name               vg_root
 PV Size               223.37 GiB / not usable 1.87 MiB
 Allocatable           yes (but full)
 PE Size               4.00 MiB
 Total PE              57183
 Free PE               0
 Allocated PE          57183
 PV UUID               drVc1G-694L-6GSl-cTNl-uINo-zbEc-ZBpILd
  
 --- Physical volume ---
 PV Name               /dev/sda1
 VG Name               vg_var
 PV Size               405.09 GiB / not usable 3.00 MiB
 Allocatable           yes (but full)
 PE Size               4.00 MiB
 Total PE              103703
 Free PE               0
 Allocated PE          103703
 PV UUID               nMZCtR-oZkS-2Ctu-9HwB-Vt5r-7iPD-vf5v7H

[root@dart bwilson]# vgdisplay

 --- Volume group ---
 VG Name               vg_black
 System ID             
 Format                lvm2
 Metadata Areas        1
 Metadata Sequence No  3
 VG Access             read/write
 VG Status             resizable
 MAX LV                0
 Cur LV                2
 Open LV               2
 Max PV                0
 Cur PV                1
 Act PV                1
 VG Size               931.51 GiB
 PE Size               4.00 MiB
 Total PE              238466
 Alloc PE / Size       238466 / 931.51 GiB
 Free  PE / Size       0 / 0   
 VG UUID               fYgPgc-cXaW-6ryU-nU1v-5m3F-Rj0r-Afe9p6
   
--- Volume group ---
 VG Name               vg_workspace
 System ID             
 Format                lvm2
 Metadata Areas        1
 Metadata Sequence No  2
 VG Access             read/write
 VG Status             resizable
 MAX LV                0
 Cur LV                1
 Open LV               0
 Max PV                0
 Cur PV                1
 Act PV                1
 VG Size               111.80 GiB
 PE Size               4.00 MiB
 Total PE              28620
 Alloc PE / Size       28620 / 111.80 GiB
 Free  PE / Size       0 / 0   
 VG UUID               9OkF0O-E5f7-bfvH-qAfb-E7Wh-cOGS-EhVydH

 --- Volume group ---
 VG Name               vg_root
 System ID             
 Format                lvm2
 Metadata Areas        1
 Metadata Sequence No  2
 VG Access             read/write
 VG Status             resizable
 MAX LV                0
 Cur LV                1
 Open LV               1
 Max PV                0
 Cur PV                1
 Act PV                1
 VG Size               223.37 GiB
 PE Size               4.00 MiB
 Total PE              57183
 Alloc PE / Size       57183 / 223.37 GiB
 Free  PE / Size       0 / 0   
 VG UUID               2Q5ooC-B0ZW-BcxA-jufX-80zh-hkQh-qYNm2G
  
 --- Volume group ---
 VG Name               vg_var
 System ID             
 Format                lvm2
 Metadata Areas        1
 Metadata Sequence No  2
 VG Access             read/write
 VG Status             resizable
 MAX LV                0
 Cur LV                1
 Open LV               1
 Max PV                0
 Cur PV                1
 Act PV                1
 VG Size               405.09 GiB
 PE Size               4.00 MiB
 Total PE              103703
 Alloc PE / Size       103703 / 405.09 GiB
 Free  PE / Size       0 / 0   
 VG UUID               mIgZdQ-4tzM-T2KP-S2f7-QA14-iwp9-cTwyly

[root@dart bwilson]# lvdisplay

 --- Logical volume ---
 LV Path                /dev/vg_black/lv_swap
 LV Name                lv_swap
 VG Name                vg_black
 LV UUID                3ZtM9o-a5Z8-iAs9-Sm1K-O1zU-jfLM-iKX7pj
 LV Write Access        read/write
 LV Creation host, time dart.alseageo.com, 2012-12-08 02:26:11 -0800
 LV Status              available
 # open                 1
 LV Size                15.62 GiB
 Current LE             4000
 Segments               1
 Allocation             inherit
 Read ahead sectors     auto
 - currently set to     256
 Block device           253:1
  
 --- Logical volume ---
 LV Path                /dev/vg_black/lv_home
 LV Name                lv_home
 VG Name                vg_black
 LV UUID                rwvORk-YHTr-jmMR-lxey-Q0Ue-NwTO-mOcrUO
 LV Write Access        read/write
 LV Creation host, time dart.alseageo.com, 2012-12-08 02:26:12 -0800
 LV Status              available
 # open                 1
 LV Size                915.88 GiB
 Current LE             234466
 Segments               1
 Allocation             inherit
 Read ahead sectors     auto
 - currently set to     256
 Block device           253:2
     
 --- Logical volume ---
 LV Path                /dev/vg_workspace/lv_workspace
 LV Name                lv_workspace
 VG Name                vg_workspace
 LV UUID                wUhofL-LtjP-p6Rr-wMj0-jKRF-X2fB-Bjyy9H
 LV Write Access        read/write
 LV Creation host, time dart.alseageo.com, 2012-12-09 18:29:17 -0800
 LV Status              available
 # open                 0
 LV Size                111.80 GiB
 Current LE             28620
 Segments               1
 Allocation             inherit
 Read ahead sectors     auto
 - currently set to     256
 Block device           253:5

 --- Logical volume ---
 LV Path                /dev/vg_root/lv_root
 LV Name                lv_root
 VG Name                vg_root
 LV UUID                5cCyeM-yuTM-GsfE-M4Xu-em5h-6olD-sMucxV
 LV Write Access        read/write
 LV Creation host, time dart.alseageo.com, 2012-12-08 02:26:00 -0800
 LV Status              available
 # open                 1
 LV Size                223.37 GiB
 Current LE             57183
 Segments               1
 Allocation             inherit
 Read ahead sectors     auto
 - currently set to     256
 Block device           253:0
  
 --- Logical volume ---
 LV Path                /dev/vg_var/lv_var
 LV Name                lv_var
 VG Name                vg_var
 LV UUID                GZFOdu-D6uN-UCAR-Gt3R-Ddni-2dGd-ZNmhNh
 LV Write Access        read/write
 LV Creation host, time dart.alseageo.com, 2012-12-08 02:25:17 -0800
 LV Status              available
 # open                 1
 LV Size                405.09 GiB
 Current LE             103703
 Segments               1
 Allocation             inherit
 Read ahead sectors     auto
 - currently set to     256
 Block device           253:3

/etc/fstab

/dev/mapper/vg_root-lv_root               /          ext4    defaults,noatime,discard   1 1
UUID=488a0c3b-689e-4f2b-8688-d8eefdf0a7f8 /boot	     ext4    defaults,noatime,discard   1 2
/dev/mapper/vg_black-lv_home              /home      ext4    defaults,noatime        	1 2
/dev/mapper/vg_var-lv_var                 /var       ext4    defaults,noatime        	1 2
/dev/mapper/vg_black-lv_swap              swap       swap    defaults        		0 0
/dev/mapper/vg_workspace-lv_workspace     /workspace ext4    defaults,noatime,discard  	1 2
tmpfs					  /dev/shm   tmpfs   defaults        0 0
devpts                  		  /dev/pts   devpts  gid=5,mode=620  0 0
sysfs                   		  /sys       sysfs   defaults        0 0
proc                    		  /proc      proc    defaults        0 0

# SSD optimizations
none					  /tmp	     tmpfs   defaults,noatime,mode=1777 0 0

CentOS install

Install server system from DVD

vi /etc/sysconfig/network-scripts/ifcfg-eth0
vi /etc/resolv.conf
service network restart
yum -y install emacs
yum -y install openssh-clients
yum -y install gcc unzip
yum -y install hdparm
yum -y install httpd
yum update
reboot

Basic software install

PostGIS

Install PostGIS 2.0 See PostGIS page

PHP 5

Build php so that you can use your own PostgreSQL build. See Ben Ramsey's page I had to add pgsql to Ben's configuration.

Using the standard php won't work because it pulls in postgresql 8.4 client library

This generates an SELINUX error though:

Cannot load /usr/lib64/httpd/modules/libphp5.so into server: libpq.so.5: failed to map segment from shared object: Permission denied

You fix this with

chcon --reference=/usr/lib64/httpd/modules/libphp5.so /usr/local/pgsql/lib/libpq.so.5.4

And then it works! The audience goes crazy!!! http://dart.geo-ceg.org/phpPgAdmin/

Install ArcGIS Server

I actually had a working copy of ArcGIS Server for a few months. As of Mar 2014 I am taking the system offline and moving it to California. Once I get it back onto the Internet, I will be putting Debian on as the primary operating system and running virtual machines if I decide I need something like ArcGIS Server + CentOS.

Mondo back up

2014-Mar-26

I had to add mondorescue.repo from here to /etc/yum.repos.d/ and then do 'yum install mondo'

This is how I build a bare metal restore boot disc that can get me going again after an SSD failure. How much space do I need? Well, 'df -h' shows this.

Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_root  220G  9.9G  199G   5% /
tmpfs                         16G     0   16G   0% /dev/shm
/dev/md0                     194M  142M   43M  77% /boot
/dev/sda                      14T  838G   13T   6% /var
none                          16G  8.2M   16G   1% /tmp

I am going to try doing a backup to DVD, hoping compression works else it will go to 2 discs. I want to exclude the bulk of /var since it's a 14 terabyte RAID!

The commands

export MRTMP="/var/tmp"
export MRSCRATCH=$MRTMP
mondoarchive -OVr -z -E "/var/home|/var/lib/pgsql|/var/lib/libvirt" -d /dev/sr0 -s 20G

What the options mean:

-OVr = Do a backup, Verify it, and output to a DVD device
-E "/var/home|/var/lib/pgsql|/var/lib/libvirt" = Ignore these, they are big
-d /dev/sr0 = Device to write backup to.
-z = use ACLs, for SELINUX
-s 20G = using 25 GB media