Overview

The following two boxes will be involved in deploying to the Production environment.

1.bsgplatform.com (72.249.21.180)

4.bsgplatform.com (72.249.20.204)

These will be referred to as boxes 1 and 4 respectively.

Box 1

This is already running all the platform services except the storage service.

Box 4

This is running the storage service. The following ports are being used by tomcat already: 4001, 8005, 8009, 8080, 8082, 8192, 8443, 45564

Deployment Steps

Ensure all required gems are installed.

For Box 1

The new version of rails needs to be installed on this box.

gem install rails -v=2.1.1

Besides this the list of gems on this server is the official list and this should be followed when installing gems on box 4. The only exceptions may be rails (version 2.1.1 needs to be installed on box 1 as well) and rubygems-update (if a later version than 0.9.4 is able to install all required gems then keeping that version will be fine)

actionmailer (2.1.0, 2.0.2, 1.3.5)
actionpack (2.1.0, 2.0.2, 1.13.5)
actionwebservice (1.2.5)
activerecord (2.1.0, 2.0.2, 1.15.5)
activeresource (2.1.0, 2.0.2)
activesupport (2.1.0, 2.0.2, 1.4.4)
asciify (0.1.0)
atom-tools (2.0.0)
cgi_multipart_eof_fix (2.5.0)
daemons (1.0.9)
ezcrypto (0.7)
fastthread (1.0.1)
gem_plugin (0.2.3)
hoe (1.3.0)
hpricot (0.6)
httpclient (2.1.2)
libxml-ruby (0.5.4)
log4r (1.0.5)
memcache-client (1.5.0)
mime-types (1.15)
mocha (0.9.0)
mongrel (1.1.3)
mongrel_cluster (1.0.5)
mysql (2.7)
needle (1.3.0)
net-sftp (1.1.0)
net-ssh (1.1.2)
rails (2.1.0, 2.0.2)
rake (0.8.1, 0.7.3)
rfacebook (0.9.8)
rubyforge (0.4.4)
rubygems-update (0.9.4)
sources (0.0.1)
uuidtools (1.0.2)
XMLCanonicalizer (1.0.1)
ZenTest (3.6.1)

Box 4

Install the following gems on Box 4. Besides XMLCanonicalizer all other gems are already installed on Box 4 but if the versions need to be matched with Box 1 gems then the following need to be installed.

actionmailer (1.3.5)

actionpack (1.13.5)

actionwebservice (1.2.5)

activerecord (1.15.5)

activesupport (1.4.4)

atom-tools (2.0.0)

daemons (1.0.9)

hoe (1.3.0)

libxml-ruby (0.5.4)

mongrel (1.1.3)

net-sftp (1.1.0)

net-ssh (1.1.2)

rake (0.7.3)

rubyforge (0.4.4)

sources (0.1.1)

uuidtools (1.0.2)

XMLCanonicalizer (1.0.1) - This is missing in 4 altogether

ZenTest (3.6.1)

Copy SSL certificates from Box 1 to Box 4 and setup the service-user.vhost file accordingly (This should be similar to the file on Box 3).

Install monit on Box 4:

Install flex (pre-requisite for monit):

wget http://prdownloads.sourceforge.net/flex/flex-2.5.35.tar.gz

tar zxvf flex-2.5.35.tar.gz

cd flex-2.5.35.tar.gz

./configure

make

make check

make install

Install byacc (Pre-requisite for monit)

yum install byacc

Install monit (This requires flex and yacc, as listed above, to be installed):

wget http://tildeslash.com/monit/dist/monit-4.10.1.tar.gz

tar zxvf monit-4.10.1.tar.gz

cd monit-4.10.1.tar.gz

./configure --without-ssl

make

make install

Copy the /usr/local/etc/monitrc file from 1.bsgplatform.com (72.249.21.180) to the server being setup

Modify monitrc accordingly.

Edit the /etc/inittab file and add the following statement

mo:2345:respawn:/usr/local/bin/monit -Ic /usr/local/etc/monitrc

Use telinit q to start up monit initially.

Use ps afx to check that monit is running with the -Ic parameters.

start monit:

/usr/local/bin/monit

stop monit:

/usr/local/bin/monit quit

Check the results in monit.log

DNS Setup

Now Box 1 will be setup first and all services will be deployed to it. At this point Box 1 will be brought down.

Change DNS entries by logging into rimuhosting:

http://rimuhosting.com

Click on "Control Panel" and Login.

Manage your DNS.

Click on zone "ngenplatform.com"

Set the IP Address to 72.249.39.81 for the following domains. This is the IP Address of the Load Balancer.

provision.ngenplatform.com

user.ngenplatform.com

content.ngenplatform.com

organization.ngenplatform.com

util.ngenplatform.com (This actually refers to the pubsub service at http://util.ngenplatform.com/pubsub)

Ping to these domains and check that the new ip address is enabled.

Load Balancer Setup

On the Load Balancer Box (72.249.39.81) go to the /etc/conf/httpd/sites-enabled folder and add files for the services by copying them from Box 2 (72.249.74.249) Since Box 2 is the load balancer in the test environment and the files there will be similarly configured.

Copy all files named as <nnn>-<service_name>.vhost using scp. Login to 72.249.39.81 and issue the commands

 scp root@72.249.74.249:/etc/conf/httpd/sites-enabled/002-service-user.vhost 002-service-user.vhost

scp root@72.249.74.249:/etc/conf/httpd/sites-enabled/003-service-content.vhost 003-service-content.vhost

scp root@72.249.74.249:/etc/conf/httpd/sites-enabled/004-service-org.vhost 004-service-org.vhost

 scp root@72.249.74.249:/etc/conf/httpd/sites-enabled/006-service-provision.vhost 006-service-provision.vhost

Now edit these files to set the correct ip addresses for the "Proxy Balancer" section. For the initial deployment everything will be pointing to Box 1. (You can add an entry for Box 4 and comment it out and it can be uncommented when Box 4 joins the cluster) So the entries for now should be:

BalancerMember http://72.249.21.180

#BalancerMember http://72.249.20.204

Also make the followign edits

DocumentRoot /var/www/apps/<service_name>/current/public

Directory "/var/www/apps/service_user/current/public"

Ensure that these changes are made for all virtual hosts in a file.

Also ensure that the service_user file also has a virtual host entry for SSL to port 443

 Database Setup and Replication

Now set MySQL Master/Slave databases as follows.

Box 1 will be configured as the master and Box 4 as the slave.

Before the Master Slave configuration allow remote connections to be made to the master as follows:

On the MySQL Server:

Edit the /etc/my.cnf file to set the mysqld section as follows.

[mysqld]
user=mysql
pid-file=/var/run/mysqld/mysqld.pid
socket=/var/lib/mysql/mysql.sock
port=3306
basedir=/usr
datadir=/var/lib/mysql
bind-address=72.249.21.180

Where the bind-address is the ip address of this MySQL server machine.

/etc/init.d/mysqld restart

Then go to mysql:

mysql -u root -p <password>

use mysql

Run the following commands specifying the remote ip address and each database that it needs to access.

update db set Host='<remote host ip address>' where Db='<db_name>';
update user set Host='<remote host ip address>' where user='root' and host = '72.249.21.180';

(The second statement should update one record)

flush privileges

If port 3306 is not open you will need to open it up using iptables using:

iptables -A INPUT -i eth0 -p tcp --destination-port 3306 -j ACCEPT

You can now test the remote connection from a remote machine using telnet and mysql as follows.

mysql -u root -h <mysql server ip address> -p

telnet <mysql server ip address> 3306

At this point also test that a mysql connection can be made from the localhost as well (i.e. on the local machine do a mysql -uroot -p and ensure that you can login and access databases) If a local connection cannot be made then you may need to run the following statements.

update user set Select_priv = 'Y', Insert_priv = 'Y', Update_priv = 'Y', Delete_priv = 'Y', Create_priv = 'Y', Drop_priv = 'Y', Reload_priv = 'Y', Shutdown_priv = 'Y', Process_priv = 'Y', File_priv = 'Y', Grant_priv = 'Y', References_priv = 'Y', Index_priv = 'Y', Alter_priv = 'Y', Show_db_priv = 'Y', Super_priv = 'Y', Create_tmp_table_priv = 'Y', Lock_tables_priv = 'Y', Execute_priv = 'Y', Repl_slave_priv = 'Y', Repl_client_priv = 'Y', Create_view_priv = 'Y', Show_view_priv = 'Y', Create_routine_priv = 'Y', Alter_routine_priv = 'Y', Create_user_priv = 'Y' where host = '72.249.21.180'

During the remote MySQL setup the following error may be encountered when connecting to mysql from the command prompt with mysql -uroot -p

Access denied for user 'root'@localhost

This can be remedied by resetting the password in the user table. This is detailed at  http://anojrs.blogspot.com/2007/11/access-denied-for-user-rootlocalhost.htm

mysqld --skip-grant-tables

mysql -u root mysql
UPDATE user SET Password=PASSWORD('my_password') where USER='root';
FLUSH PRIVILEGES;

/etc/init.d/mysql restart

This should reset the password.

 Add the following line to /usr/local/bin/mongrel_rails

ENV['PATH'] = "#{ENV['PATH']}:/usr/local/bin";

Setup up MySQL Master/Slave Replication

This section provides details on setting up master/slave replication for MySQL.

Master (1.bsgplatform.com)  ip address: 72.249.21.180

Slave (4.bsgplatform.com) ip address: 72.249.21.180

First take a backup of the master database using:

mysqldump -uroot -p --opt --single_transaction --database service_user_test > service_user_test_2008_10_08.sql

Create a new MySQL user called "replicator" and allow the slave to connect to it by running the following on the master.

grant replication slave on *.* to 'replicator'@'72.249.21.162' identified by 'slave';

On the Master machine edit the /etc/my.cnf file so that the mysqld section is set as follows. The last four lines (from log-bin) have been added for replication. The rest of the lines should be there already.

[mysqld]
user=mysql
pid-file=/var/run/mysqld/mysqld.pid
socket=/var/lib/mysql/mysql.sock
port=3306
basedir=/usr
datadir=/var/lib/mysql
bind-address=72.249.21.180
log-bin=mysql-bin
server-id=1
innodb_flush_log_at_trx_commit=1
sync_binlog=1

Now restart MySQL

/etc/init.d/mysqld restart 

 If mysql fails to restart then get the mysql process ids and kill them. This can be done as follows.

ps -ef | grep mysql

There should be two processes running mysqld_safe and mysqld. Kill them both using

kill -9 <pid>

On the Slave machine edit the /etc/my.cnf file and add the following line to the mysqld section.

server-id=4

Now take a backup of the Master databases that need to be replicated. This is done be stopping execution of statements on the master and recording the position of the log that the slave will use to perform replication.

flush tables with read lock;

show master status;

Record the "File" and "Position" values displayed by the command listed above. Keep this shell running so statements are blocked and open up a new sql window to perform a backup as follows.

mysqldump -uroot -p --databases service_content service_org service_provision  service_pubsub service_user --lock-all-tables > master_2008_10_09.sql

Remember to use the correct date when specifying the backup filename.

On the slave run the following statement. this will create the master.info file which is used by MySQL to configre the slave options (Note that if options are set and changed in my.cnf these changes will not be used, instead use the change master command below)

Remember to set the MASTER_LOG_FILE and MASTER_LOG_POS parameter values as recorded earlier from the master.

CHANGE MASTER TO
    MASTER_HOST='72.249.21.180',
    MASTER_USER='replicator',
    MASTER_PASSWORD='slave',
    MASTER_LOG_FILE='mysql-bin.000005',
    MASTER_LOG_POS=3323789;

Now on the slave start mysql with the --skip-slave-start option

/etc/init.d/mysqld start --skip-slave-start

On the slave copy the backup file from the master to the slave

scp root@72.249.21.180:/var/www/apps/database_backups/master_2008_10_09.sql master_2008_10_09.sql

Load the backup file into the slave

mysql -v -uroot -p < master_2008_10_09.sql

To ensure that only the required databases are replicated and the slave keeps running add the following entries in the master and slave my.cnf files for databases that need to be replicated.

Master my.cnf

binlog-do-db=service_content
binlog-do-db=service_org
binlog-do-db=service_provision
binlog-do-db=service_pubsub
binlog-do-db=service_user

Slave my.cnf

replicate-do-db=service_content
replicate-do-db=service_org
replicate-do-db=service_provision
replicate-do-db=service_pubsub
replicate-do-db=service_user

 Now restart mysqld on both the master and the slave

/etc/init.d/mysqld restart

Now start the slave by keying at the sql prompt:

start slave

Check the status of the slave by:

show slave status

The Slave_IO_Running and SLAVE_SQL_Running should both have values of "Yes"

Check the /var/log/mysqld.log for any errors. There may be errors due to databases that are in the master but not required to be replicated in the slave.

Also ensure that the master database server (72.249.21.180) has the correct version of rails installed. This is currently rails 2.1.1

Deploy Services

Deploy services on Box 1 initially.

Remove service folders from Box 1 in the /var/www/apps location. Folders for services_provision, service_user, service_content, service_org and service_pubsub should be removed.

Modify the databse.yml file for each service in deploy_platform/<service_name>/database.yml to ensure that the host in the production section points to the master database server:

host: 72.249.21.180

Modify the <service_name>/config/deploy.rb file for each service to set the variables for begin_port and mongrel_instances. These values should be set as follows:

service_content

set :begin_port, "8030"
set :mongrel_instances, "5"

service_org

set :begin_port, "8040"
set :mongrel_instances, "5"

service_provision

set :begin_port, "8010"
set :mongrel_instances, "10"

service_user

set :begin_port, "8020"
set :mongrel_instances, "10"

service_pubsub

set :begin_port, "8050"
set :mongrel_instances, "1"

Modify the deploy_platform/common.rb file to set the db and slave_db roles as:

role :db, "72.249.21.180", :primary => true

role :slave_db, "72.249.20.204"

On the local machine do the following to deploy:

cd \dev\deploy_platform

deploy_all

 Follow the prompts to finish the deployment

Configure Apache

On Box 1 go to the /etc/httpd/conf/sites-enabled folder and edit the vhost files for the services to ensure that all mongrel ports are specified in the "Proxy Balancer" section and the DocumentRoot and Directory parameters specify the new path i.e. /var/www/apps/<service_name>/current/public. Details are provided below.

On Box 1 create a backup of the /etc/httpd/conf/httpd.conf file.

cp httpd.conf httpd.conf.<date_time>

Edit the http.conf file to remove all the virtual host entries related to the services.

 On Box 1 create a folder /etc/httpd/conf/sites-enabled. Copy the files from Box 3 to Box 1 as these will be similar in configuration.

scp root@72.249.21.162:/etc/conf/httpd/sites-enabled/001-service-provision.vhost 001-service-provision.vhost

scp root@72.249.21.162:/etc/conf/httpd/sites-enabled/002-service-user.vhost 002-service-user.vhost

scp root@72.249.21.162:/etc/conf/httpd/sites-enabled/003-service-content.vhost 003-service-content.vhost

 scp root@72.249.21.162:/etc/conf/httpd/sites-enabled/004-service-org.vhost 004-service-org.vhost

Now edit each file and ensure that the "Proxy Cluster" section specifies the correct ports. The following ports should be set for each service.

service_provision (8010 - 8019) Running on 10 ports

service_user (8020 - 8029) Running on 10 ports

service_content (8030 - 8034) Running on 5 ports

service_org (8040 - 8044) Running on 5 ports

service_pubsub (8050 - 8054) Running on 5 ports (this may not be setup in this deployment)

Configure Monit

Now stop monit and ensure that no mongrel processes are running:

monit stop all

Check if any mongrel process is running using ps -ef | grep mongrel

kill all mongrel processes using

killall mongrel_rails

Create a new monitrc file using the monitrc.rb file provided in svn at /monitrc_generator/ by exporting it to Box 1 using svn.

mkdir /var/www/apps/monitrc_generator

cd  /var/www/apps/monitrc_generator

svn export svn://72.249.82.8/monitrc_generator/monitrc.rb monitrc.rb

ruby monitrc.rb

Follow the prompts to create a monitrc file. this file will provide entries based on the yml files in the /etc/mongrel_cluster folder. So please ensure that these files have correct entries before running the previous command.

This will create a file named monitrc in the current directory. Copy this file to Box 1 at /usr/local/etc

monit reload

monit start all

monit summary

monit status

service httpd restart (this should already have restarted when monit was started again)

Also on the Load Balancer Box restart apache

service httpd restart

 Now hit the urls specified in the Verification plan to test if everythig is working well.

For Box 4

Set the deployer user's password accordingly.

Verification Plan

This section lists some basic checks that should be performed to ensure that the deployment has gone according to the plan. This needs to be autoamted in the future.

Check the database.yml file for each service and ensure that it points to the master database.

Perform a "monit summary" or "monit status" to ensure that all processes are running.

Perform a ps -ef | grep mongrel to check that processes are running.

Kill a specific process using:

kill -9 <pid>

Ensure that it is automatically started

Once load balancing has been performed, open up the log files in both the 1 and 4 boxes at /var/log such as:

<service_name>_access_log

<service_name>_rewrite_log

And ensure that when a specific service is accessed a few times the log files on boxes 1 and 4 both get refreshed with subsequent requests i.e both the boxes are being hit and load balancing is being performed.

Listed below are some Urls that can be used to test some services.

service_organization

http://organization.ngenplatform.com/f_interestgroups?bsgra_guid=0ef2f140-bsga-serv-wik2-001b7744e275&auth_username=mschriftman@ngenera.com

service_provision

http://provision.ngenplatform.com/provisions/0ef2f140-bsga-serv-wiki-001b7744e275.xml

storage service (although this is not being deployed but it is present on box 4 and we should check that changes to that box have not effected it) Use username \: admin and no password

http://storageauth.ngenplatform.com/repository/default

http://storage2.ngenplatform.com/repository/default

http://storage2.ngenplatform.com:8080/repository/default/users/35702a08-5733-11dd-a0ef-0030488b5524/gadget_process.PNG

service_user

http://user.ngenplatform.com/accounts/login.html?SAMLRequest=eNplkEFPwzAMhf9Klfu6rICYIopU4MCkgaZtcOCCQuK2WRMnxG7Fz6caHJA4%2BfBsv%2Fe9G9LBJ9WM3OMePkcgLr6CR1JnoRZjRhU1OVKoA5Biow7N01ZVpVQpR44melE0RJDZRbyPSGOAfIA8OQMv%2B20teuaklsuTnjTH6AfHJXaAyWtuYw6liUGt5VqKYvNQC%2Bs7ewpR9yHB4MDgYNs09BjAhNYaCNZ2Kbaz6YZohA0Sa%2BRaVFKuF%2FJ6sbo8ri5UdaVk9SaK3W%2FEO4fWYfePp%2FzL8%2FGzROrxeNwt9mBdBsPnJ5OzkJ%2Fni1rMxSQP79qwm%2BYxNyeKV8g0088pSimWt9%2BeiXih&bsgra_guid=ccdfc894-f6bc-11dc-8f94-0030488b5524

service_content

Go to www.ngenera.com. Go to  My Page > My Personal Wiki. Edit and Save and retrieve it to ensure that the content service is working.

service_pubsub

http://util.ngenplatform.com/pubsub/sim/service.xml

Rollback Plan

The following will need to be ensured to conduct a rollback.

If database migrations are involved then run db migrate with the last version of the migrations such as:

rake db migrate -v=<version_number>

For each service go to the main folder i.e c:\dev\<service_name> and run the following command

cap deploy:rollback

On the server in the /usr/local/etc directory copy the latest monitrc_<date_time> file to monitrc

cp monitrc_<date_time> monitrc

Now kill all mongrel processes and restart them using:

killall mongrel_rails

monit reload

monit restart all

Go to the /etc/httpd/conf/sites-enabled folder and for each vhost file in there such as:

<nnn>-<service_name>.vhost

And ensure that the correct port numbers are defined in the "Proxy Balancer" section.

Also ensure that directory locations for each service are updarted to the new format i.e. /var/www/apps/<service_name>/current