MongoDB Clustering for CentOS 7

by ·

A quick guide to installing a MongoDB cluster on CentOS 7 hosts.

Network Configuration

10.0.10.1 config01
10.0.20.1 shard01
10.0.20.2 shard02
10.0.30.1 app01

Install MongoDB

First make sure the system is up to date and install the MongoDB binaries via the Mongo yum repository:

# update via yum
yum update -y

# install the mongodb repository
cat <<REPO > /etc/yum.repos.d/mongodb.repo
[mongodb]
name=MongoDB Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
gpgcheck=0
enabled=1
REPO

# install mongodb 
yum install -y mongodb-org

# change the block side - NOTE: use the /dev that hosts the mongo data
blockdev --setra 256 /dev/xvda1

# disable access times
sed -rn 's/( \/ \s*(ext[[:digit:]]|xfs)\s*defaults) /\1,noatime /p' -i /etc/fstab

# remount
mount -o remount /

Configure MongoDB Configuration Server

# mongod.conf: config server(s)

# use as config server, changes port to 27019
configsvr=true

# data directory
dbpath=/var/lib/mongo

####### STANDARD CONFIG #######

# fork and run in background
fork=true

# location of pidfile
pidfilepath=/var/run/mongodb/mongod.pid

# where to log
logpath=/var/log/mongodb/mongod.log

# append to log, don't create a new one on restart
logappend=true

Configure MongoDB Nodes

# mongod.conf: shard server(s)

# shard server, changes the port number to 27018
shardsvr=true

# data directory
dbpath=/var/lib/mongo

####### STANDARD CONFIG #######

# fork and run in background
fork=true

# location of pidfile
pidfilepath=/var/run/mongodb/mongod.pid

# where to log
logpath=/var/log/mongodb/mongod.log

# append to log, don't create a new one on restart
logappend=true

Configure MongoDB Query Router(s)

# mongos.conf: query server(s)

# config server IP and port, comma delimited if there is more than one
#configdb=config01:27019,config02:27019,...
configdb=config01:27019

####### STANDARD CONFIG #######

# fork and run in background
fork=true

# location of pidfile
pidfilepath=/var/run/mongodb/mongos.pid

# where to log
logpath=/var/log/mongodb/mongos.log

# append to log, don't create a new one on restart
logappend=true

We need to make a few more changes to make sure that the Query Router uses mongos – not mongod – when the MongoDB service starts. We also never want this server to start as a MongoDB node by accident, so we will rename the initialization script and make a few edits.

mv /etc/init.d/mongod /etc/init.d/mongos
vi /etc/init.d/mongos

In the mongos script you want to look for and replace the following lines

# Old - uses mongod.conf
CONFIGFILE="/etc/mongod.conf"

# New - uses mongos.conf
CONFIGFILE="/etc/mongos.conf"


# Old - uses mongod
SYSCONFIG="/etc/sysconfig/mongod"

# New - uses mongos
SYSCONFIG="/etc/sysconfig/mongos"

# replace in vi/vim with:
:%s/sysconfig\/mongod/sysconfig\/mongos/g

Below this I placed a new variable for the lockfile and updated all the instances of the hard coded values.

# replace in vi/vim with:
:%s/\var\/lock\/subsys\/mongod/"$LOCKFILE"/g

# now add this, so the variable value isn't overwritten by the regex above
LOCKFILE="/var/lock/subsys/mongos"

Next we want to change the $mongod to $mongos.

# Old - uses mongod
mongod=${MONGOD-/usr/bin/mongod}

# New - uses mongos
mongos=${MONGOD-/usr/bin/mongos}

# replace in vi/vim using
:%s/\$mongod/$mongos/g

# replace the status text in vi/vim with:
:%s/mongod: "/mongos: "/g

Leave the MONGO_USER and MONGO_GROUP as mongod to prevent permissions issues.

There is also a bug in the script where the path to the binary is used directly instead of the variable for stop), called by service mongod stop, so we need to fix it as well.

# Old - uses path
killproc -p "$PIDFILE" -d 300 /usr/bin/mongod

# New - uses variable
killproc -p "$PIDFILE" -d 300 $mongos

# replace the status text in vi/vim with:
:%s/\/usr/bin\/mongod/$mongos/g

The full customized mongos startup script follows.

#!/bin/bash

# mongos - Startup script for mongos

# chkconfig: 35 85 15
# description: Mongo is a scalable, document-oriented database.
# processname: mongos
# config: /etc/mongos.conf
# pidfile: /var/run/mongodb/mongos.pid

. /etc/rc.d/init.d/functions

# things from mongos.conf get there by mongos reading it

# NOTE: if you change any OPTIONS here, you get what you pay for:
# this script assumes all options are in the config file.
CONFIGFILE="/etc/mongos.conf"
OPTIONS=" -f $CONFIGFILE"
SYSCONFIG="/etc/sysconfig/mongos"
LOCKFILE="/var/lock/subsys/mongos"

# FIXME: 1.9.x has a --shutdown flag that parses the config file and
# shuts down the correct running pid, but that's unavailable in 1.8
# for now.  This can go away when this script stops supporting 1.8.
DBPATH=`awk -F'[:=]' -v IGNORECASE=1 '/^[[:blank:]]*dbpath[[:blank:]]*[:=][[:blank:]]*/{print $2}' "$CONFIGFILE" | tr -d '[:blank:]'`
PIDFILE=`awk -F'[:=]' -v IGNORECASE=1 '/^[[:blank:]]*pidfilepath[[:blank:]]*[:=][[:blank:]]*/{print $2}' "$CONFIGFILE" | tr -d '[:blank:]'`
PIDDIR=`dirname $PIDFILE`

mongos=${MONGOD-/usr/bin/mongos}

MONGO_USER=mongod
MONGO_GROUP=mongod

if [ -f "$SYSCONFIG" ]; then
    . "$SYSCONFIG"
fi

# Handle NUMA access to CPUs (SERVER-3574)
# This verifies the existence of numactl as well as testing that the command works
NUMACTL_ARGS="--interleave=all"
if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null
then
    NUMACTL="numactl $NUMACTL_ARGS"
else
    NUMACTL=""
fi

start()
{
  # Make sure the default pidfile directory exists
  if [ ! -d $PIDDIR ]; then
    install -d -m 0755 -o $MONGO_USER -g $MONGO_GROUP $PIDDIR
  fi

  # Recommended ulimit values for mongod or mongos
  # See http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
  #
  ulimit -f unlimited
  ulimit -t unlimited
  ulimit -v unlimited
  ulimit -n 64000
  ulimit -m unlimited
  ulimit -u 64000

  echo -n $"Starting mongos: "
  daemon --user "$MONGO_USER" --check $mongos "$NUMACTL $mongos $OPTIONS >/dev/null 2>&1"
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && touch "$LOCKFILE"
}

stop()
{
  echo -n $"Stopping mongos: "
  killproc -p "$PIDFILE" -d 300 $mongos
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && rm -f "$LOCKFILE"
}

restart () {
        stop
        start
}


RETVAL=0

case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart|reload|force-reload)
    restart
    ;;
  condrestart)
    [ -f "$LOCKFILE" ] && restart || :
    ;;
  status)
    status $mongos
    RETVAL=$?
    ;;
  *)
    echo "Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
    RETVAL=1
esac

exit $RETVAL

Now we’re ready to add the shards, which we do from the Query Router. So either log on to MongoDB using mongo from the command line, or from a different server use mongo --host app01.

use admin;
db.runCommand( { addShard: "shard01:27018", name: "shard01" } );
{ "shardAdded" : "shard01", "ok" : 1 }
db.runCommand( { addShard: "shard02:27018", name: "shard02" } );
{ "shardAdded" : "shard02", "ok" : 1 }


db.runCommand( { listShards: 1 } );
{
	"shards" : [
		{
			"_id" : "shard01",
			"host" : "shard01:27018"
		},
		{
			"_id" : "shard02",
			"host" : "shard02:27018"
		}
	],
	"ok" : 1
}


use sharded_db;

sh.enableSharding( "sharded_db" );
{ "ok" : 1 }

db.sharded_db.ensureIndex( { _id : "hashed" } );
{
	"raw" : {
		"shard01:27018" : {
			"createdCollectionAutomatically" : true,
			"numIndexesBefore" : 1,
			"numIndexesAfter" : 2,
			"ok" : 1
		}
	},
	"ok" : 1
}

sh.shardCollection( "sharded_db.sharded_collection", { "_id": "hashed" } );
{ "collectionsharded" : "sharded_db.sharded_collection", "ok" : 1 }


for ( var i=0; i<=100; i++ ) db.sharded_collection.insert( { x : i } );
WriteResult({ "nInserted" : 1 })

db.sharded_collection.find()
{ "_id" : ObjectId("571ac1ded785cd0adda63b59"), "x" : 1 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b5b"), "x" : 3 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b5a"), "x" : 2 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b5d"), "x" : 5 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b5c"), "x" : 4 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b5e"), "x" : 6 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b60"), "x" : 8 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b5f"), "x" : 7 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b62"), "x" : 10 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b61"), "x" : 9 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b69"), "x" : 17 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b63"), "x" : 11 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b6a"), "x" : 18 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b64"), "x" : 12 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b6d"), "x" : 21 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b65"), "x" : 13 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b6e"), "x" : 22 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b66"), "x" : 14 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b70"), "x" : 24 }
{ "_id" : ObjectId("571ac1ded785cd0adda63b67"), "x" : 15 }
Type "it" for more


sh.status();
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"version" : 4,
	"minCompatibleVersion" : 4,
	"currentVersion" : 5,
	"clusterId" : ObjectId("571a019ce3c89d90cb6e1715")
  }
  shards:
	{  "_id" : "shard01",  "host" : "shard01:27018" }
	{  "_id" : "shard02",  "host" : "shard02:27018" }
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "sharded_db",  "partitioned" : true,  "primary" : "shard01" }
		sharded_db.sharded_collection
			shard key: { "_id" : "hashed" }
			chunks:
				shard01	2
				shard02	2
			{ "_id" : { "$minKey" : 1 } } --&gt;&gt; { "_id" : NumberLong("-4611686018427387902") } on : shard01 Timestamp(2, 2) 
			{ "_id" : NumberLong("-4611686018427387902") } --&gt;&gt; { "_id" : NumberLong(0) } on : shard01 Timestamp(2, 3) 
			{ "_id" : NumberLong(0) } --&gt;&gt; { "_id" : NumberLong("4611686018427387902") } on : shard02 Timestamp(2, 4) 
			{ "_id" : NumberLong("4611686018427387902") } --&gt;&gt; { "_id" : { "$maxKey" : 1 } } on : shard02 Timestamp(2, 5) 

db.dropDatabase();
{ "dropped" : "sharded_db", "ok" : 1 }

Tags:

You may also like...

1 Response

  1. Vikrant says:
    April 23, 2019 at 3:40 am

    Nice tutorial but there is no /etc/init.d/mongod file available at least for the latest version 4.0. Can you provide a workaround for that? I already tried the same method to create mongos.service in “/usr/lib/systemd/system” but that did not work as I have to keep the connection open.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *