Right at the outset, an honest confession - I am no good at shell script. When I tried to deploy my play2.0 - scala application in cloudfoundry - and connect to the provisioned database service - I faced the difficult task of parsing a JSON string(cloudfoundry provides details of the services in JSON format) - extract the database credentials - and set them as environment variables so that when I start my play2.0 netty server - the server can read those environment variables and connect to the database. Following snippet shows a sample database related info that cloudfoundry provides as environment variable:
{"postgresql-9.0":[{"name":"database","label":"postgresql-9.0","plan":"free","tags":["postgresql","postgresql-9.0","relational"],"credentials":{"name":"d59efd73d7a0d458da6b156ea0ae67b6b","host":"172.30.48.124","hostname":"172.30.48.124","port":5432,"user":"u6dacee22b17d4c9d8e4aa832b7cc2947","username":"u6dacee22b17d4c9d8e4aa832b7cc2947","password":"p9cfd7252af8d4af4beb3815b38cf184e"}}]}
So, I had basically two options before me:
a) I write a shell script to parse the JSON script and extract the relevant data
b) Or I change the netty server start-up code itself - so that when launched - it reads the VCAP_SERVICES JSON string - takes out the relevant database credentials and connects to the provisioned database instance.
So, I started on my venture to find a nice little shell script parses a JSON string and dishes out the database settings when asked for. Googling gave me some scripts - but they were not very helpful - either they were too lousy or too specific to some other tasks - and I dared not to modify them for the obvious reason as I have mentioned at the outset - I am not good at shell script.
So, I decided to follow my second option. I got the play2.0 source code from github repository - modified the netty server scala code - so that it calls some other piece of scala code(which does the JSON parsing - and sets the database specific details nice and cool) - and does what it normally does. Following snippet shows the piece of code that does JSON parsing and environment variable setting.
package play.core.server
import play.api.libs.json._
import play.api.libs.json.Json._
import play.api.libs.json.Json
import java.util.{Map=>JMap}
import java.util._
import java.lang.reflect.Field
object VCAPJsonParser {
def init_env(): Unit = {
val vcap_services = System.getenv("VCAP_SERVICES")
if(vcap_services == null || "" == vcap_services){
println("vcap_services env value is not set!")
()
}else {
println("The VCAP_SERVICES json : "+ vcap_services)
val services: JsValue = Json.parse(vcap_services)
val postgresql = services \ "postgresql-9.0"
val credentials = postgresql(0) \ "credentials"
println("The credentials json : "+ credentials)
val dbname = (credentials \ "name").as[String]
val hostname = (credentials \ "hostname").as[String]
val user = (credentials \ "user").as[String]
val password = (credentials \ "password").as[String]
val port = (credentials \ "port").as[Int]
val database = "jdbc:postgresql://" +hostname+":"+port+"/" +dbname
println("database url : "+ database + ", user : "+ user+ ", password : "+ password)
val envObj = System.getenv()
val env = envObj.asInstanceOf[JMap[String,String]]
var newEnv: JMap[String,String] = new java.util.HashMap()
newEnv.putAll(env)
newEnv.put("postgres_database",database)
newEnv.put("postgres_dbuser",user)
newEnv.put("postgres_password",password)
setEnv(newEnv)
println("Env set: "+ System.getenv())
println("******************")
println("Database url : "+ System.getenv("postgres_database")+", user : "+ System.getenv("postgres_dbuser")+", password : "+ System.getenv("postgres_password"))
()
}
}
def main(args: Array[String]): Unit = {
init_env
}
def setEnv(newenv: JMap[String,String]): Unit = {
try {
val processEnvironmentClass : Class[_] = Class.forName("java.lang.ProcessEnvironment")
val theEnvironmentField : Field = processEnvironmentClass.getDeclaredField("theEnvironment")
theEnvironmentField.setAccessible(true)
val env: JMap[String,String] = (theEnvironmentField.get(null)).asInstanceOf[JMap[String,String]]
env.putAll(newenv)
val theCaseInsensitiveEnvironmentField: Field = processEnvironmentClass.getDeclaredField("theCaseInsensitiveEnvironment")
theCaseInsensitiveEnvironmentField.setAccessible(true)
val cienv: JMap[String,String] =(theCaseInsensitiveEnvironmentField.get(null)).asInstanceOf[JMap[String,String]]
cienv.putAll(newenv)
}catch {
case e1: NoSuchFieldException => try {
val classes: Array[Class[_]] = (classOf[Collections]).getDeclaredClasses()
val env: JMap[String,String] = System.getenv()
for(cl <- classes){
if("java.util.Collections$UnmodifiableMap".equals(cl.getName())) {
val field: Field = cl.getDeclaredField("m")
field.setAccessible(true)
val obj = field.get(env)
val map = obj.asInstanceOf[JMap[String,String]]
map.clear()
map.putAll(newenv)
}
}
}catch {
case e11: Exception => e11.printStackTrace(System.err)
}
case e2: Exception => e2.printStackTrace(System.err)
}
}
}
You can put this piece of code in the proper directory - call it from the main method of the "NettyServer.scala" and build the play framework from the source - and then push your application to cloudfoundry - and it will faithfully connect to your database instance.
I have done this - this is how my application(ooki.cloudfoundry.com) was running till I wrote this post.
I was not very happy to modify the netty server code just to read database specific details. Hence I rolled up my sleeves - And decided to give the shell script
approach another go. Following is shell script - that I ended up writing. Needless to say - there must be more efficient way of writing it - but it does the job currently. You are welcome to modify and make it more generic.
#!/usr/bin/env bash
function processVCAP_SERVICE_JSON_and_setDBenv(){
echo "*******************"
echo "$VCAP_SERVICES"
echo "*******************"
db_settings=()
counter=0
found="n"
IFS=',' read -ra SERVICE_SETTINGS <<< "$VCAP_SERVICES"
length="${#SERVICE_SETTINGS[@]}"
for ((i=0; i<${length}; i++ ));
do
curr_item=${SERVICE_SETTINGS[$i]}
curr_item="$(echo ${curr_item} | tr -d '\"')"
echo "position : "+ "$i" + "and item is : "+ "$curr_item"
#Take care of password:pbd0e3f367a6c4a36bcc48a3c763a929e}}]} <--
if [[ "$curr_item" == password* ]]; then
pwd_len="${#curr_item}"
pwd_len=$pwd_len-4
curr_item="${curr_item:0:$pwd_len}"
fi
if [[ "$found" = "y" ]] || [[ "$curr_item" == credentials* ]]; then
if [ "$found" = "n" ]; then
#get rid of 'credentials:{' part
len="${#curr_item}"
len=$len-1
db_settings[counter]="${curr_item:13:$len}"
else
db_settings[counter]="${curr_item}"
fi
counter=$counter+1
found="y"
echo counter is : "{$counter}"
fi
done
echo "ratul your db settings are : "
echo "${#db_settings}"
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
#prepare the database_url,database_user & database_password and set them in env
database_url=jdbc:postgresql://
#change above in case of MySQL
db_name=
host_name=
port_num=
database_user=
database_password=
echo "============================================="
for setting in "${db_settings[@]}"
do
echo ##############"${#setting}"
set_len="${#setting}"
set_len=$set_len-1;
echo "$setting and setting length=$set_len"
case "$setting" in
name:*) db_name="${setting:5:$set_len}";
echo "DB name : $db_name" ;;
hostname:*) host_name="${setting:9:$set_len}";
echo "DB host : $host_name" ;;
port:*) port_num="${setting:5:$set_len}";
echo "DB port : $port_num" ;;
username:*) database_user="${setting:9:$set_len}";
echo "DB user : $database_user" ;;
password:*) database_password="${setting:9:$set_len}";
echo "DB password : $database_password" ;;
esac
done
database_url=$database_url$host_name:$port_num/$db_name
export database_url=$database_url
export database_password=$database_password
export database_user=$database_user
}
processVCAP_SERVICE_JSON_and_setDBenv
echo "database url : $database_url"
echo "password : $database_password"
echo "user : $database_user"
exec java $JAVA_OPTS -Dhttp.port=$VCAP_APP_PORT -DapplyEvolutions.default=true -cp "`dirname $0`/lib/*" play.core.server.NettyServer `dirname $0`
Be sure to set environment variables accordingly in application.conf like so:
db.default.driver=org.postgresql.Driver
db.default.url=${database_url}
db.default.user=${database_user}
db.default.password=${database_password}
{"postgresql-9.0":[{"name":"database","label":"postgresql-9.0","plan":"free","tags":["postgresql","postgresql-9.0","relational"],"credentials":{"name":"d59efd73d7a0d458da6b156ea0ae67b6b","host":"172.30.48.124","hostname":"172.30.48.124","port":5432,"user":"u6dacee22b17d4c9d8e4aa832b7cc2947","username":"u6dacee22b17d4c9d8e4aa832b7cc2947","password":"p9cfd7252af8d4af4beb3815b38cf184e"}}]}
So, I had basically two options before me:
a) I write a shell script to parse the JSON script and extract the relevant data
b) Or I change the netty server start-up code itself - so that when launched - it reads the VCAP_SERVICES JSON string - takes out the relevant database credentials and connects to the provisioned database instance.
So, I started on my venture to find a nice little shell script parses a JSON string and dishes out the database settings when asked for. Googling gave me some scripts - but they were not very helpful - either they were too lousy or too specific to some other tasks - and I dared not to modify them for the obvious reason as I have mentioned at the outset - I am not good at shell script.
So, I decided to follow my second option. I got the play2.0 source code from github repository - modified the netty server scala code - so that it calls some other piece of scala code(which does the JSON parsing - and sets the database specific details nice and cool) - and does what it normally does. Following snippet shows the piece of code that does JSON parsing and environment variable setting.
package play.core.server
import play.api.libs.json._
import play.api.libs.json.Json._
import play.api.libs.json.Json
import java.util.{Map=>JMap}
import java.util._
import java.lang.reflect.Field
object VCAPJsonParser {
def init_env(): Unit = {
val vcap_services = System.getenv("VCAP_SERVICES")
if(vcap_services == null || "" == vcap_services){
println("vcap_services env value is not set!")
()
}else {
println("The VCAP_SERVICES json : "+ vcap_services)
val services: JsValue = Json.parse(vcap_services)
val postgresql = services \ "postgresql-9.0"
val credentials = postgresql(0) \ "credentials"
println("The credentials json : "+ credentials)
val dbname = (credentials \ "name").as[String]
val hostname = (credentials \ "hostname").as[String]
val user = (credentials \ "user").as[String]
val password = (credentials \ "password").as[String]
val port = (credentials \ "port").as[Int]
val database = "jdbc:postgresql://" +hostname+":"+port+"/" +dbname
println("database url : "+ database + ", user : "+ user+ ", password : "+ password)
val envObj = System.getenv()
val env = envObj.asInstanceOf[JMap[String,String]]
var newEnv: JMap[String,String] = new java.util.HashMap()
newEnv.putAll(env)
newEnv.put("postgres_database",database)
newEnv.put("postgres_dbuser",user)
newEnv.put("postgres_password",password)
setEnv(newEnv)
println("Env set: "+ System.getenv())
println("******************")
println("Database url : "+ System.getenv("postgres_database")+", user : "+ System.getenv("postgres_dbuser")+", password : "+ System.getenv("postgres_password"))
()
}
}
def main(args: Array[String]): Unit = {
init_env
}
def setEnv(newenv: JMap[String,String]): Unit = {
try {
val processEnvironmentClass : Class[_] = Class.forName("java.lang.ProcessEnvironment")
val theEnvironmentField : Field = processEnvironmentClass.getDeclaredField("theEnvironment")
theEnvironmentField.setAccessible(true)
val env: JMap[String,String] = (theEnvironmentField.get(null)).asInstanceOf[JMap[String,String]]
env.putAll(newenv)
val theCaseInsensitiveEnvironmentField: Field = processEnvironmentClass.getDeclaredField("theCaseInsensitiveEnvironment")
theCaseInsensitiveEnvironmentField.setAccessible(true)
val cienv: JMap[String,String] =(theCaseInsensitiveEnvironmentField.get(null)).asInstanceOf[JMap[String,String]]
cienv.putAll(newenv)
}catch {
case e1: NoSuchFieldException => try {
val classes: Array[Class[_]] = (classOf[Collections]).getDeclaredClasses()
val env: JMap[String,String] = System.getenv()
for(cl <- classes){
if("java.util.Collections$UnmodifiableMap".equals(cl.getName())) {
val field: Field = cl.getDeclaredField("m")
field.setAccessible(true)
val obj = field.get(env)
val map = obj.asInstanceOf[JMap[String,String]]
map.clear()
map.putAll(newenv)
}
}
}catch {
case e11: Exception => e11.printStackTrace(System.err)
}
case e2: Exception => e2.printStackTrace(System.err)
}
}
}
You can put this piece of code in the proper directory - call it from the main method of the "NettyServer.scala" and build the play framework from the source - and then push your application to cloudfoundry - and it will faithfully connect to your database instance.
I have done this - this is how my application(ooki.cloudfoundry.com) was running till I wrote this post.
I was not very happy to modify the netty server code just to read database specific details. Hence I rolled up my sleeves - And decided to give the shell script
approach another go. Following is shell script - that I ended up writing. Needless to say - there must be more efficient way of writing it - but it does the job currently. You are welcome to modify and make it more generic.
#!/usr/bin/env bash
function processVCAP_SERVICE_JSON_and_setDBenv(){
echo "*******************"
echo "$VCAP_SERVICES"
echo "*******************"
db_settings=()
counter=0
found="n"
IFS=',' read -ra SERVICE_SETTINGS <<< "$VCAP_SERVICES"
length="${#SERVICE_SETTINGS[@]}"
for ((i=0; i<${length}; i++ ));
do
curr_item=${SERVICE_SETTINGS[$i]}
curr_item="$(echo ${curr_item} | tr -d '\"')"
echo "position : "+ "$i" + "and item is : "+ "$curr_item"
#Take care of password:pbd0e3f367a6c4a36bcc48a3c763a929e}}]} <--
if [[ "$curr_item" == password* ]]; then
pwd_len="${#curr_item}"
pwd_len=$pwd_len-4
curr_item="${curr_item:0:$pwd_len}"
fi
if [[ "$found" = "y" ]] || [[ "$curr_item" == credentials* ]]; then
if [ "$found" = "n" ]; then
#get rid of 'credentials:{' part
len="${#curr_item}"
len=$len-1
db_settings[counter]="${curr_item:13:$len}"
else
db_settings[counter]="${curr_item}"
fi
counter=$counter+1
found="y"
echo counter is : "{$counter}"
fi
done
echo "ratul your db settings are : "
echo "${#db_settings}"
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
#prepare the database_url,database_user & database_password and set them in env
database_url=jdbc:postgresql://
#change above in case of MySQL
db_name=
host_name=
port_num=
database_user=
database_password=
echo "============================================="
for setting in "${db_settings[@]}"
do
echo ##############"${#setting}"
set_len="${#setting}"
set_len=$set_len-1;
echo "$setting and setting length=$set_len"
case "$setting" in
name:*) db_name="${setting:5:$set_len}";
echo "DB name : $db_name" ;;
hostname:*) host_name="${setting:9:$set_len}";
echo "DB host : $host_name" ;;
port:*) port_num="${setting:5:$set_len}";
echo "DB port : $port_num" ;;
username:*) database_user="${setting:9:$set_len}";
echo "DB user : $database_user" ;;
password:*) database_password="${setting:9:$set_len}";
echo "DB password : $database_password" ;;
esac
done
database_url=$database_url$host_name:$port_num/$db_name
export database_url=$database_url
export database_password=$database_password
export database_user=$database_user
}
processVCAP_SERVICE_JSON_and_setDBenv
echo "database url : $database_url"
echo "password : $database_password"
echo "user : $database_user"
exec java $JAVA_OPTS -Dhttp.port=$VCAP_APP_PORT -DapplyEvolutions.default=true -cp "`dirname $0`/lib/*" play.core.server.NettyServer `dirname $0`
Be sure to set environment variables accordingly in application.conf like so:
db.default.driver=org.postgresql.Driver
db.default.url=${database_url}
db.default.user=${database_user}
db.default.password=${database_password}