Showing posts with label Database. Show all posts
Showing posts with label Database. Show all posts

Tuesday, 25 December 2018

MongoDB Atlas - Off Premise Way (DBaaS)


MongoDb also provide cloud services (Database as a Service - DaaS), called MongoDB Atlas, to host your mongo database on cloud. Let's see how we can setup an cloud account and access the MDB from local machine.

Cluster Step :
1. Create an account on https://cloud.mongodb.com
2. Once you are in, the very first thing which it asked to choose your cluster configuration.
2a. It gives you to choose one of cloud service which are - AWS, Google, Azure, Choose whatever you like
2b. But always choose "FREE TIER CLUSTER" (M0 Instance) else there will be usage charge.
3. Once you have selected appropriate config, it will start building your mongoDB Cluster, it will take few mins to complete the setup.
4. When done, it will be like this, usually use Cluster0 as name, you can modify it though -




How to access from local system :

You need to install Mongo Shell to access cloud db which comes with Mongo DB pkg. You can download and install on your OS (Windows/Linux) from here - https://www.mongodb.com/download-center/enterprise

1. Login on https://cloud.mongodb.com and click on Clusters in left hand side list.
2. Click on Connect and follow below steps -
3a. Whitelist your id so that you can connect with your system or any ip address. Click on "Add a different IP Address" and to allow to connect from any system, Use 0.0.0.0 as IP address
3b. Create your cluster credential


4. Once done, you will see the below screen
5a. Now, Click on Choose connection method and click on "Connect with Mongo Shell" -


5b. Now, click on standard connection string


6. Copy the string and replace the <PASSWORD> with the password which you created in Cluster Setup Step #4.
7. Now, As I have installed the MongoDB Shell in Step #1, we need to add MongoDB Bin directory path into system path. You can add this path into windows env variable or Linux user profile so that you can access mongo command from any location.
8. Once path has been added, open cmd or terminal and paste the connection string which you copied and modified in step #5


9. When connected successfully, you can try to run commands as below -


10. For more commands, you can visit this link - https://www.datagenx.net/2018/12/learn-mongo-db-basics.html

11. Mongo Atlas Cloud Step has been completed and verified successfully. You can connect with the same connection string from any system (if firewall allows and have mongo shell installed)

Let me know in comments if you face any issue while doing Atlas setup.
Next Post on this Series and more on MongoDB can be find here -> LINK




Like the below page to get the update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     


Saturday, 23 June 2018

yet another In Memory DataBase - MemSQL


Writing this post after so many days, yet another IN-MEMORY database is in market which tag line promised with"The DataBase for Real-Time Applications". 

As per MemSQL site - MemSQL is a scalable SQL database that ingests data continuously to perform operational analytics for the front lines of your business. Ingest millions of events per day with ACID transactions while simultaneously analyzing billions of rows of data in relational SQL, JSON, or Geospatial formats.

In my current assignment, I've been asked to look into the capabilities of this db, so starting with very first step "Installation" - 

Installation in linux is quite simple, if you are OK with linux commands, You can follow the installation from HERE

1. Download the software (with sudo or root user) - 

sudo su - root
wget http://download.memsql.com/memsql-ops-6.0.11/memsql-ops-6.0.11.tar.gz

2. Extract the tar ball

tar zxvf memsql-ops-6.0.11.tar.gz

This command will extract lots n lots of files :-)

3. Run the installer script

cd memsql-ops-6.0.11

sudo ./install.sh --simple-cluster



By default, MemSQL supports the machine with 4 cpu core and 8 GB of RAM (which is little unfair;-)) so remove this constraints by below argument -

cd memsql-ops-6.0.11

sudo ./install.sh --simple-cluster --ignore-min-requirements


After being successful installation of MemSQL, it will start setting up MemSQL WebUI.



You can access the MemSQL WebUI on the sever's 9000 port by default. 

https://<SERVER_IP>:9000

4. To connect to MemSQL command line, execute - 

memsql


In next post, I will explain how this db is different than other in-memory db. Till then, Keep Learning , Keep Loving.




Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     


Sunday, 14 January 2018

Mongo DB - Installation and Configuration


MongoDB  is an open-source document database, and the leading NoSQL database. Written in C++.
  
MongoDB features:
    Document-Oriented Storage
    Full Index Support
    Replication & High Availability
    Auto-Sharding
    Querying
    Fast In-Place Updates
    Map/Reduce
    GridFS


Reduce cost, accelerate time to market, and mitigate risk with proactive support and enterprise-grade capabilities.


Today, We will see how to install and run the MongoDB.

MongoDB Installation on Linux


1. DOWNLOAD the stable version of MongoDB. It will a tar file
2. Extract the tar file to some directory.
 
$ tar -xvf mongodb.tar -C /learn/mongodb


3.  change the permisson of folder to user who run the db here-  In my case User - hduser and Group - hadoop
$ chown -R hduser:hadoop /learn/mongodb

4. Add the env var in .bashrc
export MONGO_HOME=/learn/mongodb
export PATH=$PATH:$MONGO_HOME/bin







5. Create the default DB directory for Mongo
$ mkdir -R /data/db
$ chown -R hduser:hadoop /data/db

This is by default, you can specify ur db path when starting the mongo db






$ mongod --dbpath /app/mongodata
this command will start the mongodb. in other terminal you can start work on db. "--dbpath /app/mongodata" is totally optional

If you just use just $ mongod , it will start n use the default db which we have defined in step 5.


Please don't close the current terminal, It can be kill the mongodb process.







6. Start working on MongoDB
$ mongo










Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Friday, 7 July 2017

ICONV mystery - the UV function


Iconv (Internal CONVersion) is a function supported by UniVerse DB (UV db) to convert the DATA, not only DATE, into internal format. DataStage Server Jobs are using lots of UV functions to manipulate the data.

Today, I will try to unwrap the mystery behind the Iconv function and try to put the details in simpler words. We will not go into data conversion details but date conversion which is used by DataStage :-)

Like most of other date functions (Parallel one), Iconv also accept the date(string) and its format.

Suppose, Date =   June 17, 2017

To Convert this date into internal format, we have to use -

Iconv("2017-06-17", D-YMD)  = 18066
Iconv("2017/06/17", D/YMD)   = 18066
Iconv("2017:17:06", D:YDM)  = 18066
Iconv("17-06-17", D-Y2MD)    = 18066



D-  --> D for Delimiter followed by delimiter char
Y --> year in YYYY
M --> month in MM
D --> date in DD

As we can see, if we provide the date format with date string, Iconv convert the date to an integer no and it is very important to do because now datastage can understand the given date and we can use Oconv function to re-format the date as required.

I will cover Oconv in next post, till then Keep Learning !!




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Thursday, 21 July 2016

DBMS Books



If you dont have the dropbox account, create from here - https://db.tt/H5VKcNA0

Database System Concepts 6E - Korth               Book Link and PPTs Link  
Database System 6E_Navathe - Navathe            Book Link and PPTs Link    





** The books and its content belongs only to its author and publication, Shared here for personal learning purpose. 



Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Thursday, 19 May 2016

5 Tips For Better DataStage Design #13



1. The query used in the database should be in such a way that required number of rows are fetched. Do not extract the columns which are not required.

2. For parallel jobs, sequential File should not be read using same partitioning.


http://www.datagenx.net/2016/05/5-tips-for-better-datastage-design-13.html


3. For huge amount of data, use of sequential file stage is not a good practice. This stage also should not be used for intermediate storage between jobs. It degrades the performance of the job.

4. The number of lookups in a job design should be minimum. Join stage is a good alternative to lookup stage.

5. For parallel jobs, proper portioning method is to be used for better job performance and accurate flow of data.





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Tuesday, 15 December 2015

How to use Universe Shell (uvsh) in DataStage?


In DataStage Administration, we have to use datastage command line (universe shell) to get the information directly from the datastage universe database.

While accessing it from command line what novice admin do is -

$ uvsh
This directory is not set up for DataStage.
Would you like to set it up(Y/N)?   
Confused ? What to do ?

Always answer that question "no", it means you're in the wrong place.
Always launch "uvsh" or "dssh" from one of two places - $DSHOME or inside a project directory. For the latter you're good to go, for the former you'll need to LOGTO your project name before you issue any sql.



How to use UVSH?

## Entered into the $DSHOME
$ cd $DSHOME

## Sourced the dsenv file
$ . dsenv

## Change directory to the project directory.
$ LOGTO <project_name>

## Run uvsh command 
$ $DSHOME/bin/uvsh

Many Datastage admin support to execute command from Datastage Administrator or use dssh instead of uvsh.

How to use DSSH?
## Sourced the dsenv file
$ . $DSHOME/dsenv

## Change directory to the project directory.
$ LOGTO <project_name>

## Run dssh command 
$ $DSHOME/bin/dssh



Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://groups.google.com/forum/#!forum/datagenx