Purging SQL Table without impacting the users

This happens very rarely but sometime I have been asked to conditional purge the SQL table without impacting the Users (So we can not use the truncate which is quick and harsh  :) ), These kind of request is pain when client says this table is business critical and used by different application 24/7 so can not take offline and delete the data. So I came up with SQL script or Stored Procedure which will purge the data without impacting the users, though this Proc can run slow or fast based on the Batch Size you have selected, Smaller (not Smallest) the batch size lower the deadlock risk. You can select the Batch and Total deleted row count based on your need. 

Hope, This proc will be useful for you and let me know if required any improvement. I have written this for SQL Server, you can rewrite for any other DB as well after small tweaking

-

Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     

Read More

Get HTTPS and SSL details of App Deployed - IIB/WMB SCRIPTS

One more to get the HTTPS and SSL details from an Application Deployed in IIB -

- -

Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     

Read More

Get Rest Flow from WMB/IIB Application - IIB/WMB SCRIPTS

Continuing the same journey with the WMN/IIB Scripts... Sharing another script which will return the REST Flow name from the application deployed on Execution Group ...

=

Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     

Read More

Get App deployed on Execution Group - IIB/WMB Scripts

Now a days, I am dwelling between Development and DevOps team (which is actually a prod support team....shhhh.....don't tell anyone ;-) ) and got to learn a lot about the team, people (employee vs manager), kind of work they are doing. I believe, In this IT era, if you want your application to survive, Teach your Prod Support Team, Let them Learn and have some techy kind of people in it. Okk...now... llots of blabbering about it.... Let's jump on the real stuff ......

So, Being a Dev/DevOps member, I have created some of IIB/WMB Scripts which I want to share with you one by one.... Sharing a script which will give you the Application Details Deployed on particular EG.....

==
= ==

Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     

Read More

Create Alias Command with Parameter in Linux

"Alias" - A shortcut for a command or set of commands

Linux OS provides the facility to have alias command to keep the long and frequent command to very few keystrokes, May be I am being a more cryptic with the lots of jargon words, let's try to understand this way -

When you execute a command "ll" (double Ele) in Linux prompt, it will return the result which is the same as "ls -l" because most of the modern linux os have this alias command (ll) for "ls -l" which is quick and easy.  You can add your customize command in ".bashrc" (dot bashrc) in case of  BASH shell or ".kshrc" in case of KORN shell.



How To:
alias pcs = 'ps -ef'  # this will create an alias named pcs which is equivalent to "ps -ef"

As we can see, we can create lots of alias of frequent commands needed in our day to day work, but avoid the alias which modifies/delete system files.

Now, Coming back to the Original Topic,  Can we create an alias which accepts the arguments, Answer is YES, lets check it our How To --

How To create ALIAS which accept command line parameter:

• Create a script which will accept the argument and do the task, such as - 

= =

• Create an alias for this script as - 

==

• How to use this alias - 

==

and voilà !! we are done, Now you can add so many scripts as alias which accept the parameters, Please share here if you have created something useful for you.

Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     

Read More

yet another In Memory DataBase - MemSQL

Writing this post after so many days, yet another IN-MEMORY database is in market which tag line promised with"The DataBase for Real-Time Applications". 

As per MemSQL site - MemSQL is a scalable SQL database that ingests data continuously to perform operational analytics for the front lines of your business. Ingest millions of events per day with ACID transactions while simultaneously analyzing billions of rows of data in relational SQL, JSON, or Geospatial formats.

In my current assignment, I've been asked to look into the capabilities of this db, so starting with very first step "Installation" - 

Installation in linux is quite simple, if you are OK with linux commands, You can follow the installation from HERE

1. Download the software (with sudo or root user) - 

sudo su - root
wget http://download.memsql.com/memsql-ops-6.0.11/memsql-ops-6.0.11.tar.gz

2. Extract the tar ball

tar zxvf memsql-ops-6.0.11.tar.gz

This command will extract lots n lots of files :-)

3. Run the installer script

cd memsql-ops-6.0.11

sudo ./install.sh --simple-cluster

By default, MemSQL supports the machine with 4 cpu core and 8 GB of RAM (which is little unfair;-)) so remove this constraints by below argument -

cd memsql-ops-6.0.11

sudo ./install.sh --simple-cluster --ignore-min-requirements

After being successful installation of MemSQL, it will start setting up MemSQL WebUI.

You can access the MemSQL WebUI on the sever's 9000 port by default. 

https://<SERVER_IP>:9000

4. To connect to MemSQL command line, execute - 

memsql

In next post, I will explain how this db is different than other in-memory db. Till then, Keep Learning , Keep Loving.

Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     

Read More

Folder Sharing in Ubuntu Linux with Samba

A Samba file server enables file sharing across different operating systems over a network. It lets you access your desktop files from a laptop and share files with Windows.



Installation -
sudo apt update
sudo apt install samba


Check Installation -
whereis samba

The following should be its output:

samba: /usr/sbin/samba /usr/lib/samba /etc/samba /usr/share/samba /usr/share/man/man7/samba.7.gz /usr/share/man/man8/samba.8.gz


Create samba conf file backup
sudo cp /etc/samba/smb.conf /etc/samba/smb.conf.orig


Add Folder entry in /etc/samba/smb.conf which you want to share as below -

[Friends]                        # share folder name
    path = /home/atul/Desktop/Friends   # folder path
    read only = no                    
    browsable = yes


Start Samba Service
sudo systemctl restart smbd
sudo service smbd restart


Check if all working as configured
testparm


Now, Set password Samba User -
sudo smbpasswd -a <user_name>   # user_name should be existing system user


Share the Folder link to respective users -
Windows User - \\ip-address\Friends
Ubuntu User  - smb://ip-address/Friends

Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     

Read More

ML-DL model as a Service - MLaaS

It is not always the case when we have to give or share the ML/DL model or algorithm with Client, sometime we want to allow users to use our model but without sharing the code or without setting up the infrastructure at Client end.

There is a way to achieve this, Convert your model as a Service, Yes, You can do that. You can expose your model as a Rest Service and user can use the method supported by your Service program. Today, we are going to learn the same.
Let's see how -

Tools Required - 
Flask - A python web server
and of course Python :-)


ML/DL Model Creation: 
For this part, I am picking very basic ML problem - IRIS dataset Classification.
Below code will create a Logistic Regression Classification Model and Save the model as a file.
==

Flask Web Server - REST Service
In below code, we are creating a REST service hosted on local system and having endpoint as "\iris" and Posting the array input to predict the IRIS species
==

How to run Flask Server:
    

Call ML Service:  From Command Line:

Call ML Service:  From Postman Client:

Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     

Read More

Let's #UnlockAI

Hi Guys, Writing this post after so many days, hoping you didn't take this absence otherwise :-)
Today, I am gonna start a new #hashtag #UnlockAI where we learn the basics of Machine Learning, Deep Learning, Concepts, Algorithm and their limitations under one umbrella. I will try to keep every topic in detail and with the proper example so that it will be easy to understand with under lying the mathematics.

Please post your queries or topics you want to discuss under this #hashtag #UnlockAI

Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     

Read More

Multiple Plots into One Figure in Python MatplotLib

Lets learn, How to plot multiple charts into one Figure. Embedding a jupyter notebook here with the examples.

Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     

Read More

Jupyter Lab Env - Quick Start Script

Every Data Analyst, who is working in Python, is very well aware of Jupyter or Jupyter Lab. 

Sometime it seems little annoying, to lazy person like me, to start the Command Prompt, go to your working code directory and type jupyter command to start the notebook.

To avoid this burden I have wrote a small batch script which will do the task for me in one click. Sharing here the same, Lazy Programmers, please share this post if you like it :-)



For Better View, Click on "View Raw"

Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
Join Telegram DataStage4You Group

Read More

Python Pickle - to save your efforts

This tutorial is for Python beginners who just started getting dirty in Python :-) Lots of time when we are working on some data set and completed lots of cleaning and pre-processing steps, It's advisable from GURUs to save that intermediary dataset to avoid the re-do all the steps if something unexpected happen with Python, Jupyter notebook or your system for rebooted without your permission, weird though :-/

So, let's learn how to save any dataset or variable with Pickle library -

To Import:

To Save: 

To Read:

Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Read More

Mongo DB - Installation and Configuration

MongoDB  is an open-source document database, and the leading NoSQL database. Written in C++.
  
MongoDB features:
    Document-Oriented Storage
    Full Index Support
    Replication & High Availability
    Auto-Sharding
    Querying
    Fast In-Place Updates
    Map/Reduce
    GridFS

Reduce cost, accelerate time to market, and mitigate risk with proactive support and enterprise-grade capabilities.


Today, We will see how to install and run the MongoDB.

MongoDB Installation on Linux

1. DOWNLOAD the stable version of MongoDB. It will a tar file
2. Extract the tar file to some directory.
 
$ tar -xvf mongodb.tar -C /learn/mongodb

3.  change the permisson of folder to user who run the db here-  In my case User - hduser and Group - hadoop
$ chown -R hduser:hadoop /learn/mongodb

4. Add the env var in .bashrc
export MONGO_HOME=/learn/mongodb
export PATH=$PATH:$MONGO_HOME/bin

5. Create the default DB directory for Mongo
$ mkdir -R /data/db
$ chown -R hduser:hadoop /data/db
This is by default, you can specify ur db path when starting the mongo db

$ mongod --dbpath /app/mongodata
this command will start the mongodb. in other terminal you can start work on db. "--dbpath /app/mongodata" is totally optional

If you just use just $ mongod , it will start n use the default db which we have defined in step 5.


Please don't close the current terminal, It can be kill the mongodb process.







6. Start working on MongoDB
$ mongo

Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Read More

Conda Commands

Sharing some CONDA commands which fasten the administration of different python environment. All these commands are available on Anaconda Site . 

Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Read More