Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Friday, 4 January 2019

How to Query on MongoDB by _id - INTERMEDIATE II


What if we have to query on MongoDB collections based on the "_id" field, Can we really query on "_id" field ? If so, what is the syntax ? Let's try this out -

Let's first fetch a document's id -

$ db.user.find_one()

{'_id': ObjectId('5c16e863817810ed3fc5e5f9'),
 'Fname': 'atul',
 'Lname': 'Singh',
 'Grade': 12.0,
 'College': 'SGM',
 'Job': 'Student',
 'Address': 'Young St.'}



Now, we will pick the object id and use this to fetch the same document from collection

$ db.user.find_one({'_id':'5c16e863817810ed3fc5e5f9'}) #this will return nothing

$ db.user.find_one({'_id':ObjectId('5c16e863817810ed3fc5e5f9')})
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
 in 
----> 1 db.user.find_one({'_id':ObjectId('5c16e863817810ed3fc5e5f9')})

NameError: name 'ObjectId' is not defined


We have received this error because ObjectId is not the same as its string representation, it must be converted to ObjectId from a string before it is passed to find command.

$ from bson.objectid import ObjectId
$ db.user.find_one({'_id':ObjectId('5c16e863817810ed3fc5e5f9')})

{'_id': ObjectId('5c16e863817810ed3fc5e5f9'),
 'Fname': 'atul',
 'Lname': 'Singh',
 'Grade': 12.0,
 'College': 'SGM',
 'Job': 'Student',
 'Address': 'Young St.'}



IPYTHON Notebook can be found HERE








Like the below page to get the update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     


Thursday, 3 January 2019

MongoDB with Python - Basics IV - Update & Delete Operation


Welcome to one more quick session on MongoDB CRUD Basics with Update and Delete Operations. MDB Provides below methods under these operations -

** Update
      * update_one
      * update_many
      * replace_one

** Delete
      * delete_one
      * delete_many

For on Mongo DB -> Link


As it is quite clear from the name itself (_one and _many) that these methods perform the operation on single or multiple records based on the passed condition.
There are so many operator supported by update statement, few are as below -

$set - Add new or update field value
$unset  - Remove field
$inc - increment the current value
$push - push element into array field
$push with $each - push multiple elements into array field
$pop - pull out last value from array field

There are many more Update operators support by MongoDB, Full list can be found HERE


CRUD Operation - (Update & Delete) : Link






Like the below page to get the update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     


Wednesday, 2 January 2019

How to iterate MongoDB Cursor in Python - Intermediate I


Whenever querying on mongodb, always store the output into a variable, called cursor, before performing any operation on data. It will keep your data into variable without messing your output ground. PyMongo Cursor variable supports few functions which helps with few information without actual seeing your data such as retrieved data count or distinct values in particular key etc.

In this session, we will learn about the mongo db cursor variables, for this exercise also we are going to use 'USER' database hosted on free tier MongoDB Atlas (M0) Server.


Jupyter Notebook can be accessed HERE also

= =


Next Post on this Series and more on MongoDB can be find here -> LINK





Like the below page to get the update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     


Monday, 31 December 2018

MongoDB with Python - Basics III - Find/Select Operation


Hoping you guys are enjoying the NoSql journey so far (previous posts links), till now we have seen basic CRUD operation. From this post onward, I am diving in details of these operations and starting with FIND or SELECT operation in MongoDB. We will learn what are the ways and options provided by MongoDB to select or project the data.

When you start working with complex queries you might {as I have said "Might"} face difficulties with tracking of braces {([ as I've experienced the same with me/my team/students and colleagues. But no worries, Jupyter Notebook provides a  couple highlighter for braces when selected or you can use notepad++ also (which I think is not so useful as you are not gonna copy/paste the syntax so frequently).

I advise everyone to avoid the writing queries directly on mongo shell prompt as it doesn't provide any intelligence and not so good in fixing queries if made mistake.

I am sure you will love this post as well and if have any question feel free to ask in comment section below.

For on Mongo DB -> Link


There are many more Find/Read Operators supported by MongoDB, Full list can be found HERE

CRUD Operation - (Read) : Link




Next Post on this Series and more on MongoDB can be find here -> LINK




Like the below page to get the update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     


Saturday, 29 December 2018

MongoDB with Python - Basics II - CRUD Operations


In this post, we will learn about the Advance Find and Create Operations with Sort, Skip and Limit functionality. Pymongo driver support almost same kind of syntax for python which mongo shell used.
The benefit of python (or any programming language) + mongo is to use both langauge/db functionality to work with mongo. Though, to perform the same operation is faster then performing by python but it depends on the activity you are performing.



CRUD Operation - (Create, Read, Update & Delete) : Link




Next Post on this Series and more on MongoDB can be find here -> LINK




Like the below page to get the update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     


Friday, 28 December 2018

MongoDB with Python - Basics I - CRUD Operations


In Previous few posts (Link), We have learnt about MongoDB Cloud Setup, Installation and Basic commands to do CRUD operation with MongoDB. It can be accessed by programming language such as python, java, and node.js by using respective native drivers. We will start with PyMongo (python driver) to access mongo from python.



CRUD Operation - (Create, Read, Update & Delete) : Link




Next Post on this Series and more on MongoDB can be find here -> LINK





Like the below page to get the update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     


Sunday, 11 November 2018

WebServices Health Check Report in Python


Continuing Converting PFX Certs to Certificate and Key Files using OpenSSL....., As discussing about a requirement to generate a health check report for web-services without much of human intervention. Though there are lots of open source and proprietary tools available which can do this stuff in few clicks but I have tried to write something in python which is capable of doing pretty much same and provide more customization.




WebService_HealthCheck.py:

 
WebService_HealthCheck_QA.config


Config file contains the columns as below -


ID|ServiceName|URL|Request


This python code contains 3 functions, 1 GET REST CALL, 1 POST REST CALL, and 1 FILE WRITE operation, we can add more functions which can parse the response and take action as defined.
While writing this code, I have assumed that every service all is HTTPS type which need a certificate to make a success call to service host server. Though, you can omit this setting if your service is simple HTTP type.

As I said, this baseline code is just a skeleton for your service health check. add more and more functions to automate your boring stuff :-)





Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     


Sunday, 4 February 2018

Multiple Plots into One Figure in Python MatplotLib


Lets learn, How to plot multiple charts into one Figure. Embedding a jupyter notebook here with the examples.








Like the below page to get update  
Facebook Page      Facebook Group      Twitter Feed      Google+ Feed      Telegram Group     


Saturday, 3 February 2018

Jupyter Lab Env - Quick Start Script


Every Data Analyst, who is working in Python, is very well aware of Jupyter or Jupyter Lab. 


Sometime it seems little annoying, to lazy person like me, to start the Command Prompt, go to your working code directory and type jupyter command to start the notebook.

To avoid this burden I have wrote a small batch script which will do the task for me in one click. Sharing here the same, Lazy Programmers, please share this post if you like it :-)



For Better View, Click on "View Raw"




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
Join Telegram DataStage4You Group

Tuesday, 16 January 2018

Python Pickle - to save your efforts


This tutorial is for Python beginners who just started getting dirty in Python :-) Lots of time when we are working on some data set and completed lots of cleaning and pre-processing steps, It's advisable from GURUs to save that intermediary dataset to avoid the re-do all the steps if something unexpected happen with Python, Jupyter notebook or your system for rebooted without your permission, weird though :-/



So, let's learn how to save any dataset or variable with Pickle library -

To Import:


To Save: 


To Read:





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Tuesday, 2 January 2018

Conda Commands


Sharing some CONDA commands which fasten the administration of different python environment. All these commands are available on Anaconda Site 








Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Thursday, 13 July 2017

How to Run Python Code in NotePad++


Can you run the python code from NotePad++ ?? Question seems to be little odd but we can tweak our notepad++ settings and configure it that way. Let's see how -



1. Write few line as your python code, You can use below lines-

print("Today we are going to learn how to use notepad++ to run the python code")
print("As first step, we have to write few python code line")

input("Press Enter to Exit..........")




2. Save this file, in my case, it is saved as "npp_run.py"
3. Check the python executable path in your system. In my case it is - C:\tools\Anaconda3\python.exe  (It can be different as per your python installation)

4. Now go to Run menu or Press F5. This will open a run window as below -

5. Python below code in 'the program to run' -
Python_Executable_Path $(FULL_CURRENT_PATH)
C:\tools\Anaconda3\python.exe $(FULL_CURRENT_PATH)



6. Save this Run configuration by clicking on Save button on same window

7. Choose Run button combination (can use Ctrl + Alt + Shift + Key) and Save.

8. You can see this combination under Run Menu.


9. Now, You can run the Python code by pressing the combination buttons (My case - F9)




Things to Remember:
1. This tweak is not replacement for Python IDE :-) such as PyCharm, Spider or many others.
2. Always put the  input("Press Enter to Exit..........")at very last line of your code else you will not able to see the python code output.




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Saturday, 25 March 2017

Check if Python Pandas DataFrame Column is having NaN or NULL


Before implementing any algorithm on the given data, It is a best practice to explore it first so that you can get an idea about the data. Today, we will learn how to check for missing/Nan/NULL values in data.

1. Reading the data
Reading the csv data into storing it into a pandas dataframe.


2. Exploring data
Checking out the data, how it looks by using head command which fetch me some top rows from dataframe.


3. Checking NULLs
Pandas is proving two methods to check NULLs - isnull() and notnull()
These two returns TRUE and FALSE respectively if the value is NULL. So let's check what it will return for our data

isnull() test

notnull() test

Check 0th row, LoanAmount Column - In isnull() test it is TRUE and in notnull() test it is FALSE. It mean, this row/column is holding null.

But we will not prefer this way for large dataset, as this will return TRUE/FALSE matrix for each data point, instead we would interested to know the counts or a simple check if dataset is holding NULL or not.

Use any()
Python also provide any() method which returns TRUE if there is at least single data point which is true for checked condition.


Use all()
Returns TRUE if all the data points follow the condition.


Now, as we know that there are some nulls/NaN values in our data frame, let's check those out - 

data.isnull().sum() - this will return the count of NULLs/NaN values in each column.


If you want to get total no of NaN values, need to take sum once again -

data.isnull().sum().sum()


If you want to get any particular column's NaN calculations - 




Here, I have attached the complete Jupyter Notebook for you -



If you want to download the data, You can get it from HERE.




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Friday, 3 February 2017

Learning Matplotlib #2

Thursday, 2 February 2017

Plotting in Python - Learning Matplotlib #1

Saturday, 14 January 2017

Learning Numpy #2

Thursday, 12 January 2017

Learning Numpy #1


Numpy is a python library used for numerical calculations and this is better performant than pure python. In this notebook, I have shared some basics of Numpy and will share more in next few posts. I hope you find these useful.





Click Here for Next Tutorial ~

Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Wednesday, 11 January 2017

My Learning Path for Machine Learning


I am a Python Lover guy so my way includes lots of Python points. If you dont know the basics of this wonderful language, start it from HERE else you can follow the links which I am going to share.

Learning ML is not only studying ML algorithms, it includes Basic Algebra, Statistics, Algorithms, Programming and lot more. But no need to afraid as such :-) we need to start from somewhere.....

This is my github repo, you can fork it and follow me with these 2 links --

Fork Fork
Follow - Follow @atulsingh0
I am still updating this list and welcome you to update this as well.



Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Friday, 6 January 2017

10 minutes with pandas library

Thursday, 5 January 2017

Learning Pandas #5 - read & write data from file