Showing posts with label IPython. Show all posts
Showing posts with label IPython. Show all posts

Saturday, 25 March 2017

Check if Python Pandas DataFrame Column is having NaN or NULL


Before implementing any algorithm on the given data, It is a best practice to explore it first so that you can get an idea about the data. Today, we will learn how to check for missing/Nan/NULL values in data.

1. Reading the data
Reading the csv data into storing it into a pandas dataframe.


2. Exploring data
Checking out the data, how it looks by using head command which fetch me some top rows from dataframe.


3. Checking NULLs
Pandas is proving two methods to check NULLs - isnull() and notnull()
These two returns TRUE and FALSE respectively if the value is NULL. So let's check what it will return for our data

isnull() test

notnull() test

Check 0th row, LoanAmount Column - In isnull() test it is TRUE and in notnull() test it is FALSE. It mean, this row/column is holding null.

But we will not prefer this way for large dataset, as this will return TRUE/FALSE matrix for each data point, instead we would interested to know the counts or a simple check if dataset is holding NULL or not.

Use any()
Python also provide any() method which returns TRUE if there is at least single data point which is true for checked condition.


Use all()
Returns TRUE if all the data points follow the condition.


Now, as we know that there are some nulls/NaN values in our data frame, let's check those out - 

data.isnull().sum() - this will return the count of NULLs/NaN values in each column.


If you want to get total no of NaN values, need to take sum once again -

data.isnull().sum().sum()


If you want to get any particular column's NaN calculations - 




Here, I have attached the complete Jupyter Notebook for you -



If you want to download the data, You can get it from HERE.




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Friday, 3 February 2017

Learning Matplotlib #2

Thursday, 2 February 2017

Plotting in Python - Learning Matplotlib #1

Saturday, 14 January 2017

Learning Numpy #2

Thursday, 12 January 2017

Learning Numpy #1


Numpy is a python library used for numerical calculations and this is better performant than pure python. In this notebook, I have shared some basics of Numpy and will share more in next few posts. I hope you find these useful.





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Wednesday, 11 January 2017

My Learning Path for Machine Learning


I am a Python Lover guy so my way includes lots of Python points. If you dont know the basics of this wonderful language, start it from HERE else you can follow the links which I am going to share.

Learning ML is not only studying ML algorithms, it includes Basic Algebra, Statistics, Algorithms, Programming and lot more. But no need to afraid as such :-) we need to start from somewhere.....

This is my github repo, you can fork it and follow me with these 2 links --

Fork Fork
Follow - Follow @atulsingh0
I am still updating this list and welcome you to update this as well.



Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Friday, 6 January 2017

10 minutes with pandas library

Thursday, 5 January 2017

Learning Pandas #5 - read & write data from file

Wednesday, 4 January 2017

Learning Pandas #4 - Hierarchical Indexing

Sunday, 1 January 2017

Learning Pandas #3 - Working on Summary & MissingData

Saturday, 31 December 2016

Learning Pandas - DataFrame #2

Friday, 30 December 2016

Learning Pandas - Series #1

Friday, 1 April 2016

jupyter notebook tip #1



How we can start jupyter notebook from a specific directory?

Below, I am sharing a very tiny batch (and linux shell) script which will kick off the jupyter notebook from a particular directory with just a double click...

I hope, this will give you a little ease while working on IPython :-)





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Thursday, 17 December 2015

Python, IPython, Jupyter notebook, Graphlab Installation on Windows


In "Python Installation from Source in Linux" and "Data Science Tools Installation in Linux" we have seen, how to install these tools on linux, Today we will learn how to setup these tools on Windows -




Python Installation:

1. Download the Python Windows installer from here -> https://www.python.org/downloads/release/python-2711/

2. Install it as we install any software on windows

3. Now, setup the Environment Variable -
a.              If you haven’t played with environment variables before, just stick to following these instructions as you can set them up through the Windows GUI.
b.             Right click on "My Computer", select "Properties" > "Advanced system settings" and click on the "Environment Variables" button
c.             In the System Variables box, find the variable called "path" and click on the "Edit…" button
d.             In the "Variable value" box, at the end of the entry, add the following text: ;C:\Python27;C:\Python27\Scripts (change the path as per your installation)
e.             Click "OK" a couple of times and hey presto, your environment variables are set up.
f.              Open cmd and type command 'python', if you get the python prompt we are good else check the steps once again.

4. The next step in the process is to set up easy_install and so we need to go to the setuptools page (links to version 0.8) and download the ez_setup.py script. You can download it from here -> https://bitbucket.org/pypa/setuptools/raw/0.8/ez_setup.py. and put this in python script directory (C:\Python27\Scripts)

5. Open a command prompt and type python ez_setup.py install – you’ll see a load of code whizz by which will hopefully end as follows;

C:\Python27> python ez_setup.py install
Processing dependencies for setuptools==0.8
Finished processing dependencies for setuptools==0.8
C:\Python27>
6. easy_install has now been set up and you can test to see if it is there, by typing easy_install in to a command prompt, which will throw an error about no URLs, you know that the tool has been set up successfully.

To use easy_install to get new libraries, just use the following syntax: easy_install <library name>


IPython Installation:

C:\Python27> easy_install ipython
Jupyter notebook Installation
C:\Python27\Scripts> pip install jupyter
You can run the jupyter notebook as below -

C:\Python27\Scripts>jupyter notebook

Graphlab Create Installation

C:\Python27\Scripts> pip install --upgrade --no-cache-dir https://pypi.python.org/packages/source/G/GraphLab-Create/GraphLab-Create-1.7.1.tar.gz#md5=caa4b1f78625a278dd016400d15bc5bd



Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://groups.google.com/forum/#!forum/datagenx

Sunday, 13 December 2015

Data Science Tools Installation in Linux