Showing posts with label Learning. Show all posts
Showing posts with label Learning. Show all posts

Wednesday, 31 May 2017

Remove ctrl-M character from all files within Directory #iLoveScripting

Continuing our journey on #iLoveScripting,..............
This script will do the same task as "" but this will accept Directory Path as an input rather than the filename. It will iterate through each file within given directory and remove all Ctrl-M characters.

If you are unable to see the Script, Please find it here - LINK

Like the below page to get update

Tuesday, 30 May 2017

Remove ctrl-M character from file #iLoveScripting

This is my first post under #iLoveScripting which will have lots of shell script which are helping me in my day to day task and sharing here for all guys for easing their work as well.

 The very magical script, which I have use, is "". This script is remove the ctrl-M characters (^M) from your windows file.

Usage: <FILE>

If you are unable to see the Script, Please find it here - LINK

Like the below page to get update

Saturday, 14 January 2017

Learning Numpy #2

Thursday, 12 January 2017

Learning Numpy #1

Numpy is a python library used for numerical calculations and this is better performant than pure python. In this notebook, I have shared some basics of Numpy and will share more in next few posts. I hope you find these useful.

Like the below page to get update

Wednesday, 11 January 2017

My Learning Path for Machine Learning

I am a Python Lover guy so my way includes lots of Python points. If you dont know the basics of this wonderful language, start it from HERE else you can follow the links which I am going to share.

Learning ML is not only studying ML algorithms, it includes Basic Algebra, Statistics, Algorithms, Programming and lot more. But no need to afraid as such :-) we need to start from somewhere.....

This is my github repo, you can fork it and follow me with these 2 links --

Fork Fork
Follow - Follow @atulsingh0
I am still updating this list and welcome you to update this as well.

Like the below page to get update

Friday, 6 January 2017

10 minutes with pandas library

Thursday, 5 January 2017

Learning Pandas #5 - read & write data from file

Wednesday, 4 January 2017

Learning Pandas #4 - Hierarchical Indexing

Sunday, 1 January 2017

Learning Pandas #3 - Working on Summary & MissingData

Saturday, 31 December 2016

Learning Pandas - DataFrame #2

Friday, 30 December 2016

Learning Pandas - Series #1

Wednesday, 28 December 2016

Learning Graphlab - SFrame #2

In last post Learning Graphlab - SFrame #1, we have learn basics of SFrame, like how to create, add or delete the columns in SFrame. In this post, we will revise it once again and learn some advance features of SFrame. Have a good learnng !!!

You can view the Jupyter Notebook for the same HERE

Wednesday, 30 November 2016

Learning Graphlab - SFrame #1

Hoping you guys went through the last post (Lnk -> Getting Started with Graphlab), In this post we will do some handson SFrame datatype of Graphlab which is same as dataframe of pandas python library.

i. Reading the CSV file

ii. save DataSet 

iii. load DataSet

iv. Check Total Rows and Columns

v. Check Columns data type and Name

vi. Add new column

vii. Delete column

viii. Rename column

ix. Column Swapping (location)

Like the below page to get update

Sunday, 27 November 2016

Getting Started with Graphlab - A Python library for Machine Learning

Before Starting with Graphlab, We have to configure our system with some basic tools such as Python, Jupyter Notebook etc. You can find 'How-To' on this link -

What is GraphLab ??
GraphLab Create is a Python library, backed by a C++ engine, for quickly building large-scale, high-performance data products. Some key features of GraphLab Create are:
  • Analyze terabyte scale data at interactive speeds, on your desktop.
  • A Single platform for tabular data, graphs, text, and images.
  • State of the art machine learning algorithms including deep learning, boosted trees, and factorization machines.
  • Run the same code on your laptop or in a distributed system, using a Hadoop Yarn or EC2 cluster.
  • Focus on tasks or machine learning with the flexible API.
  • Visualize data for exploration and production monitoring.
After the installation of Graphlab library we can use it as any python library.

Use Jupyter Notebook for starter, Open a Python notebook in Jupyter Notebook and execute below commands to see graphlab working -

 a. Importing Graphlab - 


b. Reading CSV file
This method will parse the input file and convert it into a SFrame variable


c. Getting Started with SFrame 

i. View content of SFrame variable sf


ii. View Head lines (top lines) 


ii. View Tail lines (last lines)

Like the below page to get update

Friday, 11 November 2016

DataStage Partitioning #2

Keyless partition
    Round Robin
    Entire
    Same
    Random

Round Robin
The first record goes to the first processing node, the second to the second processing node, and so on. When DataStage reaches the last processing node in the system, it starts over. This method is useful for resizing partitions of an input data set that are not equal in size. The round robin method always creates approximately equal-sized partitions. This method is the one normally used when DataStage initially partitions data.
Example: Assume 8 nodes are allocated to store the records then 1st record will go into the first node and the 2nd record will go into the second node ……8th record will go into the eighth node and the 9th record will go into the first node and so on….

Send all rows down all partitions.
Example: Assume 8 nodes are allocated, then in all the 8 nodes all the records will be passed.

Preserve the same partitioning.
Example: Two stages in a job (Sort and Dataset). In sort stage you have done “Hash” partition and in the dataset you have given “Same” partition. In the dataset the data will be preserved with the hash partition.

DataStage uses a random algorithm to choose where the rows goes. The result of Random is that you cannot know where a row will end up.

Application Execution: Parallel jobs can be executed in two ways
> Sequential
> Parallel
In the first slot of the below figure sequential execution is shown. Parallel job can be executed in two processing, SMP and MPP, in the second and third slot of the below figure it is shown.

Ref -

Like the below page to get update

Thursday, 10 November 2016

DataStage Partitioning #1

Partitioning mechanism divides a portion of data into smaller segments, which is then processed independently by each node in parallel. It helps make a benefit of parallel architectures like SMP, MPP, Grid computing and Clusters.

Partition is logical. Partition is to divide memory or mass storage into isolated sections. Memory space will be split into many partitions to have high parallelism. In DOS systems, you can partition a disk, and each partition will behave like a separate disk drive.

In hash partitioning no specified space will be allocated to a partition in the memory. The partition space is allocated depending upon the data.

Why Partition?
•    Ability to run multiple operating systems, or multiple versions of an operating system, on the same server
•    Ability to improve workload balancing and distribution by managing processor allocations across applications and users on the server
•    Ability to leverage hardware models such as “Capacity on Demand” and "Pay as You Grow.”

Types of partition
  • Hash
  • Modulus
  • DB2
  • Auto
  • Random
  • Range
  • Round Robin
  • Entire
  • Same


DataStage inserts partitioners as necessary to ensure correct result. Generally chooses Round Robin or Same. Since Datastage has limited awareness of data and business rules, best practice is to explicitly specify partitioning as per requirement when processing requires groups of related records.

Key based partition
  • Hash
  • Modulus
  • DB2
  • Range

Determines partition based on key value(s). One or more keys with different data type are supported. DataStage’s internal algorithm applied to key values determines the partition. All key values are converted to characters before the algorithm is applied.
Example: Key is State. All “CA” rows go into one partition; all “MA” rows go into one partition. Two rows of the same state never go into different partitions.


Partition based on modulus of key divided by the number of partitions. Key is an Integer type. ( partition=MOD(key_value/number of partition) )
Example: Key is OrderNumber (Integer type). Rows with the same order number will all go into the same partition.

Matches DB2 EEE partitioning, DB2 published its hashing algorithm and DataStage copies that.
Example: This partition is used when loading data into the DB2 table. It takes the partition key from the loading DB2 table and inserts the records effectively. If the partition key is defined in the DB2 database then it takes that Partition key otherwise it defaults to primary key.

The partition is chosen based on a range map, which maps ranges of values to specified partitions. This is similar to Hash, but partition mapping is user-determined and partitions are ordered. Range partitioning requires processing the data twice which makes it hard to find a reason for using it.

This figure gives the clear view of Key based Partitioning and repartitioning.

DataStage's parallel technology operates by a divide-and-conquer technique, splitting the largest integration jobs into subsets ("partition parallelism") and flowing these subsets concurrently across all available processors ("pipeline parallelism"). This combination of pipeline and partition parallelism delivers true linear scalability (defined as an increase in performance proportional to the number of processors) and makes hardware the only mitigating factor to performance.

                However, downstream processes may need data partitioned differently. Consider a transformation that is based on customer last name, but the enriching needs to occur on zip code - for house-holding purposes - with loading into the warehouse based on customer credit card number (more on parallel database interfaces below). With dynamic data re-partitioning, data is re-partitioned on-the-fly between processes - without landing the data to disk - based on the downstream process data partitioning needs.

Ref -

Like the below page to get update

Monday, 7 November 2016

Modify Stage - Drop Columns

Monday, 12 September 2016

Python Points #15 - Exceptions

Sunday, 14 August 2016

NULL behaviour in Oracle

NULL, as known in SQL, a special marker and keyword indicating that something has no value. A value of NULL is different from an empty or zero value. No two null values are equal. Comparisons between two null values, or between a NULL and any other value, return unknown because the value of each NULL is unknown. (ref -

For understanding it, I am using a practical approach where we play around NULL value, sorry No Value, with Oracle database.

** NULL value doesn't count in aggregate functions.
NULL value doesn't count by aggregate functions such as MAX, MIN, SUM, COUNT in SQL.


** Inserted null string converted to Null.


** Where can Null be compared?

Oracle SQL Reference says "In a DECODE function, Oracle considers two nulls to be equivalent. If expr is null, then Oracle returns the result of the first search that is also null."
But this therory varies with DB to DB internal architecture and the way they handled the NULL.Another place where Null can be compared is in range partition definition, where MAXVALUE is greater than Null

**  Unique constraints

You are able to insert another Null without getting ORA-1 (unique constraint violated).

So if all columns are null, the unique constraint will not be violated. If one or more columns have non-null values, the constraint takes effect.

** Unknown OR True returns True, Unknown AND False returns False

Like the below page to get update

Tuesday, 21 June 2016

DataStage Quiz #1

If you are unable to see your result, Kindly wait for 5 min then click here - 
If you need more quizs like this, Like the page or join me over these social connections