Showing posts with label Linux. Show all posts
Showing posts with label Linux. Show all posts

Thursday, 23 February 2017

Shell script to access DataStage Director ETL joblogs



We have various data warehouses hosted on AIX/Linux/Unix operating systems. DataStage Director is one of the ETL monitoring & scheduling tool used in numerous data warehouses. In case of ETL job failures we need to login to DataStage Director and check the log for error messages. The ETL job logs can tend to span into several pages with various types of informative messages. Usually we need to locate the error message under ‘Fatal’ message type. Doing this task by searching the DataStage log manually can be very time consuming and tiring.




This shell script facilitates the user to access the error messages from the ease of the  Linux screen. The script also has facility to email the filtered log messages to the user’s mail box.


Ø  Accept jobname and other parameters while script execution.
Ø  Establish proper environment settings for local DataStage use
Ø  Locate event ids for fatal errors in the ETL joblog.
Ø  Extract detail log for fatal errors
Ø  Mail the filtered job log with exact fatal error message to the user.




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Monday, 16 January 2017

Linux Shell Utilities


This shell script can be called from any other shell script as ". UXUtils.sh" and any of the functions can be called to use. Even the funcions can be kept in the .profile and called through calling .profile.


The following functions are provided here:
to_lower               [ Change the upper case word to lower case.]
to_upper              [ Change the lower case to upper case. ]
check_numeric    [ Checks whether an input is a number. ]
check_decimal     [ Checks whether an input is a floating point number. ]
check_null            [ Checks for NULLs. ]
string_length        [ Evaluates the length of a string. ]
concatAll              [ Concats 'n' number of strings. ]
token_n                [ Returns the n'th token of a string.]

How To:
This is how you have to call this script to utilize any function in your current script.

In the caller script:
----------------------
# Calling UXUtils.sh
. UXUtils.sh

x1="COUMPTER"
RV=`to_lower $x1`
RV1=`string_length $x1`


Find the complete script here -




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Wednesday, 12 October 2016

Script to Auto Compress the System Log Files


This Script is originally written by "Andy Welter" to compress the linux system log files. I have modified it to work better. You can find it below (modified version) -

Script Usage:
logroll [-compress|-nocompress]

$ logroll -compress
# compress the log files and move to archive directory

logroll -nocompress
# Move the file to archive directory with compressing it 


=================================================================== ===================================================================




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Tuesday, 11 October 2016

5 Tips For Better DataStage Design #16


1. Use 4-node configuration file for unit testing/system testing the job.
2. If there are multiple jobs to be run for the same module. Archive the source files in the after job routine of the last job.
3. Check whether the file exists in the landing directory before moving the sequential file. The ‘mv’ command will move the landing directory if the file is not found.

4. Ensure that the unix files created by any Datastage job is created by the same unix user who has run the job.
5. Make sure that the Short Job Description is filled using ‘Description Annotation’ and it contains the job name as part of the description. Don’t use Annotation for putting the job description.





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Tuesday, 31 May 2016

Python Points #13 - Comprehensions

Wednesday, 18 May 2016

Windows & Linux - Command Comparison



I am a linux command line geek and when I am working on windows I miss it as there are lot of kung-fu which we can do easily in linux terminal than any GUI tool. So thought of creating a command equivalent list of Windows and Linux.

let me know if there is any daily command which I missed to add here. Hoping this will help you guys.

www.datagenx.net/2016/05/windows-linux-command-comparison.html



Command's Purpose MS-DOS Linux Basic Linux Example
Copies files copy cp cp thisfile.txt /home/thisdirectory
Moves files move mv mv thisfile.txt /home/thisdirectory
Lists files dir ls ls
Clears screen cls clear clear
Closes shell prompt exit exit exit
Displays or sets date date date date
Deletes files del rm rm thisfile.txt
"Echoes" output to the screen echo echo echo this message
Edits text files edit gedit gedit thisfile.txt
Compares the contents of files fc diff diff file1 file2
Finds a string of text in a file find grep grep word or phrase thisfile.txt
Displays command help command /? man or info man command
Creates a directory mkdir, md mkdir mkdir directory
Deletes an existing directory rmdir, rd rmdir rmdirt directory
Views contents of a file more less less thisfile.txt
Renames a file ren mv mv thisfile.txt thatfile.txt
Displays your location in the file system chdir pwd pwd
Changes directories with a specified path (absolute path) cd pathname cd pathname cd /directory/directory
Changes directories with a relative path cd.. cd .. cd ..
Displays the time time date date
Shows amount of RAM in use mem free free
Displays the contents of a file type cat type filename
Reverse IP lookup nslookup nslookup
Checking the network status netstat netstat netstat
Pinging any system ping ping ping 127.0.0.1
Running Process tasklist ps ps -ef
Get login username net session who who 
Tracing IP  tracert traceroute
Sets file permissions Attrib chmod chmod perm1 perm2
Display OS version ver uname -a uname -a
Sorts lines in a file sort sort sort filename
Compare two files diff fc fc file1 file2
OS scheduler at crontab crontab -l
FTP ftp ftp ftp user@host




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Wednesday, 20 April 2016

Shell Script Scenario #8 - Odd and Even No



 Write a shell script to find whether an input integer is even or odd. 








Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Tuesday, 19 April 2016

Best book for Linux Shell Scripting - Intermediate User


Today, I am gonna share my secret of Linux Scripting. The book, shared below, is core responsible for my shell scripting technique and ease. I love the tips n tricks, tweaks or many more or you can say I simply love this book :-)

So here it is -- Hoping you all also get benefitted by this.

For Intermediate Linux Shell Scripting users, this book provide all the tips and tricks which we can include in out script to make it work more efficient way.

Mastering Unix Shell Scripting: Bash, Bourne, and Korn Shell Scripting for Programmers, System Administrators, and UNIX Gurus
by Randal K. Michael
Link: http://www.amazon.in/dp/0470183012


== ==




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Friday, 1 April 2016

jupyter notebook tip #1



How we can start jupyter notebook from a specific directory?

Below, I am sharing a very tiny batch (and linux shell) script which will kick off the jupyter notebook from a particular directory with just a double click...

I hope, this will give you a little ease while working on IPython :-)





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Thursday, 31 March 2016

echo the 'echo'




"echo" command is the very basic command of Linux/Unix. Most of us known this command to print whatever is being passed to it as argument. So let's revise once more if we are missing something here.


"echo" command prints its argument by single space.
[atul@localhost ~]$ echo Atul learns commands 
Atul learns commands

print argument by single space mean it will suppress the space between its arguments.
[atul@localhost ~]$ echo Atul          learns                         commands 
Atul learns commands

if you want to preserver the space between them, we need to make these 3 argument to 1 by using a quote.
[atul@localhost ~]$ echo "Atul     learns   commands" 
Atul     learns   commands



One more example :



** Need to remember -
* echo expect arguments to print whether they are separated by single space or multiple, it will print them with single space
* if you want to keep multiple space in argument string, make them one argument by quoting them.





Friday, 11 March 2016

Python Points #12 - execute OS commads

Thursday, 10 March 2016

Python Points #11 - Set

Tuesday, 8 March 2016

Python - Get line no with print statement


Sometime when we are working on some code and there are lot of print statement, It's difficult to check for which print statement produce this statement.

It helps a lot while debugging the code and helped me to create better python learning script which will generate the output with the code line no which is easy to relate.




Code:
from inspect import currentframe

def lno():
    cf = currentframe()
    val = str(cf.f_back.f_lineno)+". "
    return val

print "this is Me", lno()
print lno(), "Hi! there"

Output:

this is Me  8.
9.  Hi! there


** This is a little overhead on the code as with each print statement the function is being called. So use it wisely :-)





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Tuesday, 23 February 2016

DataStage Scenario #15 - Get First & Last Date of Last Month



Design a job which can generate the First and Last date of Last Month and pass this into an SQL which executes inside a parallel job?









Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Monday, 22 February 2016

Python Points #10d - Writing into Files

Wednesday, 17 February 2016

Python Points #10c - File Methods

Sunday, 7 February 2016

Python Points #10b - Reading Files

Saturday, 6 February 2016

Python Points #10a - Reading Files

Friday, 5 February 2016

DataStage Scenario #14 - Get Day specific file



Hi Guys,
Design jobs for one more realtime scenario -


Requirement-
Job will run once in a day, which read a file from a folder, but filename is changing on each day.

File Name -- FileName_Date.txt

Here -
Date -  Job run date
File - File Name


Example -

FileName on Monday -   InfoSphere_20160201.txt
FileName on Tuesday -   Info_Search_20160202.txt
FileName on Wednesday -   InfoLables_20160203.txt
FileName on Thursday -   InfoLocation_20160204.txt
FileName on Friday -   InfoOptions_20160205.txt




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Sunday, 31 January 2016

Create a Daemon to Trace New Processes


Description

The following code can be used to create a daemon that will watch for processes that show up in the "ps -ef" output with certain characteristics. When it identifies such processes, it will attach to them with a trace utility (i.e. strace, truss, tusc... you must change the code to reflect this on whatever platform this is run on). The tool does not follow these processes with a fork since it will trace any children that contain the same "ps -ef" characteristics. This makes it useful for tracing DS PX programs that contain rsh since truss's fork flag (i.e. "-f") blocks the rsh from executing.



Usage

The script below should be saved to a file such as /tmp/tracer.sh and given rwx permissions.  The trace utility name that is appropriate for your platform should be altered in the "ps -ef" command and in the "for" loop.  The script would then be run using this syntax:
    /tmp/tracer.sh <search string>
As mentioned above, the search string can be any value that would appear in the "ps -ef" output.  Such values might be a user id, particular time, a command, or arguments to a command.  The fifth and eight lines of this script gather lists of all commands to be traced and then attempts to remove commands that should be ignored.  If you find too many processes getting traced, identify why it was selected and then alter these two lines by adding a "grep -v" to the list of items bieng ignored.









Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://groups.google.com/forum/#!forum/datagenx