Showing posts with label Windows. Show all posts
Showing posts with label Windows. Show all posts

Sunday, 5 March 2017

How to get DataStage Project Settings


DSListProjectSettings.bat is used to report all available settings of existing projects.

USAGE: DSListProjectSettings <SERVER> <USER> <PASSWORD> <PROJECT>

The DSListProjectSettings batch file reports all available project settings and environment variable names and values to standard output. The first section shows available DataStage project properties and their settings. The second section shows environment variable names and values. When run successfully, each section will report a status code of zero.



..


Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Wednesday, 7 December 2016

Import the jobs from DS windows client


As we have discussed a script which can export the datastage jobs from your client system (http://bit.ly/2frNPKj) likewise we can write another one to import the jobs. Let's see how -


DsImportJobsClient.bat :

This Script read all the *.dsx job name from the specified Directory and Sub-Directory and import to the Specified project. It can also build (Only BUILD) the existing package created on Information Server Manager and send it to the specified location on client machine.

To use the build feature you need to make sure the package has been created with all the needed jobs, saved and closed. Only update to the selected job will be taken care automatically. To add/delete a job, you need to do manually.

Modify the Import.properties and ImportJobList.txt file and Go the .bat dir and then execute the importAndBuild.bat.




Import.properties :


ImportJobList.txt :


DsImportJobsClient.bat :





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Sunday, 6 November 2016

Export the jobs from DS windows client



Datastage jobs Export/Import are occasional activity (Deployment time :-)) for a developer But it becomes very tedious if the job list are long or it's daily routine to export or import jobs.

So I have written a batch script (windows script) which we can execute from the Client Machine (where datastage clients are installed) and automate this process.



DSjobExportClient.bat :
Export Script read the job name from the file (ExportJobList.txt) and exports the jobName.dsx from the project to the export base location and maintain the folder structure specified in the ExportJobList.txt file. File "ExportJobList.txt" and “Export.properties” should be updated before running the export script.

Copy the above file on any location of your DS windows client machine and update the "ExportJobList.txt" and “Export.properties” files.
-    Export.properties
-    ExportJobList.txt
-    ExportJobs.bat   



Export.properties :


ExportJobList.txt :


DsExportJobsClient.bat :




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Sunday, 16 October 2016

Jenkins with Windows #1


One of my team member is assigned to install and configure "jenkins" on our server so out of curiosity, I asked what is this now ?? but didn't get a satisfactory answer :-) so I thought of having my hand dirty with it. Here, I am sharing whatever I learn.

What is Jenkins:-
Wikipedia sources says,  Jenkins is an open source automation server written in Java. Jenkins helps automating the non-human part of the whole software development process, with now common things like Continuous Integration, but by further empowering teams to implement the technical part of a Continuous Delivery.

https://jenkins.io/

What is Continuous Integration & Continuous Delivery:-
CI is a process that most developers follow to keep their code base intact. It's mostly a common practice when you work in a group environment. For example, an analogy for this would be constructing a new home. There will be multiple contractors working on the site. So, if we have installed the window glasses and the paint person comes in and paints the house there are high chances that he will drop some paint on the glasses or end up breaking the glass. So, the inspector comes and checks it every day to see if something broke. The same process is applied for constructing a new code. CI system gathers all your code from different developers and makes sure it compiles and build fine. This is good. But, not complete. I will get to that once I complete talking about Jenkins.


Jenkins is the inspector in the analogy. Jenkins is nothing but a middle man between your code repo and your build server. It checks for changes on your server every few minutes. If it found them, it gathers them and sends them to your build server. That's what Jenkins is.

Basically Continuous Integration is the practice of running your tests on a non-developer machine automatically everytime someone pushes new code into the source repository.

This has the tremendous advantage of always knowing if all tests work and getting fast feedback. The fast feedback is important so you always know right after you broke the build (introduced changes that made either the compile/build cycle or the tests fail) what you did that failed and how to revert it.

If you only run your tests occasionally the problem is that a lot of code changes may have happened since the last time and it is rather hard to figure out which change introduced the problem. When it is run automatically on every push then it is always pretty obvious what and who introduced the problem.

Built on top of Continuous Integration are Continuous Deployment/Delivery where after a successful test run your instantly and automatically release the latest version of your codebase. Makes deployment a non-issue and helps you speed up your development.


                              Jenkins offers the following major features out of the box, and many more can be added through plugins:

Developer time is focused on work that matters — Much of the work of frequent integrations is handled by automated build and testing systems, meaning developer time isn't wasted on large-scale error-ridden integrations.

Software quality is improved — Any issues are detected and resolved almost immediately, keeping software in a state where it can be safely released at any time.

Faster Development - Integration costs are reduced both because serious integration issues are less likely and because much of the work of integration is automated.

Easy installation: Just run java -jar jenkins.war, deploy it in a servlet container. No additional install, no database. Prefer an installer or native package? We have those as well.

Easy configuration: Jenkins can be configured entirely from its friendly web GUI with extensive on-the-fly error checks and inline help.

Rich plugin ecosystem: Jenkins integrates with virtually every SCM or build tool that exists.

Extensibility: Most parts of Jenkins can be extended and modified, and it's easy to create new Jenkins plugins. This allows you to customize Jenkins to your needs.

Distributed builds: Jenkins can distribute build/test loads to multiple computers with different operating systems. Building software for OS X, Linux, and Windows? No problem.


Check out the part 2 for Installation.



Sources:
https://jenkins.io/ https://en.wikipedia.org/wiki/Jenkins_(software) http://stackoverflow.com https://www.quora.com



Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Wednesday, 18 May 2016

Windows & Linux - Command Comparison



I am a linux command line geek and when I am working on windows I miss it as there are lot of kung-fu which we can do easily in linux terminal than any GUI tool. So thought of creating a command equivalent list of Windows and Linux.

let me know if there is any daily command which I missed to add here. Hoping this will help you guys.

www.datagenx.net/2016/05/windows-linux-command-comparison.html



Command's Purpose MS-DOS Linux Basic Linux Example
Copies files copy cp cp thisfile.txt /home/thisdirectory
Moves files move mv mv thisfile.txt /home/thisdirectory
Lists files dir ls ls
Clears screen cls clear clear
Closes shell prompt exit exit exit
Displays or sets date date date date
Deletes files del rm rm thisfile.txt
"Echoes" output to the screen echo echo echo this message
Edits text files edit gedit gedit thisfile.txt
Compares the contents of files fc diff diff file1 file2
Finds a string of text in a file find grep grep word or phrase thisfile.txt
Displays command help command /? man or info man command
Creates a directory mkdir, md mkdir mkdir directory
Deletes an existing directory rmdir, rd rmdir rmdirt directory
Views contents of a file more less less thisfile.txt
Renames a file ren mv mv thisfile.txt thatfile.txt
Displays your location in the file system chdir pwd pwd
Changes directories with a specified path (absolute path) cd pathname cd pathname cd /directory/directory
Changes directories with a relative path cd.. cd .. cd ..
Displays the time time date date
Shows amount of RAM in use mem free free
Displays the contents of a file type cat type filename
Reverse IP lookup nslookup nslookup
Checking the network status netstat netstat netstat
Pinging any system ping ping ping 127.0.0.1
Running Process tasklist ps ps -ef
Get login username net session who who 
Tracing IP  tracert traceroute
Sets file permissions Attrib chmod chmod perm1 perm2
Display OS version ver uname -a uname -a
Sorts lines in a file sort sort sort filename
Compare two files diff fc fc file1 file2
OS scheduler at crontab crontab -l
FTP ftp ftp ftp user@host




Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Friday, 1 April 2016

jupyter notebook tip #1



How we can start jupyter notebook from a specific directory?

Below, I am sharing a very tiny batch (and linux shell) script which will kick off the jupyter notebook from a particular directory with just a double click...

I hope, this will give you a little ease while working on IPython :-)





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Tuesday, 8 March 2016

Python - Get line no with print statement


Sometime when we are working on some code and there are lot of print statement, It's difficult to check for which print statement produce this statement.

It helps a lot while debugging the code and helped me to create better python learning script which will generate the output with the code line no which is easy to relate.




Code:
from inspect import currentframe

def lno():
    cf = currentframe()
    val = str(cf.f_back.f_lineno)+". "
    return val

print "this is Me", lno()
print lno(), "Hi! there"

Output:

this is Me  8.
9.  Hi! there


** This is a little overhead on the code as with each print statement the function is being called. So use it wisely :-)





Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://datagenx.slack.com/messages/datascience/

Friday, 29 January 2016

Where to get DataStage Scheduled jobs ?


Did you guys ever try to find out how the scheduler works when we schedule jobs by DataStage scheduler? Interesting na !!

So, let's have a discussion how this is happening.

DataStage itself doesn't contain any scheduling system or application but it is using OS-level scheduling techniques which is OS-level job scheduler.



Unix/Linux-
If the DataStage server is on Linux/Unix/*nix OS. That's "at" and "cron"

'at -lv' or 'at -l' will show any jobs that are scheduled to run once, 'crontab -l' will show jobs that are scheduled to repeat


Windows-
On Windows, the jobs become entries in the Scheduled Tasks control panel. In GUI, you can't understand much. But we can extract and dump this information to see it better.

'schtasks /query /v /fo list' command give you a proper formatted information about the scheduled jobs.


Windows has nice output format options, and (with the verbose switch) lots of info, including the next run time, last run time and last result, so you can tell if a job actually ran when it was supposed to.  You may need to use a batch script to filter out the specific entry.






Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://groups.google.com/forum/#!forum/datagenx

Thursday, 17 December 2015

Python, IPython, Jupyter notebook, Graphlab Installation on Windows


In "Python Installation from Source in Linux" and "Data Science Tools Installation in Linux" we have seen, how to install these tools on linux, Today we will learn how to setup these tools on Windows -




Python Installation:

1. Download the Python Windows installer from here -> https://www.python.org/downloads/release/python-2711/

2. Install it as we install any software on windows

3. Now, setup the Environment Variable -
a.              If you haven’t played with environment variables before, just stick to following these instructions as you can set them up through the Windows GUI.
b.             Right click on "My Computer", select "Properties" > "Advanced system settings" and click on the "Environment Variables" button
c.             In the System Variables box, find the variable called "path" and click on the "Edit…" button
d.             In the "Variable value" box, at the end of the entry, add the following text: ;C:\Python27;C:\Python27\Scripts (change the path as per your installation)
e.             Click "OK" a couple of times and hey presto, your environment variables are set up.
f.              Open cmd and type command 'python', if you get the python prompt we are good else check the steps once again.

4. The next step in the process is to set up easy_install and so we need to go to the setuptools page (links to version 0.8) and download the ez_setup.py script. You can download it from here -> https://bitbucket.org/pypa/setuptools/raw/0.8/ez_setup.py. and put this in python script directory (C:\Python27\Scripts)

5. Open a command prompt and type python ez_setup.py install – you’ll see a load of code whizz by which will hopefully end as follows;

C:\Python27> python ez_setup.py install
Processing dependencies for setuptools==0.8
Finished processing dependencies for setuptools==0.8
C:\Python27>
6. easy_install has now been set up and you can test to see if it is there, by typing easy_install in to a command prompt, which will throw an error about no URLs, you know that the tool has been set up successfully.

To use easy_install to get new libraries, just use the following syntax: easy_install <library name>


IPython Installation:

C:\Python27> easy_install ipython
Jupyter notebook Installation
C:\Python27\Scripts> pip install jupyter
You can run the jupyter notebook as below -

C:\Python27\Scripts>jupyter notebook

Graphlab Create Installation

C:\Python27\Scripts> pip install --upgrade --no-cache-dir https://pypi.python.org/packages/source/G/GraphLab-Create/GraphLab-Create-1.7.1.tar.gz#md5=caa4b1f78625a278dd016400d15bc5bd



Like the below page to get update  
https://www.facebook.com/datastage4you
https://twitter.com/datagenx
https://plus.google.com/+AtulSingh0/posts
https://groups.google.com/forum/#!forum/datagenx