Hi Hi all guys~

Long time ago, we were playing around Talend (Data Integration (EP 3 end)) to make data integration. And we found we can set schedule on the jobs with crontab and Task scheduler but they might not be comfortable sometimes. Therefore, here is one of automated job tools.

Ref: https://www.rundeck.com

Rundeck has capability for managing job orchestration. In other words, it is for automatic job scenarios.

Rundeck’s features are scheduling jobs, plus:

  • Job logs and history
  • Error handling
  • Webhook supports (apply to send notifications to Slack)
  • Etc.

This time, we will use Docker (Try Docker) and Git (Try Git) together to build a simple job deployment.

All steps we need to do are:

  1. Build a Talend job and upload to Git repo
  2. Build a Rundeck container
  3. Create a job in Rundeck

Let’s go!

1. Build a Talend job and upload to Git repo

1.1 Build a job

For example, I gonna create a job displaying 10 names as below:

Use tRowGeneratator and apply function getFirstname() for 10 names
Test and get the correct results
build it

1.2 Push to Git repo

After that, we push the source code to Git repo. Done for first step.

1.3 App passwords

Because Rundeck operates job with user “Rundeck” not “Root” of the container, we need an app password to allow other user to access the repo.

Click on profile picture then Setting to access this page and select Developer settings
Personal access token >> Generate new token
Add Note and select scope by ticking repo
Got app password as this token

For more information, please visit https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line

2. Build a Rundeck container

At this step, we download Docker image from this link https://hub.docker.com/r/rundeck/rundeck. Then connect Git repo from this container.

2.1 Access Container with Root user and install Git

docker exec -it -u root [docker_container_name]
root@docker $ apt-get update
root@docker $ apt-get install git-core

2.2 create a local Git repo folder

Let’s say we are going to use folder “test”

mkdir test
cd test/

2.3 clone remote repo to the folder

After run this command, the app password must be input here.

git clone [https://github.com/username/git.git]

2.4 change owner from Root to Rundeck

We have to change the owner, or we will get the error about permission denial.

chown -R rundeck [git_folder]

2.5 update remote origin

This is for connecting remote repo with the app password.

git remote set-url origin [https://username:[email protected]/username/git.git]
git remote show origin

Trying git fetch and get no errors means completed connection.

git fetch --all

3. Create a job in Rundeck

3.1 Login to Rundeck

Rundeck provides default accounts defined in the file /home/rundeck/server/config/realm.properties as the folllowings:

  • username = admin
    password = admin
    role = admin, user
  • username = user
    password = user
    role = user

These roles are mapped with Access Control in another configuration file. I have no explanations in this time.

When the Rundeck container started, open the browser and access http://localhost:4440. If we find this page below, login with admin account.

3.2 create a project

click “New Project+”
Name it and other descriptions

3.3 create a job

click “Job”
At page “Details”, fill in job name and description
At page “Workflow”, we are setting commands here
Add steps here

3.4 Download Remote repo and update source code in local repo

We are going to add steps as scripts.

We put the commands below to update local repo and the job will be fresh to run every time.

cd /home/rundeck/test/test_rundeck/
git fetch – all
git reset – hard origin/master

git fetch --all is for retrieving all source code and git reset --hard origin/master is for cancelling all changes in local repo and keep only source code from remote repo.

Finally, we add step to run Talend source code.

sh /home/rundeck/test/test_rundeck/sample_job01_0.1/sample_job01/sample_job01_run.sh

3.5 More configurations and run

Set schedules, notification, and save it.

Done adding workflow steps
set schedule
set notifications
Saved. Try “Run Job Now”
Got 10 names as expected

Job definitions

Additionally, we can import job definition files as the following figures. The files can be obtained by export via action button at right-handed side of the specific jobs.

“Job Actions” >> “Upload Definitions” to enter import page
Select a file and operations

This is an obvious example and I would say I am currently taking care of this sort of processes. I just build a job and place it on Git repo then the flow will automatically do the rest.

That’s all for this story. Stay tuned for the next.