Hi Hi all guys~
Long time ago, we were playing around Talend (Data Integration (EP 3 end)) to make data integration. And we found we can set schedule on the jobs with crontab and Task scheduler but they might not be comfortable sometimes. Therefore, here is one of automated job tools.
Rundeck has capability for managing job orchestration. In other words, it is for automatic job scenarios.
Rundeck’s features are scheduling jobs, plus:
- Job logs and history
- Error handling
- Webhook supports (apply to send notifications to Slack)
All steps we need to do are:
- Build a Talend job and upload to Git repo
- Build a Rundeck container
- Create a job in Rundeck
1. Build a Talend job and upload to Git repo
1.1 Build a job
For example, I gonna create a job displaying 10 names as below:
1.2 Push to Git repo
After that, we push the source code to Git repo. Done for first step.
1.3 App passwords
Because Rundeck operates job with user “Rundeck” not “Root” of the container, we need an app password to allow other user to access the repo.
For more information, please visit https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line
2. Build a Rundeck container
At this step, we download Docker image from this link https://hub.docker.com/r/rundeck/rundeck. Then connect Git repo from this container.
2.1 Access Container with Root user and install Git
docker exec -it -u root [docker_container_name] root@docker $ apt-get update root@docker $ apt-get install git-core
2.2 create a local Git repo folder
Let’s say we are going to use folder “test”
mkdir test cd test/
2.3 clone remote repo to the folder
After run this command, the app password must be input here.
git clone [https://github.com/username/git.git]
2.4 change owner from Root to Rundeck
We have to change the owner, or we will get the error about permission denial.
chown -R rundeck [git_folder]
2.5 update remote origin
This is for connecting remote repo with the app password.
git remote set-url origin [https://username:[email protected]/username/git.git] git remote show origin
git fetch and get no errors means completed connection.
git fetch --all
3. Create a job in Rundeck
3.1 Login to Rundeck
Rundeck provides default accounts defined in the file
/home/rundeck/server/config/realm.properties as the folllowings:
- username = admin
password = admin
role = admin, user
- username = user
password = user
role = user
These roles are mapped with Access Control in another configuration file. I have no explanations in this time.
When the Rundeck container started, open the browser and access http://localhost:4440. If we find this page below, login with admin account.
3.2 create a project
3.3 create a job
3.4 Download Remote repo and update source code in local repo
We are going to add steps as scripts.
We put the commands below to update local repo and the job will be fresh to run every time.
cd /home/rundeck/test/test_rundeck/ git fetch – all git reset – hard origin/master
git fetch --all is for retrieving all source code and
git reset --hard origin/master is for cancelling all changes in local repo and keep only source code from remote repo.
Finally, we add step to run Talend source code.
3.5 More configurations and run
Set schedules, notification, and save it.
Additionally, we can import job definition files as the following figures. The files can be obtained by export via action button at right-handed side of the specific jobs.
This is an obvious example and I would say I am currently taking care of this sort of processes. I just build a job and place it on Git repo then the flow will automatically do the rest.
That’s all for this story. Stay tuned for the next.