Master DolphinScheduler Time Parameter With These Dynamic Configuration Secrets

In DolphinScheduler, using time parameters is very important during scheduling, especially when dealing with data processing, ETL tasks, or other scenarios that rely on time ranges. DolphinScheduler allows flexible configuration of time parameters in various ways to meet different task scheduling needs. Below are the common usage methods of time parameters in DolphinScheduler scheduling:

Time in Global Parameters

DolphinScheduler supports defining global parameters in workflows or tasks, and time parameters can be configured as global parameters and used throughout the workflow or task execution.

Defining Time Global Parameters:

In the workflow configuration interface, you can define time-related parameters in "Global Parameters". For example, define a global parameter representing the current date:

• key: current_date
• value: $[yyyy-MM-dd]

With this, you can use ${current_date} as a date parameter in the task script.

Script example:

#!/bin/bash  
echo "Current date is ${current_date}"  

When the workflow is executed, ${current_date} will be parsed into the actual date (e.g., 2024-10-14).

Time Window Parameters

Time window parameters are often used in time range-based data processing tasks, such as regularly reading data from a data source within a specific time window. DolphinScheduler provides a way to dynamically generate time window parameters through built-in expressions.

Common time window parameters include:

• ${startTime}: Represents the task's start time
• ${endTime}: Represents the task's end time

Time window expression examples:

• ${[yyyy-MM-dd 00:00:00 -1d]}: Represents the start time of the previous day (e.g., 2024-10-13 00:00:00)
• ${[yyyy-MM-dd 23:59:59 -1d]}: Represents the end time of the previous day (e.g., 2024-10-13 23:59:59)

These parameters can be used in task scripts, for example, when performing batch processing of the previous day's data, the script would look like:

#!/bin/bash  
startTime=${[yyyy-MM-dd 00:00:00 -1d]}  
endTime=${[yyyy-MM-dd 23:59:59 -1d]}  

echo "Processing data from $startTime to $endTime"  
# Commands to process data  

Dynamic Parameter Parsing:

During task execution, DolphinScheduler will parse time window parameters into actual time values based on the scheduled time. This allows the task to dynamically process data from different time periods according to the execution time.

Cron Expression

DolphinScheduler uses Cron expressions to configure the execution time or scheduling frequency of tasks. Cron expressions allow users to precisely control the scheduling time of tasks, such as executing daily, weekly, or at regular intervals.

Cron Expression Examples:

• Execute the task at 1 AM every day:

0 0 1 * * ?

• Execute the task at 2 AM every Monday:

0 0 2 ? * 1 

You can define when to run the task using Cron expressions when scheduling the task, which is suitable for configuring timed jobs.

Time Parameters in Complement Tasks

DolphinScheduler supports performing complement (backfill) operations on historical tasks that missed execution. Complement tasks often also involve time parameters. When using complement tasks, you can specify a certain time range, and the system will automatically re-execute tasks according to this range.

Complement Task Settings:

• You can select a specific date range to make up for tasks not executed during that period.
• During the complement process, DolphinScheduler will set appropriate time parameters based on the time range, such as ${startTime} and ${endTime}.

Time Formatting and Operations

DolphinScheduler provides some time formatting and operation functions that allow date addition and subtraction in time parameters. For example:

• ${[yyyy-MM-dd -1d]}: Gets the date of the previous day.
• ${[yyyy-MM-dd HH:mm:ss -7h]}: Gets the time 7 hours ago.

You can flexibly set the task scheduling time range through these time operations. For example, to process data from 7 days ago to today:

#!/bin/bash  
startTime=${[yyyy-MM-dd 00:00:00 -7d]}  
endTime=${[yyyy-MM-dd 23:59:59]}  

echo "Processing data from $startTime to $endTime"  

System Built-in Time Parameters

DolphinScheduler provides some built-in time parameters that users can directly use to implement tasks based on the current scheduling time:

• ${system.biz.date}: Business date, generally represents the current date of the scheduled task, formatted as yyyy-MM-dd.
• ${system.biz.curdate}: Complete format of the current date, formatted as yyyy-MM-dd HH:mm:ss.

These built-in parameters can be directly called in task scripts. For example:

#!/bin/bash  
echo "Business date is ${system.biz.date}"  

Time-Dependent Task Chains

In DolphinScheduler, time parameters can be used not only in a single task but also to configure time-dependent task chains through workflows. You can pass time parameters across different tasks so that downstream tasks can dynamically generate based on the output time of upstream tasks.

Functions

• Add/Subtract Months: add_months()

  • N years later: $[add_months(yyyyMMdd,12*N)]
  • N years earlier: $[add_months(yyyyMMdd,-12*N)]
  • N months later: $[add_months(yyyyMMdd,N)]
  • N months earlier: $[add_months(yyyyMMdd,-N)]

  
  

• Add/Subtract Days: +/- number

  • N weeks later: $[yyyyMMdd+7*N]
  • N weeks earlier: $[yyyyMMdd-7*N]
  • N days later: $[yyyyMMdd+N]
  • N days earlier: $[yyyyMMdd-N]
  • N hours later: $[HHmmss+N/24]
  • N hours earlier: $[HHmmss-N/24]
  • N minutes later: $[HHmmss+N/24/60]
  • N minutes earlier: $[HHmmss-N/24/60]

Summary

In DolphinScheduler, the use of time parameters is very flexible and is mainly used in the following scenarios:

• Timed Scheduling: Use Cron expressions to configure task execution time.
• Time Windows: Dynamically generate the time range for tasks, used in data processing and similar scenarios.
• Complement Operations: Reset the time range for missed tasks.
• Time Formatting and Operations: Provide rich time operation tools to perform date arithmetic.
• Global Parameters and System Built-in Parameters: Convenient for passing and using time information in workflows.

These time parameters in DolphinScheduler help users precisely control task scheduling and execution, especially in data processing, timed tasks, and stream computing scenarios.