A longitudinal force-balance model [31, 32] is introduced to describe the dominant transition dynamics. The nacelle tilt angle is denoted by λ , where λ = 0 ∘ corresponds to the rotor-borne configuration and λ f = 90 ∘ corresponds to the fully wing-borne configuration.

Let the longitudinal state vector be defined as x = s V x h V h θ q T , (1) where s denotes the longitudinal position, V x the forward velocity, h the altitude, V h the vertical velocity, θ the pitch angle, and q the pitch rate. The resultant longitudinal-plane airspeed used only in the aerodynamic lift and drag calculations is defined as V = V x 2 + V h 2 . (2)

The longitudinal state and resultant airspeed are defined in Equations 1, 2. The low-level control input ul is defined in Equation 3. u l = Ω 1 Ω 2 Ω 3 Ω 4 δ e T , (3) where Ω i is the rotational speed of the i -th rotor and δ e is the elevator deflection or an equivalent longitudinal control-surface input.

The dominant longitudinal force and moment balances are given by Equations 4 -9. m V ˙ x = F x x , u l , λ − D V , α + d V , (4) m V ˙ h = F z x , u l , λ + L V , α − mg + d h , (5) I y q ˙ = M y x , u l , λ , δ e + d q , (6) s ˙ = V x , (7) h ˙ = V h , (8) θ ˙ = q , (9) where m is the vehicle mass, I y is the pitch-axis moment of inertia, F x and F z are the longitudinal and vertical components of the rotor thrust components, L and D are the aerodynamic lift and drag, M y is the pitch moment, and d V , d h , d q are bounded disturbance terms.

The rotor thrust and its longitudinal and vertical components are modeled as T MR , i = ρ D MR 4 C T MR Ω i 2 , F x x , u l , λ = ∑ i = 1 4 sin λ T MR , i , F z x , u l , λ = ∑ i = 1 4 cos λ T MR , i , (10) where ρ is the air density, D MR is the rotor diameter, and C T MR is the rotor thrust coefficient.

The aerodynamic lift and drag are expressed as L V , α = C L α QS , D V , α = C D α QS , (11) where Q = 1 2 ρ V 2 is the dynamic pressure, S is the wing reference area, and C L α and C D α are the lift and drag coefficients, respectively.

The pitch moment is written as M y = − ∑ i = 1 4 l xi ⁡ sin λ T MR , i + C m α QS , (12) where l xi is the longitudinal moment arm from the i -th rotor to the aircraft center of gravity, and C m α QS denotes the aerodynamic pitching-moment term under the adopted simplified longitudinal model. The rotor thrust components, aerodynamic forces, and pitch moment are expressed in Equations 10-12.

The level-transition process is dominated by the coupling among rotor thrust vectoring, wing lift buildup, forward acceleration, and pitch stabilization. The longitudinal transition dynamics can be compactly written as x ˙ = f l x , u l , λ , d l , (13) where d l represents bounded longitudinal disturbances and unmodeled dynamics, including aerodynamic uncertainty, wind disturbance, and actuator-related perturbations. In this formulation, the low-level controller generates u l for altitude and attitude regulation, while the high-level scheduler determines the nacelle command λ . This separation allows different nacelle tilt schedules to be compared under the same stabilizing low-level control architecture.

Let T s denote the decision interval of the high-level scheduler and t k = k T s . The sampled nacelle command sequence is denoted by Λ N = λ 0 , λ 1 , … , λ N . (14)

For a forward level-transition task, the boundary condition is specified as λ 0 = 0 ∘ ,   λ N ≈ λ f = 90 ∘ . (15)

The sampled nacelle command sequence and boundary conditions are given by Equations 14, 15. The nacelle angle is a physical transition variable rather than an arbitrary independent command. Its evolution affects the distribution of rotor thrust between vertical support and forward propulsion. Therefore, an overly aggressive tilt command at low forward speed may reduce the vertical rotor-thrust component before sufficient wing lift has been established. Conversely, an overly conservative tilt command at high forward speed may delay transition completion and increase rotor-speed demand. These characteristics make the nacelle schedule a key high-level variable for level-transition performance.

During level-flight transition, the forward speed and nacelle angle must satisfy a feasible coupling relationship. This relationship is represented by a transition corridor in the V x , λ plane: C = V x , λ   |   λ min V x ≤ λ ≤ λ max V x , V x , 0 ≤ V x ≤ V x , f , (16) where V x , 0 and V x , f are the initial and target forward speeds, respectively. The functions λ min V x and λ max V x denote the lower and upper admissible nacelle-angle boundaries determined by aerodynamic lift capability, power margin, actuator limitations, and attitude-safety requirements.

The above formulation shows that the high-level nacelle scheduler must address two coupled requirements. First, the nacelle command should evolve as a physically realizable transition variable rather than as a sequence of independent absolute setpoints. Second, the generated command should remain compatible with the speed-dependent admissible corridor under the scheduled low-level flight-control architecture. Therefore, the problem considered in this paper is to design a feedback-capable high-level nacelle scheduler that maps measurable transition states to a smooth, actuator-compatible, and corridor-feasible nacelle command, without replacing the stabilizing low-level attitude and altitude controller.

The proposed scheduler is implemented under a hierarchical transition-control architecture. The high-level scheduler determines the evolution of the nacelle tilt angle, whereas the low-level controller tracks altitude and attitude references and allocates actuator commands to rotors and aerodynamic control surfaces. The scheduled low-level control architecture is kept unchanged for all compared tilt schedules, so that the performance differences can be attributed mainly to the high-level nacelle scheduling mechanism.

At the decision instant t k = k T s , the high-level scheduler receives measurable closed-loop transition information and generates the next nacelle command. The final command is then sent to the low-level controller, which produces the low-level actuator input u l , k . The closed-loop information flow is summarized in Equation 17 as z k → N s k → π ϕ a k → scale Δ λ k ⏟ Incremental TD 3  scheduler → rate / corridor λ k + 1 → low − level   controller u l , k → plant x k + 1 . (17)

This hierarchical separation is important for the scope of this work. The RL policy is not required to stabilize the fast attitude and altitude dynamics directly. Instead, it adapts the slow nacelle transition process according to the measured closed-loop scheduling vector. The stabilizing controller provides local attitude and altitude regulation for all tested schedules, while the proposed scheduler shapes the transition behavior through the nacelle command.

Remark 1

The proposed framework evaluates nacelle tilt scheduling rather than low-level stabilization. Therefore, the same scheduled low-level control architecture, actuator model, and simulation environment are used for all compared schedules. This setting isolates the effect of action representation and feasibility enforcement at the high-level scheduling layer.

The high-level scheduler uses measurable closed-loop transition information to determine the nacelle command evolution. The physical scheduling vector is defined as z k = e h , k V x , k Ω Σ , k λ k T , (18) where e h , k = h k − h d is the altitude tracking error, V x , k is the forward speed, Ω Σ , k = ∑ i = 1 4 Ω i , k is the total rotor-speed index, and λ k is the current nacelle angle. The altitude error characterizes the vertical tracking margin, the forward speed represents the progress of wing-lift buildup and determines the corridor interval, the rotor-speed index provides an effort-related signal, and the nacelle state is included because the scheduling decision depends on the current nacelle configuration.

For neural-network training and policy evaluation, the physical scheduling vector is normalized as s k = N z k = e ¯ h , k V ¯ x , k Ω ¯ Σ , k λ ¯ k T ∈ R 4 , (19) where e ¯ h , k = h k − h d e h , ⁡ max ,   V ¯ x , k = V x , k V x , f , (20) Ω ¯ Σ , k = ∑ i = 1 4 Ω i , k Ω Σ , ⁡ max ,   λ ¯ k = λ k λ f . (21)

The physical and normalized scheduling vectors are defined in Equations 18-21. The normalization does not introduce a new physical scheduling state; it only maps the measurable transition variables into a numerically suitable input range for the actor and critic networks.

Rapid longitudinal acceleration is not used directly as a policy observation. This choice avoids amplifying measurement noise in the high-level scheduler. Instead, acceleration and jerk are evaluated in the reward function to guide the policy toward smooth transition behavior.

The action representation is the key difference between the proposed scheduler and an absolute-action RL scheduler. A conventional absolute-action policy directly maps the scheduling state to the complete nacelle command: λ k + 1 abs = λ f π ϕ abs s k + 1 2 ,   π ϕ abs s k ∈ − 1 , 1 . (22)

Although this representation is simple, it treats the nacelle command at each decision instant as an independent setpoint. Therefore, the physical evolution between adjacent commands is not encoded in the action itself. Exploration noise, critic approximation errors, or policy updates may consequently lead to abrupt inter-sample command variations.

In contrast, the proposed policy outputs a normalized incremental action a k = π ϕ s k ,   a k ∈ − 1 , 1 . (23)

The action a k represents the intended change of the nacelle angle rather than the absolute nacelle command. It is then converted into a nacelle-angle increment and implemented through the discrete-time command-evolution law λ k + 1 = λ k + Δ λ k . (24)

Thus, the RL scheduler is converted from an absolute setpoint generator into a constrained nacelle-evolution generator. This distinction is fundamental: a smoothness penalty can discourage oscillatory actions, but it does not change the underlying action variable. By contrast, the incremental formulation directly modifies the command-generation structure itself, because the learned action is the nacelle-angle increment and the final command is produced by a constrained update.

The high-level scheduling policy is denoted by π ϕ : s k ↦ a k . (25)

After action scaling, rate saturation, and corridor projection, this policy induces a physical nacelle command sequence Λ N = λ 0 , λ 1 , … , λ N . (26)

The absolute-action and incremental-action parameterizations are given by Equations 22-26. A feasible scheduler should therefore generate a physically consistent nacelle evolution that respects both actuator-rate compatibility and the speed-dependent transition corridor, rather than a sequence of unrelated absolute setpoints.

The feasibility layer converts the normalized actor output into an executable nacelle command by enforcing two physical implementation requirements: nacelle-rate compatibility and speed-dependent corridor feasibility. This subsection describes the operational command-generation mechanism, whereas the post-training boundedness characterization later analyzes the resulting closed-loop scheduler under explicit assumptions.

The normalized incremental action is first scaled into a raw nacelle-angle increment: Δ λ k raw = T s λ ˙ ¯ a k , (27) where T s is the high-level decision interval and λ ˙ ¯ is the prescribed maximum nacelle tilt rate. For the forward level-transition task from the rotor-borne configuration to the wing-borne configuration, a monotonic rate-limited increment is adopted: Δ λ max = T s λ ˙ ¯ , Δ λ k = sat 0 , Δ λ max Δ λ k raw . (28)

Equivalently, the unidirectional saturation is expressed by Equation 29. Δ λ k = Δ λ max , Δ λ k raw > Δ λ max , Δ λ k raw , 0 ≤ Δ λ k raw ≤ Δ λ max , 0 , Δ λ k raw < 0 . (29)

The unidirectional saturation prevents reverse tilting and ensures that the command evolves consistently with the prescribed forward transition direction.

The pre-projection nacelle command is generated by bounded integration: λ k + 1 pre = λ k + Δ λ k . (30)

This step embeds the inter-sample evolution of the nacelle command into the action-generation process.

For implementation, a tightened transition corridor is introduced as C ρ c = V x , λ   |   λ min V x + ρ c ≤ λ ≤ λ max V x − ρ c , V x , 0 ≤ V x ≤ V x , f , (31) where ρ c > 0 is a corridor margin. The margin separates the commanded trajectory from the nominal corridor boundary and provides room for sampling-induced speed variation, modeling mismatch, and low-level tracking transients.

At the current forward speed V x , k , the tightened corridor induces the admissible nacelle-angle interval I ρ c V x , k = λ min V x , k + ρ c , λ max V x , k − ρ c . (32)

For the monotonic forward transition considered in this paper, the rate-compatible interval is R k + = λ k , λ k + Δ λ max . (33)

The final admissible update interval is the intersection I a V x , k , λ k = I ρ c V x , k ∩ R k + . (34)

When I a V x , k , λ k is nonempty, the final command is obtained by scalar projection: λ k + 1 = Π I a V x , k , λ k λ k + 1 pre , (35) where Π I a · denotes the Euclidean projection onto the closed interval I a .

Equivalently, if I a V x , k , λ k = λ _ a , k , λ ¯ a , k , (36) with λ _ a , k = max λ min V x , k + ρ c , λ k , (37) λ ¯ a , k = min λ max V x , k − ρ c , λ k + Δ λ max , (38) then the projection can be implemented as λ k + 1 = min max λ k + 1 pre , λ _ a , k , λ ¯ a , k . (39)

The tightened corridor, admissible intervals, and projected update are defined in Equations 31-34, 36-39. Thus, the actor output is not directly applied to the plant. Instead, it is first interpreted as an intended nacelle-angle increment and then processed by rate saturation and tightened-corridor projection. This design separates reward-level performance optimization from hard feasibility enforcement: the TD3 actor searches for a desirable transition behavior, while the feasibility layer enforces actuator-rate compatibility and practical corridor admissibility.

The transition corridor used in the feasibility layer should be interpreted as a scheduling corridor for high-level command generation, rather than as an exact representation of the full physical flight envelope. In practical tiltrotor transition, the admissible boundary may be affected by stall limits, power constraints, rotor--wing interference, attitude-safety requirements, actuator limits, and control-allocation capability. As a result, the raw transition envelope obtained from trimming, aerodynamic analysis, or offline simulation may contain piecewise segments, corner points, nonsmooth portions, or locally conservative regions.

The margin ρ c is selected to compensate for one-step boundary movement and to avoid operating exactly on the nominal corridor boundary. Let L c = max Lip λ min , Lip λ max (40) be an estimated gradient bound of the smoothed corridor boundaries over V x ∈ V x , 0 , V x , f . In a tabulated implementation, L c can be estimated from finite differences of the stored corridor table as L c = max i λ min V x , i + 1 − λ min V x , i V x , i + 1 − V x , i , λ max V x , i + 1 − λ max V x , i V x , i + 1 − V x , i . (41)

If the forward-speed variation over one high-level sampling interval is bounded by V x , k + 1 − V x , k ≤ Δ V max , (42) then a practical margin-selection rule is L c Δ V max ≤ ρ c < 1 2 min V x ∈ V x , 0 , V x , f λ max V x − λ min V x . (43)

A practical corridor-margin selection rule is given by Equations 40-43. The lower bound reserves angular margin for one-step corridor-boundary movement caused by forward-speed variation, whereas the upper bound prevents the tightened corridor from becoming empty or excessively conservative. The one-step speed bound Δ V max can be estimated either from an acceleration bound as Δ V max = a ¯ V x T s , or directly from the maximum one-step forward-speed variation observed in offline envelope-generation simulations.

If I a V x , k , λ k becomes empty in implementation, no command can simultaneously satisfy the tightened-corridor interval and the monotonic rate interval at that decision instant. This event is recorded as a feasibility-margin event, indicating that the corridor smoothing, the margin ρ c , the high-level sampling time T s , the nacelle-rate limit Δ λ max , or the nominal corridor table should be redesigned. In the simulations of this paper, these design elements are selected so that the admissible update interval remains nonempty along the evaluated transition trajectories.

This feasibility layer should be distinguished from a continuous-time safety proof for detailed nacelle-servo dynamics. It specifies how the sampled high-level command is generated to be rate-compatible and corridor-aware. Detailed servo lag, overshoot, actuator faults, and online corridor adaptation require additional actuator-level analysis or experimental validation, which are beyond the scope of the present high-level scheduling design.

The level-transition scheduler must balance several competing objectives. Increasing the nacelle angle promotes forward acceleration and transition completion, but it reduces the vertical component of rotor thrust and may increase altitude deviation. Conversely, delaying nacelle rotation preserves vertical thrust margin but may increase transition duration and rotor-speed demand. In addition, abrupt nacelle variations may induce excessive longitudinal acceleration and jerk, which are undesirable for actuator compatibility and transition smoothness.

The reward function is designed as a normalized maximization-oriented surrogate of the scheduling trade-off considered in this work. It combines mission progress, actuator-effort regulation, and handling-qualities penalties, while leaving hard rate and corridor admissibility to the feasibility layer. The altitude-error, rotor-speed, acceleration, jerk, and increment terms correspond to the penalty terms in (Equation 56) with opposite signs, whereas the forward-speed term corresponds to the negative speed term in the cost. Additional shaping terms are introduced to improve learning efficiency and to reduce unnecessary reliance on the projection layer.

The feasibility layer enforces the actuator-rate and corridor constraints independently of the reward. Therefore, the reward function primarily shapes transition performance, while hard admissibility is imposed by the command-generation mechanism in Subsection Rate-and-corridor feasibility layer.

As shown in the hierarchical decomposition, the instantaneous reward is explicitly divided into three categories: mission goal reward, actuator effort penalty, and flight safety and quality penalty, which correspond to transition completion & velocity promotion, actuator operation smoothness constraint, and flight state safety maintenance, respectively. This yields the unified structural expression r k = R mission − P actuator − P safe , qual , where R mission denotes the mission-oriented reward item, P actuator represents the penalty for consumption actuator operating effort and fluctuation, and P safe , qual stands for the comprehensive penalty related to flight safety and dynamic quality.

At each decision instant, the categorized reward components are specifically defined in Equations 44-47. r k = R mission − P actuator − P safe , qual (44) R mission = c v V ¯ x , k + c m λ ¯ k V ¯ x , k + r T , k (45) P actuator = c Ω 1 − λ ¯ k Ω ¯ Σ , k   2 + c Δ λ Δ λ k Δ λ max 2 + c c Φ c V x , k , λ k + 1 pre (46) P safe , qual = c h e ¯ h , k   2 + c a ρ a a x , k + c j ρ j j x , k (47) in which c h , c v , c m , c Ω , c a , c j , c Δ λ and c c are non-negative weighting coefficients. The mission goal reward encourages forward-speed buildup and accelerates the overall transition progress, where the mixed coupling term facilitates synchronous optimization of nacelle tilt angle and flight velocity. The actuator effort penalty regularizes the incremental nacelle command within the admissible rate interval (acting as a soft constraint rather than enforcing the hard rate limit, which is already handled by the feasibility layer), suppresses excessive rotor-speed demand, and guides the policy to generate candidate commands close to the feasible corridor region in advance. The safety and quality penalty mainly suppresses altitude tracking deviation, excessive longitudinal acceleration and severe jerk, so as to guarantee stable and comfortable transition flight.

The acceleration and jerk penalties are defined as ρ a a x , k = max 0 , a x , k − a th 2 , (48) ρ j j x , k = max 0 , j x , k − j th 2 , (49) where a th and j th are prescribed smoothness thresholds. These terms suppress aggressive longitudinal transients during the lift-redistribution process.

The corridor-related penalty is evaluated using the pre-projection command: Φ c V x , k , λ k + 1 pre = dist 2 λ k + 1 pre , I c V x , k , (50) where dist · , I c is the Euclidean distance to the tightened corridor interval. This term encourages the actor to generate commands that are naturally close to the admissible region before projection, thereby reducing reliance on the feasibility layer during training.

The terminal completion reward r T , k is given by r T , k = c T , V x , k ≥ V x , tar  and λ k ≥ λ f − ϵ λ , 0 , otherwise , (51) where V x , tar is the target transition-completion speed, ϵ λ is a small terminal nacelle-angle tolerance, and c T > 0 is the positive terminal-completion bonus. The acceleration, jerk, corridor, and terminal reward terms are given by Equations 48-51.

Remark 2

The rotor-speed-related term in the reward function serves as an operation effort proxy. It reflects the variation tendency of rotor rotational speed demand during aerodynamic lift redistribution and transition scheduling, rather than establishing a high-fidelity calibrated model of electrical energy consumption, motor operating efficiency or battery power loss.

A Beetle Swarm (BS) Optimization based reference schedule is used to provide a physically feasible benchmark and actor warm-start source. The BS optimization is performed offline over a finite-dimensional parameterization of the nacelle schedule. This reference is not imposed as a tracking command during final TD3 policy evaluation; it is used only to initialize the actor toward a feasible transition trend and to provide an interpretable non-learning baseline.

The nacelle schedule is parameterized by a normalized PCHIP interpolation curve. Let p = p 1 , p 2 , … , p M T (52) denote the free interpolation-node vector. Together with the boundary conditions λ 0 = 0 ∘ ,   λ T tilt = λ f , (53) the parameter vector p determines a smooth baseline nacelle profile λ BS t ; p .

The offline optimization problem is written as p ⋆ = arg min p ∈ P J BS p , (54) where P is the feasible parameter domain and J BS (55) is a multi-objective transition-performance index including altitude error, speed buildup, rotor-speed-related effort, acceleration, jerk, and terminal completion terms. The BS schedule parameterization and offline optimization are formulated in Equations 52-54.

For a finite transition horizon T = N T s , the transition-performance index is defined in Equations 55-57. J BS = ∑ k = 0 N − 1 l z k , Δ λ k + l f z N (55) l z k , Δ λ k = w h e h , k 2 − w v V x , k + w Ω 1 − λ k λ f Ω Σ , k 2 + w a ρ a a x , k + w j ρ j j x , k + w Δ Δ λ k Δ λ max 2 (56) l f z N = w vf V x , N − V x , f 2 + w λ f λ N − λ f 2 + w h f e h , N 2 (57) where l · is the stage cost and l f · is the terminal cost.

w h , w v , w Ω , w a , w j , w Δ ≥ 0 are weighting coefficients. The terms in (56) penalize altitude error, rotor-speed-based effort, excessive longitudinal acceleration, jerk, and aggressive nacelle increments, while rewarding forward-speed buildup through the negative V x , k term. The functions ρ a · and ρ j · are nonnegative penalty functions used to suppress acceleration and jerk beyond desired smoothness levels. The acceleration a x , k and jerk j x , k are performance-evaluation variables rather than direct components of the scheduling state z k . w vf , w λ f , w h f ≥ 0 . This term encourages terminal speed buildup, completion of nacelle transition, and bounded terminal altitude error.

The criterion in (Equation 55) specifies the desired scheduling trade-off. The offline BS criterion follows the same scheduling trade-off as the reward design, but is expressed as a deterministic finite-horizon cost for reference generation. Compared to the reinforcement-learning reward, which acts as its normalized maximization-oriented surrogate with additional shaping terms for training efficiency and feasibility, this criterion eliminates these unnecessary implementation-related terms.

The optimized reference is used in two ways. First, it serves as a deterministic benchmark in the simulation section. Second, it provides supervised samples for actor warm-starting. Specifically, state--action pairs are extracted along the BS schedule: D BS = s k BS , a k BS k = 0 N − 1 , (58) where a k BS = λ k + 1 BS − λ k BS Δ λ max . (59)

The actor can then be initialized by minimizing L pre = 1 D BS ∑ s , a ∈ D BS ∥ π ϕ s − a ∥ 2 . (60)

Supervised warm-start samples and the pretraining loss are given by Equations 58–60.

Remark 3

The BS reference provides an initial feasible tilt trend, but it does not constrain the final learned TD3 policy to reproduce the BS trajectory. After warm-starting, the actor is further optimized through closed-loop interaction with the environment, and the final policy is selected according to the fixed evaluation criterion described in Section Simulation results and performance evaluation.

The proposed scheduler is trained using the twin delayed deep deterministic policy gradient algorithm. TD3 is adopted because it is suitable for continuous-action control and mitigates the overestimation bias of deterministic actor--critic learning through clipped double critics, delayed actor updates, and target policy smoothing.

The actor network is denoted by π ϕ s , and the two critic networks are denoted by Q θ 1 s , a and Q θ 2 s , a . Their target networks are denoted by π ϕ ′ , Q θ 1 ′ , and Q θ 2 ′ . At each training step, a transition tuple s k , a k , r k , s k + 1 (61) is stored in the replay buffer B .

For a mini-batch sampled from B , the target action is computed as a ∼ k + 1 = clip π ϕ ′ s k + 1 + ϵ , − 1 , 1 ,   ϵ ∼ clip N 0 , σ tar 2 , − c ϵ , c ϵ . (62)

The target value is y k = r k + γ min i = 1 , 2 Q θ i ′ s k + 1 , a ∼ k + 1 , (63) where γ is the discount factor. Each critic is updated by minimizing L θ i = 1 N b ∑ k Q θ i s k , a k − y k 2 ,   i = 1 , 2 , (64) where N b is the mini-batch size.

The actor is updated less frequently according to the deterministic policy gradient: ∇ ϕ J π ≈ 1 N b ∑ k ∇ a Q θ 1 s k , a | a = π ϕ s k ∇ ϕ π ϕ s k . (65)

The target networks are updated by soft update: θ i ′ ← τ θ i + 1 − τ θ i ′ ,   ϕ ′ ← τ ϕ + 1 − τ ϕ ′ , (66) where τ ∈ 0 , 1 is the soft-update coefficient.

During environment interaction, the exploratory action is a k = clip π ϕ s k + η k , − 1 , 1 ,   η k ∼ N 0 , σ exp 2 . (67)

The sampled action is then passed through the same scaling, rate-saturation, and corridor-projection mechanism described in Subsection Rate-and-corridor feasibility layer. Therefore, both training and evaluation use the same physical command-generation structure.

The training procedure is summarized in Algorithm 1.

Algorithm 1

1: Initialize actor π ϕ , critics Q θ 1 , Q θ 2 , and target networks.

The preceding subsections define the command-generation and training procedure of the proposed scheduler. Before characterizing the post-training boundedness property, the scheduled low-level closed-loop representation used in the analysis is introduced below.

The role of the high-level scheduler is therefore to generate a feasible nacelle-angle evolution λ t while the scheduled low-level controller family maintains altitude and attitude tracking over the admissible transition corridor.

For each admissible nacelle command, the low-level controller generates u l to track the altitude and attitude references. Since the control effectiveness and aerodynamic characteristics vary significantly with the forward speed and nacelle angle, the low-level controller is represented as a scheduled controller family rather than a single fixed-parameter feedback law.

Let the scheduling variable be defined as σ = V x λ T ∈ C , (68) where C denotes the admissible transition corridor. In implementation, this admissible corridor is represented by a smoothed scheduling corridor constructed from the raw offline envelope, as detailed in Subsection Rate-and-corridor feasibility layer.

The low-level control law is written as u l = κ l , σ x , x r , λ , (69) where x r denotes the low-level reference signal, and the subscript σ indicates that the controller parameters are scheduled according to the current transition operating condition. This notation covers gain-scheduled, interpolated, or locally parameterized low-level controllers without requiring the high-level scheduler to redesign the stabilizing controller online.

Substituting (Equation 69) into (Equation 13) yields the scheduled closed-loop longitudinal dynamics by Equation 70. x ˙ = F l , σ x , λ , d l , (70) where F l , σ x , λ , d l = f l x , κ l , σ x , x r , λ , λ , d l . (71)

With this scheduled closed-loop representation, the trained actor can now be treated as a bounded nonlinear scheduler, and the local boundedness of the resulting tracking-error dynamics can be characterized.

The preceding subsections define how the actor output is converted into an executable nacelle command through incremental action scaling, rate saturation, bounded integration, and corridor projection. This subsection provides a post-training boundedness characterization of the resulting closed-loop scheduler. The purpose is not to prove TD3 convergence, global optimality, or global nonlinear stability. Instead, the trained actor is treated as a bounded nonlinear scheduler, and the local boundedness of the scheduled low-level tracking error is characterized under explicit assumptions.

The analysis is conducted for the implemented closed-loop system after training. Suppose that the feasibility layer and the corridor-margin design described in Subsection Rate-and-corridor feasibility layer generate an implemented nacelle schedule satisfying σ t = V x t λ t T ∈ C ,   t ∈ 0 , T , (72) over the considered level-transition interval. This condition means that the high-level scheduling trajectory remains inside the admissible speed–nacelle-angle corridor used for the scheduled low-level control design. Although the transition task is evaluated over a finite horizon, the boundedness result below is stated in the standard UUB form. It applies when the implemented schedule remains in C during transition and then stays in an admissible terminal operating region after transition completion. Over the finite transition horizon, the same differential inequality directly provides an explicit local tracking-error bound. In the simulations, corridor membership is verified by evaluating the generated transition trajectories in the forward-speed–nacelle-angle plane.

The high-level scheduler determines the nacelle-angle evolution, whereas the scheduled low-level controller family stabilizes the attitude and altitude loops over the admissible transition corridor. Let the longitudinal tracking error be defined as e = e h e ˙ h e θ e q T , (73) where e h = h − h d ,   e θ = θ − θ d ,   e q = q − q d . (74)

For the level-transition task considered in this paper, the desired altitude is constant and the nominal desired pitch motion is selected as V h d = 0 ,   θ d = 0 ,   q d = 0 . (75)

From the scheduled closed-loop longitudinal dynamics (Equation 71), the corresponding tracking-error dynamics can be written locally as e ˙ = Φ e , σ e , d l , (76) where Φ e , σ · is the error-system vector field induced by the scheduled low-level closed-loop dynamics. The nominal zero-error manifold is assumed to be locally consistent with the scheduled low-level controller, namely Φ e , σ 0 , 0 = 0 ,   ∀ σ ∈ C . (77)

The tracking-error state, reference conditions, local error dynamics, and zero-error consistency condition are defined in Equations 73 -77. Any residual trim mismatch or unmodeled bias is included in the bounded disturbance term d l . Around the nominal zero-error manifold, the scheduled tracking-error dynamics are represented by the local expansion e ˙ = A σ e + B d , σ d l + Δ σ e , (78) where A σ denotes the Jacobian matrix of the scheduled low-level closed-loop error dynamics, B d , σ is the disturbance input matrix, and Δ σ e collects higher-order nonlinear residuals.

In (Equation 78), the nacelle angle λ affects the tracking-error dynamics through the scheduling variable σ = V x , λ T and the scheduled closed-loop vector field F l , σ . Therefore, the nacelle-rate bound established by the incremental command-generation mechanism is not introduced as an independent low-level input. Instead, rate-compatible nacelle evolution supports the validity of the local scheduled model by preventing abrupt movement among operating conditions.

Assumption 1

(Uniform local stability of the scheduled low-level controller). For all σ ∈ C , there exist symmetric positive definite matrices P = P T > 0 and Q = Q T > 0 , independent of σ , such that A σ T P + P A σ ≤ − Q . (79)

Assumption 2

(Bounded disturbance and local residual). The lumped longitudinal disturbance arising from aerodynamic uncertainty, external wind interference and unmodeled dynamics is bounded as d l t ≤ d ¯ l . (80)

Herein, d ¯ l denotes the predefined upper amplitude bound of the overall lumped system disturbance. Moreover, B d , σ is uniformly bounded over the admissible transition corridor: B d , σ ≤ B ¯ d ,   ∀ σ ∈ C , (81) where B ¯ d represents the maximum spectral norm bound of the disturbance input matrix varying with scheduling states. The nonlinear residual satisfies Δ σ e ≤ c Δ e 2 (82) for all σ ∈ C and all e inside a compact neighborhood D e = e : e ≤ r e . (83)

The common-Lyapunov and local bounding assumptions are stated in Equations 79 -83. In particular, c Δ is the quadratic Lipschitz constant used to quantify the growth rate of high-order unmodeled nonlinear terms, and r e stands for the maximum allowable radius of the local error convergence domain where the above nonlinear bounding condition holds. e ˙ = A σ e + B d , σ d l + Δ σ e , (84) where σ = V x , λ T ∈ C . Suppose that the implemented schedule satisfies (Equation 72) during transition and remains in an admissible terminal operating region after completion. Suppose further that Assumptions 1 and 2 hold. Let p min = λ min P ,   p max = λ max P ,   q min = λ min Q . (85)

The eigenvalue bounds used in the theorem are defined in Equation 85.

Remark 4

Assumption 1 is a uniform local stability condition for the scheduled low-level controller family over the considered transition corridor. It is not an assumption on the convergence or stability of the TD3 learning algorithm. The trained actor provides a bounded high-level scheduling command, while the scheduled low-level controller family is responsible for stabilizing the altitude and attitude tracking dynamics. The use of a common P is conservative, but it avoids additional terms associated with parameter-dependent Lyapunov functions or switching among local Lyapunov functions. This condition is used only to obtain a local boundedness characterization of the scheduled low-level error dynamics.

Theorem 1

(Conditional uniform ultimate boundedness). Consider the scheduled tracking-error dynamics.

Choose a local radius r 0 ∈ ( 0 , r e such that 2 p max c Δ r 0 ≤ q min 4 . (86)

Then, for any solution of (Equation 84) that remains in D r 0 = e : ∥ e ∥ ≤ r 0 , (87) the tracking error is uniformly ultimately bounded. Specifically, define η = q min 4 p max , (88) and Γ d = 2 p max 2 B ¯ d 2 q min d ¯ l 2 . (89)

Then the Lyapunov function V e = e T Pe (90) satisfies V e t ≤ e − η t V e 0 + Γ d η 1 − e − η t . (91)

Consequently, limsup t → ∞ ∥ e t ∥ ≤ Γ d η p min . (92)

Moreover, if there exists a constant V ¯ e satisfying V e 0 ≤ V ¯ e ,   Γ d η < V ¯ e < p min r 0 2 , (93) then the sublevel set Ω V ¯ e = e : V e e ≤ V ¯ e (94) is positively invariant, and the bound (Equation 92) holds for all t ≥ 0 .

Proof. Choose the Lyapunov function candidate V e = e T Pe . (95)

Since P = P T > 0 , it satisfies p min ∥ e ∥ 2 ≤ V e ≤ p max ∥ e ∥ 2 . (96)

Taking the derivative of V e along (Equation 84) gives V ˙ e =   e T A σ T P + P A σ e + 2 e T P B d , σ d l + 2 e T P Δ σ e . (97)

Using Assumption 1, one obtains V ˙ e ≤ − q min ∥ e ∥ 2 + 2 p max B ¯ d ∥ e ∥ ∥ d l ∥ + 2 p max c Δ ∥ e ∥ 3 . (98)

For e ∈ D r 0 , condition (Equation 86) implies 2 p max c Δ ∥ e ∥ 3 ≤ 2 p max c Δ r 0 ∥ e ∥ 2 ≤ q min 4 ∥ e ∥ 2 . (99)

Substituting (Equation 99) into (Equation 98) yields V ˙ e ≤ − 3 q min 4 ∥ e ∥ 2 + 2 p max B ¯ d ∥ e ∥ ∥ d l ∥ . (100)

By Young’s inequality, 2 p max B ¯ d ∥ e ∥ ∥ d l ∥ ≤ q min 2 ∥ e ∥ 2 + 2 p max 2 B ¯ d 2 q min ∥ d l ∥ 2 . (101)

Therefore, V ˙ e ≤ − q min 4 ∥ e ∥ 2 + 2 p max 2 B ¯ d 2 q min ∥ d l ∥ 2 . (102)

Using ∥ d l t ∥ ≤ d ¯ l , one obtains V ˙ e ≤ − q min 4 ∥ e ∥ 2 + Γ d . (103)

From the upper quadratic bound in (Equation 96), V e ≤ p max ∥ e ∥ 2 , (104) ∥ e ∥ 2 ≥ V e p max . (105)

The intermediate Lyapunov inequalities are given by Equation 100 -105. Hence, V ˙ e ≤ − q min 4 p max V e + Γ d = − η V e + Γ d . (106)

Solving the scalar differential inequality (Equation 106) gives V e t ≤ e − η t V e 0 + Γ d η 1 − e − η t , (107) which proves (Equation 91). Taking the limit superior on both sides gives limsup t → ∞ V e t ≤ Γ d η . (108) p min ∥ e ∥ 2 ≤ V e , (109) one obtains limsup t → ∞ ∥ e t ∥ ≤ Γ d η p min , (110) which proves (92).

It remains to verify the sufficient local invariance condition. Suppose that (Equation 93) holds. On the boundary V e = V ¯ e (Equation 106), gives V ˙ e ≤ − η V ¯ e + Γ d < 0 . (111)

Therefore, the trajectory cannot leave the sublevel set Ω V ¯ e . Since V ¯ e < p min r 0 2 , every e ∈ Ω V ¯ e satisfies ∥ e ∥ 2 ≤ V e p min ≤ V ¯ e p min < r 0 2 . (112)

The remaining proof steps are given by Equations 107 -112. Thus, Ω V ¯ e ⊂ D r 0 , and the residual bound remains valid for all t ≥ 0 . This completes the proof.

Remark 5

Theorem 1 establishes a local and conditional boundedness result for the scheduled closed-loop tracking-error system. The conclusion depends on three conditions: the implemented nacelle schedule remains inside the admissible transition corridor, the scheduled low-level controller family satisfies the common Lyapunov condition over the corridor, and the disturbance and nonlinear residual remain bounded in the local operating region. The result does not prove TD3 convergence, global stability, or global optimality of the learned scheduler.

Remark 6

The nacelle-rate bound does not appear as an independent input term in (Equation 84) because the compact closed-loop model uses λ as a scheduling variable and represents the low-level controller as a σ -scheduled controller family. The role of the incremental action parameterization is to keep the evolution of σ = V x , λ T inside the admissible corridor and compatible with the nacelle actuator rate. This prevents abrupt movement among scheduled operating conditions and supports the validity of the uniform local stability condition in Assumption 1.

In summary, the boundedness characterization complements the feasibility layer. The feasibility layer defines how the actor output is converted into a corridor-aware and rate-compatible sampled command, whereas Theorem 1 characterizes the local tracking-error behavior once the implemented schedule evolves inside the admissible corridor. This result supports the mechanism of ESIT without assigning stabilizing responsibility to the TD3 actor itself.

The simulations were conducted in MATLAB/Simulink using the nonlinear quad-tiltrotor UAV model. All tilt-scheduling strategies were evaluated with the same scheduled low-level attitude/altitude control architecture, so that performance differences mainly reflect the high-level nacelle scheduling strategy.

The task is a forward level-flight transition from the rotor-borne configuration to the wing-borne configuration. The nacelle angle starts from 0 ∘ and reaches 90 ∘ at the terminal configuration. The height command is constant, and the desired attitude is set to zero during transition. The recorded variables include height error, forward velocity, nacelle angle, nacelle rate, longitudinal acceleration, longitudinal jerk, and rotor-speed-related quantities.

For deterministic policy evaluation, exploration noise was disabled. For the reinforcement-learning-based strategies, the saved agent with the highest training return was used as the evaluation policy. For TD3-incremental, the BS-generated reference was used only for actor warm-starting and did not constrain the final learned policy.

The transition corridor was constructed from offline envelope analysis in the forward-speed--nacelle-angle plane. One boundary was obtained from trim-based admissibility evaluation in Simulink, and the other was determined by the rotor-thrust limitation. The same corridor table was used for all safety-filtered policies and for corridor-feasibility verification.

The BS reference optimization and TD3 training used a 31   s horizon, whereas all final deterministic evaluations, baseline comparisons, TD3-variant comparisons, corridor evaluations, Monte Carlo tests, and observation-noise tests used a unified 34   s simulation duration. Unless otherwise stated, all reported performance indices correspond to this final evaluation horizon.

The main simulation settings are summarized in Table 1.

TD3 training was conducted in two stages: a critic burn-in stage with an almost frozen actor, followed by actor fine-tuning with a small learning rate. The main training parameters are listed in Table 2.

For each simulation, the following performance indices were calculated: RMSE e h = 1 T ∫ 0 T e h 2 t  dt (113) RMS x = 1 T ∫ 0 T x 2 t  dt , x ∈ a x , j x , λ ˙ (114) J Ω = ∫ 0 T ∑ i = 1 4 Ω i t 2 dt (115) where e h is the height tracking error, V x is the forward velocity, a x is the longitudinal acceleration, j x = a ˙ x is the longitudinal jerk, and J Ω is the rotor-speed-based effort proxy. The performance indices used in the simulations are defined in Equations 113 -115.

In addition, V x T , max e h t , and max λ ˙ t were recorded.

The transition completion time is defined as the first time when V x t reaches 0.98 V x , f = 23.52   m / s and remains above this threshold for at least 0.5   s . This avoids declaring completion from a transient velocity crossing.

Representative non-learning tilt schedules are first evaluated under the same scheduled low-level control architecture. This comparison serves two purposes: it shows the sensitivity of level-flight transition to nacelle timing, and it verifies that the shared low-level controller maintains bounded pitch and altitude responses under feasible nacelle schedules.

The evaluated schedules include a linear schedule, a slow–fast diagnostic schedule, a fast–slow diagnostic schedule, and a BS-optimized reference schedule. The predefined schedules were generated by normalized PCHIP interpolation with five free nodes between λ 0 = 0 ∘ and λ T tilt = 90 ∘ , where T tilt = 30   s . The command was held at the terminal value after T tilt . The BS-optimized reference was generated using the 31   s optimization horizon, and all schedules were evaluated under the unified 34   s deterministic horizon.

The normalized free-node vectors are as followed.

The linear baseline used p lin = 0.1667 , 0.3333 , 0.5000 , 0.6667 , 0.8333 .

The slow--fast diagnostic schedule used p SF = 0.04 , 0.09 , 0.16 , 0.30 , 0.55 .

The fast--slow diagnostic schedule used p FS = 0.40 , 0.65 , 0.80 , 0.90 , 0.95 .

The BS-optimized baseline used p opt = 0.110 , 0.166 , 0.351 , 0.709 , 0.940 .

Figure 2 shows the baseline tilt schedules and the corresponding closed-loop responses. The fast--slow schedule advances most of the nacelle rotation to the early transition stage, whereas the slow--fast schedule delays the main tilting process. These different timing patterns produce different altitude-error, forward-velocity, and total-rotor-speed responses. The result confirms that the nacelle schedule is an active high-level variable that affects altitude regulation, speed buildup, and rotor-effort redistribution during level-flight transition.

The low-level pitch and height tracking bounds under different schedules are summarized in Table 3. The maximum absolute pitch angle remains below 1.10 ∘ for all evaluated strategies. For the TD3-based strategies, the maximum absolute pitch angle is below 0.46 ∘ , and the RMS pitch angle remains small. These results indicate that the shared low-level controller remains well behaved during the evaluated transitions.

Table 3 is not intended to rank all schedules by tracking accuracy or to prove global nonlinear stability. It only verifies that the subsequent TD3 comparisons are not caused by low-level instability. The repeated maximum pitch value observed for several TD3-based cases mainly reflects a common low-level transient under the shared stabilization loop; the corresponding height-error bound, nacelle-rate demand, jerk response, and corridor trajectory still differ across high-level schedulers.

This subsection isolates the effect of action parameterization on TD3-based nacelle scheduling. Three policies are compared under the same observation variables, scheduled low-level control architecture, reward components, and deterministic evaluation protocol: absolute-action TD3, penalty-based absolute-action TD3, and the proposed incremental-action TD3.

The absolute-action policy directly outputs the complete nacelle angle, so adjacent commands are not structurally coupled. The penalty-based policy retains this absolute-action representation but adds an oscillation penalty. In contrast, the proposed policy outputs a nacelle-angle increment before command filtering. Therefore, the comparison separates action-level reparameterization from reward-level smoothness shaping.

Figure 3 compares the nacelle-angle responses of the three TD3 variants. All policies reach the terminal nacelle configuration, but their transient command evolution is different. The absolute-action TD3 policy exhibits local oscillations during the middle-to-late transition stage, especially when the aircraft shifts from rotor-dominant support to wing-lift-supported flight. TD3 + penalty attenuates this fluctuation, but local irregularities remain. By contrast, TD3-incremental produces a smoother and nearly monotonic nacelle-angle trajectory.

The nacelle-rate responses in Figure 4 further highlight the difference among the action representations. Absolute-action TD3 generates large and rapidly alternating rate peaks. TD3 + penalty reduces the peak level but does not remove local rate fluctuations. TD3-incremental keeps the nacelle-rate response within a narrow range throughout the transition, indicating better compatibility with rate-limited nacelle actuation.

These results indicate that the oscillatory behavior of absolute-action TD3 is mainly associated with the mismatch between independent absolute-angle commands and the physical continuity requirement of nacelle rotation. Reward-level oscillation penalties can mitigate this behavior, but the improvement remains tuning-dependent. The incremental action representation modifies the command structure itself and therefore provides a more direct mechanism for smooth nacelle evolution.

Command smoothness alone does not guarantee transition feasibility. A nacelle trajectory may be smooth but still leave the admissible speed--nacelle-angle corridor. Conversely, a corridor-feasible trajectory may still impose undesirable nacelle-rate demand. Therefore, the generated schedules are examined in the forward-speed--nacelle-angle plane.

Figure 5 shows the transition trajectories of the predefined baselines, the BS-optimized baseline, and the TD3-based schedulers. To match the plotting convention of the generated corridor map, the vertical axis is displayed as the converted nacelle angle 90 ∘ − λ t . This conversion is used only for visualization and does not change the feasibility criterion defined in (Equation 16). All compared strategies remain inside the prescribed transition corridor during the evaluated level-flight transition.

The corridor trajectories show that feasibility and smoothness are distinct properties. The absolute-action TD3 trajectory satisfies the corridor constraint but still contains local irregularities, consistent with the oscillatory nacelle response in Figure 3. The penalty-based variant reduces these irregularities, whereas TD3-incremental produces the smoothest feasible path among the TD3-based schedulers. Thus, the smoothness advantage of the proposed scheduler is achieved within the same admissible speed--nacelle-angle region, rather than by relaxing or leaving the prescribed transition corridor.

Monte Carlo simulations are conducted to examine the consistency of the smoothness improvement under randomized transition conditions. Each TD3-based strategy was evaluated over 50 scenarios, and the same random scenarios were reused for absolute-action TD3, TD3 + penalty, and TD3-incremental. The randomized quantities include wind amplitude, frequency, phase, initial Euler angles, and initial forward speed.

The evaluation focuses on nacelle-rate demand, longitudinal acceleration, longitudinal jerk, altitude response, terminal speed, and rotor-speed-based effort. Figure 6 shows the smoothness statistics, and Table 4 summarizes the numerical results. Absolute-action TD3 exhibits the largest nacelle-rate demand and dispersion. TD3 + penalty reduces the mean oscillation level, but its maximum rate and jerk remain more sensitive to scenario variations. TD3-incremental yields the smallest nacelle-rate demand and the most consistent smoothness metrics among the three TD3 variants.

Compared with absolute-action TD3, TD3-incremental reduces RMS λ ˙ from 22.2302   deg ⁡ / s to 2.9804   deg ⁡ / s , and reduces max λ ˙ from 100.2860   deg ⁡ / s to 4.9256   deg ⁡ / s . Compared with TD3 + penalty, it further reduces the maximum nacelle-rate demand from 29.8180   deg ⁡ / s to 4.9256   deg ⁡ / s . The standard deviations of both nacelle-rate metrics are also smaller for TD3-incremental, indicating that the smoother command behavior is less sensitive to the tested random variations.

The same trend appears in the longitudinal smoothness metrics. TD3-incremental reduces RMS j x from 3.5789   m / s 3 for absolute-action TD3 and 0.3750   m / s 3 for TD3 + penalty to 0.1720   m / s 3 . Meanwhile, all three TD3 variants maintain terminal forward speeds close to the target value. Thus, the smoother response of TD3-incremental is not obtained by failing to complete the transition, but by changing the nacelle command evolution mechanism.

These Monte Carlo results do not constitute a global robustness guarantee. They show that, under the tested randomized scenarios, the proposed incremental action structure provides more consistent nacelle-rate and jerk reduction than absolute-action TD3 and reward-penalized absolute-action TD3.

This subsection compares the overall closed-loop transition performance of representative strategies and further evaluates the sensitivity of TD3-incremental to observation noise.

The simulation results support the main mechanism of the proposed ESIT framework. The baseline comparison confirms that nacelle timing is a dominant high-level factor affecting altitude response, speed buildup, rotor-speed demand, and longitudinal smoothness. The TD3 ablation shows that the incremental action representation provides a structural mechanism for smooth command generation beyond reward-level oscillation penalties. The corridor evaluation verifies that this smoothness improvement is achieved inside the prescribed speed--nacelle-angle corridor. The Monte Carlo and observation-noise evaluations further show that the smoothness advantage remains consistent under the tested randomized scenarios and perturbed policy inputs.

These findings are consistent with the command-generation mechanism developed in Section Proposed corridor-constrained incremental TD3 tilt strategy. By learning a bounded nacelle-angle increment instead of an absolute nacelle command, the scheduler limits inter-sample command variation before the command reaches the low-level controller. The feasibility layer then enforces rate compatibility and sampled-command corridor admissibility. As a result, the learned scheduler improves nacelle-command smoothness without requiring changes to the scheduled low-level attitude/altitude control architecture.

The results should nevertheless be interpreted within the simulation scope of this paper. The rotor-speed-based index J Ω is used as an effort proxy rather than as a calibrated propulsion-power or battery-energy model. The transition corridor is constructed from the considered vehicle model and offline envelope analysis, so the feasibility conclusion depends on the fidelity of the corridor table. In addition, the Monte Carlo and observation-noise tests evaluate representative variations in wind parameters, initial conditions, and policy inputs, but they do not constitute a global robustness guarantee.