In an unknown three-dimensional environment, multi-UAV cooperative reconnaissance aims to maximize exploration coverage while ensuring flight safety and coordination and minimizing redundant exploration and path costs. Consider a swarm of N UAVs, where the position of the i-th UAV is x i , y i , z i . Each UAV acquires local environmental information through onboard sensors and updates the global map via map sharing.

This paper adopts a three-dimensional voxel map to discretely represent the environment. Let the environment map at time t be denoted as shown in Equation 1: M t = E t , O t , X t (1) where E t , O t and X t represent the free, occupied, and unknown states, respectively. During the exploration of an unknown environment, a Frontier point is defined as a boundary point between a known free region and an unknown region, as expressed in Equation 2: F t = f 1 , f 2 , . . . , f n (2)

Based on the above problem model, this paper further constructs the Voronoi-Frontier-NBV algorithm to realize candidate viewpoint generation, cooperative region partitioning, and next-best-view selection.

The algorithm detects and clusters Frontier points to generate candidate exploration targets, partitions them among UAVs using dynamic Voronoi regions, and selects the next-best viewpoints through an evaluation function. A path planner then generates the corresponding UAV trajectories.

To reduce the number of candidate points and improve decision-making efficiency, this paper clusters the Frontier point set to obtain several cluster centers which are obtained using the MeanShift algorithm. Starting from the initial seed points, each center is iteratively updated toward the mean position of the Frontier points within the specified bandwidth until convergence. The converged locations are then merged to form the final cluster centers g k , as given in Equation 3: g k = ∑ f j ∈ F t K h f j − g k f j ∑ f j ∈ F t K h f j − g k (3) where K h · is a Gaussian kernel function with bandwidth h . The reachable frontier point closest to each cluster center is selected as a candidate viewpoint, as defined in Equation 4, and these viewpoints form the candidate viewpoint set given in Equation 5: c k = arg min f j ∈ F k f j − g k 2 (4) C t = c 1 , c 2 , . . . , c k (5)

In the multi-UAV scenario, to reduce repeated exploration caused by different UAVs selecting identical or nearby targets, the concept of dynamic Voronoi partitioning is adopted to allocate candidate points. The arrival cost from the i-th UAV to the position q is defined in Equation 6: R d i q = α d d i q + α h Δ h i q + α ρ ρ i q (6) where d i q denotes the length of the shortest feasible path, Δ h i q denotes the altitude variation cost, and ρ i q denotes the local risk density. The dynamic Voronoi region is defined in Equation 7: V i t = q ∈ F t ∪ C t | R d i q ≤ R d j x , ∀ j ≠ i (7)

Each UAV preferentially performs NBV scoring within its own candidate region and selects the candidate viewpoint with the highest score as its flight target. The NBV scoring function is given in Equation 8, and the corresponding coefficient vector is defined in Equation 9: S i c , t = ω i g i t I c − ω d i s t i t D i c − ω c o o p i t P i c − ω o b s i R o b s c (8) ω i t = ω i g i , ω d i s t i , ω c o o p i , ω o b s i (9) where I c denotes information gain, D i c denotes path distance, P i c denotes coordination cost, and R o b s c denotes obstacle risk cost.

Multi-UAV target selection depends on UAV states, path costs, target distribution, unknown regions, and obstacle risks, constituting a multi-agent cooperative decision problem. Independent learning may cause non-stationarity, while the continuous NBV weights required in this study are unsuitable for discrete-action methods. Therefore, MADDPG is adopted to learn adaptive NBV coefficients through centralized training with joint states and actions and decentralized execution based on local observations.

Each UAV in the system corresponds to an agent. The local observation of the i-th UAV at time t is denoted as o i . The Actor network outputs an action according to this observation, as given in Equation 10: a i = μ i o i ; θ i (10) where μ i represents the Actor network of the i-th agent and θ i denotes its network parameters. During training, each Critic estimates the action value Q i from the joint observations and actions of all UAVs. This centralized information allows it to account for interactions among agents, improving the stability of multi-UAV cooperative policy learning.

For the sample o , a , r , o ′ in the experience replay buffer, the target Q-value of the i-th agent can be expressed as shown in Equation 11: y i = r i + γ Q i ′ o 1 ′ , . . . , o n ′ , a 1 ′ , . . . , a n ′ (11) where γ denotes the discount factor, Q i ′ denotes the target Critic network, and a i ′ denotes the action output by the target Actor network at the next state. The Critic network is updated by minimizing the mean squared error between the current Q-value and the target Q-value, as defined in Equation 12: L ϕ i = min 1 B ∑ Q i − y i 2 (12) where B denotes the batch size, and ϕ i represents the parameters of the i-th Critic network. The objective of the Actor network is to select actions that can obtain higher Q-values. Therefore, the update objective of the Actor network can be formulated as maximizing the Critic’s evaluation of the current action, as expressed in Equation 13: J θ i = max 1 B ∑ Q i (13)

To improve training stability, MADDPG introduces target networks and a soft update mechanism. The parameters of the target Actor and Critic networks are slowly updated according to Equations 14, 15, respectively: θ i ′ ← τ θ i + 1 − τ θ i ′ (14) ϕ i ′ ← τ ϕ i + 1 − τ ϕ i ′ (15) where τ denotes the soft update coefficient. This mechanism can prevent drastic fluctuations in the target values, thereby improving the stability of the training process. The framework diagram of the MADDPG algorithm is shown in Figure 1.

Let the local observation state of the i-th UAV in the system at time t be denoted as shown in Equation 16: o i t = x ¯ i , y ¯ i , z ¯ i , C o v t , N ¯ f , N ¯ c , N ¯ c i , ρ i u , ρ d , ρ i d (16) where x ¯ i , y ¯ i , and z ¯ i denote the normalized 3D coordinates of the i-th UAV; C o v t is the global map coverage; N ¯ f and N ¯ c are the normalized numbers of Frontier points and global candidate viewpoints, respectively; N ¯ c i is the normalized number of viewpoints assigned to the i-th UAV; ρ i u is the local unknown-voxel ratio; and ρ d and ρ i d represent global and local obstacle information, respectively.

The action of the i-th UAV at time t is defined in Equation 17: a i t = ω i t = ω i g i , ω d i s t i , ω c o o p i , ω o b s i (17)

During the actual training process, the action output by the Actor network is normalized using the Softmax function, which ensures that each action component satisfies the normalization and non-negativity constraints are given in Equations 18, 19, respectively: ω i g i + ω d i s t i + ω c o o p i + ω o b s i = 1 (18) ω i g i , ω d i s t i , ω c o o p i , ω o b s i > 0 (19)

The reward function of the i-th UAV at time t is defined in Equation 20, where the coverage increment and the coefficient-sum constraint are given in Equations 21, 22, respectively: r i t = λ C Δ C t − λ m L i t − λ r ρ i r e p − λ d R i o b s t − λ S I i s t u c k t (20) Δ C t = C t − C t − 1 (21) λ C + λ m + λ r + λ d + λ s = 1 (22) where Δ C t denotes the increment in map coverage, L i t denotes the travel distance, ρ i r e p t denotes the repeated exploration cost, R i o b s t denotes the obstacle risk, and I i s t u c k t denotes the stagnation cost.

Viewpoints covering more unknown areas receive higher rewards, encouraging information-rich exploration. Travel-distance and repeated-exploration penalties reduce energy consumption and redundant coverage, while obstacle-risk and stagnation costs promote flight safety and prevent prolonged hovering or repeated target selection.

The main procedure of the MADDPG-VFNBV algorithm is as follows:

Initialize the three-dimensional voxel map, UAV states, Actor network, Critic network, and experience replay buffer;

The schematic diagram of the MADDPG-VFNBV algorithm is shown in Figure 2.

The simulations are conducted in an unknown three-dimensional terrain environment covering a 3 km × 3 km area with undulating terrain, key information points, no-fly zones, dynamic obstacles, and predefined UAV initial positions. Figure 3 presents two-dimensional and three-dimensional views of the initial environment. Red dots denote key information points, yellow circles indicate no-fly zones, cyan triangles show UAV initial positions, and blue dashed circles and crosses represent dynamic obstacles and their influence areas. The top view illustrates the horizontal distribution of task elements, while the three-dimensional view shows terrain elevation and UAV initial altitudes. No-fly zones and dynamic obstacles affect exploration path planning and candidate-viewpoint selection.

Three UAVs cooperatively explore an unknown area using onboard sensors while avoiding static impassable regions and dynamic obstacles. The objective is to maximize map coverage within a specified time while minimizing path redundancy and repeated exploration.

The Voronoi-Frontier-NBV method uses fixed coefficients without training, where as MADDPG-VFNBV requires model training. The Actor network consists of two hidden layers with 128 neurons each, while the Critic network contains two hidden layers with 256 neurons each. Both networks are optimized with learning rates of 1 × 10 − 4 and 1 × 10 − 3 for the Actor and Critic, respectively. The batch size is 128, the discount factor is 0.95, and the soft-update coefficient is 0.007. Gaussian noise is adopted for exploration during training. At the end of each episode, the cumulative reward, final coverage, and repeated exploration rate are recorded, with the episode rewards shown in Figure 4.

Figure 4 shows noticeable fluctuations in episode rewards due to changes in candidate-viewpoint distribution, dynamic obstacles, and environmental noise. However, the 10-episode moving average generally increases and gradually converges, indicating that the Actor network learns an increasingly effective strategy for adjusting the NBV coefficients.

Figure 5 shows that, as training progresses, the final coverage rate increases and stabilizes at approximately 0.90, while the repeated exploration rate decreases to a low level. This indicates that the trained reinforcement learning module improves viewpoint selection and reduces competition among UAVs.

The training curves show that MADDPG-VFNBV gradually converges, with increasing episode rewards and coverage rates and decreasing repeated exploration, indicating that the reinforcement learning module effectively learns to adjust the NBV coefficients.

To evaluate the improvement of MADDPG-VFNBV over Voronoi-Frontier-NBV, both methods are tested in the same three-dimensional environment under identical conditions, including UAV number, sensor range, flight time, and path planning method.

The experimental results are shown in Figure 6. The MADDPG-VFNBV algorithm achieves higher coverage during most of the simulation, particularly in the middle and later stages. According to Table 2, compared with Voronoi-Frontier-NBV under the same flight time, it increases final coverage by approximately 6.94%, reduces total path length by 3.59% and the longest single-UAV path by 2.22%, and lowers repeated exploration by 2.90%. These results demonstrate that reinforcement learning improves the adaptability of the baseline method to different exploration stages and environmental changes, thereby enhancing multi-UAV cooperative exploration performance.

To further verify the adaptability of the MADDPG-VFNBV algorithm under different environmental conditions, this paper conducts tests in multiple randomly generated three-dimensional terrain environments. The different test environments vary in terms of UAV initial positions, terrain elevation distributions, no-fly zone locations, and dynamic obstacle distributions.

In Figure 7, square markers and solid lines denote UAVs and their trajectories, while circles and dashed lines represent dynamic obstacles and their trajectories. The UAVs avoid no-fly zones and dynamic obstacles while expanding toward unexplored regions. When key points and unknown areas are dispersed, they explore in different directions to reduce regional overlap. These results demonstrate the algorithm’s adaptability to varying terrain and obstacle distributions.