MergeSort is a general-purpose sorting algorithm based on element comparison and the divide-and-conquer strategy [2]. On a conceptual level, MergeSort operates as follows on a vector of n elements:

1. Divide the vector into n parts, each with a single element, which are considered ordered.

2. Merge the parts producing new ordered vectors until only one ordered vector remains as the final result.

MergeSort is presented in Algorithm 1. Sequential MergeSort., following the version by Cormen et al. [3]. Being an algorithm based on the divide-and-conquer strategy, its most natural description is recursive. Assuming an initial vector A=(a ₁ ,…,a_n ) , the result is obtained by invoking MergeSort (A,1,A.len) , where A.len = n , l (left) stands for the initial position, and r (right) stands for the final position.

Algorithm 1 Algorithm 1 Sequential MergeSort. Algorithm 1 Sequential MergeSort.

Algorithm 1. Sequential MergeSort.

The division of the problem is performed by two recursive invocations of the procedure MergeSort on the left part A[l,…,m] and the right part A[m+1,…,r] of the vector, respectively. The division is performed while there are at least two elements, since otherwise the vector is already sorted. After the division, the conquer action is carried out through the Merge operation to merge the parts A[l,…,m] and A[m+1,…,r] already ordered. The usual way to implement this operation follows the intuitive idea of how to mix two decks of cards already sorted where the smallest cards appear on top. The basic step consists of comparing two cards taken from each deck where we choose and remove the smallest one to build a new deck. We repeat the step until one of the decks runs out of cards in which case the remaining cards will go to the end of the new deck. The sequential merge operation is presented in Algorithm 2. Sequential merge operation.. The presented version differs slightly from the one by Cormen et al. [3] since it does not use sentinels, just to simplify the presentation.

Algorithm 2 Algorithm 2 Sequential merge operation. Algorithm 2 Sequential merge operation.

Algorithm 2. Sequential merge operation.

The computational complexity of MergeSort considers the number of comparisons performed when shuffling as the representative operation of the execution time cost. In general, mixing in MergeSort requires no more than n–1 comparisons for a vector of elements. The total cost is the sum of the cost of ordering two halves plus the cost of merging them. Therefore, in the worst case (and also in the average case) the cost of MergeSort is T(n) = 2 T(n/2)+Θ(n) = Θ(n log n).

Regarding memory cost, the typical implementation of MergeSort is not considered an in-place algorithm since the merge operation requires space Θ(n) for an auxiliary vector (vector T in Algorithm 2. Sequential merge operation.).

Since the cost of MergeSort coincides with the lower bound, MergeSort is said to be optimal. There are other optimal sorting algorithms (e.g., HeapSort), but implementations of MergeSort have the property of stability, as equal elements maintain their position. This property, together with its relative simplicity, makes MergeSort attractive for several applications.

This section describes strategies for the developed parallel MergeSort versions.

MergeSort follows a divide-and-conquer strategy, and the natural technique for parallelization is domain decomposition. All parallel versions of MergeSort follow the strategy of executing in parallel on different partitions of the input vector.

Specific instructions are applied for creating threads (fork) or processes (spawn), assuming a dynamic multithreading environment following the style presented by Cormen et al. [3]. Communications and synchronizations are performed using sync and join instructions in the shared memory implementations, or via explicit message passing (Send and Receive) in the distributed memory implementations.

The intuitive semantics of spawn f(x ₁ ,...,x_n ) indicate that the execution of f(x ₁ ,...,x_n ) can take place in a new thread of execution, and there is no need to wait for its completion to proceed. Similarly, the intuitive semantics of sync instructs the current thread of execution to wait until all previously created threads have finished their execution. Some parallel programming platforms offer similar instructions to express nested parallelism, but the implementations use them as a means to convey the expected solution at an abstract level.

From a theoretical perspective, there is a lower bound for the comparison ordering problem in the parallel paradigm. Considering n processing units to sort n elements, no algorithm using the available n processors can achieve a sorting time less than a constant multiple of n comparisons in the worst case.

The first parallel version of MergeSort (MergeSortPv1) is based on parallelizing only recursive invocations. This version uses the sequential merge algorithm described in Algorithm 2. Sequential merge operation.. Algorithm 3. Parallel MergeSort (parallel recursive invocations). presents the pseudocode of MergeSortPv1.

Algorithm 3 Algorithm 3 Parallel MergeSort (parallel recursive invocations). Algorithm 3 Parallel MergeSort (parallel recursive invocations).

Algorithm 3. Parallel MergeSort (parallel recursive invocations).

The execution cost of MergeSortPv1 on a single processor matches the execution cost of MergeSort. Ideally, for an unlimited number of processing units, recursive invocations can execute in parallel, but the merge operation imposes a linear cost. The ideal parallelizability of this approach is given by Eq.(1)where TP_i is the execution time on a parallel machine with processing units.

(1) TP 1 TP ∞ = Θ ( n log n ) Θ ( n ) = Θ ( log n )

The second version of parallel MergeSort (MergeSortPv2) is based on parallelizing the merge operation to improve the degree of parallelism of the first version of parallel MergeSort. In the previous version, the merge operation is the main limitation for parallelization, since it is inherently sequential. Thus, a new merging algorithm is used (MergeP, presented in Algorithm 4. Parallel merge operation.), which also applies the divide-andconquer strategy and is, therefore, also naturally parallelizable.

At a conceptual level, MergeP operates on two ordered vectors A and B (where A.len≥ B.len is assumed, without loss of generality) as follows:

1. Divide A into two parts A ₁ and A ₂.

2. Let x be the first element of A ₂. A binary search is performed on B to find the position h where x should be inserted into B. Then B is partitioned over h, generating two parts B ₁ and B ₂.

3. A ₁ is merged with B ₁ resulting in the first sorted half, and A ₂ is merged with B ₂ resulting in the second sorted half.

Algorithm 4 Algorithm 4 Parallel merge operation. Algorithm 4 Parallel merge operation.

Algorithm 4. Parallel merge operation.

Figure 1. Description of the logic in MergeP. shows an intuitive scheme of how the result C is constructed from the input vectors A and B in Algorithm 4. Parallel merge operation..

Figure 1 Figure 1 Description of the logic in MergeP. Figure 1 Description of the logic in MergeP.

Figure 1. Description of the logic in MergeP.

The cost of executing MergeP on a single processor is asymptotically equal to the sequential version. However, assuming an unlimited number of process units, the cost is logarithmic rather than linear, i.e., Θ(log² n), assuming that the search for position is performed in a binary way.

Thus, in the parallel version of MergeSort using MergeP instead of Merge, the cost on a single processor matches the cost of MergeSortPv1, but on an ideal machine the cost is Θ(log³ n). The ideal parallelizability of this approach is given by Eq. In comparison, the theory suggests that the second proposed version has the property of scaling better over additional computing resources, ignoring any overhead associated with parallelism..

(2) TP 1 TP ∞ = Θ ( n log n ) Θ ( log 3 n ) = Θ ( n log 2 n )

(2)

In comparison, the theory suggests that the second proposed version has the property of scaling better over additional computing resources, ignoring any overhead associated with parallelism.

This section describes the implementation details of the developed sequential and parallel versions of MergeSort. The code is publicly available on GitHub, https://github.com/josedusol/parallel-merge-sort.

MergeSort.c is the classic implementation of MergeSort in the C language, following Algorithm 1. Sequential MergeSort.. The best implementations of MergeSort typically apply another stable sorting algorithm (e.g. InsertionSort) for small inputs, taking advantage of improved efficiency on those inputs. This hybrid approach is not applied in the developed implementation, to guarantee a fair comparison with the developed parallel implementations of MergeSort.

The shared memory implementations of MergeSort were developed using the C language and the OpenMP application programming interface (API). Two approaches provided by the OpenMP API were studied for the execution of code segments in parallel:

1. sections directive. Each recursive invocation is placed in a different section. Because there is an implicit synchronization barrier at the end of this directive, the merge operation is inserted after. This alternative is only applicable if the nested parallelism option OMP_NESTED is explicitly activated.

2. task directive, available from OpenMP v3.0 onward. This is the best alternative to support unstructured parallelism, such as the one arising from parallel recursion. In this case, each recursive invocation is a different task. For the merge operation, it is necessary to also use taskwait which explicitly indicates the precedence between the split and merge tasks.

Preliminary experiments showed that task/taskwait allowed for computing better performance results. Thus, this approach was adopted for deployments.

MergeSortPv1_omp.c implements the MergeSortPv1 algorithm. The merge operation in the algorithm assumes that a new temporal vector is used for each invocation. However, a more efficient approach is to create a single vector before starting the sort and pass its address as a parameter to the involved routines. By doing so, the overhead associated with the initializing and releasing memory in each invocation is avoided. The overhead reduction implies that the same vector is used concurrently by different threads, accessing different portions during the merge process, without any risk of data races.

MergeSortPv2_omp.c implements the MergeSortPv2 algorithm, following the version presented by Cormen et al. [3]. The algorithm receives the original array A and an additional output array B to store the resulting values. However, it assumes that a new temporal vector T is used in each invocation. This vector is passed as the extra output array to the recursive calls, and the parallel merge operation is performed on T, storing the result in B. This approach has overhead due to initializing and releasing memory in each invocation, similar to the merge phase in the previous algorithm MergeSortPv1. To avoid this overhead and eliminate the need for additional vectors, a solution is to swap the roles of vectors A and B at each level of the recursion. In fact, without this optimization, the speedup achieved by this implementation is only marginally higher than that of the MergeSortPv1_omp.c implementation.

In both implementations, creating OpenMP tasks at each level of the recursion introduces an overhead that degrades the expected performance gains, even though creating a task does not mean creating a new operating system thread. To mitigate this negative factor, one approach is to limit the application of parallelism based on either the size of the input vector or the depth of the recursion. The developed implementations apply the restriction based on the size of the input vector.

On the one hand, in the implementation of MergeSortPv1_omp.c, MergeSort is invoked sequentially for sizes smaller than a threshold value CUT_OFF. On the other hand, there are two different levels of recursion to control in MergeSortPv2_omp.c: initially, MergeSort is executed sequentially for sizes less than a threshold value CUT_ OFF_1, and the sequential Merge is called for the total sizes (i.e., the sum of the sizes of two vectors) less than a threshold value CUT_OFF_2.

The optimal values for thresholds CUT_OFF, CUT_OFF_1, and CUT_OFF_2 are platformdependent and are determined through empirical testing and experimentation. These values are chosen in an ad-hoc manner to achieve the best performance on the specific computing platform.

The distributed memory implementations of MergeSort were developed using the C language and the MPI library. Three implementations were developed, applying a straightforward master-slave approach, a functional decomposition approach, and a hybrid approach combining distributed and shared memory for efficient computation. The developed implementations are described next.

MergeSortPv0_mpi.c is a distributed memory implementation of MergeSort based on worksharing. A master-slave hierarchical architecture is applied. The master evenly divides the input vector and distributes chunks using MPI_Scatter among the available processes so that each process performs a local sequential sorting in the assigned chunk. The processes return the sorted parts to the master process using MPI_ Gather, and the master performs a final sequential sort. Although this implementation does not correspond exactly to the theoretical algorithms MergeSortPv1 and MergeSortPv2, it is the most direct and elementary parallelization of the problem using MPI, and therefore it is interesting to consider it for simplicity as a baseline for the comparison of efficiency results.

MergeSortPv1_mpi.c is an implementation of the MergeSortPv1 algorithm, applying a functional decomposition technique [5].

A virtual tree of processes that represents the tree of recursive calls is built. At each level, different processes work in parallel either to divide or to merge. The depth of the process tree is limited by the number of available processes. The base case in the construction of the tree is to invoke MergeSort sequentially for the leave nodes of the tree. Figure 2. Virtual tree of four processes to sort vector . presents a scheme of the functional decomposition technique when using four processes on a vector of eight elements. The dashed lines indicate that the process retains the first half of the vector, and the bi-directional arrows indicate that the process should send the second half of the vector to an auxiliary process. The four lists of size 2 on the leaves of the tree are ordered sequentially. Asynchronous communications are performed, using MPI_Isend, to properly overlap data transmission and computation. Then, the process receives the ordered half from the auxiliary process using a standard MPI_Recv routine.

Figure 2 Figure 2 Virtual tree of four processes to sort vector. Figure 2 Virtual tree of four processes to sort vector.

Figure 2. Virtual tree of four processes to sort vector.

MergeSortPv2_hyb.c is a hybrid implementation of the MergeSortPv2 algorithm using MPI and OpenMP. The data splitting phase is implemented using MPI, as performed in MergeSortPv1_mpi.c, and the merge operation is implemented using OpenMP, as performed in MergeSortPv2_omp.c. Additionally, OpenMP is also used in the base case of the virtual process tree. Parallelism restrictions with the values CUT_OFF_1 and CUT_OFF_2 are also applied to mitigate the overhead associated with threads creation and management in OpenMP.

This section describes the empirical analysis of the developed parallel MergeSort implementations.

The execution time is used as the main metric for performance evaluation. The performance evaluation of parallelism is focused on strong scalability, i.e., how the execution time varies when using a different number of processing units for a fixed input size. The goal is to reduce the execution time.

The traditional approach applying the algorithmic speedup is applied to evaluate performance. Speedup computes the ratio of the execution time of a baseline sequential implementation T_S and the execution time of the parallel algorithm using N compute resources TP_N (Eq. (3)In this case, the sequential algorithm is MergeSort.c, described before. Computational efficiency is the normalized version of the speedup (Eq. Development and execution platform).

(3) S N = T S TP N

(3)

(4) E N = S N N

(4)

The proposed algorithms were developed in the C language, using the GCC v8 compiler. Initial local developments were performed using the MPICH v3.2 implementation of MPI, and the final development was performed using the OpenMPI implementation of MPI.

The experimental evaluation was performed on the high-performance computing infrastructure of National Supercomputing Center, Uruguay (ClusterUY) [8]. Each computing node has two Intel Xeon Gold 6138 CPUs with 20 cores each and 128 GB of RAM. Nodes are connected by 10 Gbps Ethernet. Executions were performed without using hyperthreading.

This subsection reports on the efficiency results of the developed parallel versions of MergeSort. For OpenMP-based implementations, the following values were determined in preliminary experiments to mitigate overheads: CUT_OFF = 800 in MergeSortPv1_ omp.c and the hybrid implementation MergeSortPv2_hyb.c, and CUT_OFF_1=800, CUT_OFF_2=80000 in MergeSortPv2_omp.c and the hybrid implementation MergeSortPv2_hyb.c.

Shared-memory implementations. Table 1 reports the average execution times of the shared memory implementations using OpenMP. For reference, the average execution time of the sequential implementation MergeSort.c for n=10⁹ w was 318.67s. Figure 3 graphically presents the evolution of execution times.

Table 1. Execution time (s) for different numbers of computing resources (threads) for shared memory MergeSort implementations using OpenMP.

Table 1 Table 1 Execution time (s) for different numbers of computing resources (threads) for shared memory MergeSort implementations using OpenMP. Table 1 Execution time (s) for different numbers of computing resources (threads) for shared memory MergeSort implementations using OpenMP.

implementation	1	2	computing resource 4 8		s 16	32
MergeSortPv1_omp	274.96	153.68	115.14	83.49	65.66	47.42
MergeSortPv2_omp	231.69	124.86	83.48	49.43	27.85	17.59

Figure 3 Figure 3 Execution time of shared memory OpenMP MergeSort implementations. Figure 3 Execution time of shared memory OpenMP MergeSort implementations.

Figure 3. Execution time of shared memory OpenMP MergeSort implementations.

Tables 2 and 3 report the speedup and efficiency values computed for the shared memory implementations using OpenMP. Figure 4 graphically presents the graphs of speedup and efficiency as a function of the number of threads.

Table 2. Speedup for different number of computing resources (threads) for shared memory MergeSort implementations using OpenMP.

Table 2 Table 2 Speedup for different number of computing resources (threads) for shared memory MergeSort implementations using OpenMP. Table 2 Speedup for different number of computing resources (threads) for shared memory MergeSort implementations using OpenMP.

implementation	S 1	S 2	speedup S4 S8		S 16	S 32
MergeSortPv1_omp	1.16	2.07	2.77	3.82	4.85	6.72
MergeSortPv2_omp	1.38	2.55	3.82	6.45	11.44	18.11

Table 3. Efficiency for different number of computing resources (threads) for shared memory MergeSort implementations using OpenMP.

Table 3 Table 3 Efficiency for different number of computing resources (threads) for shared memory MergeSort implementations using OpenMP. Table 3 Efficiency for different number of computing resources (threads) for shared memory MergeSort implementations using OpenMP.

implementation	E 1	E 2	efficiency E4 E8		E 16	E 32
MergeSortPv1_omp	1.16	1.04	0.69	0.48	0.30	0.21
MergeSortPv2_omp	1.38	1.28	0.95	0.81	0.72	0.57

Figure 4 Figure 4 Speedup and efficiency for the shared memory MergeSort implementations using OpenMP. The dotted line is the linear speedup. Figure 4 Speedup and efficiency for the shared memory MergeSort implementations using OpenMP. The dotted line is the linear speedup.

Figure 4. Speedup and efficiency for the shared memory MergeSort implementations using OpenMP. The dotted line is the linear speedup.

The results of the performance analysis indicate that the MergeSortPv1_omp.c implementation showed a high speedup when using a small number of threads (≤3). However, as the number of threads increased, the implementation deviated from the theoretically optimal performance given by the linear speedup. Using four threads, the efficiency dropped below 0.7. Moreover, when using more than 20 threads, the speedup tended to stagnate. This result aligns with Amdahl's Law, which suggests that sequential work becomes a limiting factor despite the availability of additional processing units. For the considered implementation, the limiting factor is the linear-cost merge operation, which acts as the bottleneck of the implementation.

In contrast, the MergeSortPv2_omp.c implementation demonstrated a remarkable speedup with consistent growth. Unlike the previous implementation, it showed no signs of stagnation and achieved the maximum speedup value when using the largest number of threads (39). However, it is anticipated that this upward trend will diminish or even reverse as the number of threads further increases. The efficiency of this implementation only dropped below 0.7 when using more than 20 threads.

Distributed-memory implementations. Table 4 reports the average execution times of the distributed memory implementations of MergeSort using MPI. Figure 5 graphically presents the evolution of execution times.

Table 4. Execution time (s) for different number of computing resources (processes) for distributed memory MPI MergeSort implementations.

Table 4 Table 4 Execution time (s) for different number of computing resources (processes) for distributed memory MPI MergeSort implementations. Table 4 Execution time (s) for different number of computing resources (processes) for distributed memory MPI MergeSort implementations.

implementation	1	computing resources 2 4 8			16
MergeSortPv0_mpi	236.06	286.17	222.13	201.62	211.97
MergeSortPv1_mpi	281.88	287.91	221.54	159.27	117.59
MergeSortPv2_hyb	254.05	264.01	73.97	54.68	41.29

Figure 5 Figure 5 Execution times for distributed memory MPI MergeSort implementations. Figure 5 Execution times for distributed memory MPI MergeSort implementations.

Figure 5. Execution times for distributed memory MPI MergeSort implementations.

Tables 5 and 6 report the speedup and efficiency of the distributed memory MergeSort implementations using MPI. Figure 6 graphically presents the speedup and efficiency as a function of the number of computing resources/processes.

Table 5. Speedup of distributed memory MPI MergeSort implementations.

Table 5 Table 5 Speedup of distributed memory MPI MergeSort implementations. Table 5 Speedup of distributed memory MPI MergeSort implementations.

implementation	S 1	S 2	speedup S 4	S 8	S 16
MergeSortPv0_mpi	1.35	1.11	1.43	1.58	1.50
MergeSortPv1_mpi	1.13	1.11	1.44	2.00	2.71
MergeSortPv2_hyb	1.25	1.21	4.31	5.83	7.72

Table 6. Efficiency of distributed memory MPI MergeSort implementations.

Table 6 Table 6 Efficiency of distributed memory MPI MergeSort implementations. Table 6 Efficiency of distributed memory MPI MergeSort implementations.

implementation	E 1	E 2	efficiency E 4	E 8	E 16
MergeSortPv0_mpi	1.35	0.56	0.36	0.10	0.09
MergeSortPv1_mpi	1.13	0.55	0.36	0.25	0.17
MergeSortPv2_hyb	1.25	0.60	1.08	0.73	0.48

Figure 6 Figure 6 Speedup and efficiency for distributed memory MergeSort implementations using MPI. The dotted line is the linear speedup. Figure 6 Speedup and efficiency for distributed memory MergeSort implementations using MPI. The dotted line is the linear speedup.

Figure 6. Speedup and efficiency for distributed memory MergeSort implementations using MPI. The dotted line is the linear speedup.

The results of the performance analysis indicate that the speedup achieved by the MergeSortPv0_mpi.c implementation was notably poor. Comparatively, the MergeSortPv1_mpi.c implementation demonstrated a slightly improved speedup, but its growth rate remained quite slow. On the other hand, the hybrid implementation MergeSortPv2_hyb.c, despite initially exhibiting unexpectedly poor performance when using only two processes, showed a better performance overall as the number of processes increased. This is a relevant result since the hybrid implementation was executed on different computing nodes, implying that the other developed implementations had not experienced significant inter-node communication costs.

When comparing the two groups of implementations, it is evident that superior results were achieved using OpenMP. In fact, the best-performing MPI-based implementation was the hybrid approach that combined MPI with OpenMP. Both groups had specific cases where superlinear speedup was achieved, i.e., the performance improvement exceeded the expected linear scaling. Overall, the utilization of OpenMP has proven to be more effective in terms of performance, and the presence of almost linear speedup values highlights the potential efficiency gains achievable through parallelization.