Mask2Former Leads
Highest AP = 0.651, recall, and F1 score, accurately segmenting small and overlapping bubbles.
A Comparative Study of Deep Learning Models for Bubble Segmentation in Gas-Liquid Stirred Tank Reactors
Undergraduate Research Project · Atharva Tagalpallewar
Under review at The Canadian Journal of Chemical Engineering
7Instance segmentation models
22,500High-speed images
400Annotated images
2Impellers, dual config
Summary
Abstract
Graphical Abstract. Comparative evaluation of seven deep learning instance segmentation models for bubble segmentation and size distribution analysis in gas-liquid dual-impeller stirred tank reactors.
Accurate characterization of bubble parameters, particularly bubble size distribution (BSD), is crucial for understanding gas-liquid hydrodynamics and optimizing the performance of stirred tank reactors (STRs). Traditional image analysis techniques face significant challenges due to bubble overlap, irregular shapes, and wide variations in bubble size.
To address this, we present a comparative evaluation of deep-learning-based instance segmentation models for automated bubble segmentation in STRs. Seven state-of-the-art models, Mask R-CNN (with and without PointRend), CondInst, SOLOv2, YOLOv8, StarDist and Mask2Former, are benchmarked on a diverse dataset of STR images obtained under a range of operating conditions. Performance is assessed on segmentation accuracy, computational efficiency, and robustness in handling bubble complexities.
Mask2Former achieved the highest precision (Average Precision = 0.651), making it suitable for high-fidelity hydrodynamics analysis. In contrast, YOLOv8 achieved the fastest inference time (69 ms), suitable for real-time applications. This comparative study highlights the trade-off between accuracy and efficiency, establishes the suitability of transformer-based architectures for detailed scientific analysis, and confirms the potential of lightweight models for practical industrial deployment.
Keywords: Deep Learning Models, Instance Segmentation, Image Post-Processing, Bubble Size Distribution (BSD), Gas-liquid STR
At a glance
Key findings
Benchmarked on high-speed stirred-tank reactor images under diverse multiphase operating conditions.
YOLOv8 Fastest
Lowest inference latency of 69 ms and shortest training time, ideal for real-time monitoring.
Accuracy–Speed
Box-free and dynamic-head models separate clusters best, while real-time models trade precision for speed.
Consistent BSDs
Architecturally distinct models recover matching modal diameters (within ~0.2 mm) across reactor zones.
Background & Literature Review
Introduction & Literature Review
Gas-liquid stirred tank reactors (STRs) are essential equipment in a wide variety of chemical, petrochemical, and biochemical reactions. The reactors provide a carefully controlled environment of gas-liquid contact and serve as catalysts for mass transfer and chemical reactions that form the basis of many industrial processes, such as chemical synthesis, polymerization, fermentation, and wastewater treatment. The flexibility of STRs allows them to support a broad range of operating conditions and impeller designs.
Understanding bubble dynamics is critical to optimizing reactor design and operation. Bubble characteristics such as size, shape, and distribution directly dictate the gas-liquid interfacial area, controlling mass transfer rates and process efficiency. Smaller bubbles produce greater interfacial area, increasing mass transfer. However, accurate measurement requires segmenting individual bubbles in reactor images, a task severely challenged by multiphase flow dynamics where bubbles constantly deform, break up, coalesce, and oscillate.
Traditional Image Processing vs Deep Learning
Digital image processing has long been used to extract quantitative bubble data. Traditional workflows apply thresholding, noise filtering, edge detection, and hole filling. However, classical geometric algorithms suffer from notable limitations:
- Hough Transform: Detects circular or elliptical geometries well but fails when bubbles are irregular or densely overlapping.
- Breakpoint Method & Ellipse Fitting: Identifies curvature maxima along bubble contours to fit ellipses, but is highly sensitive to noise and fails for non-elliptical bubble shapes.
- Watershed Segmentation: Treats images as topographic relief maps, but is notoriously prone to over-segmentation in crowded flows or low-contrast regions.
These classical algorithms require manual parameter tuning and generalize poorly when lighting, bubble density, or optical distortion vary across operating conditions. This has driven a transition toward deep learning and Convolutional Neural Networks (CNNs), which replace handcrafted rules by automatically learning hierarchical features directly from raw image pixels.
Figure 1. (A) Raw image, and (B) different types of image-based deep learning models. From a raw image, models can perform classification, object detection, semantic segmentation, or instance segmentation, each producing a different kind of output.
Gaps in Current Research & Study Objective
While deep learning has gained traction in multiphase flow analysis, notable limitations persist in the literature:
- Elliptical Assumptions: Many existing models rely on bounding box or elliptical regression (such as YOLO or basic U-Nets), failing to represent the irregular, deforming shapes produced by turbulent breakup and coalescence.
- Low-Density Datasets: Most prior studies evaluate models on sparse bubble populations with minimal overlap, which does not reflect the dense, heavily clustered bubble fields typical of real-world industrial STRs.
- High-Contrast Dependencies: Previous works often require clear contrast between bubbles and background, failing under non-uniform illumination or out-of-focus blurs.
- Narrow Architectural Scope: Modern advancements, such as transformer-based mask classification (Mask2Former) and box-free dynamic heads (CondInst), have not been systematically benchmarked against established baselines (Mask R-CNN, YOLOv8, StarDist).
This study addresses these gaps through a comprehensive comparative evaluation of seven state-of-the-art instance segmentation models on a challenging, high-speed STR image dataset under diverse operating conditions.
Methodology
Experiment & Dataset
To investigate the effectiveness of deep learning models for bubble segmentation, a series of high-speed imaging experiments were conducted in a flat-bottom cylindrical dual-impeller gas-liquid STR fabricated from Perspex. A dual-impeller configuration of a six-bladed Rushton turbine (RT-6) at the bottom and a four-bladed 45 degree pitched mixed impeller (MI) at the top was employed. Distilled water was operated batchwise as the liquid phase while compressed air was continuously introduced through a ring sparger at 1.5 bar.
Imaging measurements were acquired systematically across axial (z/H = 0.333, 0.749) and radial (r/R = 0.175, 0.525, 0.878) planes to capture representative local flow dynamics across distinct reactor zones.
Figure 2. Measurement locations (z/H & r/R).
Tank diameter, T0.19 m (height 0.415 m, H/T = 2)
Baffles4 x 0.019 m at 90 degree intervals (w = T/10)
Bottom impeller, RT-60.0633 m Rushton turbine (T/3)
Top impeller, MI0.0667 m 45 degree pitched mixed (T/2.85)
Ring sparger0.0633 m, 16 holes 1.5 mm each, 1.5 bar
Camera544x544 px 25 fps, 750 us, 2.07x2.07 cm FOV
Impeller speed, N300-500 rpm
Gas flow, Qg3.5 / 7 / 10.5 LPM
Measurement locationsr/R = 0.175, 0.525, 0.878 z/H = 0.333, 0.749
Figure 3. Sample images drawn from across the dataset, illustrating how bubble populations vary with operating condition and spatial location in the reactor.
Throughout the experiments, a total of 22,500 images were captured, about 750 images per distinct operating condition. Across the full dataset, bubble characteristics were found to vary appreciably with both spatial location and operating condition. A subset of 400 images was selected for annotation; a data-scaling analysis using Mask R-CNN showed segmentation performance plateaued beyond 400 images, indicating diminishing returns from further annotation. Images were carefully annotated with the CVAT tool, with each bubble outlined by polygons aligned to its edges, including partially separated bubbles with common boundaries as separate entities.
Architectures
The seven models
Click on any model to expand or collapse its architectural description and full-width pipeline diagram.
Mask R-CNN
Two-stage, baseline
Mask R-CNN
Two-stage, baseline
A widely adopted two-stage, detect-then-segment model extending Faster R-CNN with a parallel mask branch. Its Feature Pyramid Network produces multi-scale features relevant to a dataset where bubble size varies substantially, and its per-region mask head gives clean instance-level outputs when contrast is adequate and overlap moderate. Serves here as an established baseline.
Figure 4. Mask R-CNN Architecture pipeline.
Mask R-CNN + PointRend
Boundary refinement
Mask R-CNN + PointRend
Boundary refinement
PointRend replaces the standard mask head, generating a coarse mask and then iteratively re-predicting the lowest-confidence pixels, usually on object boundaries, at higher effective resolution. Because bubble size is derived directly from the mask boundary, sharper edges should yield more accurate area and size estimates. Included to test the gain, and its computational cost.
Figure 5. Mask R-CNN + PointRend Architecture pipeline.
CondInst
Single-stage, dynamic head
CondInst
Single-stage, dynamic head
Departs from the fixed-head design by dynamically generating a compact, instance-specific mask head for each potential bubble, applied to a shared higher-resolution feature map. Well suited to the heterogeneity of the STR bubble population: each head adapts to its own instance, accommodating the wide variation in bubble size and appearance within a single image.
Figure 6. CondInst Architecture pipeline.
SOLOv2
Single-stage, box-free
SOLOv2
Single-stage, box-free
A single-stage, box-free model that divides the image into a grid where each cell is responsible for any object whose centre falls within it, decoupling mask-kernel prediction from mask-feature learning. This location-based assignment is a conceptually different approach to instance separation, testing how well a purely location-driven scheme handles spatially clustered bubbles whose centres lie close together.
Figure 7. SOLOv2 Architecture pipeline.
YOLOv8-seg
Real-time, single-pass
YOLOv8-seg
Real-time, single-pass
Belongs to the YOLO family, framing detection as a single-pass regression over a grid of cells. The official YOLOv8-seg variant adds a segmentation-mask branch based on prototype masks while retaining the family's characteristic speed. Included to represent the efficiency end of the spectrum, quantifying how much accuracy must be traded for real-time performance on this dataset.
Figure 8. YOLOv8 Architecture pipeline.
StarDist
Geometry prior, star-convex
StarDist
Geometry prior, star-convex
Represents objects as star-convex polygons, predicting for each pixel an object probability and radial distances to the boundary along fixed directions. Developed for cell nuclei in microscopy, its assumptions map onto dense fields of smooth, roughly convex ellipse-like bubbles. Included to test whether an explicit geometry-based shape prior helps, or whether the irregular, deforming shapes from breakup and coalescence fall outside its range.
Figure 9. StarDist Architecture pipeline.
Mask2Former
Transformer, mask classification
Mask2Former
Transformer, mask classification
A transformer-based model that frames segmentation as set prediction: learnable queries each predict one mask through mask classification, combined with masked attention and multi-scale features atop a Swin Transformer backbone. Because it separates instances through attention rather than bounding boxes and requires no NMS, it is structurally suited to the dense, heavily overlapping bubbles that confound box-based methods, and scales naturally to variable bubble counts.
Training
Training details
Data augmentation included random flipping, cropping, scaling, and colour jittering to improve generalization and prevent overfitting. All models were initialized with pre-trained weights (COCO or ImageNet).
Each model was trained until its validation loss plateaued, with early stopping applied; the epoch counts in Table 1 correspond to convergence. Mask2Former, coupling a large Swin-L backbone with an iterative transformer decoder, required approximately 100 epochs, whereas remaining models converged within 50.
| Model | Epochs | GPU | Backbone |
|---|---|---|---|
| Mask2Former | 100 | Nvidia A100 | Swin-L |
| StarDist | 50 | Nvidia V100 | U-Net |
| YOLOv8 | 50 | Nvidia V100 | CSPDarknet53 |
| SOLOv2 | 50 | Nvidia V100 | ResNet-101 |
| CondInst | 50 | Nvidia V100 | ResNet-101 |
| Mask R-CNN | 50 | Nvidia V100 | ResNeXt-101 |
| PointRend | 50 | Nvidia V100 | ResNeXt-101 |
Table 1. Training details for the seven models.
Results
Evaluation metrics & Performance
To evaluate model performance, we used a comprehensive suite of metrics standard for instance segmentation tasks. The primary metric was Average Precision (AP), which evaluates detection and segmentation quality across a range of Intersection over Union (IoU) thresholds.
Intersection over Union (IoU) measures the overlap between a predicted bubble boundary and the manually annotated ground truth, normalized between 0 and 1. An IoU threshold defines the minimum overlap required for a prediction to count as a correct detection. For example, an IoU threshold of 0.5 means a prediction is considered correct only when its segmented area overlaps the manually drawn boundary by 50% or more.
IoU = Area of Intersection / Area of Union (1)
Average Precision (AP) considers both Precision (the proportion of correctly identified bubbles among all predictions) and Recall (the proportion of actual bubbles correctly identified out of all ground truth objects). AP is calculated from the Precision-Recall curve (Figure 10), which plots precision against recall at various IoU thresholds. The area under this curve represents AP; a higher value indicates a superior balance between precision and recall while minimizing false positives and false negatives.
Figure 10. Precision-Recall curve definition.
Figure 11. AP as a function of IoU threshold.
We evaluate several variants of Average Precision across the models:
- AP (0.5:0.95): Calculates the mean AP across ten IoU thresholds from 0.50 to 0.95 in steps of 0.05 (Figure 11), providing a comprehensive assessment across levels of overlap.
- AP50: Measures AP at a single moderate IoU threshold of 0.50.
- AP75: Measures AP at a strict IoU threshold of 0.75, evaluating higher-precision segmentation.
- F1 Score: The harmonic mean of precision and recall reported across the same IoU range as a complementary measure.
AP(0.5:0.95) = (1/10) · Σi=0.500.95 AP(IoU = i) (2)
Results
Qualitative evaluation
To visually evaluate the segmentation capabilities of each model under realistic multiphase flow conditions, qualitative comparisons were conducted on representative full-scale reactor images containing diverse bubble populations (Figure 12).
The models separate most distinctly by their ability to resolve two key operational challenges in gas-liquid stirred tank reactors:
- Densely Overlapping Clusters: In regions near the sparger and discharge streams, bubbles constantly collide and overlap. Box-free mask classification (Mask2Former) and dynamic head generation (CondInst) segment adjacent bubbles as individual instances without merging boundaries.
- Small & Out-of-Focus Bubbles: Faint, blurred bubbles at secondary focal depths are frequently missed by two-stage bounding-box methods like Mask R-CNN, whereas Mask2Former captures sub-millimeter fragments with high recall.
Boundary fidelity also correlates directly with architecture type. Standard Mask R-CNN yields somewhat coarse, pixelated mask contours due to its fixed-resolution ROI mask head. PointRend substantially sharpens boundary curvature around bubble edges. Conversely, StarDist forces all predictions into star-convex radial polygons, failing to capture the non-elliptical, deforming boundaries characteristic of turbulent gas-liquid shear flows.
Figure 12. Qualitative comparison of segmentation performance across all seven models on a full reactor image.
Results
Quantitative benchmarking
Table 2 and Figure 14 present a detailed quantitative overview of model performance, encompassing key metrics such as Average Precision (AP, AP50, AP75), Recall, F1 Score, Inference Time (ms), and Training Duration (minutes).
| Model | AP | AP50 | AP75 | Recall | F1 | Inference (ms) | Training (min) |
|---|---|---|---|---|---|---|---|
| Mask2Former | 0.651 | 0.934 | 0.749 | 0.721 | 0.684 | 339 | 68 |
| CondInst | 0.637 | 0.904 | 0.727 | 0.708 | 0.671 | 295 | 68 |
| PointRend | 0.631 | 0.913 | 0.711 | 0.681 | 0.655 | 188 | 63 |
| Mask R-CNN | 0.628 | 0.900 | 0.712 | 0.687 | 0.656 | 97 | 53 |
| YOLOv8 | 0.622 | 0.918 | 0.696 | 0.680 | 0.650 | 69 | 37 |
| SOLOv2 | 0.618 | 0.884 | 0.693 | 0.676 | 0.646 | 92 | 95 |
| StarDist | 0.529 | 0.842 | 0.508 | 0.435 | 0.477 | 549 | 54 |
Table 2. Quantitative comparison of model performance. Bold values represent the best result in each column.
Figure 14. Comparison of AP, AP50, and AP75 across models.
Mask2Former consistently emerged as the top-performing model across all accuracy metrics, achieving AP = 0.651, AP50 = 0.934, and AP75 = 0.749, demonstrating superior capability in segmenting dense, overlapping, and out-of-focus bubbles. Among CNN architectures, CondInst (AP = 0.637) and Mask R-CNN (AP = 0.628) served as strong, highly accurate alternatives.
While integrating the PointRend module improved boundary sharpness (raising AP to 0.631), it introduced a considerable computational overhead, doubling inference latency from 97 ms to 188 ms.
Figure 15. Accuracy versus inference time trade-off.
Figure 16. Training duration across instance segmentation models.
YOLOv8 proved to be the most efficient model, achieving an inference time of only 69 ms per image and the shortest training duration (37 min), making it ideal for real-time applications where minor accuracy trade-offs are acceptable. SOLOv2 provided a middle ground, balancing speed (92 ms) and accuracy (AP = 0.618).
In contrast, StarDist exhibited the highest inference latency (549 ms) alongside the lowest AP (0.529), indicating that rigid star-convex polygon prior assumptions do not provide an advantage for complex, deforming bubbles in STR hydrodynamics.
Results
Discussion
Mask2Former
Its transformer-based architecture captures long-range dependencies and global context, segmenting bubbles accurately even in complex overlap or irregular shapes. Mask classification handles varying bubble counts without predefined anchors or NMS.
StarDist
The steep AP decline with IoU reveals the limits of a star-convex polygon prior: efficient for smooth convex objects, but it struggles with precise boundaries of complex or overlapping bubbles, yielding lower IoU at higher thresholds.
CondInst
The strongest non-transformer model, thanks to dynamic instance-head generation that adapts to each instance, effective across the wide variation in bubble size and appearance in the dataset.
Mask R-CNN & PointRend
A reliable baseline; PointRend further refines boundaries, especially around edges, at the cost of added computation, a marginal accuracy gain with considerable overhead.
YOLOv8 & SOLOv2
Designed for real-time efficiency, they balance speed and accuracy, suitable where processing speed matters, accepting some reduction in segmentation precision.
Box-free separation
Models that separate overlapping bubbles without bounding boxes, Mask2Former and CondInst, recover the most complete bubble population and preserve the most accurate boundaries.
Results
Bubble size distribution
Each distribution comes from 50 sequential images per condition; the equivalent diameter of every segmented bubble is computed from its mask area. All distributions peak at small diameters with a long tail toward the largest bubbles.
The distributions vary systematically with location, as expected. The modal diameter is smallest near the wall (~0.6 mm), intermediate in the annular region (~1.4 mm), and largest near the centre (~1.8 mm).
The wall distribution is narrow and sharply peaked, with a secondary shoulder near 2 mm, whereas the centre distribution is broader and flatter, consistent with elevated local energy dissipation in the impeller discharge stream near the wall, where intense breakup produces a large population of sub-millimetre bubbles.
Importantly, architecturally distinct models demonstrate high agreement on modal diameter locations and general curve shapes, validating that the retrieved distributions reflect authentic hydrodynamic phenomena rather than model-specific artifacts.
Figure 17. Bubble size distributions from each model at three representative conditions: (a) 300 RPM, 3.5 LPM, z/H = 0.749 (Annular), (b) 300 RPM, 7 LPM, z/H = 0.333 (Center), and (c) 300 RPM, 10.5 LPM, z/H = 0.333 (Wall).
Takeaways
Conclusion
This study presented a comprehensive evaluation and comparison of seven state-of-the-art instance segmentation models, Mask R-CNN (with and without PointRend), CondInst, SOLOv2, YOLOv8, StarDist and Mask2Former, for bubble segmentation in gas-liquid dual-impeller stirred tank reactors. The evaluation revealed a clear performance trade-off between accuracy and computational efficiency.
Mask2Former, using a transformer-based architecture, achieved the highest score across all AP metrics, making it the top performer for precision, though its computational demands are significant, requiring longer inference times and more powerful GPUs. For applications requiring a balance between accuracy and efficiency, Mask R-CNN proved a reliable choice, with the optional PointRend module offering enhanced boundary segmentation at increased cost. YOLOv8 demonstrated exceptional efficiency with the fastest inference times, making it the most suitable model for real-time applications where a slight reduction in accuracy is acceptable.
Practical impact. Modern deep learning models can overcome the limitations of traditional image analysis, enabling large-scale, high-fidelity data on bubble dynamics to be generated automatically and accurately, removing a major bottleneck in the study of multiphase flows. This opens the door to rigorous CFD validation, data-driven process control, and the optimized design of next-generation reactors.