Overcoming computational challenges to realize meter- to submeter-scale resolution in cloud simulations using the super-droplet method

Abade, G. C., Grabowski, W. W., and Pawlowska, H.: Broadening of Cloud Droplet Spectra through Eddy Hopping: Turbulent

Entraining Parcel Simulations, J. Atmos. Sci., 75, 3365–3379,

https://doi.org/10.1175/JAS-D-18-0078.1, 2018.

Akinlabi, E. O., Wacławczyk, M., Mellado, J. P., and Malinowski,

S. P.: Estimating turbulence kinetic energy dissipation rates in

the numerically simulated stratocumulus cloud-top mixing layer:

Evaluation of different methods, J. Atmos. Sci., 76, 1471–1488,

https://doi.org/10.1175/JAS-D-18-0146.1, 2019.

Arabas, S. and Shima, S.-i.: Large-eddy simulations of

trade wind cumuli using particle-based microphysics with

Monte Carlo coalescence, J. Atmos. Sci., 70, 2768–2777,

https://doi.org/10.1175/JAS-D-12-0295.1, 2013.

Arabas, S. and Shima, S.: On the CCN (de)activation nonlinearities, Nonlin. Processes Geophys., 24, 535–542,

https://doi.org/10.5194/npg-24-535-2017, 2017.

Arabas, S., Pawlowska, H., and Grabowski, W.: Effective radius and

droplet spectral width from in-situ aircraft observations in tradewind cumuli during RICO, Geophys. Res. Lett., 36, L11803,

https://doi.org/10.1029/2009GL038257, 2009.

Arabas, S., Jaruga, A., Pawlowska, H., and Grabowski, W. W.: libcloudph++ 1.0: a single-moment bulk, double-moment bulk, and

particle-based warm-rain microphysics library in C++, Geosci.

Model Dev., 8, 1677–1707, https://doi.org/10.5194/gmd-8-16772015, 2015.

Aref, H.: Stirring by chaotic advection, J. Fluid Mech., 143, 1–21,

https://doi.org/10.1017/S0022112084001233, 1984.

Bodenschatz, E., Malinowski, S. P., Shaw, R. A., and Stratmann,

F.: Can we understand clouds without turbulence?, Science, 327,

970–971, https://doi.org/10.1126/science.1185138, 2010.

Brenguier, J.-L., Burnet, F., and Geoffroy, O.: Cloud optical thickness and liquid water path – does the k coefficient vary with

droplet concentration?, Atmos. Chem. Phys., 11, 9771–9786,

https://doi.org/10.5194/acp-11-9771-2011, 2011.

https://doi.org/10.5194/gmd-16-6211-2023

6243

Brown, A. R., Derbyshire, S., and Mason, P. J.: Large-eddy simulation of stable atmospheric boundary layers with a revised

stochastic subgrid model, Q. J. Roy. Meteor. Soc., 120, 1485–

1512, https://doi.org/10.1002/qj.49712052004, 1994.

Chang, K., Bench, J., Brege, M., Cantrell, W., Chandrakar, K., Ciochetto, D., Mazzoleni, C., Mazzoleni, L., Niedermeier, D., and

Shaw, R.: A laboratory facility to study gas–aerosol–cloud interactions in a turbulent environment: The 5 chamber, B. Am.

Meteorol. Soc., 97, 2343–2358, https://doi.org/10.1175/BAMSD-15-00203.1, 2016.

Connolly, M. P., Higham, N. J., and Mary, T.: Stochastic rounding

and its probabilistic backward error analysis, SIAM J. Sci. Comput., 43, A566–A585, https://doi.org/10.1137/20M1334796,

2021.

Cooper, W. A.: Effects of variable droplet growth histories on droplet size distributions. Part I: Theory, J. Atmos. Sci., 46, 1301–1311, https://doi.org/10.1175/15200469(1989)046<1301:EOVDGH>2.0.CO;2, 1989.

Cuturi, M.: Sinkhorn distances: Lightspeed computation of optimal

transport, in: Advances in Neural Information Processing Systems 26 (NIPS 2013), Nevada, US, 5–10 December, 2292–2300,

https://proceedings.neurips.cc/paper/2013 (last access: 24 October 2023), 2013.

Decyk, V. K. and Singh, T. V.: Particle-in-cell algorithms for emerging computer architectures, Comput. Phys. Commun., 185, 708–

719, https://doi.org/10.1016/j.cpc.2013.10.013, 2014.

Dziekan, P. and Pawlowska, H.: Stochastic coalescence in Lagrangian cloud microphysics, Atmos. Chem. Phys., 17, 13509–

13520, https://doi.org/10.5194/acp-17-13509-2017, 2017.

Dziekan, P. and Zmijewski, P.: University of Warsaw Lagrangian Cloud Model (UWLCM) 2.0: adaptation of a

mixed Eulerian–Lagrangian numerical model for heterogeneous computing clusters, Geosci. Model Dev., 15, 4489–4501,

https://doi.org/10.5194/gmd-15-4489-2022, 2022.

Dziekan, P., Waruszewski, M., and Pawlowska, H.: University of

Warsaw Lagrangian Cloud Model (UWLCM) 1.0: a modern

large-eddy simulation tool for warm cloud modeling with Lagrangian microphysics, Geosci. Model Dev., 12, 2587–2606,

https://doi.org/10.5194/gmd-12-2587-2019, 2019.

Eytan, E., Koren, I., Altaratz, O., Pinsky, M., and Khain, A.: Revisiting adiabatic fraction estimations in cumulus clouds: highresolution simulations with a passive tracer, Atmos. Chem.

Phys., 21, 16203–16217, https://doi.org/10.5194/acp-21-162032021, 2021.

Grabowski, W. W. and Abade, G. C.: Broadening of cloud

droplet spectra through eddy hopping: Turbulent adiabatic parcel simulations, J. Atmos. Sci., 74, 1485–1493,

https://doi.org/10.1175/JAS-D-17-0043.1, 2017.

Grabowski, W. W. and Jarecka, D.: Modeling condensation in shallow nonprecipitating convection, J. Atmos. Sci., 72, 4661–4679,

https://doi.org/10.1175/JAS-D-15-0091.1, 2015.

Grabowski, W. W. and Wang, L.-P.: Growth of cloud droplets in

a turbulent environment, Annu. Rev. Fluid Mech., 45, 293–324,

https://doi.org/10.1146/annurev-fluid-011212-140750, 2013.

Grabowski, W. W., Dziekan, P., and Pawlowska, H.: Lagrangian

condensation microphysics with Twomey CCN activation,

Geosci. Model Dev., 11, 103–120, https://doi.org/10.5194/gmd11-103-2018, 2018.

Geosci. Model Dev., 16, 6211–6245, 2023

6244

T. Matsushima et al.: Overcoming computational challenges for the super-droplet method

Grabowski, W. W., Morrison, H., Shima, S.-i., Abade, G. C.,

Dziekan, P., and Pawlowska, H.: Modeling of cloud microphysics: Can we do better?, B. Am. Meteorol. Soc., 100, 655–

672, https://doi.org/10.1175/BAMS-D-18-0005.1, 2019.

Grant, R. E., Levenhagen, M., Olivier, S. L., DeBonis, D., Pedretti, K. T., and Laros III, J. H. L.: Standardizing Power

Monitoring and Control at Exascale, Computer, 49, 38–46,

https://doi.org/10.1109/MC.2016.308, 2016.

Hoffmann, F.: The effect of spurious cloud edge supersaturations in Lagrangian cloud models: An analytical

and numerical study, Mon. Weather Rev., 144, 107–118,

https://doi.org/10.1175/MWR-D-15-0234.1, 2016.

Hoffmann, F., Siebert, H., Schumacher, J., Riechelmann, T.,

Katzwinkel, J., Kumar, B., Götzfried, P., and Raasch, S.: Entrainment and mixing at the interface of shallow cumulus

clouds: Results from a combination of observations and simulations, Meteorol. Z., 23, 349–368, https://doi.org/10.1127/09412948/2014/0597, 2014.

Jarecka, D., Grabowski, W. W., Morrison, H., and Pawlowska, H.:

Homogeneity of the subgrid-scale turbulent mixing in large-eddy

simulation of shallow convection, J. Atmos. Sci., 70, 2751–2767,

https://doi.org/10.1175/JAS-D-13-042.1, 2013.

Jenny, P., Pope, S. B., Muradoglu, M., and Caughey, D. A.:

A hybrid algorithm for the joint PDF equation of turbulent reactive flows, J. Comput. Phys., 166, 218–252,

https://doi.org/10.1006/jcph.2000.6646, 2001.

Lam, M. D., Rothberg, E. E., and Wolf, M. E.: The cache performance and optimizations of blocked algorithms, Oper. Syst.

Rev. (ACM), 25, 63–74, https://doi.org/10.1145/106973.106981,

1991.

Lasher-Trapp, S. G., Cooper, W. A., and Blyth, A. M.: Broadening of droplet size distributions from entrainment and mixing

in a cumulus cloud, Q. J. Roy. Meteor. Soc., 131, 195–220,

https://doi.org/10.1256/qj.03.199, 2005.

Matsushima, T., Nishizawa, S., Shima, S.-i., and Grabowski, W.:

Intra-cloud Microphysical Variability Obtained from Large-eddy

Simulations using the Super-droplet Method, ESS Open Archive

[preprint], https://doi.org/10.1002/essoar.10508672.1, 2021.

Matsushima, T., Nishizawa, S., and Shima, S.-i.: SCALESDM source code and configuration files for meter-tosubmeter-scale resolution in cloud simulations, Zenodo [code],

https://doi.org/10.5281/zenodo.8103369, 2023a.

Matsushima, T., Nishizawa, S., and Shima, S.-i.: Supplement to

the manuscript “Overcoming computational challenges to realize meter-to-submeter-scale resolution in cloud simulations using super-droplet method” (Matsushima et al., 2023), Zenodo

[data set], https://doi.org/10.5281/zenodo.8103378, 2023b.

McCann, R. J.: A convexity principle for interacting gases, Adv.

Math., 128, 153–179, https://doi.org/10.1006/aima.1997.1634,

1997.

Mellado, J.-P., Bretherton, C., Stevens, B., and Wyant,

M.: DNS and LES for simulating stratocumulus: Better together, J. Adv. Model. Earth Sy., 10, 1421–1438,

https://doi.org/10.1029/2018MS001312, 2018.

Morrison, H. and Grabowski, W. W.: Modeling supersaturation and subgrid-scale mixing with two-moment

bulk warm microphysics, J. Atmos. Sci., 65, 792–812,

https://doi.org/10.1175/2007JAS2374.1, 2008.

Geosci. Model Dev., 16, 6211–6245, 2023

Morrison, H., van Lier-Walqui, M., Fridlind, A. M., Grabowski,

W. W., Harrington, J. Y., Hoose, C., Korolev, A., Kumjian, M. R.,

Milbrandt, J. A., Pawlowska, H., Posselt, D. J., Prat, O. P.,

Reimel, K. J., Shima, S.-i., van Diedenhoven, B., and Xue, L.:

Confronting the challenge of modeling cloud and precipitation

microphysics, J. Adv. Model. Earth Sy., 12, e2019MS001689,

https://doi.org/10.1029/2019MS001689, 2020.

Nakashima, H., Miyake, Y., Usui, H., and Omura, Y.: OhHelp:

a scalable domain-decomposing dynamic load balancing for

particle-in-cell simulations, in: ICS ’09: Proceedings of the 23rd

international conference on Supercomputing, NY, USA, 8–12

June, 90–99, https://doi.org/10.1145/1542275.1542293, 2009.

Nishizawa, S., Yashiro, H., Sato, Y., Miyamoto, Y., and Tomita, H.:

Influence of grid aspect ratio on planetary boundary layer turbulence in large-eddy simulations, Geosci. Model Dev., 8, 3393–

3419, https://doi.org/10.5194/gmd-8-3393-2015, 2015.

Peyré, G. and Cuturi, M.: Computational optimal transport: With applications to data science, Foundations

and Trends® in Machine Learning, 11, 355–607,

https://doi.org/10.1561/2200000073, 2019.

Santambrogio, F.: Optimal transport for applied mathematicians, Birkäuser, Cham, Switzerland, 353 pp.,

https://doi.org/10.1007/978-3-319-20828-2, 2015.

Santos, S. P., Caldwell, P. M., and Bretherton, C. S.: Numerically relevant timescales in the MG2 microphysics

model, J. Adv. Model. Earth Sy., 12, e2019MS001972,

https://doi.org/10.1029/2019MS001972, 2020.

Sato, Y., Nakajima, T., Suzuki, K., and Iguchi, T.: Application of a Monte Carlo integration method to collision and coagulation growth processes of hydrometeors

in a bin-type model, J. Geophys. Res., 114, D09215,

https://doi.org/10.1029/2008JD011247, 2009.

Sato, Y., Nishizawa, S., Yashiro, H., Miyamoto, Y., Kajikawa, Y.,

and Tomita, H.: Impacts of cloud microphysics on trade wind

cumulus: which cloud microphysics processes contribute to the

diversity in a large eddy simulation?, Prog. Earth Planet Sci., 2,

23, https://doi.org/10.1186/s40645-015-0053-6, 2015.

Sato, Y., Shima, S.-i., and Tomita, H.: A grid refinement study of

trade wind cumuli simulated by a Lagrangian cloud microphysical model: the super-droplet method, Atmos. Sci. Lett., 18, 359–

365, https://doi.org/10.1002/asl.764, 2017.

Sato, Y., Shima, S.-i., and Tomita, H.: Numerical Convergence of Shallow Convection Cloud Field Simulations:

Comparison Between Double-Moment Eulerian and ParticleBased Lagrangian Microphysics Coupled to the Same Dynamical Core, J. Adv. Model. Earth Sy., 10, 1495–1512,

https://doi.org/10.1029/2018MS001285, 2018.

Schmitz, M. A., Heitz, M., Bonneel, N., Ngole, F., Coeurjolly,

D., Cuturi, M., Peyré, G., and Starck, J.-L.: Wasserstein dictionary learning: Optimal transport-based unsupervised nonlinear dictionary learning, SIAM J. Imaging Sci., 11, 643–678,

https://doi.org/10.1137/17M1140431, 2018.

Schulz, B. and Mellado, J. P.: Competing effects of droplet

sedimentation and wind shear on entrainment in stratocumulus, J. Adv. Model. Earth Sy., 11, 1830–1846,

https://doi.org/10.1029/2019MS001617, 2019.

Seifert, A. and Rasp, S.: Potential and limitations of machine learning for modeling warm-rain cloud microphysical

https://doi.org/10.5194/gmd-16-6211-2023

T. Matsushima et al.: Overcoming computational challenges for the super-droplet method

processes, J. Adv. Model. Earth Sy., 12, e2020MS002301,

https://doi.org/10.1029/2020MS002301, 2020.

Seiki, T. and Nakajima, T.: Aerosol effects of the condensation process on a convective cloud simulation, J. Atmos. Sci., 71, 833–

853, https://doi.org/10.1175/JAS-D-12-0195.1, 2014.

Shaw, R. A., Cantrell, W., Chen, S., Chuang, P., Donahue,

N., Feingold, G., Kollias, P., Korolev, A., Kreidenweis, S.,

Krueger, S., Mellado, J. P., Niedermeier, D., and Xue, L.:

Cloud–aerosol–turbulence interactions: Science priorities and

concepts for a large-scale laboratory facility, B. Am. Meteorol.

Soc., 101, E1026–E1035, https://doi.org/10.1175/BAMS-D-200009.1, 2020.

Shima, S., Sato, Y., Hashimoto, A., and Misumi, R.: Predicting the

morphology of ice particles in deep convection using the superdroplet method: development and evaluation of SCALE-SDM

0.2.5-2.2.0, -2.2.1, and -2.2.2, Geosci. Model Dev., 13, 4107–

4157, https://doi.org/10.5194/gmd-13-4107-2020, 2020.

Shima, S.-i., Kusano, K., Kawano, A., Sugiyama, T., and

Kawahara, S.: The super-droplet method for the numerical simulation of clouds and precipitation: A particle-based

and probabilistic microphysics model coupled with a nonhydrostatic model, Q. J. Roy. Meteor. Soc., 135, 1307–1320,

https://doi.org/10.1002/qj.441, 2009.

Siebesma, A. P., Bretherton, C. S., Brown, A., Chlond, A.,

Cuxart, J., Duynkerke, P. G., Jiang, H., Khairoutdinov,

M., Lewellen, D., Moeng, C.-H., Sanchez, E., Stevens,

B., and Stevens, D. E.: A large eddy simulation intercomparison study of shallow cumulus convection, J.

Atmos. Sci., 60, 1201–1219, https://doi.org/10.1175/15200469(2003)60<1201:ALESIS>2.0.CO;2, 2003.

Skamarock, W. C.: Evaluating mesoscale NWP models using kinetic energy spectra, Mon. Weather Rev., 132, 3019–3032,

https://doi.org/10.1175/MWR2830.1, 2004.

Sölch, I. and Kärcher, B.: A large-eddy model for cirrus clouds

with explicit aerosol and ice microphysics and Lagrangian ice

particle tracking, Q. J. Roy. Meteor. Soc., 136, 2074–2093,

https://doi.org/10.1002/qj.689, 2010.

Stevens, B., Moeng, C.-H., Ackerman, A. S., Bretherton, C. S.,

Chlond, A., de Roode, S., Edwards, J., Golaz, J.-C., Jiang, H.,

Khairoutdinov, M., Kirkpatrick, M. P., Lewellen, D. C., Lock,

A., Müller, F., Stevens, D. E., Whelan, E., and Zhu, P.: Evaluation of large-eddy simulations via observations of nocturnal marine stratocumulus, Mon. Weather Rev., 133, 1443–1462,

https://doi.org/10.1175/MWR2930.1, 2005.

Suzuki, K., Nakajima, T., Nakajima, T. Y., and Khain, A. P.: A

study of microphysical mechanisms for correlation patterns between droplet radius and optical thickness of warm clouds with a

spectral bin microphysics cloud model, J. Atmos. Sci., 67, 1126–

1141, https://doi.org/10.1175/2009JAS3283.1, 2010.

Tong, M. and Xue, M.: Simultaneous estimation of microphysical parameters and atmospheric state with simulated radar data

and ensemble square root Kalman filter. Part II: Parameter estimation experiments, Mon. Weather Rev., 136, 1649–1668,

https://doi.org/10.1175/2007MWR2071.1, 2008.

Unterstrasser, S. and Sölch, I.: Optimisation of the simulation particle number in a Lagrangian ice microphysical model, Geosci.

Model Dev., 7, 695–709, https://doi.org/10.5194/gmd-7-6952014, 2014.

https://doi.org/10.5194/gmd-16-6211-2023

6245

Unterstrasser, S., Hoffmann, F., and Lerch, M.: Collection/aggregation algorithms in Lagrangian cloud microphysical models: rigorous evaluation in box model simulations, Geosci.

Model Dev., 10, 1521–1548, https://doi.org/10.5194/gmd-101521-2017, 2017.

Unterstrasser, S., Hoffmann, F., and Lerch, M.: Collisional growth

in a particle-based cloud microphysical model: insights from

column model simulations using LCM1D (v1.0), Geosci.

Model Dev., 13, 5119–5145, https://doi.org/10.5194/gmd-135119-2020, 2020.

vanZanten, M. C., Stevens, B., Nuijens, L., Siebesma, A. P.,

Ackerman, A. S., Burnet, F., Cheng, A., Couvreux, F., Jiang,

H., Khairoutdinov, M., Kogan, Y., Lewellen, D. C., Mechem,

D., Nakamura, K., Noda, A., Shipway, B. J., Slawinska, J.,

Wang, S., and Wyszogrodzki, A.: Controls on precipitation

and cloudiness in simulations of trade-wind cumulus as observed during RICO, J. Adv. Model. Earth Sy., 3, M06001,

https://doi.org/10.1029/2011MS000056, 2011.

Wang, H., Agrusta, R., and van Hunen, J.: Advantages of

a conservative velocity interpolation (CVI) scheme for

particle-in-cell methods with application in geodynamic

modeling, Geochem. Geophy. Geosy., 16, 2015–2023,

https://doi.org/10.1002/2015GC005824, 2015.

Wicker, L. J. and Skamarock, W. C.: Time-splitting methods for elastic models using forward time schemes, Mon.

Weather Rev., 130, 2088–2097, https://doi.org/10.1175/15200493(2002)130<2088:TSMFEM>2.0.CO;2, 2002.

Xiao, J., Chen, J., Zheng, J., An, H., Huang, S., Yang, C., Li,

F., Zhang, Z., Huang, Y., Han, W., Liu, X., Chen, D., Liu,

Z., Zhuang, G., Chen, J., Li, G., Sun, X., and Chen, Q.:

Symplectic structure-preserving particle-in-cell whole-volume

simulation of tokamak plasmas to 111.3 trillion particles

and 25.7 billion grids, in: SC ’21: International Conference

for High Performance Computing, Networking, Storage and

Analysis, St. Louis, Missouri, USA, 14–19 November, 1–13,

https://doi.org/10.1145/3458817.3487398, 2021.

Xue, L., Bera, S., Chen, S., Choudhary, H., Dixit, S., Grabowski,

W. W., Jayakumar, S., Krueger, S., Kulkarni, G., Lasher-Trapp,

S., Mallinson, H., Prabhakaran, T., and Shima, S.-i.: Progress and

Challenges in Modeling Dynamics–Microphysics Interactions:

From the Pi Chamber to Monsoon Convection, B. Am. Meteorol. Soc., 103, E1413–E1420, https://doi.org/10.1175/BAMS-D22-0018.1, 2022.

Yashiro, H., Terasaki, K., Kawai, Y., Kudo, S., Miyoshi, T., Imamura, T., Minami, K., Inoue, H., Nishiki, T., Saji, T., Satoh,

M., and Tomita, H.: A 1024-member ensemble data assimilation with 3.5-km mesh global weather simulations, in: SC ’20:

International Conference for High Performance Computing, Networking, Storage and Analysis, IEEE, Atlanta, GA, USA, 9–19

November, 1–10, https://doi.org/10.1109/SC41405.2020.00005,

2020.

Zalesak, S. T.: Fully multidimensional flux-corrected transport algorithms for fluids, J. Comput. Phys., 31, 335–362,

https://doi.org/10.1016/0021-9991(79)90051-2, 1979.

Geosci. Model Dev., 16, 6211–6245, 2023

...