Increasing developer productivity by improving build performance and automating logging code injection (本文)

[1] Buildbot. http://buildbot.net/.

[2] filebench(1) - linux man page. https://linux.die.net/man/ 1/filebench.

[3] Jenkins. https://www.jenkins.io/.

[4] tpcc-mysql. https://github.com/Percona-Lab/tpcc- mysql.

[5] Travis ci. https://travis-ci.org/.

[6] C. Adam. Unity (Jumbo) build (!3611) · Merge Requests · CMake/ CMake. https://gitlab.kitware.com/cmake/cmake/ merge_requests/3611, July 2019.

[7] M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthi- tacharoen. Performance debugging for distributed systems of black boxes. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP ’03, pages 74–89, New York, NY, USA, 2003. Association for Computing Machinery.

[8] G. Altekar and I. Stoica. Odr: Output-deterministic replay for multicore debugging. In Proceedings of the ACM SIGOPS 22nd Symposium on Operat- ing Systems Principles, SOSP ’09, pages 193–206, New York, NY, USA, 2009. Association for Computing Machinery.

[9] G. Ammons and J. R. Larus. Improving data-flow analysis with path pro- files. In Proceedings of the ACM SIGPLAN 1998 Conference on Programming Language Design and Implementation, PLDI ’98, pages 72–84, New York, NY, USA, 1998. Association for Computing Machinery.

[10] Apple Inc. Technical note tn2123 crashreporter, February 2006.

[11] Apple Inc. llbuild. https://github.com/apple/swift- llbuild, 2020.

[12] Apple Inc. Webkit. https://webkit.org/, 2020.

[13] O. Arkhipova. Support for unity (jumbo) files in visual studio 2017 15.8 (experimental). https://devblogs.microsoft.com/cppblog/support-for-unity-jumbo-files-in- visual-studio-2017-15-8-experimental/, July 2018.

[14] A. H. Ashouri, W. Killian, J. Cavazos, G. Palermo, and C. Silvano. A survey on compiler autotuning using machine learning. ACM Comput. Surv., 51(5), Sept. 2018.

[15] Atlassian. Bitbucket. https://bitbucket.org/product/, 2020.

[16] C. AtLee, L. Blakk, J. O’Duinn, and A. Z. Gasparian. Firefox Release Engi- neering. The Architecture of Open Source Applications. Lulu.com, 2012.

[17] J. Atwood. The problem with logging. https://blog. codinghorror.com/the-problem-with-logging/, December 2008.

[18] A. Ayers, R. Schooler, C. Metcalf, A. Agarwal, J. Rhee, and E. Witchel. TraceBack. In Proceedings of the 2005 ACM SIGPLAN conference on Pro- gramming language design and implementation - PLDI '05. ACM Press, 2005.

[19] P. Barham, A. Donnelly, R. Isaacs, and R. Mortier. Using magpie for request extraction and workload modelling. In Proceedings of the 6th Conference on Symposium on Operating Systems Design and Implementation - Volume 6, OSDI’04, page 18, USA, 2004. USENIX Association.

[20] I. Beschastnikh, Y. Brun, S. Schneider, M. Sloan, and M. D. Ernst. Leverag- ing existing instrumentation to automatically infer invariant-constrained models. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ESEC/FSE ’11, pages 267–277, New York, NY, USA, 2011. Association for Comput- ing Machinery.

[21] S. Bhatia, A. Kumar, M. E. Fiuczynski, and L. Peterson. Lightweight, high- resolution monitoring for troubleshooting production systems. In Proceed- ings of the 8th USENIX Conference on Operating Systems Design and Imple- mentation, OSDI’08, pages 103–116, USA, 2008. USENIX Association.

[22] L. Bo. Btrfs: fix crash of btrfs_release_extent_buffer_page, 9 2014. https://github.com/torvalds/linux/commit/814650.

[23] E. Bodden, P. Lam, and L. Hendren. Finding programming errors earlier by evaluating runtime monitors ahead-of-time. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engi- neering - SIGSOFT '08/FSE-16. ACM Press, 2008.

[24] P. Bodik, M. Goldszmidt, A. Fox, D. B. Woodard, and H. Andersen. Finger- printing the datacenter. In Proceedings of the 5th European conference on Computer systems - EuroSys '10. ACM Press, 2010.

[25] R. Bodík and S. Anik. Path-sensitive value-flow analysis. In Proceedings of the 25th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’98, pages 237–251, New York, NY, USA, 1998. Associa- tion for Computing Machinery.

[26] P. Braione, G. Denaro, A. Mattavelli, and M. Pezzè. Sushi: A test generator for programs with complex structured inputs. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, ICSE ’18, pages 21–24, New York, NY, USA, 2018. Association for Comput- ing Machinery.

[27] T. Britton, L. Jeng, G. Carver, and P. Chea. Reversible debugging soft- ware “quantify the time and cost saved using reversible debuggers". http://citeseerx.ist.psu.edu/viewdoc/download? doi=10.1.1.370.9611&rep=rep1&type=pdf, 2013.

[28] Z. Brodsky. Git much? the top 10 companies contributing to open source. https://resources.whitesourcesoftware. com/blog-whitesource/git-much-the-top-10- companies-contributing-to-open-source, February 2018.

[29] B. Busjaeger and T. Xie. Learning for test prioritization: An industrial case study. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, pages 975– 980, New York, NY, USA, 2016. Association for Computing Machinery.

[30] C. Cadar, D. Dunbar, and D. Engler. Klee: Unassisted and automatic gener- ation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implemen- tation, OSDI’08, pages 209–224, USA, 2008. USENIX Association.

[31] J. Campos, Y. Ge, N. Albunian, G. Fraser, M. Eler, and A. Arcuri. An empir- ical evaluation of evolutionary algorithms for unit test suite generation. Information and Software Technology, 104:207 – 235, 2018.

[32] C. Catal and D. Mishra. Test case prioritization: a systematic mapping study. Software Quality Journal, 21(3):445–478, jul 2012.

[33] M. Catanzaro. On compiling webkit (now twice as fast!). https://blogs.gnome.org/mcatanzaro/2018/02/17/on-compiling-webkit-now-twice-as-fast/, February 2018.

[34] T. M. Chilimbi, B. Liblit, K. Mehra, A. V. Nori, and K. Vaswani. Holmes: Effective statistical debugging via efficient path profiling. In 2009 IEEE 31st International Conference on Software Engineering, pages 34–44, May 2009.

[35] M. Chow, D. Meisner, J. Flinn, D. Peek, and T. F. Wenisch. The mystery ma- chine: End-to-end performance analysis of large-scale internet services. In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation, OSDI’14, pages 217–231, USA, 2014. USENIX Association.

[36] I. Circle Internet Services. Continuous integration and delivery - circleci. https://circleci.com/, 2020.

[37] I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox. Cap- turing, indexing, clustering, and retrieving system history. In Proceedings of the Twentieth ACM Symposium on Operating Systems Principles, SOSP ’05, pages 105–118, New York, NY, USA, 2005. Association for Computing Machinery.

[38] L. Community. Clang - c++20, c++17, c++14, c++11 and c++98 status. https://clang.llvm.org/cxx_status.html, Augest 2020.

[39] O. Crameri, R. Bianchini, and W. Zwaenepoel. Striking a new balance between program instrumentation and debugging time. In Proceedings of the sixth conference on Computer systems - EuroSys '11. ACM Press, 2011.

[40] W. Cui, X. Ge, B. Kasikci, B. Niu, U. Sharma, R. Wang, and I. Yun. Rept: Reverse debugging of failures in deployed software. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI’18, pages 17–32, USA, 2018. USENIX Association.

[41] M. development team. Unity builds in meson. http://mesonbuild. com/Unity-builds.html, 2020.

[42] C. Dietrich, V. Rothberg, L. Füracker, A. Ziegler, and D. Lohmann. chash: Detection of redundant compilations via AST hashing. In 2017 USENIX Annual Technical Conference (USENIX ATC 17), pages 527–538, Santa Clara, CA, July 2017. USENIX Association.

[43] G. W. Dunlap, S. T. King, S. Cinar, M. A. Basrai, and P. M. Chen. Revirt: Enabling intrusion analysis through virtual-machine logging and replay. In Proceedings of the 5th Symposium on Operating Systems Design and Im- plementation (Copyright Restrictions Prevent ACM from Being Able to Make the PDFs for This Conference Available for Downloading), OSDI ’02, pages 211–224, USA, 2002. USENIX Association.

[44] G. W. Dunlap, D. G. Lucchetti, M. A. Fetterman, and P. M. Chen. Execu- tion replay of multiprocessor virtual machines. In Proceedings of the Fourth ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Envi- ronments, VEE ’08, pages 121–130, New York, NY, USA, 2008. Association for Computing Machinery.

[45] A. Duvall, P. Matyas, and S. Glover. Continuous Integration: Improving Software Quality and Reducing Risk. Addison-Wesley Professional, 2007.

[46] S. Elbaum, G. Rothermel, and J. Penix. Techniques for improving regres- sion testing in continuous integration development environments. In Pro- ceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, pages 235–245, New York, NY, USA, 2014. Association for Computing Machinery.

[47] S. Erdweg, M. Lichter, and M. Weiel. A sound and optimal incremental build system with dynamic dependencies. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Sys- tems, Languages, and Applications, OOPSLA 2015, pages 89–106, New York, NY, USA, 2015. Association for Computing Machinery.

[48] Facebook. Buck: A fast build tool. https://buck.build/, 2020.

[49] S. I. Feldman. Make — a program for maintaining computer programs. Software: Practice and Experience, 9(4):255–265, April 1979.

[50] K. Fisher, D. Walker, K. Q. Zhu, and P. White. From dirt to shovels: Fully automatic tool generation from ad hoc data. In Proceedings of the 35th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’08, pages 421–434, New York, NY, USA, 2008. Associa- tion for Computing Machinery.

[51] R. Fonseca, G. Porter, R. H. Katz, S. Shenker, and I. Stoica. X-trace: A perva- sive network tracing framework. In Proceedings of the 4th USENIX Confer- ence on Networked Systems Design and Implementation, NSDI’07, page 20, USA, 2007. USENIX Association.

[52] S. Foundation. Scons: A software construction tool. https://scons. org/, 2020.

[53] T. A. S. Foundation. ab - apache http server benchmarking tool. http:// httpd.apache.org/docs/2.4/ja/programs/ab.html, 2020.

[54] T. A. S. Foundation. Apache ant - welcome. https://ant.apache. org/, 2020.

[55] M. Fowler. Continuous integration (original ver- sion). https://martinfowler.com/articles/ originalContinuousIntegration.html, September 2000.

[56] G. Fraser and A. Arcuri. Whole test suite generation. IEEE Transactions on Software Engineering, 39(2):276–291, 2013.

[57] Q. Fu, J. Zhu, W. Hu, J.-G. Lou, R. Ding, Q. Lin, D. Zhang, and T. Xie. Where do developers log? an empirical study on logging practices in industry. In Companion Proceedings of the 36th International Conference on Software En- gineering, ICSE Companion 2014, pages 24–33, New York, NY, USA, 2014. Association for Computing Machinery.

[58] F. Fulin. Fastbuild - function reference - unity. https://www. fastbuild.org/docs/functions/unity.html, 2020.

[59] G. Garen. [webkit-dev] growing tired of long build times? check out this awesome new way to speed up your build... soon (hint: It’s not buying a new computer). https://lists.webkit.org/pipermail/ webkit-dev/2017-August/029508.html, Augest 2017.

[60] X. Ge, B. Niu, and W. Cui. Reverse debugging of kernel failures in deployed systems. In Proceedings of the 2020 USENIX Annual Technical Conference (ATC), July 2020.

[61] D. Geels, G. Altekar, P. Maniatis, T. Roscoe, and I. Stoica. Friday: Global comprehension for distributed replay. In Proceedings of the 4th USENIX Conference on Networked Systems Design and Implementation, NSDI’07, page 21, USA, 2007. USENIX Association.

[62] T. A. Ghaleb, D. A. da Costa, and Y. Zou. An empirical study of the long duration of continuous integration builds. Empir Software Eng, 24:2102– 2139, 2019.

[63] GitHub Inc. The state of the octoverse. https://octoverse. github.com/, 2019.

[64] GitHub Inc. Github. https://github.com/, 2020.

[65] GitLab Inc. Gitlab. https://about.gitlab.com/, 2020.

[66] K. Glerum, K. Kinshumann, S. Greenberg, G. Aul, V. Orgovan, G. Nichols, D. Grant, G. Loihle, and G. Hunt. Debugging in the (very) large: Ten years of implementation and experience. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP ’09, pages 103–116, New York, NY, USA, 2009. Association for Computing Machinery.

[67] Google. Bazel - a fast, scalable, multi-language and extensible build sys- tem". https://www.bazel.build/, Augest 2020.

[68] G. Grano, C. Laaber, A. Panichella, and S. Panichella. Testing with fewer resources: An adaptive approach to performance-aware test case genera- tion. IEEE Transactions on Software Engineering, pages 1–1, 2019.

[69] Z. Guo, X. Wang, J. Tang, X. Liu, Z. Xu, M. Wu, M. F. Kaashoek, and Z. Zhang. R2: An application-level kernel for record and replay. In Pro- ceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI’08, pages 193–208, USA, 2008. USENIX Association.

[70] P. Gupta, M. Ivey, and J. Penix. Testing at the speed and scale of google. http://google-engtools.blogspot.com/2011/06/ testing-at-speed-and-scale-of-google.html, June 2011.

[71] D. Hao, L. Zhang, and H. Mei. Test-case prioritization: achievements and challenges. Frontiers of Computer Science, 10(5):769–777, jun 2016.

[72] B. Hardekopf and C. Lin. Semi-sparse flow-sensitive pointer analysis. In Proceedings of the 36th Annual ACM SIGPLAN-SIGACT Symposium on Prin- ciples of Programming Languages, POPL ’09, pages 226–238, New York, NY, USA, 2009. Association for Computing Machinery.

[73] P. He, Z. Chen, S. He, and M. R. Lyu. Characterizing the natural lan- guage descriptions in software logging statements. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineer- ing, ASE 2018, pages 178–189, New York, NY, USA, 2018. Association for Computing Machinery.

[74] M. Hilton, N. Nelson, T. Tunnell, D. Marinov, and D. Dig. Trade-offs in continuous integration: Assurance, security, and flexibility. In Proceed- ings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pages 197–207, New York, NY, USA, 2017. Association for Computing Machinery.

[75] M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig. Usage, costs, and benefits of continuous integration in open-source projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engi- neering, ASE 2016, pages 426–437, New York, NY, USA, 2016. Association for Computing Machinery.

[76] F. Hoffa. Who contributed the most to open source in 2017 and 2018? let’s analyze github’s data and find out. https://www.freecodecamp. org/news/the-top-contributors-to-github-2017- be98ab854e87/, October 2017.

[77] S. Hore, L. Lewis, P. Palukuri, and A. Dindzans. Ontap autosupport and au- tosupport on demand. https://www.netapp.com/us/media/ tr-4444.pdf, October 2017.

[78] J. Huang, M. Allen-Bond, and X. Zhang. Pallas: Semantic-aware checking for finding deep bugs in fast path. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Lan- guages and Operating Systems, ASPLOS ’17, pages 709–722, New York, NY, USA, 2017. Association for Computing Machinery.

[79] S. Jayatheertha, L. S. Balu, and A. C. H. Streamlined trou- bleshooting with the dell system e-support tool. https://www.dell.com/downloads/global/power/ps4q08- 20080257-Jayatheertha.pdf, November 2008.

[80] B. Jiang, Z. Zhang, W. K. Chan, and T. H. Tse. Adaptive random test case prioritization. In 2009 IEEE/ACM International Conference on Automated Software Engineering, pages 233–244, 2009.

[81] H. Jula, D. Tralamazza, C. Zamfir, and G. Candea. Deadlock immu- nity: Enabling systems to defend against deadlocks. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI’08, pages 295–308, USA, 2008. USENIX Association.

[82] A. Kadav, M. J. Renzelmann, and M. M. Swift. Tolerating hardware device failures in software. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP ’09, pages 59–72, New York, NY, USA, 2009. Association for Computing Machinery.

[83] M. P. Kasick, J. Tan, R. Gandhi, and P. Narasimhan. Black-box problem diagnosis in parallel file systems. In Proceedings of the 8th USENIX Confer- ence on File and Storage Technologies, FAST’10, page 4, USA, 2010. USENIX Association.

[84] B. Kasikci, B. Schubert, C. Pereira, G. Pokam, and G. Candea. Fail- ure sketching: A technique for automated root cause diagnosis of in- production failures. In Proceedings of the 25th Symposium on Operating Systems Principles, SOSP ’15, pages 344–360, New York, NY, USA, 2015. Association for Computing Machinery.

[85] B. Kasikci, C. Zamfir, and G. Candea. Racemob: Crowdsourced data race detection. In Proceedings of the Twenty-Fourth ACM Symposium on Operat- ing Systems Principles, SOSP ’13, pages 406–422, New York, NY, USA, 2013. Association for Computing Machinery.

[86] Y. Keren. Zapcc. https://github.com/yrnkrn/zapcc, 2018.

[87] C. H. P. Kim, D. Marinov, S. Khurshid, D. Batory, S. Souto, P. Barros, and M. D’Amorim. Splat: Lightweight dynamic analysis for reducing combina- torics in testing configurable systems. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, pages 257–267, New York, NY, USA, 2013. Association for Computing Machin- ery.

[88] S. T. King, G. W. Dunlap, and P. M. Chen. Debugging operating systems with time-traveling virtual machines. In Proceedings of the Annual Con- ference on USENIX Annual Technical Conference, ATEC ’05, page 1, USA, 2005. USENIX Association.

[89] Kitware Inc. Cmake. https://cmake.org/, 2020.

[90] S. Kratky. sakra/cotire: Cmake module to speed up builds. https:// github.com/sakra/cotire, 2019.

[91] P. Krill. Software engineers spend lots of time not building soft- ware. https://www.infoworld.com/article/2613762/ software-engineers-spend-lots-of-time-not- building-software.html, April 2015.

[92] T. Kubota. Btrfs: add another missing end_page_writeback on sub- mit_extent_page failure. https://github.com/torvalds/ linux/commit/fe01aa65, February 2017.

[93] T. Kubota. [v2] btrfs: add another missing end_page_writeback on sub- mit_extent_page failure. https://patchwork.kernel.org/ patch/9564177/, February 2017.

[94] T. Kubota, Y. Suzuki, and K. Kono. To unify or not to unify: A case study on unified builds (in webkit). In Proceedings of the 28th International Con- ference on Compiler Construction, CC 2019, pages 42–52, New York, NY, USA, 2019. Association for Computing Machinery.

[95] A. Kumar. Development at the speed and scale of google. http://goo. gl/hCPQx\, December 2010.

[96] W. Landi. Undecidability of static analysis. ACM Letters on Programming Languages and Systems, 1(4):323–337, dec 1992.

[97] J.-C. Laprie. Dependable computing: Concepts, limits, challenges. In Proceedings of the Twenty-Fifth International Conference on Fault-Tolerant Computing, FTCS’95, pages 42–54, USA, 1995. IEEE Computer Society.

[98] J. Lawall and G. Muller. Coccinelle: 10 years of automated evolution in the linux kernel. In Proceedings of the 2018 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC ’18, pages 601–613, USA, 2018. USENIX Association.

[99] D. Lee, B. Wester, K. Veeraraghavan, S. Narayanasamy, P. M. Chen, and J. Flinn. Respec: Efficient online multiprocessor replayvia speculation and external determinism. In Proceedings of the Fifteenth International Confer- ence on Architectural Support for Programming Languages and Operating Systems, ASPLOS XV, pages 77–90, New York, NY, USA, 2010. Association for Computing Machinery.

[100] C. Legnitto. The facebook mobile release process. https:// www.infoq.com/presentations/Facebook-Release- Process/, February 2014.

[101] H. Li, W. Shang, Y. Zou, and A. E. Hassan. Towards just-in-time sugges- tions for log changes. Empirical Software Engineering, 22(4):1831–1865, oct 2016.

[102] S. Li, X. Niu, Z. Jia, J. Wang, H. He, and T. Wang. Logtracker: Learn- ing log revision behaviors proactively from software evolution history. In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC), pages 178–17810, 2018.

[103] B. Liblit, A. Aiken, A. X. Zheng, and M. I. Jordan. Bug isolation via remote program sampling. In Proceedings of the ACM SIGPLAN 2003 Conference on Programming Language Design and Implementation, PLDI ’03, pages 141– 154, New York, NY, USA, 2003. Association for Computing Machinery.

[104] R. Manevich, M. Sridharan, S. Adams, M. Das, and Z. Yang. Pse: Explaining program failures via postmortem static analysis. In Proceedings of the 12th ACM SIGSOFT Twelfth International Symposium on Foundations of Software Engineering, SIGSOFT ’04/FSE-12, pages 63–72, New York, NY, USA, 2004. Association for Computing Machinery.

[105] J. Mao, Y. Chen, Q. Xiao, and Y. Shi. Rid: Finding reference count bugs with inconsistent path pair checking. In Proceedings of the Twenty-First Interna- tional Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’16, pages 531–544, New York, NY, USA, 2016. Association for Computing Machinery.

[106] L. Mariani and F. Pastore. Automated identification of failure causes in system logs. In 2008 19th International Symposium on Software Reliability Engineering (ISSRE), pages 117–126, 2008.

[107] L. Mariani, F. Pastore, and M. Pezze. A toolset for automated failure anal- ysis. In Proceedings of the 31st International Conference on Software Engi- neering, ICSE ’09, pages 563–566, USA, 2009. IEEE Computer Society.

[108] D. Marijan, A. Gotlieb, and S. Sen. Test case prioritization for continuous regression testing: An industrial case study. In Proceedings of the 2013 IEEE International Conference on Software Maintenance, ICSM ’13, pages 540–543, USA, 2013. IEEE Computer Society.

[109] D. Marsh. From code to monkeys: Continuous delivery at netflix. https://www.infoq.com/presentations/netflix- continuous-delivery/, November 2013.

[110] E. Martin. Ninja, a small build system with a focus on speed. https://ninja-build.org/, 2012.

[111] A. J. Mashtizadeh, T. Garfinkel, D. Terei, D. Mazieres, and M. Rosenblum. Towards practical default-on multi-core record/replay. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’17, pages 693– 708, New York, NY, USA, 2017. Association for Computing Machinery.

[112] S. McIntosh, B. Adams, M. Nagappan, and A. E. Hassan. Identifying and understanding header file hotspots in C/C++ build processes. Automated Software Engineering, 2005.

[113] P. McMinn. Search-based software test data generation: a survey: Re- search articles. Softw. Test., Verif. Reliab., 14:105–156, 06 2004.

[114] J. Micco. Tools for continuous integration at google scale. https:// www.youtube.com/watch?v=KH2_sB1A6lA, June 2012.

[115] K. Miller. [webkit-dev] unified source builds: A new rule for static vari- ables. https://lists.webkit.org/pipermail/webkit- dev/2017-August/029465.html, Augest 2017.

[116] B. Miranda, E. Cruciani, R. Verdecchia, and A. Bertolino. FAST approaches to scalable similarity-based test case prioritization. In Proceedings of the 40th International Conference on Software Engineering. ACM, may 2018.

[117] N. Mitchell. Shake build system. https://shakebuild.com/, 2020.

[118] A. Mokhov, N. Mitchell, and S. Peyton Jones. Build systems à la carte. Proc. ACM Program. Lang., 2(ICFP), July 2018.

[119] K.-K. Muniswamy-Reddy, P. Macko, and M. Seltzer. Provenance for the cloud. In Proceedings of the 8th USENIX Conference on File and Storage Technologies, FAST’10, pages 15–14, USA, 2010. USENIX Association.

[120] K. Nagaraj, C. Killian, and J. Neville. Structured comparative analysis of systems logs to diagnose performance problems. In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, NSDI’12, page 26, USA, 2012. USENIX Association.

[121] T. Nagy. Waf: the meta build system. https://waf.io/, 2019.

[122] NetApp Inc. Savecore. ONTAP 7.3 Manual Page Reference, Volume 1, Pages 471-472, June 2014.

[123] M. Olszewski, J. Ansel, and S. Amarasinghe. Kendo: Efficient determin- istic multithreading in software. In Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Op- erating Systems, ASPLOS XIV, pages 97–108, New York, NY, USA, 2009. Association for Computing Machinery.

[124] A. Panichella, F. M. Kifetew, and P. Tonella. Automated test case gener- ation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering, 44(2):122–158, 2018.

[125] S. Park, Y. Zhou, W. Xiong, Z. Yin, R. Kaushik, K. H. Lee, and S. Lu. PRES. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles - SOSP '09. ACM Press, 2009.

[126] C. Prasad and W. Schulte. Taking control of your engineering tools. Com- puter, 46(11):63–66, 2013.

[127] T. M. D. Project. The mesa 3d graphics library. https://www. mesa3d.org/, 2020.

[128] G. Ramalingam. The undecidability of aliasing. ACM Transactions on Pro- gramming Languages and Systems, 16(5):1467–1471, sep 1994.

[129] C. Robertson, T. Lyngmo, and N. Schonning. Precompiled header files. https://docs.microsoft.com/en-US/cpp/build/ creating-precompiled-header-files?view=vs-2019, October 2019.

[130] J. M. Rojas, M. Vivanti, A. Arcuri, and G. Fraser. A detailed investigation of the effectiveness of whole test suite generation. Empirical Softw. Engg., 22(2):852–893, Apr. 2017.

[131] J. Rosdahl. ccache — compiler cache. https://ccache.dev/, 2020.

[132] N. Sidwell. cxx-modules - gcc wiki. https://gcc.gnu.org/ wiki/cxx-modules, July 2020.

[133] B. H. Sigelman, L. A. Barroso, M. Burrows, P. Stephenson, M. Plakal, D. Beaver, S. Jaspan, and C. Shanbhag. Dapper, a large-scale distributed systems tracing infrastructure. Technical report, Google Inc., 2010.

[134] R. Smith. Merging modules. http://www.open-std.org/ jtc1/sc22/wg21/docs/papers/2019/p1103r3.pdf, February 2019.

[135] R. Smith. Working draft, standard for programming language c++. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/n4849.pdf, January 2020.

[136] N. Stew. Automatically finding slow headers in c++ projects. https://nibblestew.blogspot.com/2018/02/ automatically-finding-slow-headers-in-c.html, February 2018.

[137] Y. Sui, P. Di, and J. Xue. Sparse flow-sensitive pointer analysis for multi- threaded programs. In Proceedings of the 2016 International Symposium on Code Generation and Optimization - CGO 2016. ACM Press, 2016.

[138] J. Tan, S. Kavulya, R. Gandhi, and P. Narasimhan. Visual, log-based causal tracing for performance debugging of mapreduce systems. In 2010 IEEE 30th International Conference on Distributed Computing Systems, pages 795–806, 2010.

[139] J. Tan, X. Pan, S. Kavulya, R. Gandhi, and P. Narasimhan. Salsa: Analyzing logs as state machines. In Proceedings of the First USENIX Conference on Analysis of System Logs, WASL’08, page 6, USA, 2008. USENIX Association.

[140] J. Tan, X. Pan, S. Kavulya, R. Gandhi, and P. Narasimhan. Mochi: Visual log-analysis based tools for debugging hadoop. In Proceedings of the 2009 Conference on Hot Topics in Cloud Computing, HotCloud’09, USA, 2009. USENIX Association.

[141] P.-N. Tan, M. Steinbach, A. Karpatne, and V. Kumar. Introduction to Data Mining. Pearson, 2nd edition, 2019.

[142] The Chromium Team. Breakpad. https://chromium. googlesource.com/breakpad/breakpad.

[143] The Chromium Team. Chrome infrastructure - welcome to luci! https://chromium.googlesource.com/infra/infra/+/master/doc/users/services/about_luci.md, 2020.

[144] The Chromium Team. Chromium. https://www.chromium.org/ Home, 2020.

[145] The Clang Team. Precompiled header and modules internals. https://clang.llvm.org/docs/PCHInternals.html, 2020.

[146] The GCC Team. Precompiled headers (using the gnu compiler col- lection (gcc)). https://gcc.gnu.org/onlinedocs/gcc/ Precompiled-Headers.html, 2020.

[147] The LLVM Team. Buildbot for the llvm project. http://lab.llvm. org:8011/, September 2020.

[148] The Meson Team. The meson build system. https://mesonbuild. com/, 2020.

[149] The Squid Team. Squid web proxy cache. http://www.squid- cache.org/.

[150] The WebKit Team. Buildbot: Webkit. https://build.webkit. org, September 2020.

[151] J. Thomas and A. Kumar. Google engineering tools. http://google-engtools.blogspot.com/2011/05/welcome-to-google-engineering-tools.html, May 2011.

[152] J. Tucek, S. Lu, C. Huang, S. Xanthos, and Y. Zhou. Triage: Diagnosing production run failures at the user’s site. In Proceedings of Twenty-First ACM SIGOPS Symposium on Operating Systems Principles, SOSP ’07, pages 131–144, New York, NY, USA, 2007. Association for Computing Machinery.

[153] T. Tuttle. workqueue: lock cwq access in drain_workqueue. https:// github.com/torvalds/linux/commit/fa2563e, Septem-ber 2011.

[154] K. Veeraraghavan, D. Lee, B. Wester, J. Ouyang, P. M. Chen, J. Flinn, and S. Narayanasamy. Doubleplay: Parallelizing sequential logging and re- play. In Proceedings of the Sixteenth International Conference on Architec- tural Support for Programming Languages and Operating Systems, ASPLOS XVI, pages 15–26, New York, NY, USA, 2011. Association for Computing Machinery.

[155] E. Vlachos, M. L. Goodstein, M. A. Kozuch, S. Chen, B. Falsafi, P. B. Gib- bons, and T. C. Mowry. Paralog: Enabling and accelerating online par- allel monitoring of multithreaded applications. In Proceedings of the Fif- teenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XV, pages 271–284, New York, NY, USA, 2010. Association for Computing Machinery.

[156] X. Wang, H. Chen, Z. Jia, N. Zeldovich, and M. F. Kaashoek. Improving in- teger security for systems with kint. In Proceedings of the 10th USENIX Con- ference on Operating Systems Design and Implementation, OSDI’12, pages 163–177, USA, 2012. USENIX Association.

[157] X. Wang, N. Zeldovich, M. F. Kaashoek, and A. Solar-Lezama. Towards optimization-safe systems: Analyzing the impact of undefined behavior. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, SOSP ’13, pages 260–275, New York, NY, USA, 2013. Association for Computing Machinery.

[158] D. Weeratunge, X. Zhang, and S. Jagannathan. Analyzing multicore dumps to facilitate concurrency bug reproduction. ACM SIGARCH Computer Ar- chitecture News, 38(1):155–166, mar 2010.

[159] W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan. Detecting large- scale system problems by mining console logs. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP ’09, pages 117–132, New York, NY, USA, 2009. Association for Computing Machinery.

[160] S. Yang, T. Harter, N. Agrawal, S. S. Kowsalya, A. Krishnamurthy, S. Al- Kiswany, R. T. Kaushik, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. Split-level i/o scheduling. In Proceedings of the 25th Symposium on Operat- ing Systems Principles, SOSP ’15, pages 474–489, New York, NY, USA, 2015. Association for Computing Machinery.

[161] X. Yu, P. Joshi, J. Xu, G. Jin, H. Zhang, and G. Jiang. Cloudseer: Workflow monitoring of cloud infrastructures via interleaved logs. In Proceedings of the Twenty-First International Conference on Architectural Support for Pro- gramming Languages and Operating Systems, ASPLOS ’16, pages 489–502, New York, NY, USA, 2016. Association for Computing Machinery.

[162] D. Yuan, H. Mai, W. Xiong, L. Tan, Y. Zhou, and S. Pasupathy. Sherlog: Error diagnosis by connecting clues from run-time logs. In Proceedings of the Fifteenth International Conference on Architectural Support for Program- ming Languages and Operating Systems, ASPLOS XV, pages 143–154, New York, NY, USA, 2010. Association for Computing Machinery.

[163] D. Yuan, S. Park, P. Huang, Y. Liu, M. M. Lee, X. Tang, Y. Zhou, and S. Sav- age. Be conservative: Enhancing failure diagnosis with proactive logging. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation, OSDI’12, pages 293–306, USA, 2012. USENIX Associ- ation.

[164] D. Yuan, S. Park, and Y. Zhou. Characterizing logging practices in open- source software. In Proceedings of the 34th International Conference on Soft- ware Engineering, ICSE ’12, pages 102–112. IEEE Press, 2012.

[165] D. Yuan, J. Zheng, S. Park, Y. Zhou, and S. Savage. Improving software diagnosability via log enhancement. ACM Trans. Comput. Syst., 30(1), 2012.

[166] T. Zhang, D. Lee, and C. Jung. Txrace: Efficient data race detection us- ing commodity hardware transactional memory. In Proceedings of the Twenty-First International Conference on Architectural Support for Program- ming Languages and Operating Systems, ASPLOS ’16, pages 159–173, New York, NY, USA, 2016. Association for Computing Machinery.

[167] Q. Zhao, R. Rabbah, S. Amarasinghe, L. Rudolph, and W.-F. Wong. How to do a million watchpoints: Efficient debugging using dynamic instru- mentation. In Proceedings of the Joint European Conferences on Theory and Practice of Software 17th International Conference on Compiler Construc- tion, CC’08/ETAPS’08, pages 147–162, Berlin, Heidelberg, 2008. Springer- Verlag.

[168] X. Zhao, K. Rodrigues, Y. Luo, M. Stumm, D. Yuan, and Y. Zhou. Log20: Fully automated optimal placement of log printing statements under spec- ified overhead threshold. In Proceedings of the 26th Symposium on Operat- ing Systems Principles, SOSP ’17, pages 565–581, New York, NY, USA, 2017. Association for Computing Machinery.

[169] X. Zhao, Y. Zhang, D. Lion, M. F. Ullah, Y. Luo, D. Yuan, and M. Stumm. Lprof: A non-intrusive request flow profiler for distributed systems. In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation, OSDI’14, pages 629–644, USA, 2014. USENIX Association.

[170] J. Zhu, P. He, Q. Fu, H. Zhang, M. R. Lyu, and D. Zhang. Learning to log: Helping developers make informed logging decisions. In Proceedings of the 37th International Conference on Software Engineering - Volume 1, ICSE ’15, pages 415–425. IEEE Press, 2015.