Jekyll2021-10-11T09:12:41+00:00https://onqtam.com/feed.xmlNot just gamedevBlog about game development, C++, programming, entrepreneurship, motivation, books, fitness, swimming, health and other parts of life.Viktor Kirilovvik.kirilov@gmail.comCMake 3.16 added support for precompiled headers & unity builds - what you need to know2019-12-20T13:40:34+00:002019-12-20T13:40:34+00:00https://onqtam.com/programming/pch-unity-cmake-3-16<p>Modules are coming in C++20 but it will take a while before they are widely adopted, optimized and supported by tooling - what can we do <strong>right now</strong>?</p>
<p><img src="/assets/images/compiling.png" alt="" /></p>
<p>I recently consulted a company on this exact matter - luckily <a href="https://cmake.org/download/">CMake 3.16 was just released</a> and there was no need to resort to <a href="https://github.com/sakra/cotire">3rd party CMake scripts</a> for precompiled headers and unity builds (thanks to <a href="https://twitter.com/cristianadam">Cristian Adam</a> for the hard work - <a href="https://gitlab.kitware.com/cmake/cmake/merge_requests/3553">MR 1</a>, <a href="https://gitlab.kitware.com/cmake/cmake/merge_requests/3611">MR 2</a>!!!). Here is what I told them:</p>
<h1 id="precompiled-headers-pch">Precompiled headers (PCH)</h1>
<h3 id="the-idea-is-to-precompile-a-bunch-of-common-header-files">The idea is to <em>precompile</em> a bunch of common header files</h3>
<ul>
<li>to <em>precompile</em> means that the compiler will parse the C++ headers and save its intermediate representation (IR) into a file, and then when compiling the <code class="language-plaintext highlighter-rouge">.cpp</code> files of the target that IR will be prepended to them - as if the headers were included - the contents of the PCH are the first thing each translation unit sees</li>
<li>easy to integrate - doesn’t require any C++ code changes</li>
<li>~20-30% speedup with GCC/Clang (can be up to 50%+ with MSVC)</li>
<li>for targets with at least 10 <code class="language-plaintext highlighter-rouge">.cpp</code> files (takes space & time to compile)</li>
</ul>
<h3 id="what-to-put-in-a-pch">What to put in a PCH</h3>
<ul>
<li>STL & third-party libs like boost (used in at least ~30% of the sources)</li>
<li>some <strong>rarely changing</strong> project-specific headers (at least 30% use)
<ul>
<li>for example if you have common utilities for logging/etc.</li>
</ul>
</li>
<li>each time any header which ends up in the PCH is changed - the entire PCH is recompiled along with the entire target which includes it</li>
<li>careful not to put too much into a PCH - once it reaches ~150-200MB you might start hitting diminishing returns</li>
<li>how to determine which are the most commonly used header files
<ul>
<li>option 1: do a few searches in the codebase/target
<ul>
<li><code class="language-plaintext highlighter-rouge"><algorithm></code>, <code class="language-plaintext highlighter-rouge"><vector></code>, <code class="language-plaintext highlighter-rouge"><boost/asio.hpp></code>, etc.</li>
<li>note that some header might be included only in a few other header files, but if those headers go everywhere, then the other header gets included almost everywhere as well</li>
</ul>
</li>
<li>option 2: - <a href="https://slides.com/onqtam/faster_builds#/22">use software to visualize includes & dependencies</a></li>
</ul>
</li>
</ul>
<h3 id="how-to-use">How to use</h3>
<ul>
<li><a href="https://cmake.org/cmake/help/latest/command/target_precompile_headers.html"><code class="language-plaintext highlighter-rouge">target_precompile_headers(<my_target> PRIVATE my_pch.h)</code></a></li>
<li>the PCH will be included automatically in every <code class="language-plaintext highlighter-rouge">.cpp</code> file
<ul>
<li>adding a PCH to a target doesn’t require that you remove the headers in it from all <code class="language-plaintext highlighter-rouge">.cpp</code> files - the C preprocessor is fast</li>
</ul>
</li>
<li>easiest if a single header includes the common ones - <a href="https://github.com/onqtam/game/blob/master/src/precompiled.h">example</a>
<ul>
<li>you could have per-project precompiled header files or you could <a href="https://cmake.org/cmake/help/latest/command/target_precompile_headers.html#reusing-precompile-headers">reuse a PCH from one CMake target in another</a> - remember that each PCH takes around ~50-200MB and takes some time to compile…</li>
</ul>
</li>
<li>you could list the headers which you want precompiled directly in the call to <code class="language-plaintext highlighter-rouge">target_precompile_headers</code> and even set them as <code class="language-plaintext highlighter-rouge">PUBLIC</code>/<code class="language-plaintext highlighter-rouge">PRIVATE</code> selectively so other targets which link to the current one would also precompile those, but I’m old fashioned and prefer to maintain the PCH for each target on my own.</li>
</ul>
<h3 id="some-problems">Some problems</h3>
<ul>
<li>adding a header which was used only in 30% of the <code class="language-plaintext highlighter-rouge">.cpp</code> files to the precompiled header means that all <code class="language-plaintext highlighter-rouge">.cpp</code> files in the target will now have access to it - in time more files might start depending on it without you even noticing - the code might not build without the PCH anymore
<ul>
<li>do you care if it compiles successfully without a PCH? If so, make every <code class="language-plaintext highlighter-rouge">.cpp</code> explicitly include the precompiled header. This would be problematic if the same <code class="language-plaintext highlighter-rouge">.cpp</code> file is used in 2 or more CMake targets with different PCHs but in that case you should probably move that <code class="language-plaintext highlighter-rouge">.cpp</code> into a static lib, compile it only once and link against that!</li>
</ul>
</li>
<li>if you are using GCC but are using <code class="language-plaintext highlighter-rouge">ccls</code>/<code class="language-plaintext highlighter-rouge">cquery</code>/<code class="language-plaintext highlighter-rouge">clangd</code>/<code class="language-plaintext highlighter-rouge">rtags</code> or (based on clang) as a language server - those tools might not work because they will try to read the <code class="language-plaintext highlighter-rouge">.gch</code> file produced by GCC - <a href="https://bugs.llvm.org/show_bug.cgi?id=41579">bug report</a></li>
</ul>
<h1 id="unity-builds">Unity builds</h1>
<h3 id="the-idea-is-to-cram-the-cpp-files-of-a-cmake-target-into-a-few-cpp-files-which-include-the-original-cpp-files">The idea is to cram the <code class="language-plaintext highlighter-rouge">.cpp</code> files of a CMake target into a few <code class="language-plaintext highlighter-rouge">.cpp</code> files which include the original <code class="language-plaintext highlighter-rouge">.cpp</code> files</h3>
<ul>
<li>up to 7-8 times faster builds (usually x3 or x4)</li>
<li>example: a project of 200 <code class="language-plaintext highlighter-rouge">.cpp</code> files might be divided into 16 unity <code class="language-plaintext highlighter-rouge">.cpp</code> (or <code class="language-plaintext highlighter-rouge">.cxx</code> - whatever) files each including about 13 of the original <code class="language-plaintext highlighter-rouge">.cpp</code> files => 16 <code class="language-plaintext highlighter-rouge">.cpp</code> files to build in parallel and 16 <code class="language-plaintext highlighter-rouge">.obj</code> files to link</li>
<li>the reasons for the speedup are:
<ul>
<li>common headers from the different <code class="language-plaintext highlighter-rouge">.cpp</code> files end up being included and parsed fewer times (this is beneficial even when using PCHs!)</li>
<li>common template instantiations with the same types in separate <code class="language-plaintext highlighter-rouge">.cpp</code> files (<code class="language-plaintext highlighter-rouge">vector<int></code>) end up being done in fewer places</li>
<li>the linker has to stitch much fewer <code class="language-plaintext highlighter-rouge">.obj</code> files in the end - there are a lot less <em>weak</em> symbols to deduplicate (<code class="language-plaintext highlighter-rouge">inline</code>/template functions from headers end up in every <code class="language-plaintext highlighter-rouge">.obj</code> => linkers leave just 1)
<ul>
<li>surprisingly incremental builds (changing a single <code class="language-plaintext highlighter-rouge">.cpp</code>) will probably also be faster instead of slower (even though you compile more <code class="language-plaintext highlighter-rouge">.cpp</code> files together) - precisely because of the reduced number of <em>weak</em> duplicated symbols!</li>
</ul>
</li>
<li>less compiler invocations and less <code class="language-plaintext highlighter-rouge">.obj</code> files are written to disk</li>
</ul>
</li>
<li>the most reliable way to detect <a href="https://stackoverflow.com/questions/31722473">ODR violations</a></li>
<li>runtime (final binary) might even be faster! (free <a href="https://en.wikipedia.org/wiki/Interprocedural_optimization">LTO</a>) - because the compiler sees more symbols from different <code class="language-plaintext highlighter-rouge">.cpp</code> files together</li>
<li><a href="http://onqtam.com/programming/2018-07-07-unity-builds/">A detailed blog post about unity builds and why they make builds faster</a></li>
</ul>
<h3 id="how-to-use-1">How to use</h3>
<ul>
<li>CMake 3.16 adds the <a href="https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html"><code class="language-plaintext highlighter-rouge">UNITY_BUILD</code></a> target property
<ul>
<li>you can set this property per target explicitly
<ul>
<li><code class="language-plaintext highlighter-rouge">set_target_properties(<target> PROPERTIES UNITY_BUILD ON)</code></li>
</ul>
</li>
<li>or set it globally: <code class="language-plaintext highlighter-rouge">set(CMAKE_UNITY_BUILD ON)</code> (or call CMake with <code class="language-plaintext highlighter-rouge">-DCMAKE_UNITY_BUILD=ON</code>) and then you can explicitly disable it for some targets by setting their property to <code class="language-plaintext highlighter-rouge">OFF</code></li>
</ul>
</li>
<li>the order in which the <code class="language-plaintext highlighter-rouge">.cpp</code> files go into the batches depends on the order they were given to a target in <code class="language-plaintext highlighter-rouge">add_library</code>/<code class="language-plaintext highlighter-rouge">add_executable</code></li>
<li>if for some reason 2 <code class="language-plaintext highlighter-rouge">.cpp</code> files are hard to compile together they can be separated in different batches by reordering the sources
<ul>
<li>or use <a href="https://cmake.org/cmake/help/latest/prop_sf/SKIP_UNITY_BUILD_INCLUSION.html"><code class="language-plaintext highlighter-rouge">SKIP_UNITY_BUILD_INCLUSION</code></a> to exclude one</li>
</ul>
</li>
<li>about 10-20 <code class="language-plaintext highlighter-rouge">.cpp</code> files per unity is the most optimal
<ul>
<li>this is controlled through the <a href="https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD_BATCH_SIZE.html"><code class="language-plaintext highlighter-rouge">UNITY_BUILD_BATCH_SIZE</code></a> target property - default is 8 (can be set globally with <a href="https://cmake.org/cmake/help/latest/variable/CMAKE_UNITY_BUILD_BATCH_SIZE.html"><code class="language-plaintext highlighter-rouge">CMAKE_UNITY_BUILD_BATCH_SIZE</code></a>)</li>
<li>don’t worry if a target has few <code class="language-plaintext highlighter-rouge">.cpp</code> files - if it has more than 1 it would benefit from a unity build, + decent build systems like ninja will schedule <code class="language-plaintext highlighter-rouge">.obj</code> files from different targets to be built in parallel</li>
</ul>
</li>
<li>the unity <code class="language-plaintext highlighter-rouge">.cpp</code> files will go in the build directory - you don’t have to maintain them or add them to version control</li>
</ul>
<h3 id="initial-problems-when-trying-to-compile-a-project-as-unity">Initial problems when trying to compile a project as unity</h3>
<ul>
<li>some headers will be missing include guards or <code class="language-plaintext highlighter-rouge">#pragma once</code></li>
<li>there will be static globals (or in anonymous namespaces) in different <code class="language-plaintext highlighter-rouge">.cpp</code> files with identical names which would clash
<ul>
<li>either rename them or put such globals into an additional namespace - perhaps with the name of the file: for <code class="language-plaintext highlighter-rouge">GraphVisitor.cpp</code> I would recommend <code class="language-plaintext highlighter-rouge">GRAPH_VISITOR_CPP</code> (putting static symbols inside of a named namespace in a <code class="language-plaintext highlighter-rouge">.cpp</code> keeps their linkage to <em>internal</em> , and the same is true with nesting anonymous namespaces into named ones)</li>
</ul>
</li>
<li>there will be symbol ambiguities
<ul>
<li>mostly because some <code class="language-plaintext highlighter-rouge">.cpp</code> file uses a namespace, and then some other <code class="language-plaintext highlighter-rouge">.cpp</code> file which ends up in the same unity <code class="language-plaintext highlighter-rouge">.cpp</code> cannot compile</li>
<li>either remove the <code class="language-plaintext highlighter-rouge">using namespace ...</code> stuff or fully qualify symbols where necessary (can use <code class="language-plaintext highlighter-rouge">::</code> to mean <em>from global scope</em>)</li>
</ul>
</li>
<li>some macros might have to be explicitly <code class="language-plaintext highlighter-rouge">#undef</code>-ined at the end of the <code class="language-plaintext highlighter-rouge">.cpp</code> files where they are defined/used
<ul>
<li>take a look at <a href="https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD_CODE_BEFORE_INCLUDE.html"><code class="language-plaintext highlighter-rouge">UNITY_BUILD_CODE_BEFORE_INCLUDE</code></a> and <a href="https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD_CODE_AFTER_INCLUDE.html"><code class="language-plaintext highlighter-rouge">UNITY_BUILD_CODE_AFTER_INCLUDE</code></a></li>
<li>or rewrite them to C++ constructs - macros are <strong>$h1t</strong> anyway</li>
</ul>
</li>
<li>another more obscure possible problem: if a static library gets built as a unity build => there are fewer <code class="language-plaintext highlighter-rouge">.obj</code> files stitched together, and if some other target links to the static library and uses only some symbols - from a few of the original <code class="language-plaintext highlighter-rouge">.obj</code> files, it might no longer link because now there are other symbols in the same <code class="language-plaintext highlighter-rouge">.obj</code> file where the needed symbols reside, and those other symbols might need some other symbols which are in some other static library
<ul>
<li>solution: find what else needs to be linked in</li>
</ul>
</li>
<li>unity build failures are rare after the initial cleanup (and if everyone uses unity builds locally there wouldn’t be any)</li>
<li>it took me about ~2 full days to compile about 2000 <code class="language-plaintext highlighter-rouge">.cpp</code> files in 20+ different CMake targets as unity in my current company (<a href="https://www.nuodb.com/">NuoDB</a>])</li>
</ul>
<h3 id="recommended-way-to-go-about-it">Recommended way to go about it</h3>
<ul>
<li>I would recommend trying targets 1 by 1 by setting their target property <code class="language-plaintext highlighter-rouge">UNITY_BUILD</code> to ON - and not to enable it globally directly</li>
<li>start with low batching - first try to get the project to compile with 4 <code class="language-plaintext highlighter-rouge">.cpp</code> files per unity, and then increase</li>
<li>if you desire to have 20 <code class="language-plaintext highlighter-rouge">.cpp</code> files per batch in the end - go to atleast 40 or 50, clean the errors and then move back to 20
<ul>
<li>=> future problems will be less likely - when a new <code class="language-plaintext highlighter-rouge">.cpp</code> file is added somewhere it changes which <code class="language-plaintext highlighter-rouge">.cpp</code> files get paired together</li>
</ul>
</li>
<li>unity builds should eventually become the default mode for building
<ul>
<li>there should be a separate CI build that checks that the project still compiles not as unity - checking for missing includes.</li>
</ul>
</li>
</ul>
<h3 id="some-problems-1">Some problems</h3>
<p>If you use <code class="language-plaintext highlighter-rouge">-DCMAKE_EXPORT_COMPILE_COMMANDS=ON</code> you get a file called <code class="language-plaintext highlighter-rouge">compile_commands.json</code> generated by CMake in the build folder</p>
<ul>
<li>that file contains compile commands for each translation unit - with all definitions and includes necessary for parsing the C++</li>
<li>that file is used by tools such as <code class="language-plaintext highlighter-rouge">ccls</code>/<code class="language-plaintext highlighter-rouge">cquery</code>/<code class="language-plaintext highlighter-rouge">clangd</code>/<code class="language-plaintext highlighter-rouge">rtags</code> - language servers which are usually integrated with editors and IDEs such as <code class="language-plaintext highlighter-rouge">vim</code>/<code class="language-plaintext highlighter-rouge">emacs</code>/<code class="language-plaintext highlighter-rouge">VSCode</code> for intellisense (code completion, refactoring and syntax highlighting)</li>
<li>when using unity builds the build information there will not contain compile commands for the specific <code class="language-plaintext highlighter-rouge">.cpp</code> files but only for the actual unity <code class="language-plaintext highlighter-rouge">.cpp</code> files - these tools might stop working correctly (<a href="https://gitlab.kitware.com/cmake/cmake/issues/19826">bug report</a>). In my current company we have a wrapper script which does the calls to CMake like this:
<ul>
<li>calls CMake once with disabled unity builds + generation of the <code class="language-plaintext highlighter-rouge">compile_commands.json</code> file</li>
<li>calls CMake again with unity builds enabled + no <code class="language-plaintext highlighter-rouge">compile_commands.json</code></li>
<li>=> the only downside of this is that if later we invoke our build system directly and it detects changes in CMake and reconfigures/regenerates the build files, it will be done with unity enabled & the generation of <code class="language-plaintext highlighter-rouge">compile_commands.json</code> disabled - perhaps leaving it stale</li>
</ul>
</li>
</ul>
<h1 id="some-other-good-tips-to-make-builds-faster">Some other good tips to make builds faster</h1>
<ul>
<li>use <a href="https://ninja-build.org/">ninja</a> instead of GNU make - developed for building Google Chrome
<ul>
<li>superior scheduler, dependency tracking & change detection - optimal parallelization of building object files and linking targets - just a single standalone portable binary</li>
<li>CMake can generate ninja build files instead of Makefiles
<ul>
<li>just pass <code class="language-plaintext highlighter-rouge">-G "Ninja"</code> when calling CMake</li>
<li>to build the code either call <code class="language-plaintext highlighter-rouge">cmake --build <path-to-build-dir></code> (instead of <code class="language-plaintext highlighter-rouge">make -j</code>) or just <code class="language-plaintext highlighter-rouge">ninja -C <path-to-build-dir></code></li>
</ul>
</li>
</ul>
</li>
<li>change your compilers and linkers
<ul>
<li>experiment moving from gcc to clang</li>
<li>move from <code class="language-plaintext highlighter-rouge">ld</code> (the default linker) to <code class="language-plaintext highlighter-rouge">gold</code> (<code class="language-plaintext highlighter-rouge">-fuse-ld=gold</code>) or even <code class="language-plaintext highlighter-rouge">lld</code> (part of the LLVM project)</li>
</ul>
</li>
<li>use <a href="https://slides.com/onqtam/faster_builds#/62">dynamic linking</a> instead of static - at least for internal builds if concerned about runtime performance
<ul>
<li>you could experiment with <code class="language-plaintext highlighter-rouge">-fvisibility-inlines-hidden</code> - <a href="https://gcc.gnu.org/wiki/Visibility">docs</a></li>
</ul>
</li>
<li>use <a href="https://github.com/onqtam/doctest/">doctest</a> for unit tests (<a href="https://github.com/onqtam/doctest/blob/master/doc/markdown/benchmarks.md">benchmarks</a>) - migrate from <a href="https://github.com/onqtam/doctest/blob/master/doc/markdown/faq.md#how-is-doctest-different-from-google-test">googletest</a>/<a href="https://github.com/onqtam/doctest/blob/master/doc/markdown/faq.md#how-is-doctest-different-from-catch">Catch2</a></li>
<li><a href="https://github.com/include-what-you-use/include-what-you-use">include-what-you-use</a>
<ul>
<li>based on clang - 100% correct parsing - helps identify unnecessary headers & where a simple forward declaration would do</li>
</ul>
</li>
<li><a href="https://github.com/google/bloaty">Bloaty McBloatface</a> - shows you a size profile of binaries
<ul>
<li>great to identify where code bloat is coming from and which symbols take the most space - which are the most offending templates?</li>
</ul>
</li>
<li>look into <a href="https://slides.com/onqtam/faster_builds#/41"><code class="language-plaintext highlighter-rouge">extern template</code> from C++11</a> (<a href="https://arne-mertz.de/2019/02/extern-template-reduce-compile-times/">blog</a>)
<ul>
<li>tells the compiler to not instantiate a specific template (for example <code class="language-plaintext highlighter-rouge">std::vector<int></code>) in the current translation unit</li>
<li>diagnosing which templates are a problem is easiest with:
<ul>
<li><a href="https://github.com/google/bloaty">Bloaty McBloatface</a>, or any of <a href="https://slides.com/onqtam/faster_builds#/45">these</a> <a href="https://slides.com/onqtam/faster_builds#/48">tools</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="https://slides.com/onqtam/faster_builds#/67">caching</a> & <a href="https://slides.com/onqtam/faster_builds#/70">distributed builds</a></li>
<li>inspecting the physical structure of projects - targets & dependencies
<ul>
<li><a href="http://www.graphviz.org/">Graphviz</a> (<a href="https://cmake.org/cmake/help/latest/module/CMakeGraphVizOptions.html">in CMake</a>) - <code class="language-plaintext highlighter-rouge">cmake --graphviz=<file></code></li>
<li><a href="https://www.sourcetrail.com/">sourcetrail</a>, or <a href="https://slides.com/onqtam/faster_builds#/75">other</a> <a href="https://slides.com/onqtam/faster_builds#/22">tools</a></li>
</ul>
</li>
<li>PIMPL (<a href="https://slides.com/onqtam/faster_builds#/27">1</a>, <a href="https://slides.com/onqtam/faster_builds#/28">2</a>), <a href="https://slides.com/onqtam/faster_builds#/38">disabling inlining for some functions</a>, rewriting templates… too much effort - do this as a last resort</li>
<li>on the hardware side - more cores, more RAM… Duuh :D
<ul>
<li>use RAM disks (filesystem in your RAM) - every OS supports those. Put the compiler and the temp & output directories there</li>
</ul>
</li>
</ul>
<h1 id="final-thoughts">Final thoughts</h1>
<p>If it was up to me most of the techniques listed here would be put to use - from top to bottom - they are sorted based on impact and cost to implement. Slow builds don’t just waste time - they also break the ‘flow’ (context switching) and discourage refactoring and experimentation - how do you put a price on that?</p>
<p>Based on my <a href="https://www.youtube.com/watch?v=anbOy47fBYI"><code class="language-plaintext highlighter-rouge">The Hitchhiker's Guide to Faster Builds</code></a> talk (<a href="https://slides.com/onqtam/faster_builds">slides</a>).</p>
<p>You can checkout the reddit discussion for this article <a href="https://www.reddit.com/r/cpp/comments/edaj7s/cmake_316_added_support_for_precompiled_headers/">here</a>.</p>
<p>I’m available for hire to fix your C++ build times - you can expect something along the lines of whatever is in this blog post.</p>Viktor Kirilovvik.kirilov@gmail.comA complete guide to applying the techniques + a few other tipsA guide to unity builds2018-07-07T15:01:34+00:002018-07-07T15:01:34+00:00https://onqtam.com/programming/unity-builds<p>A unity build can cut down build times dramatically and is HIGHLY underrated and easily dismissed by many senior software engineers (just like precompiled headers). In this post we will go over what it is, all its pros and cons, and why a “dirty hack” might be worth it if it speeds up your builds by at least a factor of 2 or perhaps even in the double-digits.</p>
<h2 id="why-care-about-build-times-in-the-first-place">Why care about build times in the first place</h2>
<p>Well… time is money! Let’s do the math: assuming an annual salary of 80k $ - waiting for 30 minutes extra a day for builds is 1/16 of the time of a developer ===> 5k $ per year. 200 such employees and we reach 1 million $ annually. But employees bring more value to the company than what they get as a salary (usually atleast x3 - if the company is adequate) - so the costs to the employer are actually bigger.</p>
<p><img src="/assets/images/compiling.png" alt="" /></p>
<p>Now let’s consider the facts that waiting for long builds discourages refactoring and experimentation and leads to mental context switches + distractions which are always expensive - so we reach the only possible conclusion: <strong>any time spent reducing build times is worthwhile!</strong></p>
<h2 id="introduction---a-short-summary-to-unity-builds">Introduction - a short summary to unity builds</h2>
<p>A unity build is when a bunch of source files are <code class="language-plaintext highlighter-rouge">#include</code>‘d into a single file which is then compiled:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// unity_file.cpp</span>
<span class="cp">#include "widget.cpp"
#include "gui.cpp"
#include "test.cpp"
</span></code></pre></div></div>
<p>Also known as: SCU (single compilation unit), amalgamated or jumbo.</p>
<p>The main benefit is lower build times (compile + link) because:</p>
<ul>
<li>Commonly included headers get parsed/compiled only once.</li>
<li>Less reinstantiation of the same templates: like <code class="language-plaintext highlighter-rouge">std::vector<int></code>.</li>
<li>Less work for the linker (for example not having to remove N-1 copies of the same weak symbol - an inline function defined in a header and included in N source files).</li>
<li>less compiler invocations.</li>
</ul>
<p>Note that we don’t have to include all sources in one unity file - as an example: 80 source files can be split in 8 unity files with 10 of the original sources included in each of them and then they can be built in parallel on 8 cores.</p>
<h2 id="why-redundant-header-parsingcompilation-is-slow">Why redundant header parsing/compilation is slow:</h2>
<p>Here is what happens after including a single header with 2 popular compilers and running only the preprocessor (in terms of file size and lines of code):</p>
<table>
<thead>
<tr>
<th>header</th>
<th>GCC 7 size</th>
<th>GCC 7 loc</th>
<th>MSVC 2017 size</th>
<th>MSVC 2017 loc</th>
</tr>
</thead>
<tbody>
<tr>
<td>cstdlib</td>
<td>43 kb</td>
<td>1k loc</td>
<td>158 kb</td>
<td>11k loc</td>
</tr>
<tr>
<td>cstdio</td>
<td>60 kb</td>
<td>1k loc</td>
<td>251 kb</td>
<td>12k loc</td>
</tr>
<tr>
<td>iosfwd</td>
<td>80 kb</td>
<td>1.7k loc</td>
<td>482 kb</td>
<td>23k loc</td>
</tr>
<tr>
<td>chrono</td>
<td>180 kb</td>
<td>6k loc</td>
<td>700 kb</td>
<td>31k loc</td>
</tr>
<tr>
<td>variant</td>
<td>282 kb</td>
<td>10k loc</td>
<td>1.1 mb</td>
<td>43k loc</td>
</tr>
<tr>
<td>vector</td>
<td>320 kb</td>
<td>13k loc</td>
<td>950 kb</td>
<td>45k loc</td>
</tr>
<tr>
<td>algorithm</td>
<td>446 kb</td>
<td>16k loc</td>
<td>880 kb</td>
<td>41k loc</td>
</tr>
<tr>
<td>string</td>
<td>500 kb</td>
<td>17k loc</td>
<td>1.1 mb</td>
<td>52k loc</td>
</tr>
<tr>
<td>optional</td>
<td>660 kb</td>
<td>22k loc</td>
<td>967 kb</td>
<td>37k loc</td>
</tr>
<tr>
<td>tuple</td>
<td>700 kb</td>
<td>23k loc</td>
<td>857 kb</td>
<td>33k loc</td>
</tr>
<tr>
<td>map</td>
<td>700 kb</td>
<td>24k loc</td>
<td>980 kb</td>
<td>46k loc</td>
</tr>
<tr>
<td>iostream</td>
<td>750 kb</td>
<td>26k loc</td>
<td>1.1 mb</td>
<td>52k loc</td>
</tr>
<tr>
<td>memory</td>
<td>760 kb</td>
<td>26k loc</td>
<td>857 kb</td>
<td>40k loc</td>
</tr>
<tr>
<td>random</td>
<td>1.1 mb</td>
<td>37k loc</td>
<td>1.4 mb</td>
<td>67k loc</td>
</tr>
<tr>
<td>functional</td>
<td>1.2 mb</td>
<td>42k loc</td>
<td>1.4 mb</td>
<td>58k loc</td>
</tr>
<tr>
<td><strong>all of them</strong></td>
<td><strong>2.2 mb</strong></td>
<td><strong>80k loc</strong></td>
<td><strong>2.1 mb</strong></td>
<td><strong>88k loc</strong></td>
</tr>
</tbody>
</table>
<p>And here are some (common) headers from Boost (version 1.66):</p>
<table>
<thead>
<tr>
<th>header</th>
<th>GCC 7 size</th>
<th>GCC 7 loc</th>
<th>MSVC 2017 size</th>
<th>MSVC 2017 loc</th>
</tr>
</thead>
<tbody>
<tr>
<td>hana</td>
<td>857 kb</td>
<td>24k loc</td>
<td>1.5 mb</td>
<td>69k loc</td>
</tr>
<tr>
<td>optional</td>
<td>1.6 mb</td>
<td>50k loc</td>
<td>2.2 mb</td>
<td>90k loc</td>
</tr>
<tr>
<td>variant</td>
<td>2 mb</td>
<td>65k loc</td>
<td>2.5 mb</td>
<td>124k loc</td>
</tr>
<tr>
<td>function</td>
<td>2 mb</td>
<td>68k loc</td>
<td>2.6 mb</td>
<td>118k loc</td>
</tr>
<tr>
<td>format</td>
<td>2.3 mb</td>
<td>75k loc</td>
<td>3.2 mb</td>
<td>158k loc</td>
</tr>
<tr>
<td>signals2</td>
<td>3.7 mb</td>
<td>120k loc</td>
<td>4.7 mb</td>
<td>250k loc</td>
</tr>
<tr>
<td>thread</td>
<td>5.8 mb</td>
<td>188k loc</td>
<td>4.8 mb</td>
<td>304k loc</td>
</tr>
<tr>
<td>asio</td>
<td>5.9 mb</td>
<td>194k loc</td>
<td>7.6 mb</td>
<td>513k loc</td>
</tr>
<tr>
<td>wave</td>
<td>6.5 mb</td>
<td>213k loc</td>
<td>6.7 mb</td>
<td>454k loc</td>
</tr>
<tr>
<td>spirit</td>
<td>6.6 mb</td>
<td>207k loc</td>
<td>7.8 mb</td>
<td>563k loc</td>
</tr>
<tr>
<td>geometry</td>
<td>9.6 mb</td>
<td>295k loc</td>
<td>9.8 mb</td>
<td>448k loc</td>
</tr>
<tr>
<td><strong>all of them</strong></td>
<td><strong>18 mb</strong></td>
<td><strong>560k loc</strong></td>
<td><strong>16 mb</strong></td>
<td><strong>975k loc</strong></td>
</tr>
</tbody>
</table>
<p>The point here is not to discredit Boost - this is an issue with the language itself when building zero-cost abstractions.</p>
<p>So if we have a few 5 kb source files with a 100 lines of code in each (because we write modular code) and we include some of these - we can easily get hundreds of thousands of lines of code (reaching megabyte sizes) for the compiler to go through for each source file of our tiny program. If some headers are commonly included in those source files then by employing the unity build technique we will compile the contents of each header just once - and this is where the biggest gains from unity builds come from.</p>
<p>A common misconception is that unity builds offer gains because of the reduced disk I/O - after the first time a header is read it is cached by the filesystem (they cache very aggressively since a cache miss is a huge hit).</p>
<h2 id="the-pros-of-unity-builds">The PROS of unity builds:</h2>
<ul>
<li>Up to 90+% faster (depends on modularity - stitching a few 10k loc files together wouldn’t be much beneficial) - the best gains are with short sources and lots of (heavy) includes.</li>
<li>Same as <a href="https://en.wikipedia.org/wiki/Interprocedural_optimization">LTO</a> (link-time optimizations - also LTCG) but even faster than normal full builds! Usually LTO builds take tremendously more time (but there are great improvements in that area such as clang’s <a href="https://clang.llvm.org/docs/ThinLTO.html">ThinLTO</a>).</li>
<li><a href="https://en.cppreference.com/w/cpp/language/definition#One_Definition_Rule">ODR</a> (One Definition Rule) violations get caught (see <a href="https://stackoverflow.com/questions/31722473/is-there-a-way-to-detect-inline-function-odr-violations">this</a>) - there are still no reliable tools for that. Example - the following code will result in a runtime bug since the linker will randomly remove one of the 2 methods and use the other one since they seem to be identical:
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// a.cpp</span>
<span class="k">struct</span> <span class="nc">Foo</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">method</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="mi">42</span><span class="p">;</span> <span class="p">}</span> <span class="c1">// implicitly inline</span>
<span class="p">};</span>
</code></pre></div> </div>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// b.cpp</span>
<span class="k">struct</span> <span class="nc">Foo</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">method</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="mi">666</span><span class="p">;</span> <span class="p">}</span> <span class="c1">// implicitly inline</span>
<span class="p">};</span>
</code></pre></div> </div>
</li>
<li>Enforces code hygiene such as include guards (or <code class="language-plaintext highlighter-rouge">#pragma once</code>) in headers</li>
</ul>
<h2 id="the-cons">The CONS:</h2>
<ul>
<li>Not all valid C++ continues to compile:
<ul>
<li>Clashes of symbols with identical names and internal linkage (in anonymous namespaces or static)
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// a.cpp</span>
<span class="k">namespace</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">local</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div> </div>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// b.cpp</span>
<span class="k">static</span> <span class="kt">int</span> <span class="n">local</span><span class="p">;</span>
</code></pre></div> </div>
</li>
<li>Overload ambiguities (also non-explicit 1 argument constructor…?)</li>
<li>Using namespaces in sources can be a problem</li>
<li>Leaked preprocessor identifiers after some source which defines them</li>
</ul>
</li>
<li>Might slow down some workflows:
<ul>
<li>Minimal rebuilds - but if a source file can be excluded from the unity ones for faster iteration it should all be OK</li>
<li>Might interfere with parallel compilation - but that can be tuned by better grouping of the sources to avoid “long poles” in compilation</li>
</ul>
</li>
<li>Might need a lot of RAM depending on how many sources you combine.</li>
<li>One scary caveat is a miscompilation - when the program compiles successfully but in a wrong way (perhaps a better matching overload got chosen somewhere - or something to do with the preprocessor). Example:
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// a.cpp</span>
<span class="k">struct</span> <span class="nc">MyStruct</span> <span class="p">{</span>
<span class="n">MyStruct</span><span class="p">(</span><span class="kt">int</span> <span class="n">arg</span><span class="p">)</span> <span class="o">:</span> <span class="n">data</span><span class="p">(</span><span class="n">arg</span><span class="p">)</span> <span class="p">{}</span>
<span class="kt">int</span> <span class="n">data</span><span class="p">;</span>
<span class="p">};</span>
<span class="kt">int</span> <span class="nf">func</span><span class="p">(</span><span class="n">MyStruct</span> <span class="n">arg</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="n">arg</span><span class="p">.</span><span class="n">data</span><span class="p">;</span> <span class="p">}</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="n">func</span><span class="p">(</span><span class="mi">42</span><span class="p">);</span> <span class="p">}</span>
</code></pre></div> </div>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// b.cpp</span>
<span class="kt">int</span> <span class="nf">func</span><span class="p">(</span><span class="kt">int</span> <span class="n">arg</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="n">arg</span> <span class="o">*</span> <span class="mi">2</span><span class="p">;</span> <span class="p">}</span>
</code></pre></div> </div>
<p>If <code class="language-plaintext highlighter-rouge">b.cpp</code> ends up before <code class="language-plaintext highlighter-rouge">a.cpp</code> then we would get 84 instead of 42.
However I haven’t seen this mentioned anywhere - people don’t run that much into it. Also <a href="https://github.com/onqtam/doctest">good tests</a> will definitely help.</p>
</li>
</ul>
<h2 id="how-to-maintain">How to maintain</h2>
<p>We can manually maintain a set of unity source files - or automate that:</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/CMake">CMake</a>: either <a href="https://github.com/sakra/cotire">cotire</a> or <a href="http://kecsapblog.blogspot.com/2016/03/unity-build-macro-for-cmake.html">this</a></li>
<li><a href="http://www.fastbuild.org/docs/functions/unity.html">FASTBuild</a></li>
<li><a href="http://mesonbuild.com/Unity-builds.html#unity-builds">Meson</a></li>
<li><a href="https://gitlab.com/ita1024/waf/blob/master/waflib/extras/unity.py">waf</a></li>
<li><a href="https://github.com/nmoreaud/qmake-unity">qmake</a></li>
<li><a href="https://blogs.msdn.microsoft.com/vcblog/2018/07/02/support-for-unity-jumbo-files-in-visual-studio-2017-15-8-experimental/">Visual Studio Native Support</a></li>
<li><a href="https://marketplace.visualstudio.com/items?itemName=Trass3r.RudeBuild">Visual Studio Plugin</a></li>
</ul>
<p>It is desirable to have control on:</p>
<ul>
<li>how many unity source files there are</li>
<li>the order of source files in the unity files</li>
<li>the ability to exclude certain files (if problematic or for iterating over them)</li>
</ul>
<h2 id="projects-using-this-technique">Projects using this technique</h2>
<p>Unity builds are used in Ubisoft for almost 14 years! Also <a href="https://blogs.gnome.org/mcatanzaro/2018/02/17/on-compiling-webkit-now-twice-as-fast/">WebKit</a>! And Unreal…</p>
<p>There are also efforts by people who build chrome often (and have gotten very good results - parts of it get built in 30% of the original time) to bring native support for unity builds into clang to minimize the code changes needed for the technique (gives unique names to objects with internal linkage (static or in anonymous namespaces), undefines macros between source files, etc.):</p>
<ul>
<li><a href="http://lists.llvm.org/pipermail/cfe-dev/2018-April/057579.html">http://lists.llvm.org/pipermail/cfe-dev/2018-April/057579.html</a></li>
<li><a href="http://lists.llvm.org/pipermail/cfe-dev/2018-April/057597.html">http://lists.llvm.org/pipermail/cfe-dev/2018-April/057597.html</a></li>
<li><a href="http://lists.llvm.org/pipermail/cfe-dev/2018-April/057604.html">http://lists.llvm.org/pipermail/cfe-dev/2018-April/057604.html</a></li>
</ul>Viktor Kirilovvik.kirilov@gmail.comThe what, why and how for this techniqueMy 3 week solo Thailand trip2018-04-26T11:01:34+00:002018-04-26T11:01:34+00:00https://onqtam.com/misc/thailand-solo-2018-february<p>This post has been written day by day during my 3 week solo trip to Thailand between February 15 and March 9 of 2018. I was a bit reluctant for more than 6 weeks about posting it because of some of the contents but here it is. And the image at the top has nothing in common with my experience :D - islands will have to be next time.</p>
<h2 id="background-story">Background story</h2>
<p>A very good friend of mine had gone to Thailand at the end of 2017 and hyped it up a lot for me. I decided that I hadn’t done anything interesting like this in a while and that it was the perfect time for it. In 2017 I had gone to Poland, America and China for my talk about <a href="https://www.youtube.com/watch?v=eH1CxEC29l8">doctest</a> but I consider those business trips as I didn’t have much time to explore. So my previous travel trip was in 2015 when I went to Morocco for a <a href="http://www.moroccoswimtrek.com/eng/index.php">4 day swimming trek</a> (25km total) - again alone (it was also my first trip ever). The plan was 2 weeks Chiang Mai and 1 week Bangkok - where I would meet some friends.</p>
<h2 id="plane-trip">Plane Trip</h2>
<p>On my first flight (to Doha, Qatar) I sat next to a bulgarian and we almost didn’t exchange a word - I was with my headphones listening to an audiobook about religion (by Sam Harris). Anyway - when we landed I made a remark that the trip was pleasant aaaand we struck a conversation! Until we parted ways for different gates we talked about profession, life and other interesting topics - and we both regreted not having talked to each other earlier.</p>
<p>Then at my gate I met a philippine girl on her way to Chiang Mai for an interview for a flight attendant at an airline company. She was also alone and looked gorgeous in her uniform. I was really nervous to talk to her but thought to myself that challenging myself was precisely the reason I was going to Thailand for 3 weeks alone. Unsurprisingly we had a nice conversation and exchanged contacts for later (I say unsurprisingly because thats how things actually work - you go for something and the chances of it happening are bigger than you think!).</p>
<p>Then I sat next to a 66 year old american and we talked a lot about travelling and life in general. The thing that stuck with me from that conversation was the growing (on my behalf) difference in our beliefs - I don’t think that the good life starts at 50 and that is when you should travel and experience life. He told me about how his wife had almost passed away a few times already and was currently unable to walk for the past couple of years - so he takes trips by himself. He attributed that to her not taking enough care of her body when she had to - which is most probably true. Shoot me dead if I ever end up in that situation - the choice of a partner in life is one of the most important ones a person has to make.</p>
<h2 id="day-1">Day 1</h2>
<p>The first thing I did when I landed was to get a sim card with 3G - for the price of 25 euro you get 15 GB of internet and 50 minutes of conversation time! After that I called an Uber and we had a little saga trying to locate my apartment - but that is unimportant.</p>
<p>After I got settled in my room the philippine girl contacted me and we agreed to meet in the afternoon after we rest from the long trip we both had (she had been waiting for 8 hours in Doha between her flights and the Doha-Chiang Mai flight was 6 hours).</p>
<p>Then I overslept with 2 hours and woke up at 17:00 and she was already waiting for me so I went out instantly and we met at a temple - she was inside and called me when she saw me - she didn’t get a sim card so we had no way of contacting each other unless when she had wifi somewhere. We had 2 amazing pizzas and walked for a few hours checking out temples. Turns out she looked exactly as a Thai and everybody was mistaking her for one - so each time she replied to someone in english they would get confused :D</p>
<p>We also went to a thai massage where we changed our clothes and laid on cussions on the floor with very dim lighting - not on some tables like human meat :D. The massage was very good and we laughed a lot (especially me since I’m very ticklish).</p>
<p>So all I can say about our experience is that we both like the movie <a href="https://www.imdb.com/title/tt0112471/">“Before Sunrise”</a> (where 2 single people meet on a train spontaneously and have a memorable experience together).</p>
<p>A few observations about Chiang Mai:</p>
<ul>
<li>holy shit there are so many temples!</li>
<li>crossing roads is hard - stepping on a crosswalk doesn’t mean shit to the drivers - so pay attention (turns out this is true for most of Asia)</li>
<li>with all the trash from stores and venues being put in bags at night on the road and collected later - the streets were smelly because when nasty stuff spills out of the trash and soaks into the pavement and asphalt - it cannot be cleaned easily</li>
</ul>
<h2 id="day-2">Day 2</h2>
<p>The next morning I went to my place and a few hours later we met again for day 2 - we called a driver through Grab and he took us out of town - to the “Long Neck Karen” village some 20 km away from the city. This was something she really wanted to check out - I had read some bad reviews though (how it feels like a human zoo and isn’t something people should be sponsoring) - and I gotta say I completely agree with the reviews. I felt so awkward there - and I was sure the people weren’t enjoying having pictures taken of them and having to smile and pose all the time. But she was having a great time so I played along. I refused to have a photo taken with one of the women with long necks… but I got a few hard cigars for my friends at home :D</p>
<p>The driver was waiting for us for around 4 euros per hour (…………) so we decided to check the elephant place nearby. We weren’t able to book a ride but we could feed them and take photos with them. I got a bunch of bananas and 4 really hard sticks (maybe bamboo - not sure) - and while I was giving a few bananas to one of the elephants another snatched the entire banana stack out of my other hand :D and was stuffing himself - he even continued taking more food from others while he was still chewing on the ones from me. So the food we had left for them were the 4 sticks - and the girl fed 3 of them and then we took photos with the elephants. The guards arranged for us to get kissed by the elephants - imagine a huge slimey trunk grabbing your face like a vacuum cleaner and sucking your face after disassembling kilograms of bananas! We also got hugged by their trunks ;d</p>
<p>Then we proceeded to the baby elephants and on our way there there was an old elephant with some guards next to it. When they saw that we were looking at it they instantly made the elephant get up and come to us. We sort-of felt that the elephant didn’t feel like it but the guard insisted that we take photos with it, so the girl sat on the elephant - but she was starting to feel very weird about it too - the guard was very nasty to the animal. I had read before about the treatment of elephants in such high-volume tourist attractions and I knew exactly what was going on behind the scenes… But again I played along and tried to enjoy my trip - after all I got to touch and be kissed by some of these magestic animals!</p>
<p>Then on our way back the driver suggested we visit the tiger kingdon nearby - again I had read about that as well - drugged tigers that just moan and move a bit - nothing like the wild beasts they truly are. But… we had already sponsored 2 things that I would advise against so why not make it 3 :D … So it was quite surreal to be in a cage with such animals - I was told to pet and even spank them! But they were drugged AF… and they are like that constantly throughout the day - and perhaps even their entire lives - there were also small cubs available…</p>
<p>After the obligatory mainstream tourist attractions we headed back and we got off the car when we saw some market - turns out it was the last day of the Chinese New Year! We strolled through the streets and ate a whole bunch of different foods. Sadly we finished with fried bugs and worms which had a horrible aftertaste - wouldn’t recommend… And we finished the day at some bar with live music.</p>
<h2 id="day-3">Day 3</h2>
<p>I slept for 15 hours straight because of the exhaustion from the last couple of days. Since this was her day at the interview I stayed at my place and did some laundry (wow… so exciting!) and internet stuffs ^_^</p>
<p>Then I went out to explore and get some food. At some point I arrived at the center of the old town aaand I went into a huge night market on the streets - it took me more than 1 hour to leave it - it was literally covering more than a few square kilometers! Lots of merchandise and street food - which I got to try a lot. So far I’ve been eating only once per day - but pig mode is fully on each time :D</p>
<p>Then I went past a massage shop where me and the main massage girl locked eyes and she instantly called me in - but I passed them… Exactly 3 minutes later I was back at their place setting up a massage :D
I had to wait for half an hour before it was my turn - they gave me some wine to drink and we talked a bit - but I had a hard time understanding what they were saying. Anyway during the massage the main massage lady (which I figured was more than 40 years old but was veeery preserved) came as well and started massaging my head and shoulders while the other was on my feet. Then a third girl came (their youngest masseuse) aaand started working on my thighs! I started laughing and then they started too - and the main one whispered to my ear “you are a king!” :D felt amazing for 1 hour. And all that cost me 5 euros - so I tipped them :D</p>
<p>Then I went to a bar at 23:56 aaand 3 minutes later it closed (as all bars in chiang mai are supposed to) - so most people headed to the last place still working (Spicy) - so I went along. So it wasn’t anything special - Pop and EDM. People were getting drunk left and right. After the boyfriend of one girl got annoyed with me I went out and decided to call it a night. I was walking back… When you have a bunch of ladyboys on scooters all moaning and making sexy sounds just to lure you with them - you know you are somewhere special :D</p>
<p>Then I sat down and decided to start writing this.</p>
<h2 id="day-4">Day 4</h2>
<p>Some work, unexpected massage. Found the best place for food - for 5 euros you cook your own food and eat as much as you can! Refilled my dish 5 times.</p>
<h2 id="day-5">Day 5</h2>
<p>Some work, workout, got scammed to pay for stuff for some girl. Ate at the same place. And then some drama.</p>
<h2 id="day-6">Day 6</h2>
<p>some work… Ate at the same place - did I mention that so far I’ve been eating only once a day ever since I came here? :D</p>
<h2 id="day-7">Day 7</h2>
<p>Work, workout, found even cheaper sushi - no longer eating once a day… :D And went to Warmup Cafe. Saw a bunch of tables with people and left. Turns out there is a separate space resembling a club/bar…</p>
<h2 id="day-8">Day 8</h2>
<p>Overslept again - and didn’t go to Pai again.. hopefully tomorrow! Work now…</p>
<h2 id="day-9">Day 9</h2>
<p>How do I start… The road to Pai was 3 and a half hours with a nice scenery and … 762 grueling mountainside curves - motion sickness: ON!</p>
<p>When I got there I instantly felt a much different vibe. It was so peaceful and beautiful - full of young people getting on a scooter for the first time in their lives (and many of them crashing after drinking..). After I settled I immediately went to The Sunset Bar which is apparently THE place to go for a mushroom trip. I simply said “Mushrooms” with the biggest smile in the world on my face but the bartender said there weren’t any left since its the dry season and there hasn’t been enough rain recently. As we will see later that wasn’t the case. In the span of a few minutes after I had asked for the special thing came 2 more groups of people asking for the same and getting the same response. Anyway - we found some weed and rolled with it. Talked with a bunch of stoners who had experimented with all kinds of psychedelics and were currently training muai thai in Thailand. Then I heard about a music festival the same night a few kilometers out of Pai into the jungle aaand instantly decided I wanted to go. So after having some street food I went to some place where cabs were leaving to the festival - it was 21:00. There I met 2 cool dudes and we chatted a lot for the first 30-40 mins. Turns out we were the first people there and it all looked pretty cool. There were also fires and food stands around. Sooo I got pretty wasted that night - stayed until 4-5 and met a bunch of people, most of which I didn’t remember the next day when we bumped throughout the city :D</p>
<h2 id="day-10">Day 10</h2>
<p>So the guys I met the day before had told me they would go “tubing” (floating down the river on big inflated tires) and that I should come along. I had their phones but for some reason just went to lunch and after that strolled around looking at stuff and Bam - I met them again just when they were about to leave and I instantly joined them. I also saw something on the tongue of one of them but ignored it because “how can it be acid and why would it be so visible”… All drugs are very illegal in Thailand.. But this is Pai (it has amazing history in that regard…). And a minute later the other one asked me if I wanted some. So after 0 hesitation I was along for the ride :D. We were the 3 of us and a girl. The tour guides dropped us off 2 hours upstream the river and left us - we were going to be completely alone until some bridge we were supposed to stop at. We had a few beers and a joint or 2 and then it started. We put our phones, documents and clothes in 2 water proof bags and the ride started. It was 15:00 under very strong sun but we didn’t mind it at all.</p>
<p>I gotta say during the entire trip I felt awesome - I thought to myself that was probably the best day of my life - thats how it felt! Complete relaxation… there were moments when we weren’t speaking at all - complete silence while we are going down the river with less than 2-3 km/h with amazing scenery… The river was pretty shallow and there was also some drama - sometimes we got stuck and had to wiggle to get ourselves moving :D
We also had 1 spare tube in case one gets popped… and 1 did get popped. There was also a rope which we used to hang on to and stick together for some of the time. We passed many villagers and animals. They were all minding their business and we greeted each other. At one point we heard this racket downstream and for the next 3-4 minutes we were completely silent until we got to it - a pump getting water out of the river for irrigation. We felt like in a war in the jungle - completely stealthy and about to see what was making the noise - and we gave each other these stares of excitement :D seriously… what an amazing time.
I don’t think I had felt so close to nature and so relaxed in the past couple of years - I had completely forgotten this feeling of endless bliss…
So there was also some drama - at some point we found out that one of the bags was full of water… the waterproof ones.. turns out it was the one with the clothes. I was holding the important one with the documents and phones and it felt unusually heavy as well.. PANIC! But we decided not to open it because if it was full of water it would probably be too late to save anything anyway. I worried that I had fucked shit up but later it turned out we had 2 full bottles with 3 liters of water inside so that was why it was heavy…… damn :D
And suddenly we had arrived - after 2 hours of drifting down the stream. We went out and stayed under the sun for 30 more minutes to get dry and we ended up receiving a song from these 2 musicians who were laying next to the river. It felt so good… They let loose and the 4 of us were just smiling and enjoying the heck out of that moment.</p>
<p>We left the tubes and went our separate ways to wash ourselves. I had dinner and was still tripping a bit. I spent like 20-30 minutes next to some of the most gentle sounds I’ve ever heard live <insert video="" here="">. And I was also full of thoughts... a few hours later I went along the main road of the city (village - its probably less than 10 thousand people with all the tourists...) and I hit the bars. And OMG - there were so many different and cool places with lots of cool and different music - and I found my calling... a techno/psytrance bar with a dance floor... The fact that I had destroyed myself last night in the jungle didn't stop me and instantly went in there - and what do you know - people seemed to remember me from last night but not vice versa :D. I chatted with a tour guide who said he had been doing it in Thailand for the past couple of years but that this was actually his last group and that he really got tired of it all - constantly having to drink and put up with the people (and he didn't get to choose his groups) and get up early the next day for the next paid adventure. We quickly went on to talk about music and psychedelics and he said that if anyone of his group ever tried one of those things they wouldn't know how to react or what to think about all of it. They were also pretty loud and dancing on the music but the style just didn't fit - you could see all they wanted was to get drunk. What stuck with me is the difference in mentality and how me/him and they carried ourselves. Words like "spiritual" and "consciousnes" weren't uncommon for our conversation as well.
I then talked to more people and we discussed how peaceful everything was in Pai with one of them in particular - my thesis was that Pai harbored just cool people and that it was impossible to get into a fight because people there aren't like that... and the moment I said it he started with a story about 2 nights ago where he almost was in a fight - and while he was saying that we witnessed a fight on the street between drunk tourists... So perhaps Pai isn't what it used to be and I underestimated the power of alcohol. Did I mention I met A BUNCH of people who drove a scooter for the first time in their lives in Pai and were all in bruises from drunk driving?
I went on to talk with perhaps the most open minded person ever who had been living there for a few months and had been going to exactly the kinds of festivals I'm into and I was blown away by the lack of ego in that person. We went on to talk to a 52 year old biker dude who turned out to be a hardcore fan of the music we love and told us about his past and how he transitioned to a quiet and chill life where none of his current constituents knew the level of partying and self destruction he had gone through 10-20-30 years ago on a regular basis in the UK - and he was also so open about it and seemed perfectly sane and healthy - I'm not sure why I'm mentioning this :D
The day ended at 5 A.M. for me.</insert></p>
<h2 id="day-11">Day 11</h2>
<p>After breakfast again by total chance while walking around Pai I met with the same guys from the past 2 days - 1-2 hours before they were driving off to Chiang Mai - I kept bumping up with them and this time it was on a bridge by the river - the scenery was amazing…… more dots needed………… So we chatted about our experiences the last day and we said goodbye to each other.
I had decided this would be my last full day at Pai so I had to try those mushrooms. After some fooling around it was already 6 P.M. and I went to the famous bar and got a shake - turns out 80% of the people there had come solely for that purpose - and everyone was laying on cushions and hammacks while the sun was setting down (hence the name of the bar - “Sunset”) while they were coming up. I then went on with a bunch of ppl from the UK and I continued to feel the difference in mentality of people in Bulgaria and those outside. We went around and damn - there are so many cool and chill places there… After 2 hours in I still wasn’t feeling much of anything and was pretty disappointed and went back to town to hang out with some of the people from the last day at my favourite bar. And then I got a bit anti social - I closed up quite a bit - I felt the effects of the trip. I ended up going to bed pretty late again. Was a bit disappointed of myself for doing that and not spending my last night with the amazing person from the night before.</p>
<h2 id="day-12">Day 12</h2>
<p>Trip back to Chiang Mai from Pai - I regretted going back so early - should have stayed a few more days - seriously Pai seems to me like paradise - thats either true or I need to explore more.</p>
<p>Oh - how could I forget - when I arrived at Chiang Mai some lady from the bus asked me if I smoked grass (weed). Well… yea? So she gave me some because she was going to the airport and she said a few times that “it was very special”…… Silly me - I thought she meant it was some gift to her and that it had sentimental value… Well I had never smoked anything like that before! So trippy - like it wasn’t just weed! Those thai people… :D</p>
<h2 id="day-13">Day 13</h2>
<p>alone in chiang mai - learning about bitcoin, solving puzzlez, checked out Warmup cafe - pretty cool EDM place but damn Thai people dont dance! They just stand still and drink their stuff - lame…</p>
<h2 id="day-14">Day 14</h2>
<p>Still on Bitcoin. Also just learned that I got accepted to speak about <a href="https://github.com/onqtam/rcrl">RCRL</a> at http://cppconf.ru/ after 2 months!</p>
<h2 id="day-15">Day 15</h2>
<p>Bitcoin. Also leaving for Bangkok.</p>
<p>At the airport before boarding I met this native guy who was entirely covered with tattoos. He was sitting in front of me and a few times he turned back and looked at me - weird… Then he started a video call with his wife and kid through his phone and I was somehow in the picture and since something was moving I looked that way - and he instantly turned again to me with the biggest smile and gave me his hand - so we shook hands :D
Later he looked at me again for a bit. After that again… And at one point I decided to try to talk to him - he didn’t know any english… not ANY… but really wanted to communicate - so he got his phone out and started showing me pictures of him, his wife, his kid, his brother, his motorbike, his hobbies… We also went through his backpack to see what interesting things he had. After that I decided it was my turn so I whipped my phone out as well and started showing him where I’ve been. There were a few hand gestures that we both knew (or atleast figured them out) - relating to human connections and what is important. Those were the main focus points of our interaction… I had other experiences with natives who knew 0 english but its all about the vibe and the desire to communicate - at first he looked like the biggest thug but turns out he had a tattoo artist friend who painted his entire body for free in the span of a couple of years (not sure how he managed to explain that :D) - but at the end of the day he was a normal friendly dude - so don’t be quick to judge… It was weird and memorable.</p>
<p>Arrived at Bangkok and instantly hated the place - let me elaborate:</p>
<p>BANGKOK IN A NUTSHELL</p>
<ul>
<li>10-12 degrees warmer than Chiang Mai - it was 35 at night - and the high humidity didn’t help</li>
<li>SUPER dirty streets</li>
<li>huge city - 10-15 million people</li>
<li>horrible traffic</li>
<li>lots of tourists - and tons of prostitution and ping pong shows… monkeys and pigeons crawling inside… I dont wanna see that done to a pussy</li>
<li>nasty polluted air</li>
<li>walking around makes you hate yourself - showers and air conditioning were the only thing I kept thinking about during the entire stay of 6-7 days</li>
<li>local taxi drivers are really annoying - they kept trying to get me into their cabs/tuk-tuks and were very persistent - they engage you in a way so you make eye contact and they come to you - you gotta ignore them totally - saying “NO NO NO” doesn’t help - they also have a bunch of scams where they will pressure you into taking a very cheap ride on a specific route where you will end up buying things and they get a cut of the profits - a friend of mine came back from Thailand with a suit (atleast its great ;d)</li>
</ul>
<p>So the night I got there me and a bunch of friends who had come from Singapore for 3 nights got all drunk at a tattoo shop and we were catching until 7 AM.</p>
<h2 id="day-16">Day 16</h2>
<p>Festival day….. OMG. So a friend of mine had tipped me that there would be this Dropzone festival in Bangkok when I’m there and I was able to catch the second of the 2 days. I missed Pan-Pot but whatever - they are coming to Bulgaria in 1-2 months anyway.
The entrance was through this cool green-lit tunnel - so far so good ;d
So the festival was supposed to start at 14:00 and I was a bit suspicious as to how people would feel under the sun but anyway - I got there at 15:30 and… there were only staff people… only one of the 3-4 stages was playing music and there were literally 2 fans dancing :D - I bet this was such a disappointment to most DJs who were supposed to be in the first 4-5 hours…
But everything looked promising - huge cool stages with lots of space, lots of lights, activities (VR, laser shooting, guitar hero, consoles, artsy stuff, etc…), lots of food stands.</p>
<p>I was already sweating like a pig and regretting I had come so early when someone tipped me to “cool off” in an air-conditioned dome… What followed was 2 hours of mind-blowing entertainment! So there is this “full dome labs” company and the festival organizers had set up one of their domes - it is a half sphere which you enter and lay on the ground on pillows and cussions and stare at the ceiling - which is entirely lit by a few synchronized projectors with special content for projection onto the inside of a sphere - and it was some of the most trippy shit I’ve ever seen! Not only that but there was cool weird creepy ambient and loud music inside - it felt like you were transported somewhere else. And the air conditioning… oh my gawd… Images below!</p>
<p>[INSERT IMAGES HERE]</p>
<p>I imagine that most of the images I saw wouldn’t be considered anything close to special if they were static - but when everything is moving and morphing and the camera is also changing its position within the scene - and all that spans your entire view when looking upwards - simply amazing! I would pay much more money for this than to view a standard movie in a cinema…</p>
<p>So back to reality (heat): I got out of the dome when I saw all the available 2 hour content (would have stayed more :D) and it was already 18:00 - the sun was setting down and people were starting to come.
There were 3 main stages and 1-2 smaller ones. The music ranged between EDM, (Psychedelic) Trance, Techno, Hardstyle, Hardcore and Trap (and probably a few others) - so there was something for everybody. I decided I wouldn’t drink this time and got a red bull and joined one of the stages - choice of music: trance. What followed were 6 hours of non-stop dancing - I’ve never danced so much in my life. I went from stage to stage and was loving every second of it! People were a lot looser here compared to local clubs - they were actually dancing as well.</p>
<p>If you’ve heard the term “runners high” - the good euphoric feeling people get after running - well I got that from dancing and it was non-stop. I was pretty close to overheating a few times and each time I just went to the side to gulp down a bottle of water and get right back in it. And up until now I didn’t know I like hardstyle as well :D - and the shows were pretty cool - with fire and what not being hurdled from the stages.</p>
<p>At 22:00+ the festival was packed with people - it was amazing… literally the whole place was full… and there were also atleast 50-100 military personnel. I think there were very little drugs in this festival which was cool - people were enjoying the music for what it is - fuck our pill and white powder electronic music culture where parties start at 01:00 AM or later…</p>
<p>I think it was supposed to close at 1 or 2 AM but the police shut it down at 00:00 for some reason… (not paid enough to the right people - duuh ;d)</p>
<p>I’m now officially hooked on such festivals and 2018 will be interesting : )))</p>
<h2 id="day-17">Day 17</h2>
<p>Chilling (sweating) around - food, talk, sweating, massages, boat ride in the river (canal).</p>
<p>We also went to Route 66 - a club a bit far from the touristy center. Place was pretty cool and we danced a lot. Locals liked us because we were the most energetic and moving people around - so yea… Thai ppl don’t dance in clubs - they just drink their alcohol and instagram… I’m getting pretty sick actually from women constantly taking selfies - and discarding most without even posting them… constantly on the hunt for the perfect selfie showing what a cool lifestyle they have and collecting likes….. fuck that. What made us leave was terrible DJs - at some point came the most horrible ones - I don’t consider myself an expert in music but perhaps I’m developing taste - what they were doing was absolute and utter shit. I rarely feel so strongly against something - they were literally changing tracks every 5-10 seconds. Literally. Not kidding. Not 15 seconds. Tracks that have nothing in common. And always to the best part of the tracks - so there is no build-up - its just drop after drop. And each 5 seconds they were shouting on the mic trying to get people to dance or .. “explode” as I would say it in bulgarian :D… So after that bombardment with TOTAL CHEAP CRAP onto our ears for 10-20 minutes we decided to leave… You gotta have standards yo!</p>
<h2 id="day-18">Day 18</h2>
<p>Friends left. Fooled around.</p>
<h2 id="day-19">Day 19</h2>
<p>Went to a modern art museum with a friend of mine who had come to bkk for a teambuilding of his company. We both had a nice experience of the traffic in Bangkok. Did I mention how much it sucks in this town? I don’t care that there are some cool places to check out here and there because:</p>
<ul>
<li>you waste so much time of your life in traffic</li>
<li>the air is so polluted</li>
<li>walking around is not viable (either because of distances, lack of infrastructure (sidewalks, crosswalks, etc) or gruelling heat)</li>
<li>there is very little vegetation and almost everything is made of concrete</li>
</ul>
<p>that city is shit and not worth living in!!!</p>
<h2 id="day-20">Day 20</h2>
<p>Was at the same table with the founder of the Status cryptocurrency in a 5 star hotel.. and yes - I wasn’t invited :D
Status is currently the #33 cryptocurrency as far as market capitalisation goes (almost 600 million US dollars).
Man that startup is full of talented and smart people… I chatted with a few of them and really felt out of their league - what a positive, exciting and an eye-opening experience! I’m really thankful to my friend who smuggled me in there.</p>
<h2 id="day-21">Day 21</h2>
<p>I experienced the worse flight scheduling in my life (entirely my fault) - I spent a total of 20 hours waiting at airports between 3 flights… (so with the flights it was a trip of 32+ hours) - the same as my saga in 2015 when I went to Morocco… by now Doha kinda feels like a second home to me ;d</p>
<h2 id="conclusions">Conclusions</h2>
<ul>
<li>My world was small. it still is. but a bit less…</li>
<li>Don’t be ultra cheap with places you are staying at…</li>
<li>I shouldn’t have come with a laptop…</li>
<li>Festivals are epic.</li>
<li>I should have spent WAY more time outside of cities and into the nature.</li>
<li>My regime completely fell apart and I stopped exercising… Couldn’t work properly either… and I gained 3-4 kg of weight :D food is awesome…</li>
<li>Don’t come to Bangkok for more than 2-3 days….. I personally don’t want to go there ever again.</li>
<li>I wasted quite a bit of time because I was moody and hesitant to dive into new adventures non-stop but overall I had plenty of time to think about stuff in my life. I’d say I’m ready and eager to get back to the grind… and then come back here again :D</li>
<li>Solo travel definitely is a thing but maybe I’m a bit more on the introverted side - many times I had to be pushed by something/someone just to take action - next time I’m going with friends - definitely.</li>
<li>I dont think I’ll go somewhere for an extended period of time like this alone any time soon - my conclusion is the same as that from the movie <a href="http://www.imdb.com/title/tt0758758/">“Into the Wild (2007)”</a> - “HAPPINESS ONLY REAL WHEN SHARED” (even though my experiences and those of the main character in the movie cannot be compared…)</li>
<li>Thailand is great - especially when it’s cold in your country :D</li>
<li>Don’t support cheap touristy attractions (as in “bad taste” and not very moral/cool) like drugged tigers, villages with women with long necks, ping pong shows…</li>
</ul>Viktor Kirilovvik.kirilov@gmail.comSimply put: an experience.Read-Compile-Run-Loop - a tiny REPL for C++2018-02-12T16:01:34+00:002018-02-12T16:01:34+00:00https://onqtam.com/programming/read-compile-run-loop-a-tiny-repl-for-cpp<p>Ever wanted to modify some value or execute some statement while your C++ program is running just to test it out? Something that can’t be done through the debugger or isn’t trivial? Scripting languages have a REPL (<a href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">read-eval-print-loop</a>) - frontend web developers use the “javascript console” of the browser to test things which is available after pressing F12.</p>
<p><a href="https://github.com/root-project/cling">cling</a> is an interactive C++ interpreter (developed at CERN) but since it is built on top of LLVM it <a href="https://github.com/inspector-repl/inspector">isn’t easy at all</a> to integrate in your application in an elegant way so everything is callable and works on any platform and toolchain.</p>
<p><a href="https://github.com/onqtam/rcrl">RCRL</a> is an interactive C++ compiler in a demo application with GUI which demonstrates the technique. Without further ado here is a showcasing video:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/HscxAzFc2QY?rel=0" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen=""></iframe>
<p>In the video some basic usage is shown and the scene is interacted with. The build system used is <code class="language-plaintext highlighter-rouge">Ninja</code> for the fastest possible build times. Also the compilation happens in a background process - submitting code is not blocking - our program continues to run and loads the result when ready.</p>
<h2 id="how-to-use-it">How to use it</h2>
<p>There are 3 types of sections of code that the user may submit for compilation:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">global</code> - code that should be compiled in global scope goes there (like class/function definitions, includes, etc.)</li>
<li><code class="language-plaintext highlighter-rouge">once</code> - executable statements go in there - this is compiled in function scope and executed only once</li>
<li><code class="language-plaintext highlighter-rouge">vars</code> - definitions of variables that will be used later in other sections should go there</li>
</ul>
<p>Sections are changed with single-line comments containing one of the 3 words <code class="language-plaintext highlighter-rouge">global</code>/<code class="language-plaintext highlighter-rouge">once</code>/<code class="language-plaintext highlighter-rouge">vars</code>. For example:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// global</span>
<span class="kt">int</span> <span class="nf">foo</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="mi">42</span><span class="p">;</span> <span class="p">}</span>
<span class="c1">// vars</span>
<span class="kt">int</span> <span class="n">a</span> <span class="o">=</span> <span class="n">foo</span><span class="p">();</span>
<span class="kt">int</span><span class="o">&</span> <span class="n">b</span> <span class="o">=</span> <span class="n">a</span><span class="p">;</span>
<span class="c1">// once</span>
<span class="n">a</span><span class="o">++</span><span class="p">;</span>
<span class="c1">// global</span>
<span class="cp">#include <iostream>
</span><span class="kt">void</span> <span class="nf">print</span><span class="p">()</span> <span class="p">{</span> <span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="n">a</span> <span class="o"><<</span> <span class="n">b</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span> <span class="p">}</span>
<span class="c1">// once</span>
<span class="n">print</span><span class="p">();</span> <span class="c1">// ======> will result in "4343" being printed</span>
</code></pre></div></div>
<p>In the scripting language world REPLs automatically try to print the “value” of the last statement. In <a href="https://github.com/onqtam/rcrl">RCRL</a> we need to do the printing ourselves.</p>
<p>It should be noted that in the above example the variables may have been written also in <code class="language-plaintext highlighter-rouge">global</code> sections, but then if we submit more code for compilation as a separate step, those globals would be initialized again and there would be multiple instances of them - and that is what <code class="language-plaintext highlighter-rouge">vars</code> sections are for - persistent globals which are initialized only once. Also variable definitions in <code class="language-plaintext highlighter-rouge">once</code> sections aren’t visible outside of them since they are local variables in function scope. That will make more sense after the following section.</p>
<p>When we are done we can press the <code class="language-plaintext highlighter-rouge">Cleanup</code> button which will:</p>
<ul>
<li>call the destructors of all globals defined in <code class="language-plaintext highlighter-rouge">vars</code> sections in reverse order</li>
<li>unload all the plugins (as explained later) in reverse order</li>
<li>delete all the plugins from the filesystem</li>
</ul>
<h2 id="how-it-works">How it works</h2>
<p>Here is what happens each time you submit code:</p>
<ol>
<li>A <code class="language-plaintext highlighter-rouge">.cpp</code> file is reconstructed with all previous <code class="language-plaintext highlighter-rouge">global</code> and <code class="language-plaintext highlighter-rouge">vars</code> sections (in the appropriate order) and then the newly submitted sections are also appended (including <code class="language-plaintext highlighter-rouge">once</code> sections) in their submission order.</li>
<li>The <code class="language-plaintext highlighter-rouge">.cpp</code> file is compiled as a shared object (<code class="language-plaintext highlighter-rouge">.dll</code>) and links against the executable (more on that later)</li>
<li>The plugin is then copied with a different name depending on which compilation this is (so we would end up with <code class="language-plaintext highlighter-rouge">plugin_5.dll</code> if we have previously submitted code for compilation 4 times)</li>
<li>The plugin is loaded by the host application and all globals defined in the <code class="language-plaintext highlighter-rouge">.cpp</code> file are initialized from top to bottom</li>
</ol>
<p>The <code class="language-plaintext highlighter-rouge">.cpp</code> file always includes a header called <a href="https://github.com/onqtam/rcrl/blob/master/src/rcrl/rcrl_for_plugin.h"><code class="language-plaintext highlighter-rouge">rcrl_for_plugin.h</code></a> located in <a href="https://github.com/onqtam/rcrl/tree/master/src/rcrl"><code class="language-plaintext highlighter-rouge"><repo>/src/rcrl/</code></a> which has a few macros and forward declarations to make everything work:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">RCRL_SYMBOL_IMPORT</span> <span class="kt">void</span><span class="o">*&</span> <span class="n">rcrl_get_persistence</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">var_name</span><span class="p">);</span>
<span class="n">RCRL_SYMBOL_IMPORT</span> <span class="kt">void</span> <span class="nf">rcrl_add_deleter</span><span class="p">(</span><span class="kt">void</span><span class="o">*</span> <span class="n">address</span><span class="p">,</span> <span class="kt">void</span> <span class="p">(</span><span class="o">*</span><span class="n">deleter</span><span class="p">)(</span><span class="kt">void</span><span class="o">*</span><span class="p">));</span>
<span class="c1">// macros...</span>
</code></pre></div></div>
<p>Here is how code from the different sections ends up in the source file:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">global</code> sections just go straight to the <code class="language-plaintext highlighter-rouge">.cpp</code> file being compiled</li>
<li>
<p><code class="language-plaintext highlighter-rouge">once</code> sections go in a lambda that is called while globals are initialized:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">RCRL_ONCE_BEGIN</span>
<span class="n">a</span><span class="o">++</span><span class="p">;</span>
<span class="n">RCRL_ONCE_END</span>
</code></pre></div> </div>
<p>And after the preprocessor we get something like this:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">int</span> <span class="n">rcrl_anon_12</span> <span class="o">=</span> <span class="p">[]()</span> <span class="p">{</span>
<span class="n">a</span><span class="o">++</span><span class="p">;</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span> <span class="p">}();</span>
</code></pre></div> </div>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">vars</code> sections - they are parsed so that for each variable definition the type/name/initializer are extracted. The source code for <code class="language-plaintext highlighter-rouge">int a = 5;</code> is this:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">RCRL_VAR</span><span class="p">((</span><span class="kt">int</span><span class="p">),</span> <span class="p">(</span><span class="kt">int</span><span class="p">),</span> <span class="n">RCRL_EMPTY</span><span class="p">(),</span> <span class="n">a</span><span class="p">,</span> <span class="p">(</span><span class="mi">5</span><span class="p">));</span>
</code></pre></div> </div>
<p>which expands to the following after the preprocessor:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">int</span><span class="o">&</span> <span class="n">a</span> <span class="o">=</span> <span class="o">*</span><span class="p">[]()</span> <span class="p">{</span>
<span class="k">auto</span><span class="o">&</span> <span class="n">address</span> <span class="o">=</span> <span class="n">rcrl_get_persistence</span><span class="p">(</span><span class="s">"a"</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">address</span> <span class="o">==</span> <span class="nb">nullptr</span><span class="p">)</span> <span class="p">{</span>
<span class="n">address</span> <span class="o">=</span> <span class="p">(</span><span class="kt">void</span><span class="o">*</span><span class="p">)</span><span class="k">new</span> <span class="kt">int</span><span class="p">(</span><span class="mi">5</span><span class="p">);</span>
<span class="n">rcrl_add_deleter</span><span class="p">(</span><span class="n">address</span><span class="p">,</span> <span class="p">[](</span><span class="kt">void</span><span class="o">*</span> <span class="n">ptr</span><span class="p">)</span> <span class="p">{</span> <span class="k">delete</span> <span class="k">static_cast</span><span class="o"><</span><span class="kt">int</span><span class="o">*></span><span class="p">(</span><span class="n">ptr</span><span class="p">);</span> <span class="p">});</span>
<span class="p">}</span>
<span class="k">return</span> <span class="k">static_cast</span><span class="o"><</span><span class="kt">int</span><span class="o">*></span><span class="p">(</span><span class="n">address</span><span class="p">);</span>
<span class="p">}();</span>
</code></pre></div> </div>
<ol>
<li>First we get a pointer (by ref) to the persistent variable with name <code class="language-plaintext highlighter-rouge">a</code>.</li>
<li>If that pointer is null (<code class="language-plaintext highlighter-rouge">a</code> is defined in a newly submitted <code class="language-plaintext highlighter-rouge">vars</code> section) we allocate a new integer using the appropriate initializer and then add a deleter for <code class="language-plaintext highlighter-rouge">a</code> by passing a lambda as a function pointer.</li>
<li>Then we return the pointer from the lambda and immediately dereference it to initialize a global reference with the name <code class="language-plaintext highlighter-rouge">a</code>.</li>
</ol>
<p>That way we ensure that the global persistent variable will be initialized only once and its state will be preserved through the following compilations. In code below the <code class="language-plaintext highlighter-rouge">vars</code> section we can continue using the reference <code class="language-plaintext highlighter-rouge">a</code> as if it is a global variable <code class="language-plaintext highlighter-rouge">a</code>.</p>
</li>
</ul>
<p>The parser for <code class="language-plaintext highlighter-rouge">vars</code> sections is nothing special (it’s not a <a href="https://en.wikipedia.org/wiki/Recursive_descent_parser">recursive descent parser</a> and isn’t written like usual parsers) but is just a few hundred lines of code and is surprisingly adequate - works with very complex types with lots of templates, <code class="language-plaintext highlighter-rouge">decltype()</code>, <code class="language-plaintext highlighter-rouge">auto</code>, references and complex initializers. I could have used something proper like <a href="https://clang.llvm.org/docs/Tooling.html#libclang">LibClang</a> but decided to go with something custom and tiny instead.</p>
<h2 id="restrictions">Restrictions</h2>
<p>There are 2 sources of limitations for <a href="https://github.com/onqtam/rcrl">RCRL</a>:</p>
<ul>
<li>the parser for the <code class="language-plaintext highlighter-rouge">vars</code> sections is imperfect</li>
<li>the method itself (shared objects, dynamic allocation of variables, etc.)</li>
</ul>
<p>Let’s see what C++ constructs/usages are unsupported. In <code class="language-plaintext highlighter-rouge">vars</code> sections:</p>
<ul>
<li>nothing else should go here except for variable definitions</li>
<li>[*] <code class="language-plaintext highlighter-rouge">alignas()</code> cannot be used</li>
<li>[*] C arrays can’t be used - use <code class="language-plaintext highlighter-rouge">std::array<></code> instead</li>
<li>[*] multiple variables cannot be defined at once - like <code class="language-plaintext highlighter-rouge">int a, b;</code></li>
<li>[*] don’t use <code class="language-plaintext highlighter-rouge">auto*</code> - use <code class="language-plaintext highlighter-rouge">auto</code> directly and let it deduce pointer types</li>
<li>[*] raw string literals shouldn’t be used</li>
<li>cannot assign lambdas to auto - should use <code class="language-plaintext highlighter-rouge">std::function<></code> instead</li>
<li>no deleted operator <code class="language-plaintext highlighter-rouge">new</code>/<code class="language-plaintext highlighter-rouge">delete</code> for types - they should be allocatable</li>
<li>temporaries cannot bind to const references (and have their lifetime extended) because pointers are used under the hood - will get a compiler error when trying to get the address of a temporary</li>
<li>rvalue references as variables are disallowed by the parser itself - related to the const reference restriction</li>
</ul>
<p>The <code class="language-plaintext highlighter-rouge">[*]</code> entries can be removed by improving the parser or by using <a href="https://clang.llvm.org/docs/Tooling.html#libclang">LibClang</a>.</p>
<p>And here is a list of general restrictions to keep in mind:</p>
<ul>
<li>don’t rely on the address of functions - it will be different after each recompilation and reloading</li>
<li>don’t pass pointers to functions or globals (and persistent globals from a <code class="language-plaintext highlighter-rouge">vars</code> section) to the host app without a way to remove them before doing a cleanup (or you would end up with dangling pointers)</li>
<li>don’t use the <code class="language-plaintext highlighter-rouge">static</code> keyword - it won’t work as expected for local variables in functions and it doesn’t make sense for functions and globals</li>
<li>don’t use <code class="language-plaintext highlighter-rouge">goto</code> in <code class="language-plaintext highlighter-rouge">once</code> sections…</li>
<li><code class="language-plaintext highlighter-rouge">decltype()</code> of names from <code class="language-plaintext highlighter-rouge">vars</code> sections will return a reference to the type</li>
<li>constexpr variables should be in <code class="language-plaintext highlighter-rouge">global</code> sections and not in <code class="language-plaintext highlighter-rouge">vars</code> because it wouldn’t make any sense</li>
<li>preprocessor use is allowed only in the <code class="language-plaintext highlighter-rouge">once</code> and <code class="language-plaintext highlighter-rouge">global</code> sections - and should be kept to a minimum</li>
<li>global non-constexpr variables in the <code class="language-plaintext highlighter-rouge">global</code> section will be initialized (and have their state reset) each time code is submitted and a new plugin is compiled and loaded (also there will be multiple instances of them and the initializing code will be executed many times so if it has side effects they will accumulate) - use <code class="language-plaintext highlighter-rouge">vars</code> sections for proper persistence.</li>
<li>class static variables should go into <code class="language-plaintext highlighter-rouge">global</code> sections outside of the class definition, but that means they will always be initialized - currently there isn’t a way to make them persistent like globals from a <code class="language-plaintext highlighter-rouge">vars</code> section</li>
<li>C++14 is required by the <a href="https://github.com/onqtam/rcrl">RCRL</a> engine itself only for support of <code class="language-plaintext highlighter-rouge">auto</code> variables in <code class="language-plaintext highlighter-rouge">vars</code> sections (otherwise C++11 is enough)</li>
</ul>
<p>Perhaps there are other issues I haven’t found yet but all these seem minor.</p>
<h2 id="how-to-integrate">How to integrate</h2>
<p>The <a href="https://github.com/onqtam/rcrl">RCRL</a> repository is mainly a demo (tested on Windows/Linux/MacOS and uses OpenGL 2 - to build it follow the instructions in the repository). The important parts are the <a href="https://github.com/onqtam/rcrl">RCRL</a> “engine” itself which is located in <a href="https://github.com/onqtam/rcrl/tree/master/src/rcrl"><code class="language-plaintext highlighter-rouge"><repo>/src/rcrl/</code></a> and it depends only on the <a href="https://github.com/eidheim/tiny-process-library">tiny-process-library</a> third party which is a submodule of the repo and is located in <a href="https://github.com/onqtam/rcrl/tree/master/src/third_party"><code class="language-plaintext highlighter-rouge"><repo>/src/rcrl/third_party/</code></a>. Everything else is for the demo project and the GUI. The plugin which is compiled by the engine also has a precompiled header (<a href="https://github.com/onqtam/rcrl/blob/master/src/precompiled_for_plugin.h"><code class="language-plaintext highlighter-rouge"><repo>/src/precompiled_for_plugin.h</code></a>) for speed of compilation.</p>
<p>The purpose of the whole repository is not to take the sources from <a href="https://github.com/onqtam/rcrl/tree/master/src/rcrl"><code class="language-plaintext highlighter-rouge"><repo>/src/rcrl/</code></a> as they are but to adapt them to your needs - the goal wasn’t to create a one-size-fits-all solution because that is hardly possible.</p>
<p>The <a href="https://github.com/onqtam/rcrl">RCRL</a> integration in the demo project is by no means optimal:</p>
<ul>
<li>it invokes CMake instead of the compiler directly</li>
<li>the plugin is part of the whole CMake setup instead of being separated - and slow build systems like <code class="language-plaintext highlighter-rouge">make</code> and <code class="language-plaintext highlighter-rouge">MSBuild</code> take a lot of time to scan the dependencies (can be more than half a second) - unlike <code class="language-plaintext highlighter-rouge">Ninja</code></li>
</ul>
<p>Integrating the <a href="https://github.com/onqtam/rcrl">RCRL</a> engine properly and optimally requires knowledge of build systems, compilers, static/dynamic libraries and more!</p>
<p>Currently there are a few preprocessor identifiers setup from CMake for the <a href="https://github.com/onqtam/rcrl">RCRL</a> engine to use for convenience (used in <a href="https://github.com/onqtam/rcrl/blob/master/src/rcrl/rcrl.cpp"><code class="language-plaintext highlighter-rouge"><repo>/src/rcrl/rcrl.cpp</code></a>):</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">RCRL_PLUGIN_FILE</code> - full path to the <code class="language-plaintext highlighter-rouge">.cpp</code> plugin source file for use by <a href="https://github.com/onqtam/rcrl">RCRL</a></li>
<li><code class="language-plaintext highlighter-rouge">RCRL_PLUGIN_NAME</code> - the name of the plugin target in CMake</li>
<li><code class="language-plaintext highlighter-rouge">RCRL_BUILD_FOLDER</code> - the root build directory of the whole CMake project</li>
<li><code class="language-plaintext highlighter-rouge">RCRL_BIN_FOLDER</code> - the folder where the plugin will be after compilation</li>
<li><code class="language-plaintext highlighter-rouge">RCRL_EXTENSION</code> - the platform-specific shared object extension (<code class="language-plaintext highlighter-rouge">.dll</code> for Windows, <code class="language-plaintext highlighter-rouge">.so</code> for Linux, <code class="language-plaintext highlighter-rouge">.dylib</code> for MacOS)</li>
<li><code class="language-plaintext highlighter-rouge">RCRL_CONFIG</code> - only for multi-config IDEs like Visual Studio and XCode - the identifier represents the current configuration (Debug, Release, etc.)</li>
</ul>
<p>The plugin needs to link to the executable so it can interact with the host application through the API with exported symbols. It also needs to link for the 2 functions exported by the <a href="https://github.com/onqtam/rcrl">RCRL</a> engine for <code class="language-plaintext highlighter-rouge">vars</code> sections (<code class="language-plaintext highlighter-rouge">rcrl_get_persistence()</code> and <code class="language-plaintext highlighter-rouge">rcrl_add_deleter()</code>). In CMake executable targets cannot be linked against by default but this can be enabled by setting the <code class="language-plaintext highlighter-rouge">ENABLE_EXPORTS</code> target property to <code class="language-plaintext highlighter-rouge">ON</code> of the executable.</p>
<p>Actually the <a href="https://github.com/onqtam/rcrl">RCRL</a> engine can reside also in some shared object and it is possible that the important parts of the host application API are implemented in other shared objects and not in the executable - in which case linking to it would be unnecessary (but still linking to the appropriate modules will be).</p>
<p>The entire “API” of <a href="https://github.com/onqtam/rcrl">RCRL</a> is just a few functions in <a href="https://github.com/onqtam/rcrl/blob/master/src/rcrl/rcrl.h"><code class="language-plaintext highlighter-rouge"><repo>/src/rcrl/rcrl.h</code></a> which have a lot of comments for them and it is used in <a href="https://github.com/onqtam/rcrl/blob/master/src/main.cpp"><code class="language-plaintext highlighter-rouge"><repo>/src/main.cpp</code></a>.</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="nf">cleanup_plugins</span><span class="p">(</span><span class="kt">bool</span> <span class="n">redirect_stdout</span> <span class="o">=</span> <span class="nb">false</span><span class="p">);</span>
<span class="kt">bool</span> <span class="nf">submit_code</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">code</span><span class="p">,</span> <span class="n">Mode</span> <span class="n">default_mode</span><span class="p">,</span> <span class="kt">bool</span><span class="o">*</span> <span class="n">used_default_mode</span> <span class="o">=</span> <span class="mi">0</span><span class="p">);</span>
<span class="kt">bool</span> <span class="nf">is_compiling</span><span class="p">();</span>
<span class="kt">bool</span> <span class="nf">try_get_exit_status_from_compile</span><span class="p">(</span><span class="kt">int</span><span class="o">&</span> <span class="n">exitcode</span><span class="p">);</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="nf">get_new_compiler_output</span><span class="p">();</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="nf">copy_and_load_new_plugin</span><span class="p">(</span><span class="kt">bool</span> <span class="n">redirect_stdout</span> <span class="o">=</span> <span class="nb">false</span><span class="p">);</span>
</code></pre></div></div>
<p>Some ideas about the integration of the <a href="https://github.com/onqtam/rcrl">RCRL</a> engine:</p>
<ul>
<li>the code editor may be separate from the host application - perhaps vim or whatever you’d like (so you can also have auto completion and whatever) - it doesn’t have to be integrated into the host application</li>
<li>the entire <a href="https://github.com/onqtam/rcrl">RCRL</a> engine can be also outside of the host application - there should only be a way for the host application to be notified that a new plugin needs to be loaded (can be even done with a filesystem watcher)</li>
<li>for performance: try to avoid linking the plugin to static libraries as build times will increase and global state in them might lead to problems</li>
<li>for easy interaction with the host application most symbols should be exported - on Unix platforms all are exported by default from shared objects and linkable executables (unless built with <code class="language-plaintext highlighter-rouge">-fvisibility=hidden</code>) but on Windows the opposite is default - so unless everything is annotated with <a href="https://msdn.microsoft.com/en-us/library/a90k134d.aspx"><code class="language-plaintext highlighter-rouge">__declspec(dllexport)</code></a> properly <a href="https://cmake.org/cmake/help/latest/prop_tgt/WINDOWS_EXPORT_ALL_SYMBOLS.html">this target property</a> can be used in CMake to have everything exported by default (or <a href="https://stackoverflow.com/questions/225432/export-all-symbols-when-creating-a-dll">hacked</a> for not CMake)</li>
</ul>
<h2 id="room-for-improvement">Room for improvement</h2>
<ul>
<li><code class="language-plaintext highlighter-rouge">global</code> and <code class="language-plaintext highlighter-rouge">vars</code> sections can be merged if the parser is improved to be able to handle not just variable definitions (or perhaps it should be ditched entirely in favor of <a href="https://clang.llvm.org/docs/Tooling.html#libclang">LibClang</a> - in which case we might even infer which pieces of the code are statements intended for function scope - to be executed only once - and ditch also the <code class="language-plaintext highlighter-rouge">once</code> sections)</li>
<li>auto complete - but this is a big topic and is hard to make a universal solution that would fit everyone’s needs</li>
<li>crash handling - perhaps with structured exceptions under Windows when loading the plugin - in case anything happens while the code is executed</li>
<li>compiler error messages - a mapping of lines between the plugin <code class="language-plaintext highlighter-rouge">.cpp</code> file and the submitted code can be made so errors can be highlighted in the original submitted source directly</li>
<li>debugging support - with breakpoints and etc. - no idea about this…</li>
<li>build system integration - as mentioned in the previous section</li>
</ul>
<h2 id="random-further-thoughts">Random further thoughts</h2>
<ul>
<li>
<p>This technique can be used for other compiled languages as well.</p>
</li>
<li>
<p>It would be really cool to see something like this integrated into some big project like the <a href="https://www.unrealengine.com">Unreal Engine</a> :)</p>
</li>
<li>Wouldn’t it be cool if applications had an optional module which enables interacting with them with C++? The module would be comprised of:
<ul>
<li>a custom version of the <a href="https://github.com/onqtam/rcrl">RCRL</a> engine</li>
<li>a C++ compiler (perhaps the same version used for building them)</li>
<li>application API headers with exported symbols</li>
<li>application export lib so <a href="https://github.com/onqtam/rcrl">RCRL</a> can link to it</li>
</ul>
<p>There can be different code “snippets” for applications that modify them.</p>
</li>
<li>I’m betting that an awesome C++ game engine that enables an incredibly fast and flexible workflow can be developed without the need for scripting if 3 things are in place:
<ul>
<li>something like <a href="https://github.com/iboB/dynamix">dynamix</a> is used for the object model which would allow for very flexible and elegant implementation of the business logic - one of the 2 main reasons people turn to scripting languages</li>
<li>hot-reloading of any module (including whole subsystems like <em>rendering</em> and <em>physics</em>) and the ability to change the layout of types (like adding a new field in a class) at runtime is possible - perhaps with the help of a <a href="http://onqtam.com/programming/2017-09-02-simple-cpp-reflection-with-cmake/">practical generic reflection system</a> - this is the second main reason people turn to scripting languages for parts of the business logic - reloadability</li>
<li>something like a REPL for convenience (<a href="https://github.com/onqtam/rcrl">RCRL</a> fills this gap)</li>
</ul>
<p>What we lose by eliminating scripting is:</p>
<ul>
<li>the ability to update clients remotely without modifying executables</li>
<li>game designers can no longer tinker in something easier than C++ - but in many studios this is the case anyway (or perhaps they can - with something like <a href="https://docs.unrealengine.com/latest/INT/Engine/Blueprints/">Unreal’s Blueprints</a> - later compiled to C++)</li>
</ul>
<p>What we gain from eliminating scripting is:</p>
<ul>
<li>the (imperfect) binding layer between C++ and the scripting is gone</li>
<li>no need for a virtual machine</li>
<li>optimal performance</li>
<li>programmers work in only 1 language</li>
<li>no code duplication (which is atleast in part inevitable otherwise)</li>
</ul>
</li>
</ul>
<p>Anyway - I’m eager to see what the C++ community thinks of this project/technique and what comes out of it!</p>Viktor Kirilovvik.kirilov@gmail.comHow the cling alternative RCRL works and how to integrate itCppCon 2017 trip report2017-10-04T12:01:34+00:002017-10-04T12:01:34+00:00https://onqtam.com/misc/cppcon-2017-trip-report<p>Attending CppCon was a great experience which I’d recommend to anyone in this field. Everyone I interacted with was really welcoming and nice to talk to - nobody annoyed me in the entire conference. I don’t get why there needs to be a <a href="https://cppcon.org/codeofconduct/" target="_blank">code of conduct</a> since people are following it anyway!</p>
<p>It was great to meet in person all the legends I’ve been watching on youtube and to be surrounded by the world’s top C++ engineers!</p>
<p>It was also very exciting for me to visit the states for the first time - quite the enriching experience. The <a href="http://www.meydenbauer.com/" target="_blank">Meydenbauer Center</a> is probably the best place for hosting conferences!</p>
<table>
<tr>
<td>
<a href="/assets/images/cppcon_2017/IMG_1577.JPG"><img src="/assets/images/cppcon_2017/IMG_1577.JPG" /></a>
</td>
<td>
<a href="/assets/images/cppcon_2017/IMG_1585.JPG"><img src="/assets/images/cppcon_2017/IMG_1585.JPG" /></a>
</td>
</tr>
<tr>
<td>
<a href="/assets/images/cppcon_2017/IMG_1588.JPG"><img src="/assets/images/cppcon_2017/IMG_1588.JPG" /></a>
</td>
<td>
<a href="/assets/images/cppcon_2017/IMG_1658.JPG"><img src="/assets/images/cppcon_2017/IMG_1658.JPG" /></a>
</td>
</tr>
</table>
<p>The rest of the photos I took can be found on <a href="https://www.facebook.com/viktor.i.kirilov/media_set?set=a.1951239845141786.1073741836.100007673674779" target="_blank">facebook</a>.</p>
<h1 id="my-experience">My experience</h1>
<p>I attended only 2-3 sessions before mine (which was about <a href="https://github.com/onqtam/doctest" target="_blank">doctest</a> and was <a href="https://cppcon2017.sched.com/event/BgsI/mix-tests-and-production-code-with-doctest-implementing-and-using-the-fastest-modern-c-testing-framework" target="_blank">on the 4-th day from 2 pm.</a>) and the reason for that was that I wasn’t fully prepared with my talk. I stayed many hours in my hotel room fixing slides and rehearsing - things I really should have done in advance. I didn’t get to meet awesome people as much as I wanted to precisely because of that and I missed a bunch of talks I wanted to see and now I’ll be watching them on youtube from home… After my presentation I went to talks non-stop and finally enjoyed the conference to the fullest extent possible.</p>
<p>My session was recieved very positively (even though the room wasn’t full) - I think everyone that attended got something out of it - some people even mentioned my session in <a href="http://web.cynd.net/~willo/cppcon-2017-trip-report/" target="_blank">their trip reports</a> :) - the slides can be found <a href="http://slides.com/onqtam/2017_cppcon_doctest" target="_blank">here</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/eH1CxEC29l8" frameborder="0" allowfullscreen=""></iframe>
<p>I had a lightning talk on the 2-nd day and I thought to myself: “It’s just 5 minutes - what can possibly go wrong!”. Well on the 2-nd minute my mouth was already super dry and I had trouble speaking. On the 3-rd minute I wanted to quit so badly - I even started mumbling and made a comment about the dryness of my mouth - so unprofessional. I made it through the 5 minutes, but all I wanted was for it to be over - I <strong>HATED</strong> each second of that experience - I could hear myself talk in the worse possible accent because I couldn’t move my tongue freely - every word was a struggle. When the session was over I quickly hid somewhere and stayed there for 10-20 minutes - I was so ashamed. The next 20 hours I felt like complete shit (even in the next day) - I hadn’t felt so low in the past couple of years - not even after personal dramas and rejections - nothing got to my self-esteem as much as failing in front of a couple houndred people + being filmed. I tried cheering up myself by saying “it can’t have been that bad” and “I won’t forget to bring water for my big talk” but you can’t rationalize yourself out of feelings… Feelings are king. I also had trouble sleeping and had nightmares how I fail much more spectacularly on my upcoming talk and I quit programming for good… So what I did on the next day was I made a 4 hour walk around Bellevue - went to a few parks, touched the water in the bay. After 2-3 hours of walking under the sunshine in the perfect weather - I smiled for the first time.</p>
<p>So needless to say I drank a full bottle before my main talk and had 2 more ready to go at the stand (1 of which was completely gone by the end of the talk - I felt the need to drink quite regularly). I don’t know if it’s because of the anxiety from public speaking or it’s from all the speaking. I also don’t think everyone has this problem - but it’s a big one for me.</p>
<p>It’s also interesting how much my heart rate increases when I make comments on the microphone during sessions - the adrenaline rush I get is so huge - much more than when I’m presenting. I don’t know why it is such a big deal for me to speak up in such a way - perhaps it’s the possibility to have an unrehearsed conversation in front of everybody - or just that I can say something dumb - I don’t know… But I rarely speak up like that precisely because of that <strong>HUGE</strong> anxiety I feel - and oftentimes I talk to the presenters after everything ends - not on tape and not being actively listened to by hundreds of people.</p>
<h1 id="my-advice">My advice</h1>
<p>So here is my advice:</p>
<ul>
<li>
<p>For attendees: make sure you have read Kate Gregory’s excellent blog post <a href="http://www.gregcons.com/KateBlog/SurvivingAnIntenseConference.aspx" target="_blank">“Surviving an intense conference”</a> - it’s full of advice on how to plan for the conference so you get the most out of it!</p>
</li>
<li>
<p>For speakers: everything for attendees applies, with some additions:</p>
<ul>
<li>make sure you have read all from the <a href="https://cppcon.org/presenterinfo/" target="_blank">“Presenter Information”</a> page</li>
<li>be fully prepared with your talk (slides + rehearsing)! I mean it!</li>
<li>have <strong>at least</strong> one bottle of water next to you when you’re speaking</li>
</ul>
</li>
</ul>
<h1 id="the-talks-i-went-to">The talks I went to</h1>
<h3 id="jason-turner---practical-c17">Jason Turner - <a href="https://cppcon2017.sched.com/event/Bgu8/practical-c17" target="_blank">“Practical C++17”</a></h3>
<p>If you’re wondering which C++17 features are worth adopting and how to evaluate them - this is the talk!</p>
<h3 id="panel-with-chandler-carruth-matt-godbolt-jim-radigan---optimizing-code-speed-and-space-with-build-time-switches">Panel with Chandler Carruth, Matt Godbolt, Jim Radigan - <a href="https://cppcon2017.sched.com/event/C8Bh/optimizing-code-speed-and-space-with-build-time-switches" target="_blank">“Optimizing Code Speed and Space with Build Time Switches”</a></h3>
<p>This was so great! So many niche details and insights into how and why compiler implementers make the choices they make - we need more sessions of this kind - such information is so hard to find and is also so entertaining!</p>
<h3 id="nicholas-ormrod---fantastic-algorithms-and-where-to-find-them">Nicholas Ormrod - <a href="https://cppcon2017.sched.com/event/Bgtx/fantastic-algorithms-and-where-to-find-them" target="_blank">“Fantastic Algorithms and Where To Find Them”</a></h3>
<p>A few niche but interesting algorithms - and a good conclusion about algorithms in general.</p>
<h3 id="david-watson---c-exceptions-and-stack-unwinding">David Watson - <a href="https://cppcon2017.sched.com/event/Bgss/c-exceptions-and-stack-unwinding" target="_blank">“C++ Exceptions and Stack Unwinding”</a></h3>
<p>Everything you need to know about how exceptions are actually implemented - pretty deep stuff!</p>
<h3 id="nir-friedman---what-c-developers-should-know-about-globals-and-the-linker">Nir Friedman - <a href="https://cppcon2017.sched.com/event/Bgtn/what-c-developers-should-know-about-globals-and-the-linker" target="_blank">“What C++ developers should know about globals (and the linker)”</a></h3>
<p>A nice compact presentation about globals and some underwater rocks you might hit.</p>
<h3 id="andreas-weis---howling-at-the-moon-lua-for-c-programmers">Andreas Weis - <a href="https://cppcon2017.sched.com/event/Bgt9/howling-at-the-moon-lua-for-c-programmers" target="_blank">“Howling at the Moon: Lua for C++ Programmers”</a></h3>
<p>If you want to learn about Lua and how to integrate it easily with modern C++ - this is the talk for you :)</p>
<h3 id="chandler-carruth---going-nowhere-faster">Chandler Carruth - <a href="https://cppcon2017.sched.com/event/Bgu7/going-nowhere-faster" target="_blank">“Going Nowhere Faster”</a></h3>
<p>Chandler made a very compelling presentation - as usual! Make sure to watch the video when it’s up</p>
<h3 id="matt-godbolt---unbolting-the-compilers-lid-what-has-my-compiler-done-for-me-lately">Matt Godbolt - <a href="https://cppcon2017.sched.com/event/BguG/unbolting-the-compilers-lid-what-has-my-compiler-done-for-me-lately" target="_blank">“Unbolting the Compiler’s Lid: What Has My Compiler Done for Me Lately?”</a></h3>
<p>This was an excellent talk targeting both beginners and advanced coders showing us a thing or two about the assembly the compilers generate and also showcasing the <a href="https://godbolt.org/">compiler explorer</a>.</p>
<h3 id="michael-spencer---my-little-object-file-how-linkers-implement-c">Michael Spencer - <a href="https://cppcon2017.sched.com/event/C5ow/my-little-object-file-how-linkers-implement-c" target="_blank">“My Little Object File: How Linkers Implement C++”</a></h3>
<p>Everything about linkers - a very good summary!</p>
<hr />
<p>All talks are coming up on the <a href="https://www.youtube.com/user/CppCon/videos" target="_blank">youtube channel</a> - I missed a bunch of great presentations - like the <a href="https://cppcon2017.sched.com/event/BguH/meta-thoughts-on-generative-c" target="_blank">“Meta: Thoughts on generative C++”</a> one from Herb Sutter and many many others!</p>
<p>Thanks for reading :)</p>Viktor Kirilovvik.kirilov@gmail.comLessons learned from attending and speaking at CppCon for the first timeSimple C++ reflection with CMake2017-09-02T19:04:34+00:002017-09-02T19:04:34+00:00https://onqtam.com/programming/simple-cpp-reflection-with-cmake<p>Reflection is a very useful tool and if you aren’t familiar with the state of affairs in C++ I’d suggest reading <a href="http://jackieokay.com/2017/04/13/reflection1.html" target="_blank">these</a> <a href="http://jackieokay.com/2017/05/06/reflection2.html" target="_blank">two</a> excellent blog posts by <a href="https://twitter.com/jackayline" target="_blank">@jackie</a>.</p>
<p>Unfortunately none of the current solutions really clicks with me:</p>
<ul>
<li>using <a href="http://www.boost.org/doc/libs/1_65_0/libs/fusion/doc/html/fusion/adapted/define_struct.html" target="_blank">Boost.Fusion</a> or <a href="http://www.boost.org/doc/libs/1_65_0/libs/hana/doc/html/index.html#tutorial-introspection-adapting" target="_blank">Boost.Hana</a> to annotate my classes:
<ul>
<li>seems a bit ugly and intrusive</li>
<li>raises concerns about compile times</li>
<li>no way to annotate fields in a custom way</li>
</ul>
</li>
<li>external tooling (such as libClang - see <a href="https://github.com/Manu343726/siplasplas" target="_blank">siplasplas</a> and <a href="https://github.com/AustinBrunkhorst/CPP-Reflection" target="_blank">CPP-Reflection</a>):
<ul>
<li>makes build setups complicated (especially if your main compiler is not Clang (say MSVC))</li>
<li>not sure about the impact on compile times</li>
<li>annotating fields in a custom way is not straightforward (although some developers have reported having success using <code class="language-plaintext highlighter-rouge">__attribute__</code>)</li>
</ul>
</li>
<li>explicitly registering each field outside of the class definition:
<ul>
<li>violates the <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself" target="_blank">DRY principle</a></li>
<li>some implementations are intrusive - like <a href="https://github.com/simonask/reflect" target="_blank">this one</a></li>
</ul>
</li>
</ul>
<p>I also tried with a DSL (not in C++) which got parsed by a python script and C++ code got emitted but found it too much work to make it support complex C++ types - teaching it about templates, variants and typedefs just got hairy very quickly (not a parser expert).</p>
<h2 id="what-i-came-up-with">What I came up with</h2>
<p>The <a href="https://github.com/onqtam/cmake-reflection-template" target="_blank">cmake-reflection-template</a> repository is a small working example of a few source files with added reflection which generates serialization and deserialization routines (using <code class="language-plaintext highlighter-rouge">std::any<></code> for simplicity - so it requires C++17 - but it can be rewritten to serialize to JSON instead).</p>
<ul>
<li>each CMake target that wants to have reflection should have the <a href="https://github.com/onqtam/cmake-reflection-template/blob/master/scripts/utils.cmake#L16" target="_blank"><code class="language-plaintext highlighter-rouge">target_parse_sources()</code></a> CMake function called on it <a href="https://github.com/onqtam/cmake-reflection-template/blob/master/CMakeLists.txt#L10" target="_blank">like so</a></li>
<li>each source file in the reflected projects has an attached custom CMake command so when it gets modified that command gets ran</li>
<li>that command runs the parser on the file - named for example <code class="language-plaintext highlighter-rouge">my_type.h</code> - which generates code and dumps it in a file called <code class="language-plaintext highlighter-rouge">my_type.h.inl</code> in a <code class="language-plaintext highlighter-rouge">gen</code> folder inside <a href="https://cmake.org/cmake/help/latest/variable/CMAKE_BINARY_DIR.html" target="_blank"><code class="language-plaintext highlighter-rouge">CMAKE_BINARY_DIR</code></a></li>
<li>the resulting <code class="language-plaintext highlighter-rouge">my_type.h.inl</code> can be included either directly in <code class="language-plaintext highlighter-rouge">my_type.h</code> or perhaps elsewhere - the forward declarations of the generated functions are written inside the classes in <code class="language-plaintext highlighter-rouge">my_type.h</code> using the helper <code class="language-plaintext highlighter-rouge">FRIENDS_OF_TYPE(MY_TYPE);</code> macro.</li>
</ul>
<p>It is a bit like what <a href="https://www.unrealengine.com/en-US/blog/unreal-property-system-reflection" target="_blank">Unreal is doing for reflection of properties</a> - C++ source code is parsed (and annotated with preprocessor identifiers) and each source file includes the generated code for itself.</p>
<p>Here is what C++ code looks like with my solution:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Foo</span> <span class="p">{</span>
<span class="n">FRIENDS_OF_TYPE</span><span class="p">(</span><span class="n">Foo</span><span class="p">);</span> <span class="c1">// friend declarations of the generated functions</span>
<span class="n">FIELD</span> <span class="kt">int</span> <span class="n">a</span><span class="p">;</span>
<span class="n">FIELD</span> <span class="kt">float</span> <span class="n">b</span> <span class="o">=</span> <span class="mf">42.</span><span class="n">f</span><span class="p">;</span>
<span class="n">FIELD</span> <span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">c</span> <span class="p">{</span><span class="s">""</span><span class="p">};</span>
<span class="n">FIELD</span> <span class="n">std</span><span class="o">::</span><span class="n">map</span><span class="o"><</span><span class="kt">int</span><span class="p">,</span> <span class="kt">int</span><span class="o">></span> <span class="n">field_with_spaces_in_its_type</span><span class="p">;</span>
<span class="n">FIELD</span> <span class="kt">int</span>
<span class="n">field_on_the_next_line_of_its_type</span><span class="p">;</span>
<span class="p">};</span>
<span class="cp">#include <gen/my_type.h.inl>
</span></code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">FIELD</code> is a preprocessor identifier that expands to nothing - used for easier parsing - otherwise detecting comments or distinguishing between class fields and local variables in inline methods would be very complicated.</p>
<p>What gets generated for this class entirely depends on the parser. <code class="language-plaintext highlighter-rouge">FIELD</code>, <code class="language-plaintext highlighter-rouge">FRIENDS_OF_TYPE()</code> and a few other preprocessor identifiers have to be visible in any source file that wants to use reflection - see them <a href="https://github.com/onqtam/cmake-reflection-template/blob/master/src/common.h#L10-L19" target="_blank">here</a>.</p>
<p>Currently the parser from the example project would generate the following code for the type <code class="language-plaintext highlighter-rouge">Foo</code> written above:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">any</span> <span class="nf">serialize</span><span class="p">(</span><span class="k">const</span> <span class="n">Foo</span><span class="o">&</span> <span class="n">src</span><span class="p">)</span> <span class="p">{</span>
<span class="n">map</span><span class="o"><</span><span class="n">string</span><span class="p">,</span> <span class="n">any</span><span class="o">></span> <span class="n">out</span><span class="p">;</span>
<span class="n">out</span><span class="p">[</span><span class="s">"a"</span><span class="p">]</span> <span class="o">=</span> <span class="n">serialize</span><span class="p">(</span><span class="n">src</span><span class="p">.</span><span class="n">a</span><span class="p">);</span>
<span class="n">out</span><span class="p">[</span><span class="s">"b"</span><span class="p">]</span> <span class="o">=</span> <span class="n">serialize</span><span class="p">(</span><span class="n">src</span><span class="p">.</span><span class="n">b</span><span class="p">);</span>
<span class="n">out</span><span class="p">[</span><span class="s">"c"</span><span class="p">]</span> <span class="o">=</span> <span class="n">serialize</span><span class="p">(</span><span class="n">src</span><span class="p">.</span><span class="n">c</span><span class="p">);</span>
<span class="n">out</span><span class="p">[</span><span class="s">"field_with_spaces_in_its_type"</span><span class="p">]</span> <span class="o">=</span> <span class="n">serialize</span><span class="p">(</span><span class="n">src</span><span class="p">.</span><span class="n">field_with_spaces_in_its_type</span><span class="p">);</span>
<span class="n">out</span><span class="p">[</span><span class="s">"field_on_the_next_line_of_its_type"</span><span class="p">]</span> <span class="o">=</span> <span class="n">serialize</span><span class="p">(</span><span class="n">src</span><span class="p">.</span><span class="n">field_on_the_next_line_of_its_type</span><span class="p">);</span>
<span class="k">return</span> <span class="n">out</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">void</span> <span class="nf">deserialize</span><span class="p">(</span><span class="k">const</span> <span class="n">any</span><span class="o">&</span> <span class="n">src</span><span class="p">,</span> <span class="n">Foo</span><span class="o">&</span> <span class="n">dst</span><span class="p">)</span> <span class="p">{</span>
<span class="k">const</span> <span class="k">auto</span><span class="o">&</span> <span class="n">data</span> <span class="o">=</span> <span class="n">any_cast</span><span class="o"><</span><span class="k">const</span> <span class="n">map</span><span class="o"><</span><span class="n">string</span><span class="p">,</span> <span class="n">any</span><span class="o">>&></span><span class="p">(</span><span class="n">src</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">count</span><span class="p">(</span><span class="s">"a"</span><span class="p">))</span> <span class="p">{</span> <span class="n">deserialize</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">at</span><span class="p">(</span><span class="s">"a"</span><span class="p">),</span> <span class="n">dst</span><span class="p">.</span><span class="n">a</span><span class="p">);</span> <span class="p">}</span>
<span class="k">if</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">count</span><span class="p">(</span><span class="s">"b"</span><span class="p">))</span> <span class="p">{</span> <span class="n">deserialize</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">at</span><span class="p">(</span><span class="s">"b"</span><span class="p">),</span> <span class="n">dst</span><span class="p">.</span><span class="n">b</span><span class="p">);</span> <span class="p">}</span>
<span class="k">if</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">count</span><span class="p">(</span><span class="s">"c"</span><span class="p">))</span> <span class="p">{</span> <span class="n">deserialize</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">at</span><span class="p">(</span><span class="s">"c"</span><span class="p">),</span> <span class="n">dst</span><span class="p">.</span><span class="n">c</span><span class="p">);</span> <span class="p">}</span>
<span class="k">if</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">count</span><span class="p">(</span><span class="s">"field_with_spaces_in_its_type"</span><span class="p">))</span> <span class="p">{</span> <span class="n">deserialize</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">at</span><span class="p">(</span><span class="s">"field_with_spaces_in_its_type"</span><span class="p">),</span> <span class="n">dst</span><span class="p">.</span><span class="n">field_with_spaces_in_its_type</span><span class="p">);</span> <span class="p">}</span>
<span class="k">if</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">count</span><span class="p">(</span><span class="s">"field_on_the_next_line_of_its_type"</span><span class="p">))</span> <span class="p">{</span> <span class="n">deserialize</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">at</span><span class="p">(</span><span class="s">"field_on_the_next_line_of_its_type"</span><span class="p">),</span> <span class="n">dst</span><span class="p">.</span><span class="n">field_on_the_next_line_of_its_type</span><span class="p">);</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a href="https://github.com/onqtam/cmake-reflection-template" target="_blank">cmake-reflection-template</a> is meant to be used as an initial starting point and it is expected to be modified by anyone using it to better suit their needs - perhaps to tweak the parsing, to use JSON instead of <code class="language-plaintext highlighter-rouge">std::any<></code>, to add prefixes to the macros, to add the notion of class descriptions or to change what code gets emitted. The CMake part is solid and the rest can be viewed as a proof of concept. One might even rewrite the parser in a different language! I’m no expert in templating engines or writing parsers so the python script (located in <a href="https://github.com/onqtam/cmake-reflection-template/tree/master/scripts" target="_blank"><code class="language-plaintext highlighter-rouge">/scripts/</code></a>) is nothing special but what I’ve got is good enough for my needs so far.</p>
<h2 id="support-for-attributes-and-tags">Support for attributes and tags</h2>
<p>Currently besides <code class="language-plaintext highlighter-rouge">FIELD</code> there are a few other preprocessor identifiers in a common header that expand to nothing - used for annotating:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#define FIELD // indicates the start of a field definition inside of a type
#define INLINE // class attribute - emitted functions should be marked as inline
#define CALLBACK // field attribute - call the callback after the field changes
#define ATTRIBUTES(...) // comma-separated list of attributes and tags into this
</span></code></pre></div></div>
<p>And they are used like this:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ATTRIBUTES</span><span class="p">(</span><span class="n">INLINE</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Foo</span> <span class="p">{</span>
<span class="n">FRIENDS_OF_TYPE</span><span class="p">(</span><span class="n">Foo</span><span class="p">);</span>
<span class="n">ATTRIBUTES</span><span class="p">(</span><span class="n">tag</span><span class="o">::</span><span class="n">special</span><span class="p">,</span> <span class="n">CALLBACK</span><span class="p">(</span><span class="n">Foo</span><span class="o">::</span><span class="n">some_callback</span><span class="p">))</span>
<span class="n">FIELD</span> <span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">tagged</span><span class="p">;</span>
<span class="n">FIELD</span> <span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">normal</span><span class="p">;</span>
<span class="k">static</span> <span class="kt">void</span> <span class="n">some_callback</span><span class="p">(</span><span class="n">Foo</span><span class="o">&</span> <span class="n">src</span><span class="p">)</span> <span class="p">{</span> <span class="n">cout</span> <span class="o"><<</span> <span class="s">"callback called!"</span> <span class="o"><<</span> <span class="n">endl</span><span class="p">;</span> <span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>
<p>This will result in the following codegen:</p>
<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kr">inline</span> <span class="kt">void</span> <span class="nf">print</span><span class="p">(</span><span class="k">const</span> <span class="n">Foo</span><span class="o">&</span> <span class="n">in</span><span class="p">)</span> <span class="p">{</span>
<span class="n">print</span><span class="p">(</span><span class="n">in</span><span class="p">.</span><span class="n">tagged</span><span class="p">,</span> <span class="n">tag</span><span class="o">::</span><span class="n">special</span><span class="p">());</span> <span class="c1">// tag given to the print() function</span>
<span class="n">print</span><span class="p">(</span><span class="n">in</span><span class="p">.</span><span class="n">normal</span><span class="p">);</span>
<span class="p">}</span>
<span class="kr">inline</span> <span class="kt">void</span> <span class="nf">deserialize</span><span class="p">(</span><span class="k">const</span> <span class="n">any</span><span class="o">&</span> <span class="n">src</span><span class="p">,</span> <span class="n">Foo</span><span class="o">&</span> <span class="n">dst</span><span class="p">)</span> <span class="p">{</span>
<span class="k">const</span> <span class="k">auto</span><span class="o">&</span> <span class="n">data</span> <span class="o">=</span> <span class="n">any_cast</span><span class="o"><</span><span class="k">const</span> <span class="n">map</span><span class="o"><</span><span class="n">string</span><span class="p">,</span> <span class="n">any</span><span class="o">>&></span><span class="p">(</span><span class="n">src</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">count</span><span class="p">(</span><span class="s">"tagged"</span><span class="p">))</span> <span class="p">{</span> <span class="n">deserialize</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">at</span><span class="p">(</span><span class="s">"tagged"</span><span class="p">),</span> <span class="n">dst</span><span class="p">.</span><span class="n">tagged</span><span class="p">);</span> <span class="n">Foo</span><span class="o">::</span><span class="n">some_callback</span><span class="p">(</span><span class="n">dst</span><span class="p">);</span> <span class="p">}</span>
<span class="k">if</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">count</span><span class="p">(</span><span class="s">"normal"</span><span class="p">))</span> <span class="p">{</span> <span class="n">deserialize</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">at</span><span class="p">(</span><span class="s">"normal"</span><span class="p">),</span> <span class="n">dst</span><span class="p">.</span><span class="n">normal</span><span class="p">);</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<ul>
<li>The 2 routines are inline because of the class attribute - the codegen can be included in a header, which in turn can be included in many places.</li>
<li>Both fields are of type <code class="language-plaintext highlighter-rouge">std::string</code> but <a href="https://github.com/onqtam/cmake-reflection-template/blob/master/src/common.h#L22" target="_blank">tagging</a> would allow for different overloads to be chosen - for example consider there are 2 string fields which represent paths for assets of different types (meshes and textures) - if the reflection generates GUI for editing those fields one might want to have different filters for the 2 kinds of assets (<code class="language-plaintext highlighter-rouge">.jpg</code> vs <code class="language-plaintext highlighter-rouge">.mesh</code>)</li>
<li>There is a callback attached to the <code class="language-plaintext highlighter-rouge">tagged</code> field and it will be called any time that field is changed in a routine generated by the reflection - this might be useful when 2 fields are related and changing one must result in changes to the other (the other might not even be annotated with <code class="language-plaintext highlighter-rouge">FIELD</code>).</li>
</ul>
<h2 id="dependencies-and-minimal-rebuild">Dependencies and minimal rebuild</h2>
<p>The CMake macro <code class="language-plaintext highlighter-rouge">target_parse_sources()</code> adds custom CMake commands on each source file and when a source file is modified - the parser is ran only on that file - that way there is no explosion in build times. Also the parser is very light and its result will most likely be included by the changed file anyway so rebuilds will stay minimal.</p>
<p>Also when the parser itself is modified - all generated code in <code class="language-plaintext highlighter-rouge">${CMAKE_BINARY_DIR}/gen</code> is deleted since the parsing rules and codegen are most likely changed.</p>
<h2 id="current-limitations">Current limitations</h2>
<ul>
<li>Types in namespaces not supported (but should be easy to extend).</li>
<li>Nested types not supported (maybe easy).</li>
<li>Inheritance not supported (but should be easy with class attributes)</li>
<li>Templated types - haven’t thought about it but a solution must exists</li>
<li><code class="language-plaintext highlighter-rouge">FIELD</code> has to be used when declaring class fields - otherwise they will be skipped. This greatly simplifies the parser - but I’d be happy if the parser improved and all fields got parsed by default - and perhaps skipping fields should happen only when annotated explicitly to be skipped.</li>
<li>Currently all generated <code class="language-plaintext highlighter-rouge">.inl</code> files go into the <code class="language-plaintext highlighter-rouge">gen</code> folder of the CMake build folder - that means that if 2 files in any of the projects (CMake targets) are named identically - there would be a clash. This can be alleviated if the CMake script is improved a bit.</li>
<li>Cannot declare multiple fields on the same line: <code class="language-plaintext highlighter-rouge">FIELD int x, y, z;</code> - but I don’t like that style anyway.</li>
<li>Parser is not multi-line comment aware - will parse types in <code class="language-plaintext highlighter-rouge">/*...*/</code> and emit code for them</li>
</ul>
<h2 id="how-i-use-this">How I use this</h2>
<p>In my (cleverly titled) <a href="https://github.com/onqtam/game" target="_blank">game</a> project I use reflection extensively. So far I generate only serialization/deserialization and GUI binding routines (3 functions per type) on top of which I’ve built the following:</p>
<ul>
<li>reloadable .dll plugins (with the help of <a href="https://github.com/iboB/dynamix" target="_blank">dynamix</a>) - even with the ability to change the layout of types at runtime! (by serializing object state, recreating the instances and then deserializing the old state)</li>
<li>generic undo/redo system</li>
</ul>
<p>I might expand on that in a future post.</p>
<p>I also like how composable everything is with the serialization/deserialization routines. In a few headers I provide overloads for the primitive types and for a few containers like <code class="language-plaintext highlighter-rouge">std::vector<></code> (and maybe types from third party libraries) - and from then on the generated routines for all my types call overloads for each of their fields - composes quite nicely - I was originally exposed to the idea of such composability in <a href="https://www.youtube.com/watch?v=Njjp_MJsgt8" target="_blank">this talk</a> about <a href="https://github.com/HowardHinnant/hash_append" target="_blank"><code class="language-plaintext highlighter-rouge">hash_append</code></a>!</p>
<p>Thanks for reading :)</p>Viktor Kirilovvik.kirilov@gmail.comAdding reflection to codebases without heavy template metaprogramming or external tools such as LibClangMy hello world!2017-03-08T16:04:34+00:002017-03-08T16:04:34+00:00https://onqtam.com/misc/my-hello-world<p>My name is Viktor Kirilov (25 y/o) and I have 3.5 years of professional experience as a C++ developer in both the games and the VFX industries.</p>
<p>15 months ago I quit my job at <a href="https://www.chaosgroup.com/" target="_blank">ChaosGroup</a> to pursue personal projects - mainly a game (and game engine). I will write about that in future posts. Before that I worked in <a href="https://en.wikipedia.org/wiki/Gameloft" target="_blank">Gameloft</a> for 1 year as a game programmer.</p>
<p>There is also my open source <a href="https://github.com/onqtam/doctest" target="_blank">doctest</a> testing framework which is me being a geek - I learned (and continue to do so) a lot about software development, open source, marketing and the internet from it, but it will in no way financially justify the 4+ months of work (and more to come) I’ve put into it.</p>
<p>There are a couple of recent changes in my life:</p>
<ul>
<li>I started reading books - about self improvement, economics, motivation, fitness, entrepreneurship, business, software development, biographies and others - but not fiction (not enough time for that).</li>
<li>Since the 4th of November 2016 I no longer play or follow <a href="https://en.wikipedia.org/wiki/Defense_of_the_Ancients" target="_blank">DotA</a>.</li>
<li>Since the 1st of January 2017 I stopped drinking alcohol completely and started exercising regularly.</li>
<li>Since the 10th of February 2017 I stopped smoking weed for good because I lied to a friend - bailed on an agreed upon meeting being in no condition to talk about anything… I don’t remember the last time I had lied about anything - and suddenly I had compromised my integrity. I felt <strong>huge regret and shame</strong>. I hated myself for it. No more wasted life.</li>
</ul>
<h2 id="the-purpose-of-this-post">The purpose of this post</h2>
<p>I would say that for the past 15 months I’ve worked about 5 or 6 months combined - and the rest was me gaming, growing up (no longer living with my parents) and catching up in life.</p>
<p>I got money for a maximum of 2 more years before I return to a normal job and given that I want to be building the tech along with the games - panic mode is long overdue. And I ain’t getting any younger either.</p>
<p>The last change that is required for my entrepreneurial dreams to come true is to commit to a work schedule - the <strong>hardest</strong> thing when working from home with <strong>no deadlines</strong>. This post will mark the start of precisely that.</p>
<p><em>I pledge to work <strong>at least</strong> 40 hours per week in the following manner</em>:</p>
<ul>
<li>Monday to Friday - at least 8 hours of work with no distractions.</li>
<li>Work starts in between 10 and 11 AM and ends between 7 and 8 PM - with roughly 1 hour for lunch - like a normal person.</li>
<li>Work outside of this range is OK. Whatever I do in the weekend is fine.</li>
<li>Sports should be done in the morning - and <strong>shall not</strong> mess with these working hours. If there is no time - simply skip the workouts - there shall be <strong>no compromises</strong> with the work ethic.</li>
<li>Anything that will interfere with this work schedule should be planned at least 1 day ahead - anything spontaneous <strong>must</strong> be rejected.</li>
<li>No reading of books during work hours.</li>
</ul>
<p>No more excuses.</p>
<p>Being on a diet and thus not feeling energised for work is <strong>just a lousy excuse</strong>.</p>
<p>I am deeply unhappy when I’m not performing accordingly and living up to my own standards. Just existing is simply not good enough. Casual is not my style.</p>
<p>I also pledge to read at least 1 book each month.</p>
<h2 id="what-you-can-expect-from-this-blog">What you can expect from this blog</h2>
<p>Head over to the <a href="/about">About</a> page for details about that.</p>Viktor Kirilovvik.kirilov@gmail.comThe beginning of my new work ethic I pledge to follow from the early days of March 2017. The goal is to get more shit done and startup a game studio.