https://blog.readthedocs.com 2023-07-08T10:50:30.109425+00:00 ABlog https://blog.readthedocs.com/better-conda-support/ 2020-04-28T00:00:00+00:00 Manuel Kaufmann <section id="better-support-for-scientific-project-documentation"> <p>In the past year, we’ve been having issues when building projects’ documentation using <code class="docutils literal notranslate"><span class="pre">conda</span></code>. Our build servers were running out of memory and failing projects’ builds. Read the Docs has this memory constraint to avoid misuse of the platform, causing a poor experience for other users.</p> <p>Our first solution was a dedicated server for these kind of projects. We would manually assign them to this server on user requests. This workaround worked okay, but it involves a bad experience for the user and also us doing a manual step each time. Over time, we hit again the same issue of OOM, even giving all the memory available to one project to build its documentation. After some research, we found that <a class="reference external" href="https://www.anaconda.com/understanding-and-improving-condas-performance/">this is a known issue in the conda community</a> and there are some different attempts to fix it (like <a class="reference external" href="https://quantstack.net/mamba.html">mamba</a>). Unfortunately, none of them became the standard yet and the problem is still there.</p> <p>Meanwhile, in another completely different set of work, we were migrating all of our infrastructure to a different architecture:</p> <ul class="simple"> <li><p>Azure <em>Virtual machine scale sets</em> for autoscaling our servers,</p></li> <li><p>Azure <em>storage accounts</em> to store all the user’s documentation</p></li> <li><p>Proxito, an internal service to remove a lot of the state from our servers (more about this migration coming in a future post)</p></li> </ul> <p>This helped us to <em>reduce costs</em> and allowed us to spin up <em>bigger instances</em> for the builders. We have also made some other important operational changes:</p> <ul class="simple"> <li><p>Our builders are now single-process, giving all the memory available to only one project without worrying about affecting others.</p></li> <li><p>We added <a class="reference external" href="https://github.com/readthedocs/readthedocs.org/blob/8e78d680d02aeba12644796b979ef62459f64932/readthedocs/builds/tasks.py#L11">custom task router</a> that routes small builds to small servers (3GB RAM), and big builds to larger servers (7GB RAM). This removes the need for users to ask us to upgrade their isntances.</p></li> <li><p>Assigned all <code class="docutils literal notranslate"><span class="pre">conda</span></code> projects to be built by big servers by default.</p></li> </ul> <p>If you ever had a memory issue on Read the Docs, we’d appreciate if you give it a try again. Please <a class="reference external" href="mailto:support&#37;&#52;&#48;readthedocs&#46;org">let us know</a> about your experience when building scientific documentation. If you know that any of your friends were hitting this issue in the past, we encourage you to talk to them and tell them to give it a try.</p> <p>We are excited to have more scientific documentation on our platform and we are doing our small part to make this happen and have better science in the world.</p> </section>

In the past year, we’ve been having issues when building projects’ documentation using conda. Our build servers were running out of memory and failing projects’ builds. Read the Docs has this memory constraint to avoid misuse of the platform, causing a poor experience for other users.Our first solution was a dedicated server for these kind of projects. We would manually assign them to this server on user requests. This workaround worked okay, but it involves a bad experience for the user and also us doing a manual step each time. Over time, we hit again the same issue of OOM, even giving all the memory available to one project to build its documentation. After some research, we found that this is a known issue in the conda community and there are some different attempts to fix it (like mamba). Unfortunately, none of them became the standard yet and the problem is still there.

2020-04-28T00:00:00+00:00