IMAXT Repository Documentation ============================== September 2022 Release -- What is new ------------------------------------- Notebooks and Tutorials ''''''''''''''''''''''' * Added a repository of useful `Jupyter Notebooks and Tutorials <https://gitlab.developers.cam.ac.uk/astronomy/camcead/imaxt/useful-notebooks>`_. Updated software '''''''''''''''' * JupyterLab updated to 3.4 * The `Napari 2D Viewer for STPT and AXIOScan <https://www.napari-hub.org/plugins/imaxt-multiscale-plugin>`_ allows to visualize the STPT and AXIOScan mosaics. Future updates will include visualization of resmapled AxioScan and IMC images, bead catalogues and nuclei segmentation results. New JupyterLab Server Options ''''''''''''''''''''''''''''' Now you can select the reservation time for your session as well as directly launch the Desktop environment or use the classic notebook. .. figure:: jlab_spawn.png Archive improvements '''''''''''''''''''' * New `imaxt-mosaic-pipeline` that can process STPT and AXIOScan samples. * Automatic pipeline processing activated for AXIOScan (and already working for STPT and IMC). * The pipelines view page has been improved with extra information and a link to visualize the datasets. .. figure:: pipelines_view.png .. figure:: slide_view.png December 2021 Release -- What is new ------------------------------------ .. figure:: xmaslab.png .. note:: Please read the :ref:`migration_2021` when accessing this new release for the first time. We will keep that page updated when issues appear. Updated software '''''''''''''''' * JupyterLab updated to 3.2 * Code debugger included in Notebooks * Tensorflow 2.7 is now default * Python 3.9 is now default * R updated to version 4.1 * RStudio updated to version 1.4 * Updated software stack to latest versions Graphics card data visualization '''''''''''''''''''''''''''''''' The GPU cards are now available from the virtual desktops to run OpenGL applications. See :ref:`gpu_display`. Owl Scheduler Updated ''''''''''''''''''''' * Owl is now open sourced. Check documentation in `Owl Scheduler <https://eddienko.github.io/owl-pipeline/>`_. * Possible to submit SLURM jobs * Email notification of completed jobs * Improved stability Storage updates ''''''''''''''' * Quotas for user homes and user storage. Data access updates ''''''''''''''''''' * S3 protocol now default data access mechanism. April 2021 Release -- What is new --------------------------------- VNC client access to remote Linux desktops '''''''''''''''''''''''''''''''''''''''''' Access to remote Linux desktops using a VNC client is now available. See :ref:`remote_access_linux` for more details. .. figure:: rdlinux/vnc.png Support for custom data pipelines ''''''''''''''''''''''''''''''''' Users can now submit their own data pipelines and long running jobs to the cluster using Owl. We support jobs that can run in a terminal (e.g. bash, Python, ...) as well as parameterized notebooks. See :ref:`pipelines` for more details. Submit pipelines from JupyterLab '''''''''''''''''''''''''''''''' Pipeline definition files can be submitted to Owl using a simple JupyterLab interface. See :ref:`pipelines_jlab` for more details. .. figure:: pipelines/jlab3.png New data access gateway ''''''''''''''''''''''' Data access should be now using the new IMAXT gateway. See instructions in :ref:`sftp`. New application: Cirrocumulus ''''''''''''''''''''''''''''' Cirrocumulus, an interactive visualization tool for large-scale single-cell genomics data has been installed and available from the Applications category in the JupyterLab launcher. .. figure:: cirrocumulus.png Minor changes ''''''''''''' * JupyterLab upgraded to 3.0.12. * Tensorflow default version is now 2.4 (with 2.2 and 1.15 available). * Due to popularity of dark themes, two new ones have been installed. ---- February 2021 Release - What is new ----------------------------------- There are a lot of new features in this release as well as setting the ground to future improvements. While we document all features, see below a summary of what is new. New repository interface '''''''''''''''''''''''' The new archive interface is now online and used as default for all authentication accross the different services that we provide. We are in the process of ingesting the available data in the repository. The old archive interface has been deprecated. For more information head to :ref:`Operational Repository<operational_repository>`. JupyterLab 3.0 '''''''''''''' JupyterLab has been upgraded from version 2 to version 3. Some of the main improvements for the user are: * **Table of Contents.** A table of contents extension now ships with JupyterLab. This makes it easy to see and navigate the structure of a document. * **Simple Interface Mode.** The Simple Interface mode has been significantly updated to have a more streamlined, document-oriented feel. * **Visual filter in file browser.** The file browser now has a filter input which filters the list of files using fuzzy matching. * **Command Palette.** The command palette is now a floating window that appears on top of your JupyterLab workspace. This enables users to quickly invoke a command while keeping the sidebar closed or switching sidebar panels. Many other small changes regarding stability and easy of use have been included. Improved launcher catagories '''''''''''''''''''''''''''' The icons in the Laucher have been organized in more catagories, separating Notebooks from so called Applications. The latter now include remote desktop, RStudio and Visual Studio Code among others. The Console category has been removed (although this is user configurable). A new IMAXT category contains links to external resources. .. figure:: newlauncher.png On demand linux desktop enviroments ''''''''''''''''''''''''''''''''''' In browser Linux Desktop Environments are now available from JupyterLab. To access click on the Desktop in the Applications category of the Launcher and use your archive password to login. .. figure:: rd_linux.png This provides the same directory structure and data access as the Jupyter notebooks. Additionally there are a few applications installed in the system (ImageJ, TopCat, Firefox, VSCode, Napari). The desktop shares resources with the Jupyter notebooks and has access to the same number of CPUs, RAM and GPU access as specified when the Jupyter server was started. Access to the desktop using a VNC client for improved performance will be made available in future releases. In browser Visual Studio Code ''''''''''''''''''''''''''''' A browser version of the popular Visual Studio Code editor is now available from the Applications category in the Launcher. This greatly improves the code editing capabilities. .. figure:: vscode.png Code Snippets ''''''''''''' JupyterLab Code Snippets allows you to create and store code snippets that can be inserted into any JupyterLab workspace and save you from the hassle of typing repetitive code. Code snippets are pieces of code or individual cells that are frequently used. Simply browse or search snippets in the Snippets panel to use wherever you need in JupyterLab. Using JupyterLab code snippets is easy! Simply open the snippet explorer and browse through the provided predefined code snippets. Still can’t find what you need? Simply highlight code and right-click to save as a snippet or drag cells into the panel. Find snippets by utilizing the search, filter, and preview features. For a fresh start, select the plus icon in the snippet panel to construct a completely new, custom snippet. Also, edit your snippet quickly at any time by clicking the edit icon on the snippet. To use your snippets, you can drag and drop a code snippet to insert it as a cell or press the insert icon on the snippet to inject the code where your cursor is located within the JupyterLab workspace. .. figure:: snippets.gif Breaking changes '''''''''''''''' The directory containing the conda environments ``/opt/conda`` is not longer user writable. This means that users need to install their python packages in their home space. .. ----------------------------------------- .. Page TOC .. ----------------------------------------- .. toctree:: :maxdepth: 1 :hidden: :includehidden: :caption: Operational Repository archive/index archive/dop archive/analysis archive/query archive/data archive/sqlexplorer .. toctree:: :maxdepth: 1 :hidden: :includehidden: :caption: Interactive Environment jupyter/index jupyter/workspaces jupyter/imaxt jupyter/imaxtml jupyter/dask/dask jupyter/magic jupyter/customization jupyter/cheatsheet .. toctree:: :maxdepth: 1 :hidden: :includehidden: :caption: Data Pipelines pipelines/index pipelines/setup pipelines/running pipelines/jlab pipelines/shellpipe pipelines/paperpipe .. toctree:: :maxdepth: 1 :hidden: :includehidden: :caption: Remote access rdlinux/index vmware/index .. toctree:: :maxdepth: 1 :hidden: :includehidden: :caption: IMAXT Tools tools/stpt_viewer .. toctree:: :maxdepth: 1 :hidden: :includehidden: :caption: Data Access data/index data/winscp data/sftp .. toctree:: :maxdepth: 1 :hidden: :includehidden: faq/index.rst