Skip to content

Elaborate on managing Conda python, channels #606

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Mar 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions Topics/Development_Process/Conda/Creating_projects.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Creating Pojects
Conda provides a convenient way to manage dependencies and environments for your projects. A project in Conda is a directory that contains your project code along with a configuration file (environment.yml) specifying the dependencies required to run the project.

## Creating the Project's file

Start by creating a directory for the project. It can named it whatever you like, for example:

![makedir](mkdir.png)

Create an environment.yml File: Inside the project directory, create an environment.yml file. This file will specify the dependencies for the project. Here's an basic example:

![env](env_file.png)

#Add more dependencies as needed

## Creating our enviroment

Create Conda Environment: Once you've defined their dependencies, you can create a Conda environment using the conda env create command:

![conda_env_create](conda_create_example.png)

After creating the environment, activate it using the conda activate command:

![conda_create](act_example.png)

## Creating our Python application

Once the project environment is activated, the user can work on the project as usual, using the installed dependencies. The user can install additional packages, update existing ones, or remove packages as needed. Conda will manage the environment to ensure consistency and reproducibility across different systems.

![new_depen](new_depen.png)

Update Conda Environment: After adding the new dependencies to environment.yml, the user need to update the Conda environment. Activatethe project environment using conda activate if it's not already activated, and then run:

![update_depen](update_depen.png)

Once the update is complete, the user can verify that the new dependencies have been installed by checking the environment:

![conda_list](conda_list.png)


After updating the dependencies, it's a good practice to test the project to ensure that everything is working as expected with the new dependencies.

If the user is working in a version-controlled environment, don't forget to commit the changes to environment.yml to keep track of the updates

Creating projects with Conda is a straightforward process that enables the user to manage dependencies and environments effectively. By defining project dependencies in an environment.yml file, the user can ensure that your project is reproducible and easily shareable with others. Conda's environment management capabilities simplify the process of setting up and managing project environments, making it an ideal choice for Python developers.

### Updating our project with new dependencies

Edit environment.yml: Open the environment.yml file in project directory and add the new dependencies under the dependencies section. For example:


## Reference

1. Conda Official Documentation. Available at: [Conda Documentation](https://docs.conda.io/projects/conda/en/latest/index.html)
23 changes: 23 additions & 0 deletions Topics/Development_Process/Conda/Custom_Channels.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Creating Custom Channels

Before proceeding, ensure that you have conda-build installed. If not, the user can install it with the following command:

![conda_build](conda_build.png)

Next, organize all the packages the user wish to include in subdirectories based on the platforms intend to serve. Here's an example structure:

![conda_channel](conda_channels.png)

Run conda index command on the root directory :

![conda_channel_index](conda_channel_index.png)

The conda index command generates a repodata.json file in each repository directory, which contains metadata for the packages in the channel.

To test custom channel, the user can serve it using a web server or via a file:// URL to the channel directory. The user can then test by sending a search command to the custom channel. For example, if the user want to search for files in the custom channel location /opt/channel/linux-64/, the user can use the following command:

![conda_channel_search](conda_channel_search.png)

## Reference

1. Conda Official Documentation. Available at: [Conda Documentation](https://docs.conda.io/projects/conda/en/latest/index.html)
89 changes: 89 additions & 0 deletions Topics/Development_Process/Conda/Managing_Python using Conda.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Managing Python using Conda
Conda is a versatile package and environment management system designed to simplify the process of managing Python environments and dependencies. With Conda, you can create isolated environments with specific Python versions and packages, facilitating project reproducibility and compatibility.

### Viewing a list of available Python Versions
Conda provides users with the ability to view a list of available Python versions that can be installed within Conda environments. This feature is useful for selecting the appropriate Python version for specific project requirements or ensuring compatibility with existing codebases. Different codebases might conduct different version of python. By checking out the list of python version, conda gives you a easier and convenient way to manage your codesbases. Here's how to you view the list of available Python versions with Conda by typing `conda search python`:

![search_conda_python_available_list](conda_search_python.png)

Running this command will display a list of available Python versions along with their respective package versions. The output will include information such as the Python version number, package build number, and channel from which the package is available.

![Conda_python_version](conda_version.png)

## Installing a different version of Python:
Conda can create a new python environment without overwriting the current one, which gives the you a convenient way to use different python environment with different projects you are working on the same time in the same machine.

With previous step of finding all the python version available, the first step of creating a new environment would be determine the python version you would wish to work on:

![Conda_create_example](conda_create_example.png)

In this given example, `example` would be the name of the new environment and `python=3.10.8` would be the version of the python you would wish to work on the new environment.

Activate the environment you created:

![conda_act](conda_act.png)

This command activates the example environment, ensuring that subsequent Python commands use the Python version installed within this environment.

Verify that the new environment is your current environment.
Note: In MacOS terminal, the user can check the front of the command line to check the enviroment.

![conda_ver1](conda_ver1.png)

To verify the current environment’s version of the python, run this in the terminal:

![conda_ver2](conda_ver2.png)

The output in the terminal would be the current version of the python in your current environment.

## Installing Pypy

To install PyPy, a fast, compliant alternative implementation of the Python language. It is a drop-in replacement for the stock Python interpreter, CPython. Whereas CPython compiles Python to intermediate bytecode that is then interpreted by a virtual machine, PyPy uses just-in-time (JIT) compilation to translate Python code into machine-native assembly language. To install you can follow these steps:

Add conda-forge Channel by typing `conda config --add channels conda-force`:

![pypy_conda](pypy_force.png)

This command adds the conda-forge channel to the list of channels that Conda searches for packages.

Set Channel Priority by typing `conda config --set channel_priority strict`:

![pypy_strict](conda_strict.png)

This command sets the channel priority to strict, ensuring that Conda prioritizes packages from the channels in the order they are listed. This helps avoid unexpected package conflicts.

Create pypy Environment and activate pypy:

![pypy_setup](conda_set_py.png)
![pypy_act](conda_py_act.png)

It creates a new Conda environment named `pypy` and installs PyPy into it. The pypy package provided by the conda-forge channel will be installed into this environment. Then it activates the newly created pypy environment, allowing you to use PyPy for Python development within this environment.

## Using a different version of python
To use a different version of Python in the current environment, you'll typically create a new environment with the desired Python version and activate it. Here's how you can do it using Conda:

Create a New Environment: Use the conda create command to create a new environment with the desired Python version. For example, to create an environment named myenv with Python version 3.9, you can run:

![create_new_py](conda_new_create.png)

Replace myenv with preferred environment name and 3.9 with the desired Python version.

Activate the New Environment and Verify the Python version.: Once the environment is created, activate it using the conda activate command. To verify that you are now using the desired Python version, you can run.

![new_env](conda_new_act.png)

![new_ver](conda_ver3.png)

## Updating Python:

This command will updating the latest version of python.

![conda_update1](conda_update1.png)

If you want to install a minor change, not the newest one, or there is a specific version of python you want use:

![conda_update](conda_update2.png)

## Reference

1. Conda Official Documentation. Available at: [Conda Documentation](https://docs.conda.io/projects/conda/en/latest/index.html)
25 changes: 25 additions & 0 deletions Topics/Development_Process/Conda/Managing_Virtual.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Managing Virtual Packages

"Virtual" packages are injected into the conda solver to allow real packages to depend on features present on the system that cannot be managed directly by conda, like system driver versions or CPU features. Virtual packages are not real packages and not displayed by conda list. Instead conda runs a small bit of code to detect the presence or absence of the system feature that corresponds to the package. The currently supported list of virtual packages includes:

![conda_ver_list](conda_ver_list.png)

## Listed detected Virtual Packages
To view the list of detected virtual packages using the terminal, type `conda info` in the terminal:

Output would be a list of info, virtual packages would be under virtual packages section:

![conda_info](conda_info.png)

## Overriding detected packages
To override virtual package detection for troubleshooting purposes, you can use environment variables. Supported variables include:

CONDA_OVERRIDE_CUDA: Set this variable to the CUDA version number you want to override the detection with. If you want to indicate that no CUDA is detected, set it to an empty string ("").

CONDA_OVERRIDE_OSX: Set this variable to the OSX version number you want to override the detection with. If you want to indicate that no OSX is detected, set it to an empty string ("").

CONDA_OVERRIDE_GLIBC: Set this variable to the GLIBC version number you want to override the detection with. This only applies on Linux. If you want to indicate that no GLIBC is detected, set it to an empty string ("").

## Reference

1. Conda Official Documentation. Available at: [Conda Documentation](https://docs.conda.io/projects/conda/en/latest/index.html)
5 changes: 5 additions & 0 deletions Topics/Development_Process/Conda/Setup_Conda.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,3 +206,8 @@ After everything is done, you should not be able to find the environment named "
1. [**Conda Official Documentation**](https://docs.conda.io/en/latest/): This is the primary source of information about Conda, covering installation, commands, package management, and environment management.

2. [**Conda Cheat Sheet**](https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html): A quick reference guide provided by the official Conda documentation, summarizing the most common Conda commands and their usage.


## Reference

1. Conda Official Documentation. Available at: [Conda Documentation](https://docs.conda.io/projects/conda/en/latest/index.html)
21 changes: 21 additions & 0 deletions Topics/Development_Process/Conda/Viewing_command_line.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Viewing command-line help

Conda provides comprehensive command-line help to guide users through various tasks and operations. Here's how you can access and utilize command-line help in Conda:

## Viewing General Help

To access general help and get an overview of available commands and options, the user can use the conda --help command. This command displays a summary of Conda's main commands and provides guidance on usage.

![conda help](conda_help.png)

conda --h will do the same effect as conda --help which will return a list of conda help list.

![conda h](conda_h.png)

To get help of a specific command, try running conda xxx(the command the user wish to work on) -h. For example: conda create -h, which will output the help menu of the specific command of conda create:

![conda_create_h](conda_create_h.png)

## Reference

1. Conda Official Documentation. Available at: [Conda Documentation](https://docs.conda.io/projects/conda/en/latest/index.html)
Binary file added Topics/Development_Process/Conda/act_example.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_act.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_build.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_h.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_help.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_info.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_list.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_py_act.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_set_py.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_strict.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_ver1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_ver2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/conda_ver3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Topics/Development_Process/Conda/env_file.png
Binary file added Topics/Development_Process/Conda/mkdir.png
Binary file added Topics/Development_Process/Conda/new_depen.png
Binary file added Topics/Development_Process/Conda/update_depen.png