Installation on Colab or Windows

[jtoerber]

Dear all,

1. I tried to install Pose2Sim Version 0.10 on Google Colab.
   Installation of conda is a little different:
   https://saturncloud.io/blog/how-to-install-conda-package-to-google-colab/
   !pip install -q condacolab
   import condacolab
   condacolab.install()

!conda --version

import condacolab
condacolab.check()

switch in Colab to , not only in the bash executing the command
%cd /usr/local/lib/python3.10/site-packages/Pose2Sim/Demo_SinglePerson

Colab has no GUI, so we have to work headless or with a virtual GUI:
There seems to be a workaround (Xvfb):
https://colab.research.google.com/drive/1flu31ulJlgiRL1dnN2ir8wGh9p7Zij2t#scrollTo=8-AxnvAVyzQQ
!pip install pyvirtualdisplay
!apt-get install -y xvfb python-opengl ffmpeg
on Ubuntu: huggingface/deep-rl-class#275
!apt-get install -y xvfb python3-opengl ffmpeg

I then had to this code before running the Demo_SinglePerson:
from pyvirtualdisplay import Display
display = Display(visible=0, size=(1400, 900))
display.start()

from Pose2Sim import Pose2Sim
Pose2Sim.calibration()
Pose2Sim.poseEstimation()
Pose2Sim.synchronization()
Pose2Sim.personAssociation()
Pose2Sim.triangulation()
Pose2Sim.filtering()
Pose2Sim.markerAugmentation()
Pose2Sim.kinematics()

Then I get:
DisabledFunctionError: cv2.imshow() is disabled in Colab, because it causes Jupyter sessions
to crash; see jupyter/notebook#3935.
As a substitution, consider using
from google.colab.patches import cv2_imshow

Well, It seems that then all imports would have been to be replaced. Any chances to run it headless?

2. I then tried to install Pose2Sim Version 0.10 on Windows 10:

Installation seemed ok. After trying Demo_SinglePerson with the provided script:

from Pose2Sim import Pose2Sim
Pose2Sim.calibration()
Pose2Sim.poseEstimation()
Pose2Sim.synchronization()
Pose2Sim.personAssociation()
Pose2Sim.triangulation()
Pose2Sim.filtering()
Pose2Sim.markerAugmentation()
Pose2Sim.kinematics()

I got:
No valid CUDA installation found: using OpenVINO backend with CPU.
Inference run on every single frame.
Using HALPE_26 model (body and feet) for pose estimation.
Mode: balanced.

Traceback (most recent call last):
File "c:\Users<osuser>.conda\envs\Pose2Sim\Lib\site-packages\Pose2Sim\Demo_SinglePerson\test.py", line 3, in
Pose2Sim.poseEstimation()
File "C:\Users<osuser>.conda\envs\Pose2Sim\lib\site-packages\Pose2Sim\Pose2Sim.py", line 238, in poseEstimation
rtm_estimator(config_dict)
File "C:\Users<osuser>.conda\envs\Pose2Sim\lib\site-packages\Pose2Sim\poseEstimation.py", line 450, in rtm_estimator
pose_tracker = PoseTracker(
File "C:\Users<osuser>.conda\envs\Pose2Sim\lib\site-packages\rtmlib\tools\solution\pose_tracker.py", line 121, in init
model = solution(mode=mode,
File "C:\Users<osuser>.conda\envs\Pose2Sim\lib\site-packages\rtmlib\tools\solution\body_with_feet.py", line 104, in init
self.det_model = YOLOX(det,
File "C:\Users<osuser>.conda\envs\Pose2Sim\lib\site-packages\rtmlib\tools\object_detection\yolox.py", line 20, in init
super().init(onnx_model,
File "C:\Users<osuser>.conda\envs\Pose2Sim\lib\site-packages\rtmlib\tools\base.py", line 43, in init
onnx_model = download_checkpoint(onnx_model)
File "C:\Users<osuser>.conda\envs\Pose2Sim\lib\site-packages\rtmlib\tools\file.py", line 149, in download_checkpoint
extract_zip(cached_file, tmp_dir)
File "C:\Users<osuser>.conda\envs\Pose2Sim\lib\site-packages\rtmlib\tools\file.py", line 28, in extract_zip
with zipfile.ZipFile(zip_file_path, 'r') as zip_ref:
File "C:\Users<osuser>.conda\envs\Pose2Sim\lib\zipfile.py", line 1268, in init
self._RealGetContents()
File "C:\Users<osuser>.conda\envs\Pose2Sim\lib\zipfile.py", line 1335, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

Any help or suggestions welcome.
Currently I have no own Linux machine available.
Thanks in advance.

[davidpagnon]

Hi, sorry I'm away from the computer for a week, I'll have a look next week!

[jtoerber]

Hi, thanks for your immediate feedback.
Found Windows problem:
Seems, that something went wrong during download of yolox_m_8xb8-300e_humanart-c2c7a14a.zip during
...
Pose2Sim.poseEstimation()
...
I removed the old downloaded file from
C:\Users<osuser>.cache\rtmlib\hub\checkpoints
(previously I added a print(url) in file.py and a print(fp) in zipfile.py)
Now I can see the above mentioned cv2.imshow(...) ;-).
As my computer is slow Colab would be better. Perhaps some kind of Parameter "enableHeadless" would be helpful.
Progressing further.

[jtoerber]

Hi,
in Config.toml I set:
display_detection = false
display_sync_plots = false
display_figures = false

I can choose one of three Colab runtime environments: CPU, T4 GPU, TPU v2-8
On TPU v2-8 I could not do the above mentioned !apt-get install -y xvfb python-opengl ffmpeg
Last time I used CPU Colab runtime environment. Not quite sure if they are identically configured each time.
Perhaps some kind of dynamic Import dependent on some configuration parameter(s) could do the Job.

On Windows display_detection=false disabled the cv2.imshow(...): poseEstimation.py
The others disable the mathplotlib figures (?).

[davidpagnon]

Hi @jtoerber, apologies for the delay.

Glad to see you managed to run it within Windows!
I have never tried Pose2Sim in Colab but that's an interesting use case.

You set all the visualization options to false, which is what I would have first suggested. This is good if everything runs smoothly, but in case of challenging capture, you may want to be able to show the images and plots. Does it only fail with poseEstimation (which uses cv2.imshow()), or does it also fail with plots (in calibration, synchronization, and filtering)?

---

If it only fails with poseEstimation and you want to display detection, then check the location of your installation with pip show pose2sim. Then you can add in common.py:

try:
    from google.colab.patches import cv2_imshow
    COLAB = True
except ImportError:
    COLAB = False

    
def show_img(img_name, img):
    '''
    Display an image in Colab or a local environment
    '''
    if COLAB:
        cv2_imshow(img)  # Use cv2_imshow in Colab
    else:
        cv2.imshow(img_name, img)    

In poseEstimation.py, you can add:

from Pose2Sim.common import show_img

And you can replace both occurrences of cv2.imshow with:

show_img(f"Pose Estimation {os.path.basename(video_path)}", img_show)
# AND
show_img(f"Pose Estimation {os.path.basename(image_folder_path)}", img_show)

Whether this works or fails, can you tell me? I would like to enable this feature in the next release

[jtoerber]

Hi,

thanks for the code snippets.

First I did this according to https://stackoverflow.com/questions/53581278/test-if-notebook-is-running-on-google-colab:

try:
  import google.colab
  IN_COLAB = True
except:
  IN_COLAB = False

print(IN_COLAB)
IN_COLAB=None
print(IN_COLAB)

import sys
IN_COLAB = 'google.colab' in sys.modules
print(IN_COLAB)

if 'google.colab' in str(get_ipython()):
  print('Running on CoLab')
else:
  print('Not running on CoLab')

import os

if os.getenv("COLAB_RELEASE_TAG"):
  print("Running in Colab")
  print(os.getenv("COLAB_RELEASE_TAG"))
else:
  print("NOT in Colab")

import os
if 'COLAB_GPU' in os.environ:
  print("I'm running on Colab")
  # CPU: set but empty, T4 GPU: 1, TPU v2-8:?
  print("Hello:" + os.getenv("COLAB_GPU") + ":Hello")

import os
if os.path.exists('/colabtools'):
  print("/colabtools exists")
else:
  print("/colabtools does not exist")

with the following output on CPU(!):
True
None
True
Running on CoLab
Running in Colab
release-colab_20241031-060132_RC00
I'm running on Colab
Hello::Hello
/colabtools does not exist

On a T4 GPU I get the same except:
...
Hello:1:Hello
...
I personally use:

import pkgutil

mediapipe_loader = pkgutil.find_loader('mediapipe')

Regarding the imshow problem:
In common.py there is nearly at the top:

mpl.use('qt5agg')

In Colab executing

%matplotlib --list

gives me on a CPU-Colab-Runtime:
Available matplotlib backends: ['tk', 'gtk', 'gtk3', 'gtk4', 'wx', 'qt4', 'qt5', 'qt6', 'qt', 'osx', 'nbagg', 'notebook', 'agg', 'svg', 'pdf', 'ps', 'inline', 'ipympl', 'widget']
Perhaps you can use a different backend (see also below: inline)
I commented the line on Colab:

# mpl.use('qt5agg')

then did the additional function in common.py and changes in poseEstimation.py
poseEstimation is now working

I find on https://github.com/alankritmishra/DL-Notebooks/blob/master/Packt_CV_w_Python3_Getting_Started_with_Jupyter_notebooks_Done.ipynb:

# ...
%matplotlib inline
# ...

, which should help even if the display_... Parameters are set to true.

Currently I am working in Demo_SinglePerson so no calibration needed, but

mpl.use('qt5agg')

is also executed for Calibration and leads to an "qt5agg"-error:
ImportError: Cannot load backend 'Qt5Agg' which requires the 'qt' interactive framework, as 'headless' is currently running

Addition: poseEstimation took about 2:30 per Video on a CPU-Colab-Runtime. Will try the other ones as well.
Progressing further with a modified project needing a complete run including calibration.

At least on the free Colab, every time you use it, you have to set everything up. But you are able to mount a Google-Drive, where you maybe should put your conda environment, if you have enough space.

You may even share Colab-Notebooks as in this sample:
https://github.com/nicknochnack/ActionDetectionforSignLanguage/blob/main/Action%20Detection%20Refined.ipynb
(if only with code and without the output it is only some bytes)

[davidpagnon]

Hi, it is a bit hard to read, could you put the code inside back-quotes and indent it?

` ` ` python 
YOUR CODE
(remove the spaces between back-quotes)
` ` `

[jtoerber]

Done.

[davidpagnon]

Thanks for the formatting!
I'm afraid I do not understand what your question is exactly (is there any?)

[jtoerber]

Hi,
still in Demo_SinglePerson.
Still all display_... Parameters set to false.

All calls except

Pose2Sim.kinematics()

could be executed. The last one gives:
ImportError: Cannot load backend 'Qt5Agg' which requires the 'qt' interactive framework, as 'headless' is currently running
on import of opensim in kinematics.py (-> module opensim could not be loaded)
Found something similar:
QuantEcon/MatchingMarkets.py#1

Addition:
Today I was (automatically assigned) on a TPU.
Pose2Sim.poseEstimation()
took about 3 seconds per Video!!
This

print(torch.cuda.is_available(), ort.get_available_providers())

gave on CPU, T4 GPU and TPU always False.

With above mentioned environment variables you may detect COLAB and/or COLAB with CPU, GPU or TPU.

Proposal:
calibration on PC
poseEstimation on Colab if long running and/or multiple runs required
synchronization - marker augmentation on PC or Colab, dependent on UI-Display required or not
kinematics on PC

is one way of working with Colab reducing poseEstimation running times.

Changes attached.

I consider this as an enhancement.

poseEstimation.zip

[davidpagnon]

Okay, I get it, thanks for that!

Would you be mind proposing a pull request so that changes are easier to track?
If you could also share your Colab file once you are happy with it, that would be great as well :)

I don't have time for it right now, but I'll try to check what's wrong with OpenSim and make kinematics() work.
I'm surprised that it gives a Qt5 error, since it should not use it (unlike other scripts when using display). Are you sure this is not an error in the OpenSim installation? Can you try to run import opensim?

[davidpagnon]

1. Did you manage to install opensim? I just did within Colab with the instruction from your link and it works:

   !pip install -q condacolab
   import condacolab
   condacolab.install()

   !conda install -c opensim-org opensim -y
   import opensim
   

2. torch and CUDA are installed by default on Colab when running a GPU session, so you should not have to install torch and this should return True

   import torch
   print(torch.cuda.is_available())

   I then installed onnxruntime-gpu, and the following returns ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']

   import onnxruntime as ort
   print(ort.get_available_providers())

Do you think you could have made a mistake somewhere?

[jtoerber]

Yes. Call Pose2Sim.kinematics and see what happens.Am 06.11.2024 16:55 schrieb David PAGNON ***@***.***>: Did you manage to install opensim? I just did within Colab with the instruction from your link and it works: !pip install -q condacolab import condacolab condacolab.install() !conda install -c opensim-org opensim -y import opensim torch and CUDA are installed by default on Colab when running a GPU session, so you should not have to install torch and this should return True import torch print(torch.cuda.is_available()) I then installed onnxruntime-gpu, and the following returns ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] import onnxruntime as ort print(ort.get_available_providers()) Do you think you could have made a mistake somewhere? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[davidpagnon]

Can you share your Colab file?
In order to help you, I need to work on the same configuration as you :)

[jtoerber]

ModuleNotFoundError                       Traceback (most recent call last)
[](https://localhost:8080/#) in ()
     15 # Pose2Sim.filtering()
     16 # Pose2Sim.markerAugmentation()
---> 17 Pose2Sim.kinematics()
     18 
     19 # gives: ImportError: Cannot load backend 'Qt5Agg' which requires the 'qt' interactive framework, as 'headless' is currently running

1 frames
[/usr/local/lib/python3.10/site-packages/Pose2Sim/kinematics.py](https://localhost:8080/#) in 
     43 from anytree import PreOrderIter
     44 
---> 45 import opensim
     46 
     47 from Pose2Sim.common import natural_sort_key, euclidean_distance, trimmed_mean

ModuleNotFoundError: No module named 'opensim'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.

or with a little bit of color:

and my eperimental Colab-Notebook
Pose2Sim.zip
I store the *.py on my Google Drive and copy them to the Demo_SinglePerson folder.