Skip to content

Conversation

@OutlyingWest
Copy link
Collaborator

@OutlyingWest OutlyingWest commented Jun 26, 2025

Summary

This PR addresses issues with animation display when using tqdm in conjunction with PyTorch and Score-P logging.

Problems Fixed

  1. Spinner animation worked with the PyTorch progress bar (tested in examples/gpt-demo/demonstrator.ipynb), but was broken with the default tqdm.

  2. When scorep_jupyter_DISABLE_PROCESSING_ANIMATIONS=1 is set, spinner output overlapped with tqdm.

Changes Made

  1. Refactored reading of Score-P process streams: both stdout and stderr are now read in chunks using separate threads.

  2. Stream reader threads now wait on a shared threading.Event that signals the termination of the process_busy_spinner animation when scorep_jupyter_DISABLE_PROCESSING_ANIMATIONS=0.

  3. Added a is_multicell_final=True flag as a scorep_execute() function argument to suppress spinner creation during %%finalize_multicellmode when scorep_jupyter_DISABLE_PROCESSING_ANIMATIONS=0. This ensures create_busy_spinner() produces a dummy spinner without capturing the event.

Notes

  • Fix ensures tqdm works with both PyTorch and default modes.

  • When scorep_jupyter_DISABLE_PROCESSING_ANIMATIONS=0 is set (animation is enabled) - no Score-P child process stderr output in console during animation. In other words, if user's code in cell raises an exception - no detailed error message is seen but only error log is captured. This behavior is described in README.md

@OutlyingWest OutlyingWest requested a review from elwer July 2, 2025 14:53
@OutlyingWest OutlyingWest self-assigned this Jul 2, 2025
@OutlyingWest OutlyingWest force-pushed the fix/animation-with-tqdm-repairing branch 3 times, most recently from f541eca to c7f6dd6 Compare July 3, 2025 08:48
…ading

- Spinner no longer overlaps with tqdm due to proper threading.Event handling
- Score-P stdout and stderr are read in separate threads with chunked reading
- Animation is disabled in multicell mode when needed
- README.md and error logging improved
@OutlyingWest OutlyingWest force-pushed the fix/animation-with-tqdm-repairing branch from c7f6dd6 to 3e4f44d Compare July 8, 2025 11:39
decoded_line = line.decode(
sys.getdefaultencoding(), errors="ignore"
)
multicellmode_timestamps = self.read_scorep_stdout(proc.stdout, stdout_lock, spinner_stop_event)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what exactly is the purpose of the multicellmode_timestamps in the current version? In the past, we used them to visualize the different cells in the diagrams for coarse-grained measurements, e.g. CPU utilization. Since this was outsourced to ipython extension, I am not sure if we need the multicellmode_timestamps in the current version. I guess, you are disabling the spinner in case multiple cells should be tracked by Score-P (multicellmode) but perhaps, we do not need the timestamps anymore but just a flag for that case, right?

Copy link
Collaborator Author

@OutlyingWest OutlyingWest Jul 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, We do not need multicellmode_timestamps there anymore. I removed them in the last commit #45. Regarding a flag, you are right, it is still necessary to suppress spinner animation in case of %%finalize_multicellmode command execution

if is_enabled:
return BusySpinner(lock)
def create_busy_spinner(lock=None, stop_event=None, is_multicell_final=False):
is_enabled = os.getenv("SCOREP_JUPYTER_DISABLE_PROCESSING_ANIMATIONS") != "1"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a central definition for "True" values that we can use everywhere for checking the env variables?
e.g.
common.py
TRUTHY_VALS=['true', '1', 't']

somewhere in the code:
is_enabled = os.getenv("SCOREP_JUPYTER_DISABLE_PROCESSING_ANIMATIONS").lower() in TRUTHY_VALS

Copy link
Collaborator Author

@OutlyingWest OutlyingWest Jul 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this I'd suggest implementing something similar to Django-style settings.py. Since we already have a number of environment variables (e.g., flags, paths to results, SCOREP_ variables, etc.) and are likely to introduce many more in the future, it makes sense to define a central configuration file - not just for flags, but for all project settings.

The benefits of this approach:

  • All settings can be shared across all parts of application (e.g. kernel, extensions)
  • Environment variables and other settings are defined and parsed only once - in a single place: settings.py.
  • No more repetitive os.getenv(SOME_VAR) calls scattered throughout the codebase - just from conf import settings and settings.SOME_VAR to access any variable.
  • (Optional) Extend settings.py by logging config so we would have full configuration and control on our project in one place.

Finally (if needed) it can be extended to:
conf/
└── settings/
├── init.py - to choose between environments (prod - chosen by default with env var)
├── test.py - for unit tests (animations suppressed, SCOREP_* configured for tests, logging.DEBUG = True)
├── dev.py - e.g. logging.DEBUG = True by default
└── prod.py - e.g. logging.DEBUG = False

@OutlyingWest OutlyingWest force-pushed the fix/animation-with-tqdm-repairing branch from a180104 to e1fa1b5 Compare July 11, 2025 10:28
@OutlyingWest OutlyingWest force-pushed the fix/animation-with-tqdm-repairing branch from e1fa1b5 to 00e237a Compare July 11, 2025 10:30
@elwer elwer merged commit aa93005 into master Aug 4, 2025
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants