feat(fastplotlib): stream Fastplotlib plots via WebSocket using GPU-rendered PNGs#511
Conversation
…fscreen rendering - Introduced `fastplotlib()` backend component for rendering high-performance scatter plots using Fastplotlib and WGPU - Added support for server-side rendering and frontend display via base64-encoded PNG - Created new frontend component `FastplotlibWidget` to embed generated plots - Extended the SDK docs (`docs/sdk/fastplotlib.mdx`) with usage and parameters - Updated docs, CLI guide, and examples to include Fastplotlib usage - Registered `fastplotlib` in the public SDK interface and setup.py dependencies - Bumped package version to 0.2.0
Adds explicit PropTypes validation for label, src, and className props in the FastplotlibWidget component. This resolves ESLint prop-types warnings and improves type safety for future contributors.
- Moved the `fastplotlib` component function to its correct alphabetical position in `components.py` - No functional changes made, just reordering for maintainability and consistency - Reverted version bump in setup.py from 0.2.0 to 0.1.44 to follow the release bump protocol
…ocessing inputs Improves performance by using np.asarray(...).astype(np.float32) instead of np.array(...).tolist() when extracting x, y, and color values. This avoids unnecessary memory allocations when the inputs are already NumPy arrays (e.g. from DataFrames). Functionality remains unchanged.
…ubplot reference Replaced `fig._subplots[0, 0]` with the already defined `subplot` variable to avoid accessing internal attributes. This follows best practices and improves code clarity.
- Introduces a new Fastplotlib component for GPU-accelerated scatter plots using WGPU. - Renders images offscreen and streams PNGs to the frontend via WebSocket using MessagePack. - Implements automatic hashing of input data to skip re-renders when unchanged. - Adds support for deterministic component IDs via label-based hashing. - Updates frontend WebSocket handler to decode MessagePack image messages. - FastplotlibWidget listens for real-time updates and swaps image seamlessly. - Client ID is passed reactively and injected into backend state for routing image streams. This component is optimized for large datasets and improves rendering efficiency while maintaining compatibility with Preswald's declarative component architecture.
- Removed unused imports: base64 and zlib - Fixed double assignment of service in hello.py - Replaced en dash with hyphen in docstring range (0.0–1.0 → 0.0-1.0) - Suppressed RUF006 lint warning for asyncio.create_task with # noqa: RUF006
…ly section Reorganized dependency declarations in `setup.py` to group `fastplotlib[imgui]` and `msgpack` under the server-only section using appropriate platform markers.
…rmance and correctness - Avoid redundant list-to-array conversions when handling color input - Use np.full instead of list multiplication for default RGBA fallback - Replace deprecated rendering/export pipeline with framebuffer draw and alpha blending - Ensure axis visibility and auto-scaling for correct offscreen layout - Improve error handling and logging for image shape validation - Keeps existing WebSocket image streaming and hash-based invalidation logic intact
|
Hi @amrutha97 and @kushalkolar, The Fastplotlib chart now correctly displays the axes. I also revisited the performance concerns and made improvements around color mapping and array handling. The next step could be exposing chart type and configuration options through the component to unlock more of Fastplotlib’s capabilities for users. Looking forward to your feedback! |
|
@BloggerBust great PR, reviewing it rn. Is #491 still relevant? Looking through both this and that one, and noticing that this is just the updated version. Any reason for keeping that one open? or good to close that and work on this? |
|
Thanks so much, @shivam-singhal — really appreciate the kind words! You're right, this PR is the updated version of #491 with WebSocket streaming and improved rendering logic. I’ve closed the old one to keep things tidy. Let’s continue here! 🙌 |
|
Copying from #491 @BloggerBust @kushalkolar what would the delta be here in adding a remote frame buffer / what's the difference between sending raw jpeg bytes over ws and the rfb? |
| from preswald import connect, fastplotlib, get_df, plotly, sidebar, table, text | ||
| from preswald.engine.service import PreswaldService | ||
|
|
||
| service = service = PreswaldService.get_instance() |
There was a problem hiding this comment.
@BloggerBust while this is allowed cause of python's scoping, we don't want users to interface w/ the service itself. Is passing in the client this way needed?
There was a problem hiding this comment.
You're right — directly accessing PreswaldService in user code isn’t ideal. I only used it here to retrieve the client_id, which was needed to route the PNG data over WebSocket to the correct frontend client.
I looked for a more "official" way to access the current session’s client ID from within the script context, but didn’t find one. If there’s a supported or preferred mechanism I missed, I’d be very happy to update the implementation to use it instead.
| } | ||
| console.log('[WebSocket] Component state updated:', { | ||
| componentId: data.component_id, | ||
| value: data.value, |
There was a problem hiding this comment.
@BloggerBust have you confirmed this works? this is changing functionality because a state update will no longer also trigger connections update
There was a problem hiding this comment.
Hey Shivam, thanks for flagging!
Just to clarify, both state_update and connections_update are still handled in the updated code. They were just reorganized into the new switch inside the onmessage async handler.
state_update is now handled here:
https://github.com/StructuredLabs/preswald/pull/511/files#diff-613e8cd1b202632268f0cb482c51eabe6c4155905d03a8677cacbb3124a5a66cR78-R86
connections_update is now handled here:
https://github.com/StructuredLabs/preswald/pull/511/files#diff-613e8cd1b202632268f0cb482c51eabe6c4155905d03a8677cacbb3124a5a66cR104-R107
this._notifySubscribers(data) is still called after the switch, so all message types (including both state and connection updates) trigger downstream updates as before.
Let me know if there’s something subtle I might be missing! Happy to tweak further if needed.
Great question @shivam-singhal! The current WebSocket implementation streams raw PNG image bytes (encoded via MessagePack) to the frontend whenever the data hash changes. This works well for low-frequency, server-rendered plots (like a scatter plot update on rerun), but it's essentially a one-way push-based image delivery. In contrast, a remote framebuffer (RFB) approach like VNC or WebGPU-over-WS would provide a persistent shared rendering surface. The key differences:
In short: current model is simple and works well for static or occasional updates; RFB would be a much more powerful (but also complex) alternative for interactive GPU-backed plots. I haven’t explored implementing RFB directly, so hard to give a concrete delta. But directionally, it’d be a more significant architectural shift compared to the current MessagePack-over-WebSocket model. You’d likely need a separate RFB server, client integration, input handling, and probably more resource management on the backend to maintain persistent rendering contexts. Happy to explore what level of interactivity you’re aiming for — I’d love to help push toward that! |
|
@BloggerBust there's a black space at the bottom because you're not calling the imgui renderer. @shivam-singhal as @BloggerBust mentioned above, a remote frame buffer implementation means sending events from the client back to the server, while the server sends an image byte stream. This is quite involved and you might just want to wait until rendercanvas offers it, it's on the roadmap. Almar is experienced in this. |
|
@BloggerBust @kushalkolar thanks for the detailed responses! I'm thinking of merging in #498 first (@bhavyagada) - it stays consistent with the matplotlib and plotly wrapper apis we have rn. Though agreed on this direction (re messagepack + jpeg bytes for now) - have to think more on how to avoid exposing any ws/preswaldservice information to the users explicitly. @BloggerBust how easy/hard would it be for you to rebase these changes on top of #498 ? |
I can handle the rebase, no problem! |
A RFB doesn't require a separate server other than the server that is already performing the rendering. It would be similar to how you're already using the Offscreen canvas, the difference is that the Offscreen canvas instance would also be able to listen to events sent from the client. Also the offscreen canvas does not incorporate an async loop so you have to manually trigger updates, a RFB would have the option of using an async event loop. Many details on this issue: pygfx/rendercanvas#40
Not sure how preswald works, but my advice would be that users use the fastplotlib API as-is; it can get complicated once you introduce events and interaction. |
|
I’m totally on board with prioritizing consistency in the API surface for now. Merging #498 first makes a lot of sense, especially to establish the baseline wrapper interface that users will expect (just like with Plotly and Matplotlib). I’ll rebase this PR on top of that and adapt the WebSocket + image streaming logic to fit cleanly into the new structure. It should be straightforward. On the topic of avoiding user-facing WebSocket details: I completely agree that we shouldn’t expose internals like client_id or PreswaldService to end users. In this PR, I reached for client_id via service.get_component_state() as a temporary workaround. It felt like the least-bad option given the current session handling. Looking ahead, one idea that might help is introducing a lightweight server-side session context (e.g., current_session.client_id). That would let us access session-specific values from within components without needing user code to pass them around manually. It feels like a clean and scalable way to support more advanced component behavior, like routing WebSocket messages or coordinating future RFB-style updates, while keeping the public API clean. I am definitely open to prototyping or exploring this more if helpful! Also, I really appreciated the notes from @kushalkolar around RFB. That direction sounds powerful, and it’s great to hear that rendercanvas could eventually offer interactivity without surfacing WebSocket details at all. I am excited to keep pushing in that direction once we’ve got a solid non-interactive foundation. Let me know once #498 lands and I’ll get the rebase started right after that! |
|
@BloggerBust @bhavyagada just merged in #498 |
- Refactor `fastplotlib` component backend to stream rendered figures asynchronously over WebSockets. - Update `hello.py` example with multiple Fastplotlib examples (Image Plot, Line Plot, Scatter Plot). - Simplify `FastplotlibWidget` frontend component by directly rendering images instead of using canvas. - Include `imageio` dependency in setup for image processing. This improves performance, simplifies frontend complexity, and provides clearer examples for developers.
Frontend (FastplotlibWidget): Implement persistent image state to prevent flicker when image updates are missing data. Display a subtle warning banner if an update arrives without image data, avoiding sudden disappearance of charts. Adjust loading state to improve initial rendering experience. Backend (fastplotlib component): Include client_id in the figure state hash to automatically re-render images on page refreshes or reconnections. Extract rendering logic into a dedicated asynchronous helper function (render_and_send_fastplotlib) for clarity and concurrent rendering. Add detailed docstrings explaining the rendering, hashing, and WebSocket data transmission logic. These changes ensure smoother, flicker-free UI interactions and robust, efficient backend rendering for Fastplotlib components.
- Fix ambiguous en dash (–) in docstrings (RUF002) - Silence asyncio task reference warning (RUF006) - Improve imports formatting in hello.py
|
Just rebased this branch on top of the Fastplotlib API integration from PR #498. After the rebase, I updated our rendering logic to run asynchronously so multiple Fastplotlib components can render concurrently. This avoids bottlenecks we previously encountered with serial rendering. Also added some fallback behavior to the frontend widget so charts persist even if a future update fails to include image data.
|
Clarifies that Fastplotlib charts are rendered in real time using offscreen GPU acceleration and streamed to the browser via WebSocket. Updated in both the README and introduction page for consistency. Removed mention of scatter plot.
| <img | ||
| src={currentSrc} | ||
| alt={label || 'Fastplotlib chart'} | ||
| className="max-w-full h-auto" | ||
| /> |
There was a problem hiding this comment.
The problem with rendering the plot as an image is that you won't be able to make it interactive in the future. That's why I chose to use canvas, although svg is also an option.
There was a problem hiding this comment.
Thanks for flagging this, @bhavyagada!
I actually started off trying to get the canvas-based approach working, specifically by decoding and drawing PNGs directly onto a <canvas> element over WebSocket. Unfortunately, I ran into several tricky issues during that implementation:
img.onloadwas never firing in some cases (likely due to timing or object URL revocation issues), so plots weren't rendering reliably.- the canvas was only used to render a static image, so while the output element was
<canvas>, it didn’t offer any real interactivity. - Debugging was tough because I had no baseline confirmation that the rendering pipeline worked end-to-end.
So to unblock things, I opted to simplify the rendering path to just use a base64-encoded <img>. Doing this allowed me to get deterministic rendering working, verify image payloads, and improve reliability across refreshes and re-renders.
That said, you're totally right about the importance of interactivity. I think a good long-term plan is to reintroduce WebGPU based rendering in the frontend using Fastplotlib's upcoming runtime support. That would give us actual interactivity (zooming, tooltips, hover, etc.) and better GPU acceleration and not just static PNGs. I'm treating this current version as a baseline MVP and plan to iterate toward full interactivity soon. @shivam-singhal, this isn’t my call though, so feel free to weigh in here.
Let me know if you have thoughts on how to approach that WebGPU integration, I'd love to collaborate!
There was a problem hiding this comment.
@BloggerBust if you want to join forces on prototyping some generalized RFB stuff that could evolve into an implementation for rendercanvas let me know. My browser and JS knowledge is limited but I want to start playing with these ideas.
There was a problem hiding this comment.
Thanks, @kushalkolar,
I’d love to join forces on that! It sounds like a great direction, and collaborating on a generalized RFB-based approach could definitely pave the way for rendercanvas or something like it.
My availability might shift a bit next week depending on how some other commitments evolve, but I’m definitely excited about the idea and will circle back as soon as I know what’s feasible on my end.
Let me know how you're thinking of kicking things off.
There was a problem hiding this comment.
Feel free to post any questions or ideas to the rendercanvas repo! Almar's been playing around quite a bit with optimizing things on the server end, if you're familiar with browsers and JS it would be useful to explore what are the best ways to present stuff (webgpu canvas in the browser vs image elements etc.) if you're experienced on that end.
There was a problem hiding this comment.
Thank you Kushalkolar, I’ll take a look at the rendercanvas repo and see how things are shaping up!
|
@BloggerBust reviewing this shortly! |
shivam-singhal
left a comment
shivam-singhal
left a comment
There was a problem hiding this comment.
Looks good! I'll create a follow up task to clean up the examples code and remove the dependency on the websocket client id
|
great work on this pr @BloggerBust! created a follow up #558 for handling both the output from fastplotlib (or maybe it's pygfx) + remove the dependency on the preswald service client id. |
4 participants
name: Pull Request
about: Create a pull request to contribute to the project
title: 'feat: stream Fastplotlib plots via WebSocket using GPU-rendered PNGs'
labels: 'enhancement'
assignees: ''
Related Issue
Fixes #467
Builds on: #491
Description of Changes
This PR enhances the previously added
fastplotlib()component to support real-time WebSocket streaming of backend-rendered GPU plots.Instead of base64-encoding static images, this implementation encodes PNGs with MessagePack and streams them over WebSocket to the client. This unlocks more efficient rendering and prepares us for future interactive features.
Changes include:
client_idis now injected into thedatadictionary and used to route the image to the correct sessionFastplotlibWidgetto subscribe toimage_updatemessages and display images reactivelyDynamicComponents.jsxto passclientIdand avoid render-time side effects@msgpack/msgpackfrontend dependencygenerate_id_by_label()for consistent ID reuse across rerendersexamples/iris/hello.pyto demonstrate WebSocket-enabled Fastplotlib usageType of Change
Testing
fastplotlib()plots stream to the frontend via WebSocket on data changeChecklist