Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[windows] wezterm hangs randomly, mostly but not exclusively, when closing panes. #5882

Open
FrancescElies opened this issue Jul 31, 2024 · 8 comments · May be fixed by #5977
Open

[windows] wezterm hangs randomly, mostly but not exclusively, when closing panes. #5882

FrancescElies opened this issue Jul 31, 2024 · 8 comments · May be fixed by #5977
Labels
bug Something isn't working

Comments

@FrancescElies
Copy link

FrancescElies commented Jul 31, 2024 •
edited
Loading

What Operating System(s) are you seeing this problem on?

Windows

WezTerm version

20240203-110809-5046fc22

Happens with latest master wezterm 20240807-131622-ee063330 too (see last stack traces in this issue)

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

No, but I will now download latest nightly and see if it happens there too, but I am guessing there might be some weird interaction with windows git credential manager going on.

Describe the bug

When closing a pane wezterm hangs.
If I remember correctly happened to me too while doing something else.
I'll be glad to get my hands dirty to debug this with some hints :)

To Reproduce

I don't know yet, so far I could not identify any pattern, just opening an issue to collect ideas,

Configuration

I don't think my config has to do with wezterm hanging since it seems to hang in git_credential_username_new (see below) but added a link to it.

If there might be anything to blame from my config would be this part where there is some logic to close a wezterm or a nvim pane.

Logs

I couldn't see any errors in my logs except the following,

ERROR  wezterm_mux_server_impl::local > encoding PDU to client: writing pdu data buffer: An existing connection was forcibly closed by the remote host. (os error 10054)

Anything else?

I attached a debugger to wezterm after it hanged and could see the following.

image

Wezterm seems to hang in git_credential_username_new, I guess is this is this function from libgit2, but this is a wild guess from my side
@FrancescElies FrancescElies added the bug Something isn't working label Jul 31, 2024
@FrancescElies FrancescElies changed the title [windows] wezterm hangs randomly mostly when closing panes [windows] WezTerm hangs randomly, mostly but not exclusively, when closing panes. Jul 31, 2024
@FrancescElies FrancescElies changed the title [windows] WezTerm hangs randomly, mostly but not exclusively, when closing panes. [windows] wezterm hangs randomly, mostly but not exclusively, when closing panes. Jul 31, 2024
@RasmusN
Copy link

RasmusN commented Aug 7, 2024

Perhaps same as these two?

#5496
#5432

This was referenced Aug 7, 2024
Open
Open
@FrancescElies
Copy link
Author

Here a dump
image

It seems to hang at ntwaitforsingleobject

@FrancescElies
Copy link
Author

FrancescElies commented Aug 8, 2024 •
edited
Loading

I built wezterm in debug mode and run it, managed to hang it again and generated a memory dump.

image

The full stack trace doesn't fit in the images above, below the copy pasta.

PROCESS_NAME:  wezterm-gui.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE_STR:  80000003

STACK_TEXT:  
0000001d`828f8e58 00007ffd`86db9a9e     : 00007ff6`33eb2a28 00007ff6`333e4898 00000000`00000000 00000000`00000000 : ntdll!NtWaitForSingleObject+0x14
0000001d`828f8e60 00007ffd`7d986390     : 0000016c`1d14c530 00007ffd`8942c350 00007ff6`00000000 00000000`00000670 : KERNELBASE!WaitForSingleObjectEx+0x8e
0000001d`828f8f00 00007ff6`30615308     : 00000000`828f0000 00007ff6`30e1472e 0000016c`1d14c02f 00000000`00000014 : conpty!ConptyClosePseudoConsole+0x60
0000001d`828f8f30 00007ff6`3061328e     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : wezterm_gui!portable_pty::win::psuedocon::impl$2::drop+0x28
0000001d`828f8f70 00007ff6`306130f3     : 00000000`00000000 0000001d`828fa480 00000000`00000000 00000000`00000000 : wezterm_gui!core::ptr::drop_in_place<portable_pty::win::psuedocon::PsuedoCon>+0xe
0000001d`828f8fa0 00007ff6`3061377e     : 00000000`00000000 ffffffff`fffffffe 00000000`00000001 00007ff6`2f9692db : wezterm_gui!core::ptr::drop_in_place<portable_pty::win::conpty::Inner>+0x23
0000001d`828f8ff0 00007ff6`306137e2     : 00000000`00000000 00000000`00000000 00000000`00000000 0000016c`1d14c030 : wezterm_gui!core::ptr::drop_in_place<core::cell::UnsafeCell<portable_pty::win::conpty::Inner> >+0xe
0000001d`828f9020 00007ff6`3062e16f     : 00000000`00000010 00000000`00000008 00000000`00000010 00000000`00000008 : wezterm_gui!core::ptr::drop_in_place<std::sync::mutex::Mutex<portable_pty::win::conpty::Inner> >+0x12
0000001d`828f9050 00007ff6`30613f0a     : 00000000`00000008 00007ff6`2f892194 0000016c`1d208b20 00000000`00000001 : wezterm_gui!alloc::sync::Arc<std::sync::mutex::Mutex<portable_pty::win::conpty::Inner>,alloc::alloc::Global>::drop_slow<std::sync::mutex::Mutex<portable_pty::win::conpty::Inner>,alloc::alloc::Global>+0x1f
0000001d`828f90a0 00007ff6`306120de     : 0000016c`1d14c030 00007ff6`2f87bfd7 0000016c`1d208b28 00007ff6`2f87f8b9 : wezterm_gui!alloc::sync::impl$35::drop<std::sync::mutex::Mutex<portable_pty::win::conpty::Inner>,alloc::alloc::Global>+0x9a
0000001d`828f9140 00007ff6`3061339e     : 0000016c`1d208b10 0000016c`1d208b30 0000016c`1d14c030 00007ff6`30613190 : wezterm_gui!core::ptr::drop_in_place<alloc::sync::Arc<std::sync::mutex::Mutex<portable_pty::win::conpty::Inner>,alloc::alloc::Global> >+0xe
0000001d`828f9170 00007ff6`2f88bbfc     : 0000001d`828f91f0 00007ff6`2f889a38 0000001d`828f9228 00007ff6`2f87d6ae : wezterm_gui!core::ptr::drop_in_place<portable_pty::win::conpty::ConPtyMasterPty>+0xe
0000001d`828f91a0 00007ff6`2f87c58e     : 0000016c`05e74280 0000016c`05e74280 0000016c`05e74280 00000000`00000000 : wezterm_gui!core::ptr::drop_in_place<alloc::boxed::Box<dyn$<portable_pty::MasterPty>,alloc::alloc::Global> >+0x3c
0000001d`828f9200 00007ff6`2f880de2     : 00000000`00000000 0000016c`1d206f50 0000016c`1d208b30 ffffffff`fffffffe : wezterm_gui!core::ptr::drop_in_place<core::cell::UnsafeCell<alloc::boxed::Box<dyn$<portable_pty::MasterPty>,alloc::alloc::Global> > >+0xe
0000001d`828f9230 00007ff6`2f889392     : 0000016c`09b47428 0000016c`09b47500 00000000`00000001 0000016c`09b47350 : wezterm_gui!core::ptr::drop_in_place<lock_api::mutex::Mutex<parking_lot::raw_mutex::RawMutex,alloc::boxed::Box<dyn$<portable_pty::MasterPty>,alloc::alloc::Global> > >+0x12
0000001d`828f9260 00007ff6`2f88904d     : 0000016c`080e2310 0000016c`09b47358 00000000`00000007 0000016c`09b47428 : wezterm_gui!core::ptr::drop_in_place<mux::localpane::LocalPane>+0x52
0000001d`828f92b0 00007ff6`2f86b79a     : 0000016c`1d206f40 0000016c`0799c210 00000000`00000000 00000000`00000000 : wezterm_gui!core::ptr::drop_in_place<dyn$<mux::pane::Pane> >+0x2d
0000001d`828f9300 00007ff6`2f88fe51     : 00007ff6`33eaffe0 00000000`00000000 00007ff6`33560b88 00007ff6`33560bc0 : wezterm_gui!alloc::sync::Arc<dyn$<mux::pane::Pane>,alloc::alloc::Global>::drop_slow<dyn$<mux::pane::Pane>,alloc::alloc::Global>+0x3a
0000001d`828f9360 00007ff6`2f88aebe     : 0000016c`1d206f40 00007ff6`3241e960 ffffffff`fffffffe 00000000`00000000 : wezterm_gui!alloc::sync::impl$35::drop<dyn$<mux::pane::Pane>,alloc::alloc::Global>+0xb1
0000001d`828f9410 00007ff6`2de43689     : ff5cf4d3`3701519e 00007ff6`31b2e698 00000000`00000001 00000000`00000001 : wezterm_gui!core::ptr::drop_in_place<alloc::sync::Arc<dyn$<mux::pane::Pane>,alloc::alloc::Global> >+0xe
0000001d`828f9440 00007ff6`2de7098e     : 0000016c`00000002 0000016c`09994350 0000001d`828fa088 00007ff6`2e0a311e : wezterm_gui!wezterm_gui::termwindow::TermWindow::key_event_impl+0x629
0000001d`828f9f10 00007ff6`2e26ebb1     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : wezterm_gui!wezterm_gui::termwindow::TermWindow::dispatch_window_event+0x47e
0000001d`828faa30 00007ff6`2fca49a5     : 00000000`00000000 0000001d`828fae49 0000016c`05e70000 00007ffd`8942cc9a : wezterm_gui!wezterm_gui::termwindow::impl$4::new_window::async_fn$0::closure$7+0xd1
0000001d`828fad90 00007ff6`2fc8d123     : 00000000`00000000 00000000`00000000 00005228`23b2cbd9 0000016c`07dcfaa8 : wezterm_gui!alloc::boxed::impl$49::call_mut<tuple$<enum2$<window::WindowEvent>,ref$<window::os::windows::window::Window> >,dyn$<core::ops::function::FnMut<tuple$<enum2$<window::WindowEvent>,ref$<window::os::windows::window::Window> >,assoc$<Output,tuple$<> > > >,alloc::alloc::Global>+0x55
0000001d`828faec0 00007ff6`2fc84169     : 00000000`00000000 00007ffd`877542b1 0000016c`07e17c20 0000016c`09290068 : wezterm_gui!window::WindowEventSender::dispatch+0x113
0000001d`828fb200 00007ff6`2fc84c60     : 00000000`00000000 00007ffd`8942d68a 00000000`00080006 00000000`00000120 : wezterm_gui!window::os::windows::window::key+0x2299
0000001d`828fcbb0 00007ff6`2fcc8756     : 0000001d`828fceb0 00000000`00000000 00000000`00000000 00000000`00000000 : wezterm_gui!window::os::windows::window::do_wnd_proc+0x340
0000001d`828fcd80 00007ff6`2fd291da     : 0000016c`00000000 00007ffd`069e65af 0000016c`083d6800 00000000`00000000 : wezterm_gui!window::os::windows::window::wnd_proc::closure$0+0x46
0000001d`828fcdf0 00007ff6`2fd2a8d3     : 00000000`00000000 0000001d`828fcf30 00000000`00000000 00000100`00000101 : wezterm_gui!std::panicking::try::do_call<window::os::windows::window::wnd_proc::closure_env$0,isize>+0x4a
0000001d`828fce70 00007ff6`2fd2871e     : 00000000`00000000 00010101`01000000 00000001`01008100 00000000`00000000 : wezterm_gui!std::panicking::try::do_catch<core::panic::unwind_safe::AssertUnwindSafe<async_task::raw::impl$3::run::closure_env$1<enum2$<async_executor::impl$5::spawn_inner::async_block_env$0<tuple$<>,enum2$<window::os::windows::connection::impl$1::with_window_inner::async_block_env$0<tuple$<>,window::os::windows::window::impl$41::set_maximize_button_position::closure_env$0> > > >,tuple$<>,async_executor::impl$5::schedule::closure_env$0,tuple$<> > >,enum2$<core::task::poll::Poll<tuple$<> > > >+0xa3
0000001d`828fcec0 00007ff6`2fcc5961     : 00000000`00000001 00000000`00000000 00000000`00000000 00000000`00000001 : wezterm_gui!std::panicking::try<isize,window::os::windows::window::wnd_proc::closure_env$0>+0x7e
0000001d`828fcf60 00007ff6`2fc85057     : 00000000`0271de1a 00000000`00000000 0000001d`828fd050 00000000`00000000 : wezterm_gui!std::panic::catch_unwind<window::os::windows::window::wnd_proc::closure_env$0,isize>+0x11
0000001d`828fcf90 00007ffd`88d389a1     : 0000016c`08014009 00007ff6`2fd1c76c 00000000`00000000 00000000`80000022 : wezterm_gui!window::os::windows::window::wnd_proc+0x57
0000001d`828fd100 00007ffd`88d381eb     : 00000000`452906c4 00007ff6`2fc85000 00000000`452906c4 00000000`00000104 : user32!UserCallWinProcCheckWow+0x2d1
0000001d`828fd260 00007ffc`fa0e1eb4     : 0000016c`08306160 00000000`452906c4 00007ff6`2fc85000 00007ffd`88d454bf : user32!CallWindowProcW+0x8b
0000001d`828fd2b0 00007ffd`88d389a1     : 00000000`00000000 0000001d`828fd580 00000000`80000022 00000000`80000022 : opengl32!wglWndProc+0x294
0000001d`828fd320 00007ffd`88d38461     : 00000000`00000000 00007ffc`fa0e1c20 00000000`452906c4 0000001d`828fd540 : user32!UserCallWinProcCheckWow+0x2d1
0000001d`828fd480 00007ff6`2fc8a83e     : 0000001d`828fd540 0000001d`00000000 0000016c`05e8b660 00000000`00000000 : user32!DispatchMessageWorker+0x1f1
0000001d`828fd500 00007ff6`2e03a6e9     : 01000000`00000001 00000000`00000000 00000000`00000008 00000000`00000000 : wezterm_gui!window::os::windows::connection::impl$0::run_message_loop+0x11e
0000001d`828fd670 00007ff6`2ddcbfd2     : 00000000`00000008 00007ff6`2fbb5689 0000001d`828fda20 00000000`00000000 : wezterm_gui!wezterm_gui::frontend::GuiFrontEnd::run_forever+0x29
0000001d`828fd6c0 00007ff6`2ddd3e40     : 0000016c`05e75250 0000001d`828feb69 0000016c`05e70000 00730073`0000020d : wezterm_gui!wezterm_gui::run_terminal_gui+0xca2
0000001d`828fe830 00007ff6`2ddcc9ff     : 0000016c`05ea0b80 00007ffd`8942aff1 000067af`c7c31691 0000016c`05e70000 : wezterm_gui!wezterm_gui::run+0xb80
0000001d`828ffa30 00007ff6`2e09b31b     : 0000001d`828ffb00 00007ff6`311302f4 00000000`00000000 00000000`00000000 : wezterm_gui!wezterm_gui::main+0x1f
0000001d`828ffa70 00007ff6`2e0e2f0e     : 0000001d`828ffb10 00007ff6`31130a08 0000016c`00000000 0000016c`05e8b660 : wezterm_gui!core::ops::function::FnOnce::call_once<void (*)(),tuple$<> >+0xb
0000001d`828ffab0 00007ff6`2e01ff91     : 0000016c`05e8b660 0000016c`05eb75a0 0100001d`828ffb10 01ffffff`fffffffe : wezterm_gui!std::sys_common::backtrace::__rust_begin_short_backtrace<void (*)(),tuple$<> >+0xe
0000001d`828ffaf0 00007ff6`3112fe0e     : 0000016c`05e8b660 00000000`00000000 0000016c`05e85130 0000001d`828ffb50 : wezterm_gui!std::rt::lang_start::closure$0<tuple$<> >+0x11
0000001d`828ffb30 00007ff6`2e01ff6a     : 00000000`00000000 00007ff6`3333b800 0000001d`82683000 00000000`00000000 : wezterm_gui!std::rt::lang_start_internal+0x7e
0000001d`828ffc10 00007ff6`2ddd7769     : 00007ff6`31a89018 00000000`00000000 00000000`00000000 00007ff6`31a89020 : wezterm_gui!std::rt::lang_start<tuple$<> >+0x3a
0000001d`828ffc80 00007ff6`31a3930c     : 00000000`00000000 00007ff6`31a39385 00000000`00000000 00000000`00000000 : wezterm_gui!main+0x19
0000001d`828ffcb0 00007ffd`88c6257d     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : wezterm_gui!__scrt_common_main_seh+0x10c
0000001d`828ffcf0 00007ffd`8944af28     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x1d
0000001d`828ffd20 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x28

@FrancescElies
Copy link
Author

FrancescElies commented Aug 8, 2024 •
edited
Loading

It seems to hang in wezterm_gui::termwindow::TermWindow::key_event_impl, where exactly not clear to me ... probably I will end up adding some logging.

Finally if I understand the dump correctly it hangs at portable_pty::win::psuedocon::impl$2::drop+0x28

See src\win\psuedocon.rs:43
image

Which calls a shared library function ClosePseudoConsole.

@RasmusN maybe you have an idea of what's going on? Because me, I reached the limit of where I am able to go.

@RasmusN
Copy link

RasmusN commented Aug 8, 2024

Rust is unfortunately outside of my comfort zone so I'm unable to help analyzing the code.

I've made efforts to try to find a way to consistently replicate the crash but I haven't even been able to do that. Have you found a way to trigger the crash?

@FrancescElies
Copy link
Author

I didn't manage to consistently replicate that

The rust syntax doesn't matter that much @RasmusN if you can be a partner in this journey I will be happy.

We have this macro which expands into something else.
image

See part of this expansion below
image

I don't understand all the details, and at the moment I also don't care, the interesting part is that is that we call ClosePseudoConsole from a dylib, dynamic library, a dll.

This dll is this one down here
image

What this dll does and how it does it and why the call to close hangs, no idea.
But hey, here it is Admiral Grace Hopper, ready to give us a lesson on terminals in windows-command-line-introducing-the-windows-pseudo-console-conpty

Not sure if the article is of any help but to understand the problem probably we will have to understand conpty.dll better and see why it sometimes hangs.

Maybe conpty.dll is expecting something from us, maybe wezterm sends a weird ansi escape thing to it and it hangs, these are wild guesses, I have no idea what about those topics.

@FrancescElies
Copy link
Author

FrancescElies commented Aug 9, 2024 •
edited
Loading

This issue might be similart to microsoft/terminal#1810

I still have to read this in detail but this comment points out to a race condition, which kind of fits with this problem being hard to reproduce.

Apparently this is because This is because the ConPTY is being started with CreatePseudoConsole() with the flag PSEUDOCONSOLE_INHERIT_CURSOR (which wezterm does)

Here the start call in wezterm.
image

@FrancescElies
Copy link
Author

I also see this flag was added three months ago, in this commit

@Cammisuli maybe you could shed some light into this?

@FrancescElies FrancescElies mentioned this issue Aug 9, 2024
Closed
@FrancescElies FrancescElies linked a pull request Aug 16, 2024 that will close this issue
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(windows): closing panes sometimes freezes wezterm FrancescElies/wezterm
2 participants
@FrancescElies @RasmusN