Get yarn test-devtools to work on linux

[connorjclark]

This never worked because there was a crash, and it was decided to just focus on Mac. However, @adamraine needs to run this on GCP to do collect data at scale, so let's revisit.

Currently yarn test-devtools errors loudly complaining that "crashpad_handler does not exist". This is easy to silence with --disable-breakpad to the webserver python program. However, the crash is still there, this just removes the complaining. (Note: unnecessary, but if we wanted to we could download the crashpad handler from chrome-PLATFORM.zip and place it next to the content shell).

So with that said, here's the true error. Still investigating.

[501216:501216:1124/122624.392648:WARNING:vaapi_wrapper.cc(534)] VAAPI video acceleration not available for disabled
[501132:501132:1124/122624.384668:FATAL:platform_font_skia.cc(97)] Check failed: InitDefaultFont(). Could not find the default font
#0 0x5633328ce399 base::debug::CollectStackTrace()
#1 0x56333284af13 base::debug::StackTrace::StackTrace()
#2 0x56333285a3f0 logging::LogMessage::~LogMessage()
#3 0x56333285afbe logging::LogMessage::~LogMessage()
#4 0x56333345a3af gfx::PlatformFontSkia::PlatformFontSkia()
#5 0x56333345be4b gfx::PlatformFont::CreateDefault()
#6 0x56333344a3ce gfx::Font::Font()
#7 0x56333237931a content::RenderViewHostImpl::GetPlatformSpecificPrefs()
#8 0x5633324acf96 content::WebContentsImpl::SyncRendererPrefs()
#9 0x5633328390a3 content::Shell::CreateShell()
#10 0x563332839727 content::Shell::CreateNewWindow()
#11 0x56333617a66e content::WebTestControlHost::PrepareForWebTest()
#12 0x56333614b383 content::WebTestBrowserMainRunner::RunBrowserMain()
#13 0x563332835e50 content::ShellMainDelegate::RunProcess()
#14 0x563331f1090c content::ContentMainRunnerImpl::RunServiceManager()
#15 0x563331f10533 content::ContentMainRunnerImpl::Run()
#16 0x563331275d74 content::RunContentProcess()
#17 0x56333127675c content::ContentMain()
#18 0x5633305d44cc main
#19 0x7f2c44825cca __libc_start_main
#20 0x5633305d436a _start

Received signal 6
#0 0x5633328ce399 base::debug::CollectStackTrace()
#1 0x56333284af13 base::debug::StackTrace::StackTrace()
#2 0x5633328cdf3b base::debug::(anonymous namespace)::StackDumpSignalHandler()
#3 0x7f2c45ee1140 (/lib/x86_64-linux-gnu/libpthread-2.31.so+0x1413f)
#4 0x7f2c4483adb1 gsignal
#5 0x7f2c44824537 abort
#6 0x5633328ccec5 base::debug::BreakDebugger()
#7 0x56333285a862 logging::LogMessage::~LogMessage()
#8 0x56333285afbe logging::LogMessage::~LogMessage()
#9 0x56333345a3af gfx::PlatformFontSkia::PlatformFontSkia()
#10 0x56333345be4b gfx::PlatformFont::CreateDefault()
#11 0x56333344a3ce gfx::Font::Font()
#12 0x56333237931a content::RenderViewHostImpl::GetPlatformSpecificPrefs()
#13 0x5633324acf96 content::WebContentsImpl::SyncRendererPrefs()
#14 0x5633328390a3 content::Shell::CreateShell()
#15 0x563332839727 content::Shell::CreateNewWindow()
#16 0x56333617a66e content::WebTestControlHost::PrepareForWebTest()
#17 0x56333614b383 content::WebTestBrowserMainRunner::RunBrowserMain()
#18 0x563332835e50 content::ShellMainDelegate::RunProcess()
#19 0x563331f1090c content::ContentMainRunnerImpl::RunServiceManager()
#20 0x563331f10533 content::ContentMainRunnerImpl::Run()
#21 0x563331275d74 content::RunContentProcess()
#22 0x56333127675c content::ContentMain()
#23 0x5633305d44cc main
#24 0x7f2c44825cca __libc_start_main
#25 0x5633305d436a _start
  r8: 0000000000000000  r9: 00007fffb5beaf20 r10: 0000000000000008 r11: 0000000000000246
 r12: 00003249fade1000 r13: aaaaaaaaaaaaaaaa r14: 00003249fade1010 r15: 00007fffb5beb9c0
  di: 0000000000000002  si: 00007fffb5beaf20  bp: 00007fffb5beb170  bx: 00007f2c43d25b80
  dx: 0000000000000000  ax: 0000000000000000  cx: 00007f2c4483adb1  sp: 00007fffb5beaf20
  ip: 00007f2c4483adb1 efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000
 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.

[paulirish]

I remember seeing this with puppeteer stuff. puppeteer/puppeteer#391 has the sameish stack trace

https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md has a bunch of mitigations. look for "font"

[connorjclark]

After finding no solutions for content shell's font issue, I attempted to run the webtests in Docker, hoping the same issue wouldn't occur there (or at least, might be easier to fix). Unfortunately the same issue happened there and I haven't found a solution in Docker either.

It was non trivial to get this far with Docker so I've pushed the code to webtests-docker https://github.com/GoogleChrome/lighthouse/blob/webtests-docker/lighthouse-core/test/chromium-web-tests/Dockerfile. Maybe they'll be reason to revisit using Docker in the future.

[connorjclark]

@adamraine I'm wondering if for purposes of #11539 / #11529 we could just use test-page-devtools.sh locally with a bunch of URLs, instead of trying to get things to work in GCP. That may be good enough for detecting changes in protocol timeout rate. Currently am not hopeful we'll get webtests working in linux.

[adamraine]

The current script runs through a bunch of overhead for every single URL. I'm thinking we create another version the script to handle multiple URLs efficiently. Essentially the script would inject a single "web test" which runs Lighthouse on every URL.

Wdyt?

[connorjclark]

Where is most of the overhead?

1. how much time do all the download scripts take when everything is already downloaded?
2. overhead of "$SCRIPT_DIR/web-test-server.sh" ?

Given the results are held in memory until the test ends (TestRunner.addResult(JSON.stringify(lhr));), would probably need to batch x at a time (x = 25? just so a random crash doesn't waste tons of work)

Alternatively, instead of a single big webtest, maybe create one for every url?

[adamraine]

how much time do all the download scripts take when everything is already downloaded?

These don't take much time once everything is downloaded.

overhead of $SCRIPT_DIR/web-test-server.sh ?

This is what I was mostly referring to. There is overhead to set up the server, wait for the server to be ready, and run the "web test".

Alternatively, instead of a single big webtest, maybe create one for every url?

This sounds like a good idea. It reduces the overhead per url to whatever run_web_tests.py does between tests.

[connorjclark]

Still doesn't work on linux, but the need isn't really there so closing.

[connorjclark]

got a better crash log

which led me to https://bugs.chromium.org/p/chromium/issues/detail?id=925161

tried --no-xvfb but still no luck even on desktop linux. will try those other flags tomorrow

[connorjclark]

even on desktop linux

actually I was on a cloud vm w/ no gpu.

For some reason these web tests work on mac in GHA but not ubuntu, and if the reason is "because no gpu" then how the heck does the mac one work? none of GHA's VMs have gpus.

[connorjclark]

I noticed that the content shell we are downloading doesn't have webgl enabled (and I could find no flag that would change that). https://webglreport.com/ is useful for a quick test. I built content_shell locally on my linux cloudtop and webgl is enabled.

we should ask if this is expected behavior of the https://source.corp.google.com/piper///depot/google3/testing/web/browsers/chrome-linux.json distribution

[connorjclark]

https://groups.google.com/a/chromium.org/g/graphics-dev/c/wn54-oYmRpM

[connorjclark]

https://bugs.chromium.org/p/chromium/issues/detail?id=1320508

[paulirish]

While we completely changed the devtools tests we run.... I believe this still isn't the case.
We run our smoketests with the devtools runner. But we do not run the devtools repo's LH e2e tests in our CI.
Though we should. This should be significantly easier now than it was before.

[paulirish]

the e2e tests run on linux. (the webtests never did)

https://github.com/GoogleChrome/lighthouse/actions/runs/3992078236/workflow#L81-L118