misc: add script to automatically test lighthouse on a page from devtools #11539
Conversation
I don't remember (or maybe we never knew) why this happens, but the current layout tests do mark |
IIRC it had something to do with the content shell used by layout tests, but it's been a very long time since we looked into it AFAIK. |
There was a problem hiding this comment.
thanks @adamraine this will be great for tracking down our PROTOCOL_TIMEOUT nemesis (and just general explorations into DevTools errors compared to CLI 🎉 )!
| @@ -6,81 +6,13 @@ | |||
| # Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. | |||
| ## | |||
|
|
|||
| set -euo pipefail | |||
There was a problem hiding this comment.
set -u at least still seems relevant, no?
There was a problem hiding this comment.
@patrickhulce Adding set -u causes the DevTools check to fail in GitHub check. Seems to be failing on the yarn --frozen-lockfile step: https://github.com/GoogleChrome/lighthouse/runs/1327351848.
This appears to be a failure of the script but I can't tell where. What do you make of this?
There was a problem hiding this comment.
Hmm maybe set -u was not the problem?
There was a problem hiding this comment.
Yeah that error has been happening for the past several weeks (maybe even months?) and seems like we hit some sort of github network troubles after downloading all of chromium build tooling all the time heh 😄
I think this is unrelated to set -u
There was a problem hiding this comment.
@adamraine any chance you could add a brief note about this somewhere? https://github.com/GoogleChrome/lighthouse/blob/master/lighthouse-core/test/chromium-web-tests/README.md maybe?
(we should probably put together a "so you're trying to test/debug lighthouse in devtools..." doc soon)
There was a problem hiding this comment.
LGTM though a comment in the readme about our devtools suite that this thing exists like @brendankenny suggested would be cool too :)
This script is based heavily on. It's a hacky solution that executes a single DevTools web test which runs lighthouse on a page and then prints the LHR. The solution dividesrun-web-tests.shrun-web-tests.shinto two parts so the phony web test can be inserted after Lighthouse is rolled into DevTools.This should help test #11508 in GCP.
Currently, the script always has a
NO_SCREENSHOTSerror for Speed Index. Will this be a problem when searching forPROTOCOL_TIMEOUTerrors?Fixes #11529
Part of #6512