KittyCAMP 2024 lessons learned

[franknoirot]

Our annual company meeting was a big success in a lot of ways, and it was exciting to see users actually trying to use our CAD application nearly "in the wild".

Our team took a lot of notes individually over the course of the week, both about the user feedback they saw on the modeling app and on pieces of the systems it depends on. I'm starting this discussion here so we can see our cleaned-up notes together, and derive some more meaning and next steps from them.

Zoo teammates, please include any notes you can as a single comment in a numbered list (which is easier to refer back to), and please refer to each other's notes in follow-up comments with the numbered item and any sub-items, eg "3.viii.a", or include a block quote if that's easier.

@Irev-Dev will be at work tomorrow and Wednesday so I'd love to have some discussion underway here to get him up to speed on this illuminating event!

[franknoirot]

1. Releasing is far too hard at the moment. Need to:
   1. Block merge on out-of-date or red (done)
   2. Get all tests to not be flaky at all
      1. Jon has some good ideas for how to not require an engine connection for most
      2. Consider removing automatic commit of snapshot images since that halts the pipeline, and requires a re-run? If a snapshot test is flaky it's really painful.
   3. What, if any, of our playwright tests could be nightly and non-blocking on PRs?
   4. We have unit tests that will occasionally timeout and run over
   5. NodeJS registration step can fail
   6. MacOS builds are recently flaky due to an electron-builder issue
   7. We need to segregate between the release-related GH Action items and the test-related ones (Pierre noted this while doing a quick debugging of the v0.26.0 release CI)
   8. We should consider doing blue-green release strategies for dependencies like the engine that can break underneath us, or break on rollback.
   9. We should set up a mechanism to alert devs when their merged PR fails some CI after merging into main so that they can retry it, to preserve the sanctity of main's green status.
2. KCL
   1. Need a arc by center function (makeathon lesson)
   2. Need an ellipse function (makeathon lesson)
   3. Need an elliptical curve function (makeathon lesson)
   4. Need a three-point arc function (makeathon lesson)
   5. Hardest current problem with KCL is how to represent dimensions that span multiple sketch segments
   6. Need an explicit representation of points (Allen discussion, investor Solidworks walkthrough)
   7. Need an explicit representation of constraints (Allen discussion, Onshape investigations)
   8. Need these because codemods should be as small and straightforward as possible. Our current trajectory will require ever-increasingly complex codemods, which will make end users even less likely to hand-write code.
   9. All sketch lines should be defined using relative coordinates unless a "fixed" constraint is applied
3. User requests today are primarily to get the app to behave like traditional CAD
   1. Snapping behavior in sketch mode (2 users)
   2. Transform bodies and sketches (2 users)
   3. Highlighting sketch regions (makeathon participant)
   4. Don't fade out engine in sketch mode (investor)
   5. Don't orient to sketch in sketch mode (investor)
   6. Constraint snapping for horizontal, perpendicular, etc (2 users)
   7. Need some way to hide previously-created geometry while you work on some detail
      1. I'm categorizing this here because it would also fall under a "feature tree" or "operation list" functionality
   8. Currently no visual for offset plane, even when defined in code (makeathon participant)
   9. Solid 2Ds should be a color other than the bg color, multiples should be different colors. We can do this today with engine capabilities (makeathon participant)
   10. 2D offset of arcs would remove a lot of the math we had to manually do (makeathon participant)
   11. Don’t change button to "edit sketch" on the selection of the face. User almost always wants to sketch on the face. Relegate "edit sketch" to right-click menu and feature tree (investor) (The "Sketch" toolbar button should always trigger Start Sketch, never Edit Sketch #4273)
4. Onboarding
   1. Oscar (makeathon user) asked to skip
   2. An investor walked me through his experience (it was painful)
   3. Way too much text
   4. Back button, not dismiss button (put dismiss in upper-right corner)
   5. Download CTA should be in like the website nav (shoutout Lee for saying this a lot)
   6. Onboarding is doing four things at once:
      1. Hey what is this app? (this part is not necessary)
      2. Set up / establish settings
      3. Some stuff like investor pitch
      4. Play with a thing to learn how to use it
   7. Open-source might be scary to enterprise users and mechies, they don’t want to know that. "Does that mean my data is going to get stolen?"
   8. Way too long. Needs to be simplified and purposeful, preferably hint-based.
5. Minor or otherwise uncategorized UX notes for modeling app
   1. Sketch on XZ plane would occasionally rotate 45º when entered
   2. .step files are not showing up in the project directory
   3. In-app support form via a command bar flow
   4. Text-to-CAD health bar of remaining in the status bar when we get that
   5. Add Text-to-CAD prompt history to modeling app
   6. When hitting Top view, the X axis should be to the right
   7. XYZ labels in the meantime until we have a view cube
   8. Tooltips not disappearing on click of sketch tools confused two separate users
   9. Command bar search is too fuzzy, and somehow "text" doesn't show text-to-cad first
   10. Gear icon or hamburger menu for user settings, not an avatar (investor)
   11. Need system menus, need to figure out how to not toggle it all the time needlessly with Alt key on Windows (makeathon participant, Ryan made an issue a while back)
   12. Support click and drag as well as click and click for sketch tools (investor showed Solidworks UX)
   13. Add a timeout on the make command with error toast (makeathon participant and Jordan, when machine API was resetting)
   14. Jordan's QR code example breaks the variables pane
6. Non-modeling app feedback
   1. "Assemble these for me" is a great ML task
   2. "Complete the design, given an existing file" is a great ML task
      1. I would call this "draw the rest of the horse"
   3. Non-manifold empty scene generated by Oscar (makeathon participant's) Text-to-CAD
   4. Typing "pattern" doesn't show pattern functions in website docs
   5. The docs have the sample code in context, but people don't know what to copy out. Maybe we highlight the line of code containing the concept being detailed? We could do it procedurally.

[franknoirot]

Adding this under point 1.viv

We should set up a mechanism to alert devs when their merged PR fails some CI after merging into main so that they can retry it. We really want to preserve the sanctity of main's green status.

One idea for this: maybe we send a slack message when things fail on main (and send to Axiom)?

[Irev-Dev]

1.ii.b, this was always a "least bad situation", I still kinda back it but happy for someone to come up with something else.
We could have the test just fail instead when snapshots don't match tomorrow just by removing --update-snapshots but I don't see why for two reasons. 1. you than have to figure out a way of updating the snapshots, which ultimately does still need to be done in CI because the snapshots put the OS in the snapshot name, so can't really be done locally. 2. it doesn't stop you from re-running the tests, if the snapshot tests fail, vs auto-commiting, what's the difference? you either need to re-run or fix something right?

1.iii feels like an anti-pattern, delaying feedback that you introduced a regression is a pretty big loss. I think the suite would have to be really slow to justify this. Totaly separate note, I think we could figure out what tests are safe to run in parallel (with workers) and give them tags such that we can run more in parallel, but we still have the fall back for the tests that still needed to be run by themselves, i.e. I think there's more levers we can pull to speed up the tests.

1.ix I don't understand, don't we want to stop failing CI before it's merged?

[franknoirot]

2. it doesn't stop you from re-running the tests, if the snapshot tests fail, vs auto-commiting, what's the difference? you either need to re-run or fix something right?

This is a good point. I think making these not flaky and staying that way might be the hard but correct problem.

1.iii feels like an anti-pattern, delaying feedback that you introduced a regression is a pretty big loss.

You're right. We talked today about maybe not needing PW tests on release branches, since if main is green it has to be too (it's the equivalent of a PR that changes a comment).

1.ix I don't understand, don't we want to stop failing CI before it's merged?

Ah yes, I'm referring to the CI run that occurs on main after a merge. Flakes there can lead to a trail of red Xes even when all the PRs have merged green. Here's an example:

The PR for #4287 merged green:

The CI check on main after merging failed due to a flake, making it harder to diagnose where things actually went awry in the commit history:

Totaly separate note, I think we could figure out what tests are safe to run in parallel (with workers) and give them tags such that we can run more in parallel, but we still have the fall back for the tests that still needed to be run by themselves, i.e. I think there's more levers we can pull to speed up the tests.

Love this idea.

[jtran]

Some notes I took during the makeathon. I de-duplicated from the above.

1. Confirm extrude. People tried to click that, but it's just a label.
2. "Is there a way to snap to the origin?"
3. Stdlib function for projecting a point into a sketch. AKA pierce.
4. Would be nice to add constraints on final segment of a sketch that closes. Can't because it's already fully constrained. But a user wanted to set a vertical constraint. I think this would be good to capture their intent in case something else changed.
5. Reset View should be in the Command Palette Reset view from command palette #4189
6. Tangential arc tool should write relative commands
7. Slot/Pill shape sketch tool

[franknoirot]

6. Tangential arc tool should write relative commands

I want to check in just in case @Irev-Dev remembers some reason that tangential arc needs to be absolute under the current paradigm. Otherwise I think making the default standard library function that is applied should be trivial.

[Irev-Dev]

3. Are we talking about the same thing here? https://kittycadworkspace.slack.com/archives/C06P9L5MCUD/p1727830062908229

Frank, I kinda think we need a bunch of arc functions the same way as there are lots of straight segment functions, that helps with the code-mods constraints, but there seems to be a bit of a shake up with constraints so maybe that's not valid, but not against relative stuff for arcs.

[franknoirot]

3. Are we talking about the same thing here? https://kittycadworkspace.slack.com/archives/C06P9L5MCUD/p1727830062908229

Frank, I kinda think we need a bunch of arc functions the same way as there are lots of straight segment functions, that helps with the code-mods constraints...

Yeah you're totally right, relative versions of all of them will be helpful.

...but there seems to be a bit of a shake up with constraints so maybe that's not valid, but not against relative stuff for arcs.

Oh yeah totally! The "constraint language" approach is not a sure-thing, sorry it's not supposed to be a shakeup, just one of the areas @nrc, @adamchalmers and @jtran are taking hard looks at while they try to firm up the v1.0 KCL implementation.

Jon and I saw some stuff that worry us about the code mod approach while we were there that I'd like to talk to you about. Maybe I'll add those notes to the "constraint language" PR discussion as motivation. The brief summary is:

1. I don't think there is a way in KCL to describe overarching dimensions that span multiple lines, which is super common in CAD. There may be a way forward under the current paradigm there, applying a code mod that modifies the lengths of each of the segments between the dimensioned ones while preserving their angles and any named values.
2. I believe applying more than one constraint to a segment is not possible currently, and forces the user to think hard about what order they do things in.
3. Having constraints be a feature of not the language but our code mod system makes them implicit, hiding the users intent and editing more of their code automatically, which makes writing your own "parallel" relationship in handwritten KCL nontrivial, as well as other smaller knock-on effects.

[Irev-Dev]

Yeah code modes have limitations