Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Naive attempt at keeping crc cache around between runs #28

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

evs-chris
Copy link
Contributor

I needed a good distraction for an hour or two, so I thought I'd take a swing at #25. It did knock a "cold" build of Ractive from ~15 seconds to ~4 on my machine. I feel like I shouldn't make gobble keep .cache directories around, but without a bigger refactor than I'm capable of at this point, I couldn't see a way around it.

Anyways, I figure this is at least a fair proof of concept.

@evs-chris evs-chris closed this Jan 1, 2015
@Rich-Harris
Copy link
Contributor

@evs-chris did you mean to close this? Seems worth following up

@evs-chris
Copy link
Contributor Author

Yeah. I don't think keeping the .cache around is the best approach. I ran with it for a little while, and it caused some weirdness here and there. After pondering a bit, I think a better approach is:

  • keep a map file of relative filename to {crc,target} in each gobble stage directory
  • when starting a file transformer run, run through the input and check the files against the map collecting the differences and then process them and building a new map
  • run through the output and check for files that should no longer be present and remove them
  • save the new map to the stage directory

I think with this, directory transforms could possibly also opt in to crc tracking to make skipping directory transforms possible.

@evs-chris evs-chris reopened this Jan 7, 2015
@evs-chris
Copy link
Contributor Author

I've pondered this a bit more, and when I get some breathing room (probably a month or three), I'll try roughly this:

  • Don't automatically clean output directories on start
  • Create an observer node that watches source(s) and maintains a (cached to disk) map of input files to CRCs.
  • When the observer is triggered, it checks for changes and calls its given function with what has changed (lsr with change flags), its input dir, and options
  • The observer function can request an output dir, which turns the observer into a source
    • Either way, the observer can be an input node, it will just make downstream nodes wait for it to finish if it has no output (as far as gobble is concerned, anyways)
  • Transformers then become specialized observers that automatically request an output dir and automatically receive an lsr with info on what has changed based on CRC
  • File transforms use their one-to-one mapping knowledge to automatically maintain their output dir as a cache, selectively updating and removing files based on the cache info

That would open up an avenue for directory transformers to take advantage of caching if they so desired. It would also open up some possibilities for non-source-transforming code to run with input into the build process. It's also pretty easy to write an observer that just dumps its input files to an external directory for inspection.

The basis for my line of thought here is that the observer is the simplest unit of action rather than the transform.

Does any of that sound at all reasonable?

@Rich-Harris
Copy link
Contributor

It sounds more than reasonable. I ported everything to ES6 recently (because Babel has spoiled me, and my fingers can no longer bear to write code that Node understands natively) but didn't change the architecture in any fundamental way, because I wasn't sure I could see a clear path. But I think this ('the observer is the simplest unit of action') could be that path.

With fast cold builds and non-transforming observer nodes, I think we go from 'broccoli, but with a nicer API and more Windows bugs' to something that genuinely fills an underserved niche.

@Rich-Harris Rich-Harris mentioned this pull request Apr 4, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants