findPath async version

[andrewrk]

that way we don't hog the CPU while pathfinding

[andrewrk]

child process fork won't have access to the parent process's memory. But we can still provide an async version.

[yocontra]

Why not use something like asyncjs for this? If you have tests on the pathfinding code I'd be glad to lend a hand

[andrewrk]

Are you talking about this? https://npmjs.org/package/asyncjs

The problem is not how to organize callbacks; it's more fundamental than that. The problem is that we need to run the CPU for a long time, and we have no threads.

[yocontra]

asyncjs lets you do loops without "hogging" the CPU - it puts every iteration in a nextTick so other things have a chance at the CPU (right now it will block it for who knows how long). If you want to just throw it in a thread and forget about it you can always use threads-a-go-go or write the A* stuff as a native module and use libuv threads (that's what I would recommend)

[andrewrk]

neither of these libraries bring anything new to the table; we're left with the same problem that a child process leaves us. the problem is that what's taking a long time is searching memory. the child process, or the thread, whichever you decide to use, still has to communicate with the main thread via IPC or events/messages for block data.

it's not to say that we can't provide an async version. The question is whether to accept the overhead of starting a thread or process or not. the simplest implementation is to call process.nextTick every once in a while to let other events run.

[yocontra]

So why not write the A* stuff as a native module and use the libuv thread pool to do the CPU intensive work?

[andrewrk]

Because the problem is not just a bunch of number-crunching. it's a problem of memory sharing. We could do a native A* module, but then we'd have to store the block data in a different way so that both threads could get access to it.

You're welcome to try to tackle the problem and submit a pull request. If it's clean, maintainable, and solves the problem, I'll merge it. However I think if you do you'll discover it's a bit more tricky than you think.

Here is one new idea I haven't considered until now though: When mineflayer starts it could start a child process and when mineflayer gets new data that the subprocess needs, it will send it via IPC. So the subprocess keeps a copy of mineflayer's state; this will take up twice as much memory, but there will be a free process ready to do navigation or whatever. And when actually called upon, you won't waste all that time initializing a process and sending data back and forth; it will already have the data cached.

No matter how you choose to solve it, it gets messy.

[Darthfett]

I think this is approaching the subject that you, Josh, and I discussed a bit, of having a shared block memory for bots running on the same machine.

[yocontra]

Native modules have access to the same memory though - it's the same process.

[andrewrk]

It might be possible to create an addon, make it be in charge of the raw chunk buffers, and then be able to spawn threads to do things with the chunk buffers.