Friday, March 22, 2019

Running a Function on Dask Workers At Startup

I made the mistake of thinking that client.run() would run a function on each Dask worker regardless if the worker hadn't started yet. In the case where new workers come online that function won't run. Instead you'll need to take advantage of the register_worker_callbacks() function that will register a function to run on all of the workers at startup. It looks like this:

cluster = LocalCluster()
client = Client(cluster)

I found this by looking through the tests for this function's pull request. It's otherwise undocumented.

No comments:

Post a Comment