It is a Python only, implementation of the MapReduce framework, to allow developers to write code that is broken up into tasks and run different events on different processors.
It has two storage backends: One uses the filesystem to communicate between processes and works correctly over NFS, so you can coordinate processes on different machines.
The other uses a redis database and all it needs is for different processes to be able to communicate with a common redis server.
Here are some key features of "Jug":
· Persistent data across runs
· Re-use partial results if you change the algorithms (for example, if you search over a few more parameters for the best, then it will reuse the pre-computed values). Normally, I have a main computation script and then write a second visualisation script to plot out the results or compute some summary statistics and it's good if the second script is easy to write, easy to change, and reuses all computational results seamlessly.
· Supports concurrency with a very flexible system: CPUs can join the computation at any time. In some environments this is a necessity.
· Can check up on the status of the computation at any time (jug status)
· Two backends: file-based if all the processors share a filesystem (works over NFS too) or redis based if they can all connect to the same redis server.
What's New in This Release: [ read full changelog ]
· Added ``webstatus`` subcommand
· Added bvalue() function
· Fixed bug in ``shell`` subcommand (``value`` was not in global namespace)
· Improved identity()
· Fixed bug in using Tasklets and --aggressive-unload
· Fixed bug with Tasklets and sleep-until/check
Find us on Google+
