tomevault-io Sciris Parallelization and Profiling
Reference for parallel execution and performance analysis. See full tutorial: docs/tutorials/tut_parallel.ipynb.
If you need more detail, use your MCP tools (Context7 or GitMCP) to look up current Sciris documentation, or consult the other Sciris skills.
Basic Parallelization
# Simple: iterate over one argument
results = sc.parallelize(func, [1, 2, 3])
# With keyword arguments
results = sc.parallelize(complex_func, [1, 2, 3], arg2=10)
# Multiple iteration arguments (dict of lists)
results = sc.parallelize(mult, iterkwargs={'x': [1,2,3], 'y': [2,3,4]})
# Multiple iteration arguments (list of tuples)
results = sc.parallelize(mult, iterarg=[(1,2), (2,3), (3,4)])
# Multiple iteration arguments (list of dicts)
results = sc.parallelize(mult, iterkwargs=[{'x':1, 'y':2}, {'x':2, 'y':3}])
Resource-Limited Parallelization
results = sc.parallelize(
func = my_func,
iterarg = range(200),
maxcpu = 0.8, # Don't exceed 80% CPU
maxmem = 0.9, # Don't exceed 90% memory
interval = 0.2, # Re-check interval (seconds)
)
Advanced: Parallel Class
P = sc.Parallel(
func = slow_func,
iterarg = range(10),
parallelizer = 'multiprocess-async', # Async execution
die = False, # Continue on failure
)
P.run_async()
P.monitor() # Watch progress
P.finalize() # Collect results
print(P.success) # Per-job success flags
print(P.exceptions) # Any exceptions that occurred
print(P.results) # Results (None for failed jobs)
print(P.times) # Per-job timing
Profiling
Benchmarking
bm = sc.benchmark() # CPU performance in MOPS
ml = sc.memload() # System memory load
ram = sc.checkram() # Python process RAM usage
Line Profiling
sc.profile(my_func) # Shows time per line (like line_profiler)
Source: sciris/sciris — distributed by TomeVault.