📦 python-parallelization — Python 并行化

Overview

This skill enables transformation of sequential Python code into parallel/concurrent implementations to improve performance and efficiency.

Available Techniques

1. Multiprocessing

• Use for CPU-bound tasks
• Bypasses GIL (Global Interpreter Lock)
• Creates separate Python processes

2. Threading

• Use for I/O-bound tasks
• Shared memory space
• Subject to GIL for CPU operations

3. AsyncIO

• Asynchronous I/O operations
• Event-loop based
• Best for high-concurrency network operations

Usage Examples

Parallel TF-IDF Search

from concurrent.futures import ProcessPoolExecutor
import numpy as np
def compute_tfidf_chunk(documents):
    # TF-IDF computation for a chunk of documents
    pass
def parallel_tfidf_search(documents, num_workers=4):
    chunks = np.array_split(documents, num_workers)
    
    with ProcessPoolExecutor(max_workers=num_workers) as executor:
        results = list(executor.map(compute_tfidf_chunk, chunks))
    
    return combine_results(results)

Best Practices

• Choose the right concurrency model for your task
• Monitor resource usage
• Handle exceptions in worker processes/threads
• Consider memory overhead
• Test thoroughly for race conditions