~/problems / Pools & pipelines / Thread pool / concurrent crawler

Futures from scratch

On a phone? Coding is easier on a laptop: email this problem to yourself . Meanwhile: quiz this topic or fight a boss.

hard assessment 3 levels ~55 min

Level 1 A future you can wait on

A future is a box for a result that doesn't exist yet: one thread fills it in later, any number of threads wait for it. Build one with a lock and a condition variable.

class Future:
    def set_result(self, value) -> None: ...
    def set_exception(self, exc: Exception) -> None: ...
    def done(self) -> bool: ...
    def result(self, timeout: float | None = None): ...
    def add_done_callback(self, fn) -> None: ...
  • A new Future() is not done. set_result or set_exception completes it; completing it a second time raises RuntimeError.
  • result(timeout) blocks until the future is done, then returns the value or raises the stored exception (the same object). If timeout seconds pass first, it raises TimeoutError. None waits forever. Any number of threads may wait at once.
  • add_done_callback(fn): call fn(future) once the future is done, in the order the callbacks were added, on the thread that completes it. If the future is already done, call fn straight away on the calling thread.
  • Inside a callback the future is already done, so result() returns at once. Callbacks may call any method of the future (so don't hold your lock while calling them). If a callback raises, ignore it and run the rest.
f = Future()
f.add_done_callback(lambda fut: print("got", fut.result()))
threading.Thread(target=lambda: f.set_result(42)).start()
f.result(timeout=2)        # 42 (and the callback printed "got 42")
f.done()                   # True
Future().result(timeout=0.1)   # TimeoutError after 0.1 s

C++ and Java hold int results and take timeouts in milliseconds; the starters show the exact types.

Show hint

one threading.Condition guards the state. Waiters use wait_for(predicate, timeout). To complete, take the lock, store the outcome, grab the callback list, notify_all(), release, and only then call the callbacks.

Level 2 unlocks when level 1 passes.

Level 3 unlocks when level 2 passes.

Topic: Thread pool / concurrent crawler. ThreadPoolExecutor, asyncio, thread-safe visited set.

Read the visual guide
0:00
Ctrl ' run · Ctrl ↵ submit
esc