coding/

A contributor's claimed Big-O complexity class for their own implementation, checked EMPIRICALLY -- solve(n) is actually run at multiple real, increasing input sizes and its growth curve (CPU time, not just wall-clock, to defeat a time.sleep()-based fake) is compared against the theoretically-expected ratio for the claim. Different from `performance` (a single-scale, two-implementation relative-speed-ratio check): this category has ONE implementation, MULTIPLE input sizes, and checks the GROWTH CURVE against an asymptotic class, never a fixed relative-speed threshold.

strategy_code (Python) defines a class Strategy with a no-argument constructor and a method decide(price_history, portfolio) that the harness calls exactly once per bar of a harness-generated, curator-seeded synthetic price path -- NEVER receiving any bar beyond today's, by construction (see strategy_code's own help text). The harness owns the entire walk-forward event loop: it generates the price path itself from market_params' own pinned Geometric Brownian Motion formula, drives strategy_code bar-by-bar in a SEPARATE OS process that never receives future prices, builds the real resulting equity curve itself from real portfolio state after every bar, and independently computes Sharpe ratio / Sortino ratio / max drawdown / CAGR / Calmar ratio from that real curve using the harness's own from-scratch formula implementation (see harness.js's own module doc comment for the exact pinned formulas) -- never trusting any number strategy_code itself might compute or print. This is run against EVERY seed market_params declares (at least 2 independent seeds required, mechanically enforced) so a curator cannot author a row against a single lucky/tuned price path; the curator's own expected_metrics must match what the harness independently computes for EVERY seed, within a curator-declared tolerance. A row whose real performance is implausibly good on EVERY seed simultaneously (a red flag on what is, by construction, a random walk with no genuine exploitable edge) is rejected outright as a likely lookahead/measurement bug rather than silently accepted. This category is coding-domain, not finance-domain: the subject matter is trading-strategy CODE, execution-verified exactly like every other category in this registry, not a financial-advice or real-market-data category (the sandbox has no network egress, so real historical market data is never used or needed).

An LLM agent tool-use trace: the generated tool call must match the available schema, and the final answer must accurately state the facts the tool returned.

transformed_code (THE GRADED ARTIFACT) must be a genuine, mechanically-verifiable AST-level transformation of original_code, per one of a small fixed set of transformation_type values -- checked TWO ways, never just one: (1) STRUCTURALLY, by parsing BOTH original_code and transformed_code with Python's real stdlib `ast` module and mechanically confirming the SPECIFIC claimed transformation actually happened (never a text/regex diff, which would be trivially gameable and would not actually confirm an AST-level transformation occurred); (2) BEHAVIORALLY, by executing a fixed entry-point function from both original_code and transformed_code against a shared set of curator-declared behavior_check test cases and confirming the results match -- for 7 of the 8 transformation_type values this means pure behavior PRESERVATION (a real refactor must not change what the code computes); for the 8th (except_pass_to_logging), the return value/control-flow is still required to match exactly, but the transformation's OWN point is a new, mechanically-verified observable side effect (a real logging call, confirmed via runtime capture, not just claimed) -- see harness.js's own module doc comment for why this single semantic-changing type was included alongside 7 purely behavior-preserving ones, and the reasoning behind each transformation_type's own structural check.

A Python auth/authorization handler is run against a synthetically-constructed JWT matching the described identity-provider and test scenario; the actual returned status is compared against the claimed access result.

A real package manifest that must resolve (or fail to resolve) exactly as claimed when installed for real against the live registry.

solution_code (Python) defines a Cache class driven, in order, through a harness-controlled operation sequence -- get / put / advance_time / check_store -- and its real hit/miss result, returned value, and evicted-key identity at every step must match that operation's curator-declared expected outcome. Ground truth is never trusted blindly: the harness FIRST independently recomputes, from cache_policy + policy_params alone (via its own reference implementation of the declared algorithm -- lru / lfu / fifo / ttl / write_through -- never by inspecting solution_code or trusting the row's own claims), what every operation's outcome must be, and rejects the row outright if the curator's own declared sequence does not match that independent computation. solution_code is mechanically forbidden from reading the real system clock: any TTL logic must decide purely from the `now` argument the harness passes into get()/put(), never a real sleep or a real clock read.

solution_code (Python) defines a single top-level function `build_parser()` (no arguments) that constructs and returns a real, fully-configured `argparse.ArgumentParser` instance implementing the CLI interface described in task_description -- flags, options, required/optional args, type coercion, choices, mutually-exclusive groups, and/or subcommands via add_subparsers(). The harness calls build_parser() EXACTLY ONCE, then drives the SAME parser instance through every one of invocations' own ordered argv lists via real, unmodified `parser.parse_args(argv)` calls (confirmed empirically safe -- CPython's argparse has copied action='append'/'extend' default lists per call since 3.8, so no cross-invocation state leaks through the shared instance for any standard action). Each invocation declares its own required outcome: EITHER a successful parse (expected_namespace, matched via exact-value equality against parse_args' own real returned Namespace.__dict__) OR a real parse failure (argparse's own convention: a validation error inside parse_args always raises SystemExit via self.error(), never a catchable exception -- expected_exit_code plus an expected_error_substring that must literally appear in the real captured stderr text, chosen deliberately over exact stderr matching since argparse's own message wording is verbose/formatting-sensitive but its VALIDATION-KIND vocabulary -- 'required', 'invalid choice', 'invalid int value', 'not allowed with argument', 'unrecognized arguments' -- is stable stdlib convention, empirically confirmed against a live Python 3 argparse rather than assumed). There is no independent oracle re-deriving what a CORRECT parser for this row's own natural-language task_description should be (task_description is open-ended prose, exactly the kind of thing this registry already trusts curator-authored ground truth for elsewhere -- numerical_precision's expected_output, retry_backoff_resilience's expected_result -- rather than mechanically re-deriving); invocations' own expected_namespace/expected_exit_code/expected_error_substring are curator-authored claims, checked directly against solution_code's own REAL argparse execution, mitigated by this pipeline's own llm + human_audit stages downstream of execution, the same trust model and residual already accepted for those two sibling categories.

Code translated from one language to another — target_code must pass the same logical tests as source_code, must implement the same function (not a substitute), the declared source_language must match what source_code actually parses as, and the tests must assert on values (not merely a type).

An algorithmic problem with a reference solution checked against every supplied input/output case.

Compression + decompression code that must round-trip the sample input byte-for-byte.

A buggy (genuinely racy) Go program paired with a fixed (race-free) implementation of the same concurrent task. Verified by actually compiling and running BOTH under Go's real `go run -race` data-race detector -- never by static inspection, never by 'run it N times and eyeball the output' -- buggy_code must trigger a real `WARNING: DATA RACE` report at least once across a bounded number of attempts, and fixed_code must run clean (no race reported) AND produce exactly expected_functional_output on every attempt tried.

A cryptographic primitive implementation (hash, HMAC, block cipher, stream cipher, KDF, checksum, or encoding) is actually run against its own documented official test vector(s); the claimed match is checked against the real output.

A transform() script that receives the raw input sample text and must convert it into exactly the expected output.

A SQL migration script that must correctly transform a documented 'before' SQLite state into a documented 'after' state. Verified by actually building initial_schema_ddl + initial_fixture_data, executing migration_script against it for real (executescript -- multiple DDL/DML statements are the normal, expected shape here, unlike sql_query_correctness's single-SELECT contract), then running verification_query against the now-migrated database and comparing its real result to expected_verification_result -- the same execution-accuracy methodology as sql_query_correctness, applied to a before/after state transition instead of a single read.

Broken code paired with its corrected version, driven by failing tests. broken_code must FAIL the tests and fixed_code must PASS them — the flagship execution-verified type.

A single-package manifest audited by a real vulnerability-scanning tool (npm audit / pip-audit) against the live registry advisory database, checked against the claimed audit outcome's polarity.

A contributor-authored unified diff (patch_diff) that must apply cleanly (via real `git apply`) to a fixed base file and produce EXACTLY the documented final content -- single-file, conflict-free patch APPLICATION, not merge resolution. The harness always writes base_file_content to one harness-chosen filename ('target.txt') in a fresh workdir; patch_diff is rejected outright if its own diff headers name any other file, or more than one file.

Code paired with a predicted output — the code must actually produce that output when run.

A generated test that fails against the buggy version of a function and passes against the fixed version — proof the test actually exercises the bug.

Flag-evaluation code run against a simulated population — the observed enabled rate must match the documented rollout percentage within tolerance.

A working, tested implementation generated from a natural-language requirement. solution_code must PASS the provided tests, executed in the row's declared language.

Two genuinely conflicting branches over a common ancestor, plus the correct merged result.

A GraphQL schema (SDL) and its resolver map, executed against a sample query with real graphql-js; the result must match the claimed response exactly. Resolvers that reference an external data fixture not supplied by the row are checked structurally (every resolver key must wire to a real schema field) rather than executed against fabricated data.

Contributor-authored Python (stdlib-only) HTTP server code is actually started as a REAL, separate OS process, bound to a real loopback TCP port (127.0.0.1 -- precedented by websocket_realtime's own real loopback server; this is not network egress). The harness polls for genuine readiness with real connection attempts (never a fixed sleep), then sends a REAL, curator-authored sequence of HTTP requests against it via Python's stdlib http.client -- never by inspecting server_implementation's source, never by trusting anything the server process prints. Every request's real status code / selected response headers / body is compared against that request's own declared expected values. The contract requires at least one pair of requests to the same route with genuinely different input and genuinely different expected output, specifically so a hardcoded/constant-response server cannot pass -- every declared expected value is real, independently-checked ground truth, not merely a consistency check.

Intl.* formatting code that must produce the specific documented output for every locale it claims to support.

Parsing or query code that extracts a metric from a log fixture, checked against the exact expected value.

A claim about what a memory/resource-safety tool (Valgrind or AddressSanitizer) reports for a given C program, checked against what the tool actually reports when the program is compiled and run.

A state-machine implementation driven through a reference harness; the observed transitions must match the spec.

Notebook cells that build a data fixture matching the row count and shape they're documented to produce.

solution_code (Python) defines a solve(inputs, rounding_mode) function that must reproduce the curator's own precomputed expected_output BYTE-FOR-BYTE (exact string equality, never numeric/float tolerance) for a realistic financial/scientific/measurement computation -- compound interest, exact-cent bill splitting, currency conversion, per-line vs per-total tax allocation, weighted averages, and similar tasks whose entire point is that a float literal cannot exactly represent the decimal value being computed. inputs are curator-authored, as JSON strings (never float literals) for anything decimal-precision-sensitive, or plain JSON integers for counts -- so both the row's own authoring and solution_code's own computation can use Python's real decimal.Decimal type end to end. There is no independent oracle: unlike this registry's fixed-small-algorithm-set categories (rate_limiting_policy_simulation's 5 policy types, caching_strategy's 5 eviction disciplines), this category's computation varies arbitrarily per row, so expected_output is trusted directly (the same trust model diff_patch_application and data_transformation already use for their own curator-authored expected_final_content/output_data_sample) -- mitigated by this pipeline's own llm + human_audit stages, not by mechanical re-derivation. What IS mechanically enforced: solve() must return a decimal.Decimal or a str (never a bare float/int, so its exact string form is never left to an uncontrolled float repr), and a harness-computed perturbation of inputs (every numeric field nudged by a real, exact amount -- BigInt-precise in this file, never floating point) must make solve()'s own output change, so a solution_code that simply returns the literal expected_output string regardless of input is mechanically rejected.

A real package (pip, npm, or cargo) that must build, install, and pass its smoke test exactly as claimed.

An optimized version of working code that must still pass the same tests and run measurably faster on the given benchmark input.

A contributor-authored property-based test, written with Python's real Hypothesis library, that must genuinely discriminate a correct reference implementation from a deliberately broken one. Verified by actually invoking Hypothesis's own @given example-generation and shrinking machinery twice -- once with the correct implementation bound to IMPL, once with the broken one -- never by static inspection and never by example-based (fixed-input) testing.

solution_code (Python) defines a RateLimiter class driven, in order, through a harness-controlled virtual timeline -- key/timestamp pairs, never real wall-clock time and never a real sleep -- and its real admit/reject decision at every event must match that event's curator-declared expected outcome. Ground truth is never trusted blindly: the harness FIRST independently recomputes, from policy_type + policy_params alone (via its own reference implementation of the declared algorithm, never by inspecting solution_code or trusting the row's own claims), what every event's outcome must be, and rejects the row outright if the curator's own declared timeline does not match that independent computation -- solution_code is only ever run against a timeline already proven internally consistent with its own declared policy. solution_code is mechanically forbidden from reading the real system clock: it must decide every outcome purely from the timestamp argument the harness passes in.

Python (redis-py) code that manipulates a REAL, freshly-started redis-server process for this row -- sorted sets, hashes, TTL/expiry, lists, atomic counters/transactions. Verified by actually running solution_code against that live server, then independently re-querying the server's real post-execution state through a dataset-curator-authored verification_code on a SEPARATE connection, and comparing the observed result to expected_verification_result -- never by inspecting solution_code's source or trusting anything it prints. The whole scenario is additionally re-run end-to-end against a second, independently-started server to confirm the observed behavior is reproducible and not an artifact of incidental state/timing on one particular server instance.

A refactor of working code that must pass the exact same tests as the original — proof the refactor preserved behavior.

Regex patterns for a stated intent, with positive and negative match sets — mechanically verified in milliseconds, including a catastrophic-backtracking check.

solution_code (Python) defines a single top-level function retry_call(dependency, sleep) that wraps a harness-injected, harness-scripted flaky dependency callable and must retry it per the row's declared strategy_type/strategy_params, using ONLY the harness-injected sleep(seconds) callable for backoff delays -- never a real time.sleep(). The harness drives ONE call to retry_call() per row and checks three things: the exact sequence of delay values requested (bounds-checked, not exact-matched, for the one strategy_type whose delays are genuinely randomized), the exact number of real calls made to the dependency, and the final outcome (the dependency's own successful result value returned, or an exception propagated after every allowed attempt is exhausted). Ground truth is never trusted blindly: the harness FIRST independently recomputes, from strategy_type + strategy_params + dependency_behavior alone (via its own reference state-machine simulation of the declared strategy, never by inspecting solution_code or trusting the row's own claims), what the real call count/delay sequence/outcome must be, and rejects the row outright if the curator's own declared expected_result does not match that independent computation. Distinct from rate_limiting_policy_simulation (admission control under a fixed policy, never any retrying/waiting) and caching_strategy (storage/eviction, no failure/retry semantics at all): this category is specifically about resilience against a dependency that FAILS and must be retried, with correctly-computed backoff between attempts.

A contributor-authored validate(payload_text) -> bool function that must correctly classify a SET of test documents as schema-valid or schema-invalid, checked against a REAL, trusted validator's independently computed ground truth (Python's jsonschema library for JSON Schema rows, lxml.etree.XMLSchema for XSD rows) -- never against a row-declared 'expected' label. One category, one 'schema_language' enum selecting the format-specific verification path, the same pattern serialization's own 'format' enum already uses for its own multi-format design.

Serialization + deserialization code that must reproduce the original object, or the specific documented format limitation (e.g. JSON coercing tuples to lists).

A shell script whose recorded second-run behavior (idempotent no-op, or a described change) must match what actually happens when it is run twice.

A natural-language question over a SQLite fixture, answered with a single read-only SELECT statement. Verified by actually executing sql_query against a fresh database built from schema_ddl + fixture_data and comparing the real result set to expected_result -- the Spider/BIRD 'execution accuracy' methodology, not a text/AST comparison against a reference query.

Code that must either compile/type-check cleanly, or fail with the exact error code claimed for it.

A buggy_code -> fixed_code pair checked against a REAL linter (eslint for 'JavaScript/TypeScript' rows, ruff for 'Python' rows), never a simulated/regex-scraped one. buggy_code must genuinely trip a specific, named rule_id when linted with ONLY that one rule enabled; fixed_code must not trip that same rule_id and must remain lint-runnable (a syntax error is a disqualifying non-fix, not a pass). The linter is configured to check EXACTLY rule_id and nothing else, so the verdict can never be satisfied by an unrelated rule happening to differ between the two snippets.

A shell command sequence that must leave the filesystem in the state a verification script checks for.

A generated test suite that actually passes when run against the source code, using the framework declared for the row (pytest / jest / go test / rustc --test).

A working exploit that succeeds against vulnerable code and is neutralized by the patched version — proof the patch actually closes the hole, not just that it looks different.

Parsing code run against a supplied HTML page; the extracted structure must match exactly. Nothing is fetched — the page is provided inline.

The server's behavior is classified into one of a small set of canonical archetypes (broadcast, private routing, room isolation, idempotent dedup, seq-id replay, capacity limit, connection-count query) and faithfully reimplemented as a real asyncio `websockets` server; the scripted client scenario is driven against it for real, and the actually-observed message delivery / connection outcome is compared against the claimed outcome.