Repository Overview
The
Contents and Structure
The README organizes entries into sections that separate starter readings from deeper topics. The first section lists seven must-read items covering why evals matter, the link between evaluation and capability development, model-harness-skill decomposition, observability surfaces, infrastructure components, benchmark integrity issues such as contamination and saturation, and the relationship between evals and RL environments.
Subsequent sections address LLM-as-judge methods, verifier design, trajectory grading, and CI gating. Each entry includes a short description of its contribution and any noted limitations marked with ⚠️. New or updated resources from 2025–2026 carry a 🆕 flag. The SCAN.md file records the methodology used to surface and prune content.
Playbook and Practical Examples
PATTERNS.md supplies runnable code for several evaluation techniques. It demonstrates LLM-as-judge setups aligned to human labels, pass@k and pass^k calculations, error analysis pipelines, trajectory and world-state grading, and verifiable reward functions. The examples include concrete scoring logic and CI integration steps rather than abstract descriptions.
Developers can copy the provided harness patterns directly into Node.js or Python projects. The file also covers difficulty calibration for RL environments and lifecycle management for evolving benchmarks. These patterns focus on measurable outputs instead of subjective quality signals.
Evaluation Infrastructure Details
The list highlights tools for dataset versioning, online versus offline scoring, tracing, and automated gating. It distinguishes benchmark suites from custom eval harnesses and notes common failure modes such as label errors and leaderboard gaming. Practitioners working with React or Next.js front ends can use the tracing recommendations to surface agent outputs for grading without additional UI layers.
The repo encourages contributions through its CONTRIBUTING.md file, which specifies annotation requirements and verification steps. Dead links or unmaintained tools are removed on discovery rather than retained with warnings.
FAQs
What is the main difference between this list and other awesome lists? It requires every entry to state its purpose and include verified links, with abandoned resources pruned.
Does the repo include code for running evals? Yes, PATTERNS.md contains concrete examples for pass@k scoring, LLM judges, and CI integration.
How often is the list updated? Updates occur when new resources pass the citation and verification process; 2025–2026 items are flagged explicitly.
---
📖 Related articles
- Meta e Google siglano accordo miliardario per chip AI
- Agentic Coding: Una Trappola per lo Sviluppo Software?
- AI Generativa e Fisica: Come Cambia il Design di Oggetti Reali
Need a consultation?
I help companies and startups build software, automate workflows, and integrate AI. Let's talk.
Get in touch