OpenJudgeagentscope-ai/OpenJudge
Provides reusable grader capabilities for LLM and agent outputs, turning "is it good" into computable scores and rationa…
No ratings yet
Rate this skill
Stars851
Forks73
Watchers851
Open Issues14
InstallOne-line install
UnconfirmedWhether this repository follows the Agent Skills spec (that is, whether it contains a SKILL.md) has not been verified yet. Until that is settled, clone it and use what you need:
git clone https://github.com/agentscope-ai/OpenJudge.git
OverviewOverview
Key Features
Provides reusable grader capabilities for LLM and agent outputs, turning "is it good" into computable scores and rationales. Key capabilities include: 1. Multi-dimensional scoring: scores model responses item by item across correctness, relevance and safety, with a stated rationale for each judgment, replacing manual line-by-line review; 2. Agent trajectory evaluation: performs process-level judging of multi-step agent execution (tool calls, intermediate decisions) to locate the failing step rather than only checking the final answer; 3. Reward signal generation: outputs scores/preference data usable for reward model training and RLHF preference alignment, feeding evaluation results directly into the training loop; 4. Skill-based packaging: organizes judging rules and prompts as skills so grading criteria can be reused and swapped across evaluation tasks.
Use Cases
pre-release response quality spot checks, agent task success rate and process compliance evaluation, reward model training set construction, RLHF preference data annotation and validation, automated grading and regression comparison of evaluation sets.
Target Roles
AI / Algorithm Engineer, Data Analyst / Data Engineer, Test / QA Engineer.
TopicsTopics
RatingUser ratings
Nobody has rated this skill yet. Yours would be the first on the site — your vote becomes the initial average.
RepositoryRepository
| Skill ID | 1104 |
|---|---|
| Data source | GitHub |
| Repository | agentscope-ai/OpenJudge |
| Category | AI & Agent Skills / Model Evaluation & Alignment |
| Primary language | Python |
| License | Apache License 2.0 |
| Default branch | main |
| GitHub repository ID | 1015852565 |
| Node ID | R_kgDOPIyuFQ |
| Rank | #733 (position within the source ranking) |
| Site-wide popularity rank | #977 (computed live from Stars, across all 5833 entries) |
GrowthStar / Fork growth
Still collecting
First record taken on 2026-09-23. Two days of data are needed before a trend line can be drawn.
TimelineTimeline
Repository created2025-07-08 06:17:012025-07-08
Data last updated2026-10-26 12:05:1323 days ago
Last commit2026-09-11 07:11:092 months ago
CommitLatest commit
Committed by chr6192. Update Category: New Features & Capability Expansion. Commit details: feat(skills): reorganize skills/ into domain suites (#189). This update introduces new capabilities and expands supported use cases.