# Galileo vs LM Evaluation Harness

| Attribute | Galileo | LM Evaluation Harness |
|---|---|---|
| **vioscaleAI score** | 47.4 (7% (low)) | 48.2 (35% (low)) |
| activity.commits_last_30d | - | 57 |
| adoption.dependent_repos | - | 252 |
| adoption.github_stars | - | 14,221 |
| content.faq | - | `[{"answer":"A Python-based evaluation framework that enables testing of language models against 60+ standard academic benchmarks with support for various model formats, APIs, and custom evaluation metrics. It is indexed under AI Evals Testing.","source":"https://github.com/pricing","question":"What is LM Evaluation Harness?","confidence":0.6},{"answer":"LM Evaluation Harness is open source, so it can be self-hosted and used at no licence cost. It is released under the MIT licence. Pricing changes often, so verify at source before relying on it.","source":"https://github.com/pricing","question":"Is LM Evaluation Harness free to use?","confidence":0.6},{"answer":"LM Evaluation Harness supports a command-line interface. Platforms we have not confirmed are simply not listed here rather than ruled out.","source":"https://github.com/pricing","question":"What platforms does LM Evaluation Harness support?","confidence":0.6},{"answer":"Yes. LM Evaluation Harness can be deployed self-hosted, so it does not have to run on the vendor's infrastructure.","source":"https://github.com/pricing","question":"Can LM Evaluation Harness be self-hosted?","confidence":0.6},{"answer":"We have confirmed 5 integrations for LM Evaluation Harness, including Hugging Face, PyTorch, VLLM, GitHub and OpenAI-compliant APIs. This is what we could verify from public sources, so the vendor may support others we have not indexed.","source":"https://github.com/pricing","question":"What does LM Evaluation Harness integrate with?","confidence":0.6},{"answer":"Yes. LM Evaluation Harness is published under the MIT licence, a permissive licence that generally allows commercial use and modification. Licence terms can change between releases, so verify against the repository for the version you intend to use.","source":"https://github.com/EleutherAI/lm-evaluation-harness","question":"Is LM Evaluation Harness open source?","confidence":0.95}]` |
| deployment.options | - | `{"self_hosted":true}` |
| description.long | A platform for building and evaluating AI systems that synthesizes test datasets from multiple sources, compresses expensive LLM evaluators into efficient models, and provides production monitoring with real-time observability. | A framework for conducting few-shot evaluation of language models against various tasks and benchmarks. Supports multiple model backends, custom evaluation metrics, and extensible task definitions for assessing model capabilities and performance. |
| features.capabilities | `{"architecture_model":"managed_enterprise_saas","human_in_the_loop_hitl_annotation_ui":true,"llm_as_a_judge_prompt_grading_framework":true,"ci_cd_github_actions_pipeline_blocking_gates":true,"dashboard_analytics_for_metric_drift_over_time":true,"synthetic_test_dataset_generation_from_documents":true}` | `{"pricing_model":"free_open_source","architecture_model":"open_source_cli","multi_model_side_by_side_ab_regression_testing":true}` |
| integrations.count | 1 | 5 |
| integrations.list | `[{"name":"NVIDIA NeMo"}]` | `[{"name":"Hugging Face"},{"name":"PyTorch"},{"name":"VLLM"},{"name":"GitHub"},{"name":"OpenAI-compliant APIs"}]` |
| language.primary | - | Python |
| license.spdx | - | MIT |
| platform.support | - | `{"cli":true,"mac":true,"linux":true,"windows":true}` |
| pricing | - | `{"type":"open_source","freeTier":true,"sourceUrl":"https://github.com/pricing","retrievedAt":"2026-10-14T22:30:23.897Z"}` |
| pricing.free_tier | - | yes |
| pricing.model | - | open_source |
| pricing.price_level | - | free |
| pricing.transparent | - | yes |
| release.cadence_days | - | 72 |
| release.history | - | `[{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.13","date":"2026-08-31T20:42:30Z","type":"stable","version":"v0.4.13"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.12","date":"2026-05-11T13:04:19Z","type":"stable","version":"v0.4.12"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.11","date":"2026-02-13T20:21:47Z","type":"stable","version":"v0.4.11"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.10","date":"2026-01-27T19:56:53Z","type":"stable","version":"v0.4.10"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.9.2","date":"2025-11-26T23:27:06Z","type":"stable","version":"v0.4.9.2"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.9.1","date":"2025-08-04T11:36:05Z","type":"stable","version":"v0.4.9.1"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.9","date":"2025-06-19T14:18:27Z","type":"stable","version":"v0.4.9"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.8","date":"2025-03-05T07:49:46Z","type":"stable","version":"v0.4.8"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.7","date":"2024-12-17T10:37:09Z","type":"stable","version":"v0.4.7"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.6","date":"2024-11-25T13:38:29Z","type":"stable","version":"v0.4.6"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.5","date":"2024-10-08T21:06:05Z","type":"stable","version":"v0.4.5"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.4","date":"2024-09-05T15:13:13Z","type":"stable","version":"v0.4.4"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.3","date":"2024-07-01T14:00:36Z","type":"stable","version":"v0.4.3"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.2","date":"2024-03-18T13:07:28Z","type":"stable","version":"v0.4.2"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.1","date":"2024-01-31T15:29:14Z","type":"stable","version":"v0.4.1"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.0","date":"2023-12-04T15:08:53Z","type":"stable","version":"v0.4.0"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.3.0","date":"2022-12-08T08:34:37Z","type":"stable","version":"v0.3.0"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.2.0","date":"2022-03-07T02:12:23Z","type":"stable","version":"v0.2.0"},{"url":"https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.0.1","date":"2021-09-02T02:28:08Z","type":"stable","version":"v0.0.1"}]` |
| security.scorecard | - | 5.5 |
| security.trust_center | - | https://github.com/security |
| security.vulnerabilities | - | `{"count":0,"source":"https://advisories.ecosyste.ms/api/v1/advisories?ecosystem=pypi&package_name=lm-eval&per_page=100","last_12m":0,"max_severity":null}` |

## Capabilities (AI Evals Testing)

| Capability | Galileo | LM Evaluation Harness |
|---|:--:|:--:|
| **Capabilities** |  |  |
| Architecture model | Managed enterprise saas | Open source CLI |
| LLM as a judge prompt grading framework | ✓ | - |
| Specialized rag metrics faithfulness context relevance | - | - |
| Deterministic regex and json schema assertions | - | - |
| Synthetic test dataset generation from documents | ✓ | - |
| Ci cd github actions pipeline blocking gates | ✓ | - |
| Red teaming and adversarial vulnerability scanning | - | - |
| Multi model side by side ab regression testing | - | ✓ |
| Human in the loop hitl annotation UI | ✓ | - |
| Dashboard analytics for metric drift over time | ✓ | - |
| SOC2 type ii | - | - |
| Mit or apache permissive oss license | - | - |
| Pricing model | - | Free open source |

*Source: vioscaleAI. Generated 2026-10-16T20:16:21.823Z. "-" = undocumented, not absent.*