# Helm vs OpenAI Evals

**Leader by Vioscale score:** Helm

| Attribute | Helm | OpenAI Evals |
|---|---|---|
| **Vioscale score** | 66.9 (60% (medium)) | 44.8 (30% (low)) |
| activity.commits_last_30d | 100 | 0 |
| adoption.dependent_repos | 5,000 | 1 |
| adoption.github_stars | 30,177 | 19,257 |
| deployment.options | `{"self_hosted":true}` | `{"cloud":true,"self_hosted":true}` |
| description.long | Helm provides a templated configuration system and package manager for Kubernetes applications, allowing users to define complex multi-component deployments as reusable, versioned, and shareable packages. | A framework that lets developers create and run evaluations to measure LLM performance, providing both pre-built benchmarks and tools to write custom tests tailored to specific use cases without requiring proprietary evaluation infrastructure. |
| features.capabilities | `{"arm_support":true,"cncf_maturity":"graduated","pricing_model":"free_open_source","architecture_model":"open_source_cli"}` | `{"pricing_model":"free_open_source","architecture_model":"open_source_cli","mit_or_apache_permissive_oss_license":true,"llm_as_a_judge_prompt_grading_framework":true,"ci_cd_github_actions_pipeline_blocking_gates":true,"deterministic_regex_and_json_schema_assertions":true,"synthetic_test_dataset_generation_from_documents":true}` |
| integrations.count | - | 3 |
| integrations.list | - | `[{"name":"OpenAI API"},{"name":"Snowflake"},{"name":"GitHub"}]` |
| language.primary | Go | Python |
| license.spdx | Apache-2.0 | - |
| market.availability | `{"primaryMarkets":[],"availabilityScope":"global","availableCountries":[],"notAvailableCountries":[]}` | - |
| platform.support | `{"cli":true,"mac":true,"linux":true,"windows":true}` | `{"cli":true,"web":true}` |
| pricing | `{"type":"open_source","freeTier":true,"sourceUrl":"https://helm.sh","retrievedAt":"2026-08-21T13:42:13.702Z"}` | `{"type":"free","summary":"Free and open-source","freeTier":true,"sourceUrl":"https://github.com/pricing","retrievedAt":"2026-08-21T11:08:00.118Z"}` |
| pricing.free_tier | yes | yes |
| pricing.model | open_source | commercial |
| pricing.price_level | free | free |
| pricing.transparent | yes | yes |
| release.cadence_days | 2 | - |
| release.history | `[{"url":"https://github.com/helm/helm/releases/tag/v4.2.4","date":"2026-08-13T15:33:04Z","type":"stable","version":"v4.2.4"},{"url":"https://github.com/helm/helm/releases/tag/v3.21.4","date":"2026-08-14T14:55:42Z","type":"stable","version":"v3.21.4"},{"url":"https://github.com/helm/helm/releases/tag/v4.2.3","date":"2026-07-09T20:56:43Z","type":"stable","version":"v4.2.3"},{"url":"https://github.com/helm/helm/releases/tag/v3.21.3","date":"2026-07-09T21:20:27Z","type":"stable","version":"v3.21.3"},{"url":"https://github.com/helm/helm/releases/tag/v3.21.2","date":"2026-06-20T04:36:23Z","type":"stable","version":"v3.21.2"},{"url":"https://github.com/helm/helm/releases/tag/v4.2.2","date":"2026-06-17T20:50:16Z","type":"stable","version":"v4.2.2"},{"url":"https://github.com/helm/helm/releases/tag/v4.2.1","date":"2026-06-12T15:06:26Z","type":"stable","version":"v4.2.1"},{"url":"https://github.com/helm/helm/releases/tag/v3.21.1","date":"2026-06-12T14:51:31Z","type":"stable","version":"v3.21.1"},{"url":"https://github.com/helm/helm/releases/tag/v4.2.0","date":"2026-05-14T01:13:00Z","type":"stable","version":"v4.2.0"},{"url":"https://github.com/helm/helm/releases/tag/v3.21.0","date":"2026-05-14T01:22:37Z","type":"stable","version":"v3.21.0"},{"url":"https://github.com/helm/helm/releases/tag/v4.1.4","date":"2026-04-09T06:01:47Z","type":"stable","version":"v4.1.4"},{"url":"https://github.com/helm/helm/releases/tag/v3.20.2","date":"2026-04-09T21:17:14Z","type":"stable","version":"v3.20.2"},{"url":"https://github.com/helm/helm/releases/tag/v4.1.3","date":"2026-03-11T23:01:58Z","type":"stable","version":"v4.1.3"},{"url":"https://github.com/helm/helm/releases/tag/v3.20.1","date":"2026-03-12T00:26:02Z","type":"stable","version":"v3.20.1"},{"url":"https://github.com/helm/helm/releases/tag/v4.1.1","date":"2026-02-09T18:33:58Z","type":"stable","version":"v4.1.1"},{"url":"https://github.com/helm/helm/releases/tag/v4.1.0","date":"2026-01-21T22:31:55Z","type":"stable","version":"v4.1.0"},{"url":"https://github.com/helm/helm/releases/tag/v3.20.0","date":"2026-01-21T23:27:45Z","type":"stable","version":"v3.20.0"},{"url":"https://github.com/helm/helm/releases/tag/v4.0.5","date":"2026-01-14T23:46:31Z","type":"stable","version":"v4.0.5"},{"url":"https://github.com/helm/helm/releases/tag/v3.19.5","date":"2026-01-15T00:30:06Z","type":"stable","version":"v3.19.5"},{"url":"https://github.com/helm/helm/releases/tag/v4.1.0-rc.1","date":"2026-01-12T19:54:05Z","type":"prerelease","version":"v4.1.0-rc.1"}]` | - |
| security.disclosure_policy | yes | yes |
| security.scorecard | 9 | - |
| security.vulnerabilities | `{"count":23,"source":"https://advisories.ecosyste.ms/api/v1/advisories?ecosystem=go&package_name=helm.sh%2Fhelm%2Fv3&per_page=100","last_12m":1,"max_severity":"HIGH"}` | `{"count":0,"source":"https://advisories.ecosyste.ms/api/v1/advisories?ecosystem=go&package_name=github.com%2Fopenai%2Fevals&per_page=100","last_12m":0,"max_severity":null}` |

## Capabilities (AI Evals Testing)

| Capability | Helm | OpenAI Evals |
|---|:--:|:--:|
| **Capabilities** |  |  |
| Architecture model | Open source CLI | Open source CLI |
| LLM as a judge prompt grading framework | - | ✓ |
| Specialized rag metrics faithfulness context relevance | - | - |
| Deterministic regex and json schema assertions | - | ✓ |
| Synthetic test dataset generation from documents | - | ✓ |
| Ci cd github actions pipeline blocking gates | - | ✓ |
| Red teaming and adversarial vulnerability scanning | - | - |
| Multi model side by side ab regression testing | - | - |
| Human in the loop hitl annotation UI | - | - |
| Dashboard analytics for metric drift over time | - | - |
| SOC2 type ii | - | - |
| Mit or apache permissive oss license | - | ✓ |
| Pricing model | Free open source | Free open source |

*Source: Vioscale. Generated 2026-09-01T15:05:00.537Z. "-" = undocumented, not absent.*