# Promptfoo vs Ragas

| Attribute | Promptfoo | Ragas |
|---|---|---|
| **Vioscale score** | 54.4 (55% (medium)) | 58.4 (27% (low)) |
| adoption.dependent_repos | - | 1 |
| adoption.github_stars | - | 15,484 |
| deployment.options | `{"cloud":true,"hybrid":true,"on_prem":true,"self_hosted":true}` | `{"self_hosted":true}` |
| description.long | A comprehensive LLM security testing solution that automatically scans for vulnerabilities such as prompt injections and jailbreaks through dynamic red teaming. Offers both open-source software for local testing and enterprise SaaS with team collaboration, continuous monitoring, and compliance dashboards. | A Python-based framework that evaluates RAG applications through automatic metrics covering faithfulness, relevance, and recall. Includes tools to synthetically generate test datasets customized for specific use cases, enabling developers to assess LLM application performance at both component and end-to-end levels. |
| features.capabilities | `{"soc2_type_ii":true,"self_hostable":true,"versioning_model":"UI-hosted registry","non_engineer_editing":false,"llm_judge_or_human_eval":true,"llm_as_a_judge_prompt_grading_framework":true,"ci_cd_github_actions_pipeline_blocking_gates":true,"deterministic_regex_and_json_schema_assertions":true,"red_teaming_and_adversarial_vulnerability_scanning":true}` | `{"open_source":true,"pricing_model":"free_open_source","architecture_model":"open_source_cli","built_in_rag_eval_metrics":true,"llm_as_a_judge_prompt_grading_framework":true,"synthetic_test_dataset_generation_from_documents":true}` |
| integrations.count | 6 | 3 |
| integrations.list | `[{"name":"OpenAI"},{"name":"Anthropic"},{"name":"Google (Gemini)"},{"name":"DeepSeek"},{"name":"MCP (Model Context Protocol)"},{"name":"CI/CD pipelines"},{"name":"Webhooks"},{"name":"Agent frameworks"},{"name":"CI/CD platforms"},{"name":"GitHub"},{"name":"Python providers"}]` | `[{"name":"LlamaIndex"},{"name":"LangSmith"},{"name":"OpenAI"}]` |
| language.primary | - | Python |
| license.spdx | - | Apache-2.0 |
| market.availability | `{"hqCountry":"US","primaryMarkets":["US"],"availabilityScope":"global","availableCountries":[],"notAvailableCountries":[]}` | - |
| platform.support | `{"cli":true,"web":true}` | `{"cli":true}` |
| pricing | `{"type":"hybrid","plans":[{"free":true,"name":"Community","features":["All LLM evaluation features","All model providers and integrations","Red teaming (10k probes/month)","Custom integration with your own app","Run locally or self-host on your own infrastructure","Vulnerability scanning","Community support"],"components":[{"kind":"fixed","amount":0,"period":"month","currency":"USD"}],"description":"Open-source version for individual developers and small teams","contactSales":false,"includedLimits":{"red_teaming_probes":"10k/mo"}},{"free":false,"name":"Enterprise","features":["All Community features","Custom red teaming limits","Team sharing & collaboration","Continuous monitoring","Centralized security/compliance dashboard","Customizable attack profiles and target settings","SSO and granular permission profiles","Promptfoo API access","Managed cloud deployment","Professional services support","Priority support & SLA guarantees"],"description":"Managed cloud platform with team collaboration, continuous monitoring, and compliance features","contactSales":true},{"free":false,"name":"On-Premise","features":["Complete data isolation","On-premise deployment"],"description":"Custom deployment for organizations requiring full control over infrastructure","contactSales":true}],"summary":"Free open-source Community tier; Enterprise and On-Premise plans with custom pricing","currency":"USD","freeTier":true,"sourceUrl":"https://www.promptfoo.dev/pricing/","retrievedAt":"2026-08-19T18:21:49.377Z"}` | `{"type":"open_source","freeTier":true,"sourceUrl":"https://ragas.io","retrievedAt":"2026-08-21T13:52:19.928Z"}` |
| pricing.free_tier | yes | yes |
| pricing.model | freemium | commercial |
| pricing.price_level | low | free |
| pricing.transparent | no | yes |
| release.cadence_days | - | 7 |
| release.history | - | `[{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.4.3","date":"2026-01-13T17:47:29Z","type":"stable","version":"v0.4.3"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.4.2","date":"2025-12-23T17:13:41Z","type":"stable","version":"v0.4.2"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.4.1","date":"2025-12-10T16:28:51Z","type":"stable","version":"v0.4.1"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.4.0","date":"2025-12-03T16:22:29Z","type":"stable","version":"v0.4.0"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.9","date":"2025-11-11T17:24:46Z","type":"stable","version":"v0.3.9"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.8","date":"2025-10-28T19:09:17Z","type":"stable","version":"v0.3.8"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.7","date":"2025-10-14T16:21:37Z","type":"stable","version":"v0.3.7"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.6","date":"2025-10-03T03:56:31Z","type":"stable","version":"v0.3.6"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.5","date":"2025-09-17T19:13:15Z","type":"stable","version":"v0.3.5"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.5rc2","date":"2025-09-17T17:40:25Z","type":"prerelease","version":"v0.3.5rc2"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.5rc1","date":"2025-09-17T17:31:23Z","type":"prerelease","version":"v0.3.5rc1"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.4","date":"2025-09-10T23:57:28Z","type":"stable","version":"v0.3.4"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.3","date":"2025-09-04T17:59:51Z","type":"stable","version":"v0.3.3"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.3rc1","date":"2025-09-04T17:46:09Z","type":"prerelease","version":"v0.3.3rc1"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.2","date":"2025-08-19T12:03:50Z","type":"stable","version":"v0.3.2"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.2rc3","date":"2025-08-19T12:01:12Z","type":"prerelease","version":"v0.3.2rc3"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.2-rc2","date":"2025-08-19T11:49:23Z","type":"prerelease","version":"v0.3.2-rc2"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.2-rc1","date":"2025-08-19T10:28:33Z","type":"prerelease","version":"v0.3.2-rc1"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.1","date":"2025-08-11T11:15:03Z","type":"stable","version":"v0.3.1"},{"url":"https://github.com/vibrantlabsai/ragas/releases/tag/v0.3.0","date":"2025-07-17T05:32:24Z","type":"stable","version":"v0.3.0"}]` |
| security.iso27001 | yes | - |
| security.soc2 | yes | - |
| security.vulnerabilities | - | `{"count":2,"source":"https://advisories.ecosyste.ms/api/v1/advisories?ecosystem=pypi&package_name=ragas&per_page=100","last_12m":2,"max_severity":"HIGH"}` |

## Capabilities (AI Evals Testing)

| Capability | Promptfoo | Ragas |
|---|:--:|:--:|
| **Capabilities** |  |  |
| Architecture model | - | Open source CLI |
| LLM as a judge prompt grading framework | ✓ | ✓ |
| Specialized rag metrics faithfulness context relevance | - | - |
| Deterministic regex and json schema assertions | ✓ | - |
| Synthetic test dataset generation from documents | - | ✓ |
| Ci cd github actions pipeline blocking gates | ✓ | - |
| Red teaming and adversarial vulnerability scanning | ✓ | - |
| Multi model side by side ab regression testing | - | - |
| Human in the loop hitl annotation UI | - | - |
| Dashboard analytics for metric drift over time | - | - |
| SOC2 type ii | ✓ | - |
| Mit or apache permissive oss license | - | - |
| Pricing model | - | Free open source |

*Source: Vioscale. Generated 2026-09-01T17:09:17.521Z. "-" = undocumented, not absent.*