About

CGM Comparator

CGM stands for Code Generation Model. CGM Comparator compares the code that several such models write for the same task.

What it does

You describe a programming or data-science task, optionally with a dataset, and choose up to five models. Each model generates code for the same requirement. The platform then runs every candidate in an isolated sandbox, scores it, annotates it by functional block, and shows the candidates side by side. You can refine a candidate and compare the new version against the earlier ones.

  • Tasks without a dataset are checked against one shared set of test cases, derived from your requirement before generation and applied to every model.
  • Tasks with a dataset are executed with your files. The result tells you whether the script ran to completion; it is not a check that the analysis is correct.
  • Alongside execution results, each candidate gets efficiency, reusability and editability scores.

What it is not

CGM Comparator is a developer tool. It is not a benchmark, a certification, or a guarantee that generated code is correct, secure or fit for production. Review generated code before you rely on it.

Where it comes from

CGM Comparator was built by Xinxin Yu. It grew out of research at the University of Waterloo on helping people understand and compare AI-generated code; the research prototype was called DSCode Comparator.

Contact: x57yu@uwaterloo.ca

How it is built

A web app for submitting tasks and reading results, a Python (FastAPI) backend that orchestrates generation and evaluation, an executor that runs generated code in Docker containers with time, memory and output limits, and a PostgreSQL database. Generation requests go to models from OpenAI, Anthropic, Google, Cohere, DeepSeek. Payments are handled by Stripe.

Plans

One credit = one multi-model run or one refinement. The free plan includes 10 one-time credits; paid plans add credits per billing period. See pricing.

Open the app. You'll sign in with the same Google account in the app.