At a glance
- Test feasibility, quality, and explainability separately; a schedule can score well and still be one staff won't trust.
- Build the baseline from your own last manually built schedule, not from a vendor's demo or a generic benchmark.
- Agree on pass criteria before you see the results, or the evaluation quietly becomes a negotiation over what counts as good enough.
- Test how the system handles a small change after publication, not only how it performs on the first solve.
What a Good Schedule Means, in Measurable Terms
"Better schedule" isn't a metric until you define it. One course-timetabling solver's documentation separates two questions in sequence: whether a placement can be made at all, and only once that's settled, how well it meets everyone's preferences [1]. Your evaluation should keep the same order: judge feasibility on its own before you look at any quality score.
In that documentation, a feasibility run is one where the solver "ignores any preferences entered ... and only checks if it is possible to find a solution that meets all of the problem requirements" [1]. Quality is scored afterward, once the system "attempts to optimize satisfaction of the preferences ... as well as minimize student conflicts" [1]. Don't accept a single blended number; ask for the feasibility result and the quality result reported separately.
Separating feasibility from quality matters because they fail differently. An infeasible schedule is unusable no matter how well it scores on preferences. A feasible-but-low-quality schedule is usable but will generate complaints. Score them separately in your test plan, so a vendor, or an internal team, can't offset a feasibility problem with a strong preference score.
Build a Baseline From the Schedule You Already Have
The only fair comparison for a new optimizer is your own institution's last manually built schedule, scored against the same metrics you're about to apply to the new system. A vendor's demo data or a published benchmark tells you how the solver performs on someone else's problem, not yours.
- Pick a term that's already been taught and settled, so you know the schedule actually worked, not just that it was published.
- Score that manual schedule against your hard-constraint list and count every soft-constraint violation and student conflict it contains, exactly as you'll score the optimizer's output.
- Keep this baseline written down before you run anything new. Once you've seen the optimizer's results, it's tempting to remember the manual schedule as worse than it was.
- Where you can, also replay using the data as it stood when the manual schedule was originally built, not only the final, settled data the optimizer sees now. The optimizer has an information advantage the manual planners didn't have at the time, and comparing it only against final data overstates that advantage.
The Test Plan
Run each test below and agree the pass criterion before you see results. Sign-off responsibility should sit with someone accountable for the outcome, not with whoever is running the technical evaluation.
| Test | How to run it | Metric | Pass criterion (agree in advance) | Sign-off |
|---|---|---|---|---|
| Replay last term | Feed the optimizer your validated last-term data as if building it fresh | Feasible schedule produced (yes/no) | Must produce a feasible schedule; a failure here stops the evaluation | Registrar |
| Hard-constraint audit | Check the output against every hard constraint using a second, separately built checking tool that did not itself generate the schedule | Count of hard-constraint violations | Zero violations, verified independently, not just reported by the solver | IT / independent reviewer |
| Student conflict rate vs. manual schedule | Count conflicts in the optimizer's output and compare to the baseline | Conflict count for the optimizer's output and for the baseline, both counted with the same student-demand data | At minimum, no worse than the baseline; define a target improvement only if you can defend the number | Registrar / institutional research |
| Room utilization | Compare assigned room capacity against enrolled or expected section size | Percentage of sections in an appropriately sized room (room capacity at or above enrolled/expected size, within a fill-ratio band your institution sets) | Agreed threshold based on your own facilities constraints, not a generic target | Facilities |
| Instructor preference satisfaction | Score soft-constraint penalty for instructor-related preferences against the baseline | Total penalty, and count of preferences fully satisfied | No worse than the baseline on the preferences your institution weighted as high-tier | Department chairs |
| Change-request re-solve | Introduce one realistic change (an instructor becomes unavailable, a section is added) and re-solve | Number of previously placed sections that moved | Movement limited to what the change actually required; large unexplained reshuffling fails | Registrar |
| Explainability of a placement | Ask the system, or the team operating it, to justify a specific placement and every soft constraint it broke | Staff can explain the placement without developer help (yes/no) | Every sampled placement is explainable in plain terms by registrar staff | Registrar |
| Runtime at full scale | Run the full institution's data, not a subset | Wall-clock time to a feasible, scored solution | Fits your actual scheduling calendar, including time for staff review before publication | IT |
| Stability after a small edit | Make one minor manual correction to the output and re-solve | Number of additional sections that move beyond the one corrected | Minimal; large ripple effects from a small edit fail this test | Registrar |
Test Explainability, Not Just Scores
Ask whether the system records why each alternative assignment was rejected, not just why the final placement was chosen. One course-timetabling solver's documentation describes a per-class report listing possible times together with "the student conflicts and violated distribution preferences that will occur if the time is selected for the class" [1]. Ask any system you evaluate for the same kind of report, in language your staff can read without a developer.
Pick several placements at random, including at least one that broke a soft constraint, and ask the system or the team running it to state which rule was broken and why the alternative was worse.
Test How It Handles Change
A schedule is rarely used exactly as first solved: instructors change, sections get added, rooms become unavailable. A re-solve has a different goal from the first solve, one course-timetabling research defines directly: "A minimal perturbation problem incorporates these changes ... as a new problem whose solution must be as close as possible to the initial solution" [3].
One open-source university timetabling system's lead developer, in a separate conference abstract, describes the software as supporting "managing changes to these timetables" alongside developing the original course and exam timetables in the first place [2].
That's the specific behavior to test for: introduce one small, realistic change and count how many other, unrelated sections move as a result. A well-behaved re-solve holds everything fixed except what the change actually requires; a system that reshuffles broadly for a single edit is optimizing for the wrong objective on a re-solve.
Red Flags in a Demo
- A demo run only on the vendor's own sample data, with no offer to run your actual data before you commit.
- A quality score presented without a definition of what it measures or how penalties are weighted.
- No answer to "why was this section placed here" beyond "the optimizer decided."
- No way to independently check hard-constraint violations outside the vendor's own reporting.
- A change-handling demo that reshuffles the whole schedule for one small edit, framed as a feature rather than a problem.
- Any performance number quoted from another institution's deployment, presented as what you should expect at yours.
Deciding With the Results
Score every test as pass, fail, or conditional, against the criteria you set before you started. A single strong quality score doesn't offset a failed hard-constraint audit or an unexplainable placement; treat feasibility and explainability as gating tests, and quality and preference scores as the basis for choosing between systems that already pass the gates.
- Score every test against its pre-agreed pass criterion
Not against how it compares to a competing system; that comparison comes next.
- Treat feasibility and hard-constraint audits as gates
A system that fails either doesn't advance, regardless of its quality score.
- Compare passing systems on quality, explainability, and change-handling
This is where a genuine trade-off analysis belongs.
- Record the decision and the test results together
Whoever operates the system in two years should be able to see why it was chosen, not just that it was.
When This Evaluation Says You Don't Need a New Optimizer
A pass isn't the only useful outcome. If your current tool or your manual process already passes the replay and hard-constraint tests, and your baseline scores as well as anything a new system would be held to, that's a valid result: it tells you the mechanical evaluation isn't where the case for change lives, and you should look elsewhere (staff time, error-proneness) before investing in a new optimizer.
Frequently asked questions
What features matter in course scheduling software?
For evaluation purposes, the features that matter most are the ones this test plan depends on: an independently checkable hard-constraint definition, a documented soft-constraint or quality score, a conflict report tied to each placement, and a re-solve mode that limits change after publication. A generic feature list matters less than whether you can actually run these tests.
How do you test a scheduling optimizer before buying?
Run it on your own data, not a demo dataset. Replay a term you've already taught, audit the output against your hard constraints independently, compare quality and student-conflict counts against a baseline built from your last manual schedule, and test how a small change after publication affects the rest of the schedule.
What should a scheduling RFP include?
Beyond features and pricing, require the vendor to run your own historical data, name every hard and soft constraint the system enforces, describe how it computes a quality score, show how it explains a specific placement, and demonstrate a change-handling re-solve using a scenario you supply, not one they picked.
How do you compare output quality?
Compare feasibility first, since an infeasible schedule isn't usable regardless of its score. For feasible outputs, compare the quality or penalty score and the student-conflict count against a baseline built from your last manually built, already-taught term, not against a vendor's own sample data or a generic benchmark.
What SIS integration questions should you ask?
Ask what data the system needs from your SIS and in what format, whether write-backs have been proven against a full non-production copy of your data before going live, how the system handles a schedule that changes between export and import, and who is notified if a write-back fails partway through.
What does course scheduling software cost?
This article doesn't set a price: scheduling software is priced differently by different vendors and scopes, and no single figure applies to every institution. Ask each vendor for cost tied to your evaluation's scope, and weigh it against how the system performs on the feasibility-and-quality test plan, not on price alone.
Related reading and next steps
- For the underlying workflow and its acceptance criteria, see academic scheduling optimization.
- For background on why manual, one-constraint-at-a-time scheduling breaks down at scale, read academic scheduling optimization at universities.
- For how a system like this is scoped, tested, and handed over, see AI agent implementation.
Sources
- [1] — Course Timetabling Solver Manual, UniTime official help site (accessed 2026-09-29)
- [2] — Tomáš Müller, "ITC 2019: Preliminary Results Using the UniTime Solver" (PATAT 2020 preliminary-results abstract; unitime.org) (accessed 2026-09-29)
- [3] — Tomáš Müller, Hana Rudová and Roman Barták, "Minimal Perturbation Problem in Course Timetabling" (PATAT 2004; published 2005, unitime.org) (accessed 2026-09-28)
More in Higher education
AI Agents vs RPA in Higher Education: Choosing for Banner, Workday Student and Legacy SIS Workflows
AI agents vs RPA in higher education: a vendor-neutral decision table for CIOs choosing between RPA, AI agents or both for Banner and Workday Student workflows.
AI Transcript Processing for Admissions: Extraction, GPA Recalculation and the Exception Queue
AI transcript processing for admissions: how extraction and GPA recalculation work, an exception-routing table, and what should always reach a reviewer.
How to Automate Financial Aid Verification: Document Intake, Mismatch Flags and Where Counselors Decide
How to automate financial aid verification: a document intake and mismatch-flag workflow matrix, plus exactly where a counselor still decides.