Adaptive Diagnostic Test: Determine Proficiency in 15 Questions with IRT and CAT
Learn about the adaptive diagnostic approach, which selects the next item based on each participant’s response instead of giving everyone the same test. IRT, CAT, the limits of a 15-question design, and enterprise governance criteria—all in one guide.
Long tests administered in the same way to everyone are not always a good starting point for understanding an employee’s or student’s proficiency. Very easy items provide less measurement information, while very difficult items may not distinguish the participant’s actual level. An adaptive diagnostic test, by selecting the next question based on information from the previous response, brings the test experience and measurement focus closer to the individual.
In this article, we explain Item Response Theory (IRT) and Computerized Adaptive Testing (CAT) in straightforward terms. We also treat “determining proficiency in 15 questions” not as a marketing promise, but as a design goal that may be considered when there is a properly calibrated item bank and a clearly defined purpose. A diagnostic result should not, on its own, determine hiring, promotion, certification, or disciplinary decisions.
If your organization plans to consider skills, roles, and curriculum data together, contact the Qulak team to discuss the scope of an adaptive diagnostic scenario, integration needs, and validation steps. This invitation does not mean that any particular module or outcome is guaranteed.
What is IRT, and how does it contribute to proficiency measurement?
Rather than reducing an observed response directly to a total score, Item Response Theory relates it to the latent trait underlying the response. In education, this trait may be a level of knowledge or skill. An IRT model estimates the probability that a person at a given ability level will answer a particular item correctly; this allows item characteristics and individual proficiency to be examined within the same measurement framework.
Three key signals in an item bank
- Difficulty: Indicates the level at which an item is challenging for participants.
- Discrimination: Shows how well an item distinguishes between participants at different levels.
- Guessing probability: Particularly for multiple-choice items, this can help model the chance of a correct response due to guessing by participants with lower ability levels.
These parameters cannot simply be assumed when writing items. The item bank should be calibrated using a sufficient and appropriate sample; subject-matter experts should review content accuracy, and measurement experts should examine model fit. In a new skill area, conventional test forms, a pilot, and expert review may be needed initially.
How does CAT work? The five-step measurement cycle
CAT does not simply mean “AI selects the questions.” The items the algorithm selects, the order in which it selects them, and the stopping rule are based on a predefined measurement design. A typical cycle is as follows:
Initial estimate
A neutral initial ability estimate is used for the participant, or reliable information previously collected with consent. This information should not include unnecessary personal data.
Selecting an item with high information value
The algorithm selects an item from those expected to best distinguish the participant based on the current estimate. Content area, item exposure, and security constraints limit this selection.
Recording the response
In addition to correct or incorrect responses, data such as response time or hint use should be considered only when the measurement purpose is clearly defined and the need for the data has been demonstrated.
Updating ability and error estimates
As each response comes in, the estimate and standard error are recalculated. CAT’s value lies not only in being short, but also in making measurement uncertainty visible.
Stopping and reporting
The test ends when the error threshold, minimum or maximum number of items, content coverage, or time limit is reached. The report should present uncertainty and coverage alongside the estimate.
When does 15 questions make sense?
Fifteen items may be a reasonable initial hypothesis for a quick diagnostic screening in a narrow, well-defined skill area. However, the number alone is not a measure of reliability. A broad competency area, multiple subskills, low-quality items, or a decision requiring high precision may call for more items and different sources of evidence.
Therefore, the report should do more than state “Level 3”; it should explain which skills framework was used, the estimation error, any areas not covered by responses, and which decisions the result is not suitable for. Adaptive design has been used in PISA’s computer-based assessments to better measure performance at the extremes; this example shows that CAT depends on the context and design details.
Measurement architecture for enterprise implementation
The success of a CAT project depends on measurement design before it depends on the algorithm. Before asking “how many questions?”, the organization should answer: “Which skill will we measure, what decision will it support, and what evidence will we use?” The framework below can be used to plan a pilot.
| Layer | Review question | Output |
|---|---|---|
| Skills model | Have proficiency levels and subskills been clearly defined by experts? | Competency glossary and proficiency rubric |
| Item bank | Have items been reviewed for content, language, accessibility, and IRT evidence? | Calibrated and versioned item pool |
| CAT rules | Have content coverage, item exposure, and stopping criteria been defined? | Selection algorithm and security constraints |
| Results layer | Are uncertainty, explanations, and human review reflected in the report? | Diagnostic report, recommendation, and appeals process |
| Governance | Are data access, retention, model changes, and responsibilities documented? | Access matrix, record policy, and audit trail |
Qulak’s skills-competencies-roles-curriculum chain approach can provide a framework for discussing these data relationships. However, each organization should validate its own skill definitions, content ownership, and measurement evidence.
Neutral checklist for procurement and pilot decisions
At the BoFU stage, making a decision based only on the “adaptive” label or a short test duration is risky. Request the following evidence from the vendor and, where possible, validate it through a limited pilot:
| Criterion | Evidence to request | Red flag |
|---|---|---|
| Measurement validity | Item calibration, sample, error analysis, and update method | Only a total score or an accuracy claim without a source |
| Content coverage | Blueprint, balance of subskills, expert approval, and accessibility testing | Assumption that the algorithm will automatically resolve content gaps |
| Transparency | Selection, stopping, level mapping, and human appeals processes | “Black box” score or unexplained level |
| Data and security | Data minimization, retention, access, deletion, and subprocessor terms | Collection of unnecessary personal data or an unclear deletion policy |
| Integration | SSO, LMS/LXP, API, export, and fallback plan | Inability to transfer data or reporting available only through the vendor |
| Total cost of ownership | Item development, calibration, maintenance, support, licensing, and pilot costs | Comparison of only the initial license price |
Qulak may be considered by organizations that want to connect skills architecture with diagnostic results and can establish internal ownership of data governance. Before evaluating it, confirm in writing Qulak’s current scope, the conditions under which IRT/CAT capabilities are offered, integration limits, and data responsibilities under the contract. This article does not guarantee any specific product feature, performance percentage, or procurement outcome.
AI, data minimization, and the limits of human oversight
Limit data to its purpose
In most scenarios, the core data required for adaptive diagnostics are the participant’s system-level identifier, item ID, response, timestamp, and measurement context. Identity, demographics, behavioral signals, or free-text responses should be considered only when there is a clear purpose, legal basis, and benefits analysis. Pseudonymizing data, restricting access by role, and setting the retention period in advance are good starting points; the organization’s obligations under KVKK and relevant contracts should also be reviewed separately.
Keep a human as the decision-maker
A model may classify a participant as “low proficiency,” but this result should be treated as a signal for personalizing learning support, not as a final judgment about the person’s worth or career. An instructor or measurement expert should be able to review unusual responses, accessibility needs, and language effects, correct the result where necessary, and record any appeal.
Governance is an ongoing process
When the item bank, proficiency thresholds, or selection algorithm changes, results may not be directly comparable with previous results. Versioning, change approval, retrospective reporting, bias monitoring, and periodic recalibration are part of ongoing operations, not one-time checks. If AI-assisted item generation is used, each item should be reviewed by a subject-matter expert and a measurement expert before publication; automated generation does not replace human approval.
Checklist before a 15-question pilot
- Have the skills to be measured and observable behaviors for each level been defined?
- Is each subskill represented sufficiently and evenly in the item bank?
- Have the calibration sample, model selection, and error criteria been documented?
- Is it documented that the 15-item target is a screening tool and is not used for high-stakes decisions?
- Have content constraints, item exposure, and test security been tested?
- Were accessibility, language, device, and connectivity conditions observed during the pilot?
- Does the participant know what data are processed, why they are processed, and how long they will be retained?
- Are human review, appeals, corrections, and exception workflows in place?
- Do results show uncertainty and a coverage note alongside learning recommendations?
- Is there an exit plan for API access, data export, data deletion, and changing vendors?
Evaluate the adaptive diagnostic approach for your organization
Whether a CAT design is appropriate depends less on the 15 questions than on the skills model, item evidence, data governance, and human oversight. If you are evaluating Qulak, clarify your purpose, current systems, and validation criteria together.
Discuss the conditions with QulakFrequently Asked Questions
What is the difference between IRT and classical scoring?
Can 15 questions reliably determine everyone’s proficiency?
What data should be retained when using CAT?
When should Qulak be considered for this approach?
Sources
The key definitions and implementation limits in this article have been reviewed against the following reliable sources:
- PubMed: Item response theory, computerized adaptive testing, and PROMIS — Explains the relationship between IRT, item banks, and CAT’s selection of items based on previous responses.
- OECD PISA 2022: Adaptive testing — Explains how adaptive design is used in reading and mathematics assessments and its purpose.
- Standards for Educational and Psychological Testing — Provides access to fundamental standards for validity, fairness, and responsible testing.
Related Qulak guides
Plan the next step with evidence
Share the target skills, pilot scope, data boundaries, and procurement criteria for your adaptive diagnostic project. If appropriate, let’s consider Qulak using a neutral evaluation framework.
Contact us