Angoff for Test Revisions – What Data Should Judges Get and Review?

We’re revising our certification with some new items, a few revised items, and a large number of existing items with data. For the revised and existing items, should we give the Angoff judges the pValues of the items?

What Data Should Judges Get?

The answer is a qualified ‘no’.

Think of it this way.

Suppose you have a cutscore of 70 on your existing test. And you’d like to give your Angoff judges the pValues of revised & existing items. Why not?

The answer is because for an Angoff rating, you’re asking the judges to give their passing percent estimate of minimally qualified candidates.

But the pValue data from your test is from all candidates – including failing candidates and highly qualified candidates, neither of whom are minimally qualified.

If you want to give your Angoff judges helpful data, give them the item passing percents only for candidates who scored between 70 and 75 on the test. These are justifiably minimally qualified candidates.

What Should Judges Review?

The recommended Angoff procedure is for the judges to take the items themselves and reflect on the difficulty they can anticipate minimally qualified candidates will have in taking the items.

But there is no recommended process for presenting judges the items. The items should have choices in the order in which they are presented to candidates, with no indication of which choice is correct.

If the item choices are randomized in the test, then randomize the items as presented to judges.

Some item banking systems have the first choice as the item key, and it’s convenient to send items listed this way to the judges. But the choices should be randomized so the judges aren’t tipped off to the correct answer.

Can a member of the Board of Directors sit on a certification board (or is this, as we suspect, a threat to impartiality)?

I agree with your potential threat. Usually the Certification Advisory Board will have a non-voting liaison member on the Board of Directors for communication purposes. But having someone from the BOD on your Certification Advisory Board could be intimidating.

Can I Post Relative Domain Difficulties?

I have one question that touches on the ISO standards surrounding the separation of training and certification. As you know, your health checks identify the relative difficulties of the exam blueprint domains. Would we be able to make that information available to the public or could it be perceived as helping candidates prepare for the exam?

We were thinking about something broad like: “Top three most difficult domains on the CIPP/US exam.” Obviously we don’t want to jeopardize our accreditation, but we thought it might be interesting information for our candidates to know.

Thank you for your insight!


Great question, Doug.

You could put the relative difficulties on your website as studying information for all candidates.  As long as the information was freely available to all candidates, there shouldn’t be a problem.

You may also want to point to the online info in your candidate handbook, so as it’s revised online you don’t have to revise the candidate handbook every time the relative difficulties change.

Nice idea!

Can our annual internal audits be conducted by an external auditor (say…yourself), or must they truly be conducted internally by a member of staff not directly responsible for certification?

It can’t be conducted by me… I’d be auditing much of my own work (a no-no).  I can recommend several good external auditors, or you can have external staff conduct the audit.  The problem with external staff is that 1) they typically don’t know much about certification and 2) they aren’t familiar with the 17024:12 standard.

Can staff members work on both training and testing?

The staff members can work on both training and testing for the following functions:

Back office
Appeals Registration & Scheduling (not hearings)
Recertification verification
Credentials management

Staff members cannot work on all of the following.  They must work on either training or testing, but not both:

            Training                                               Testing                                         

Course Content                                   Item Review & Selection
Course Development                          Item Writing
Classroom Teaching                           Proctoring Exams
Writing Course Quizzes                     Evaluating Item Parameters
Creating Examples                             Writing Item Scenarios

The general rule is that people involved in training cannot be exposed to the specific content of the test.  This is to keep them from writing / designing / lecturing to the test, instead of to the general content to be mastered.

Can the Appeals Board be comprised of members of the Certification Advisory Board?

Not a good idea.  The problem is that the CAB is responsible (ultimately) for the development of the cert, and may feel defensive of it.  Better to have one CAB member, perhaps corporate counsel, and someone else clearly a disinterested party.

Can the Certification Advisory Board have authority for suspending, withdrawing, or changing the scope of certification or must this authority lie with the Scheme Committees?

That responsibility can lie with any committee.

9.5.1 The certification body shall have a policy and (a) documented procedure(s) for suspension or withdrawal of the certification, or reduction of the scope of certification, which shall specify the subsequent actions by the certification body.

Can the Exam Development Board and the Scheme Committee be comprised of the same members and run by the same chairpersons?

They can, if the same expertise is required for development of all exams.  Usually, the Scheme Committee has oversight, and the people on the EDB are subject matter experts on the scope of one, or possibly two, exams.

Can the exam include graphics (charts, photos and other images)?

Yes, assuming your delivery vendor can handle them.  These are usually termed ‘exhibits’ in the parlance of test administration.

Can we develop questions on our own and then share these questions when we meet as a group (in the same physical location)?

Yes. Good working process.

Can we modify questions we have developed from other trainings so long as the specifics of the question are changed? Or do we have to create all questions from scratch?

You can adapt items from other trainings, but that tends to break down the wall between training & testing because both items would evaluate the same ability.  A test is a sample of all possible items.  If the training and the test sample the same abilities it compromises the generalizability of the exam.

Can we review the legacy training materials as we develop questions?


Can we strike “physical and mental abilities” and “prerequisites” from the certification scheme elements if they don’t apply, or should they stay as placeholders?

Strike abilities… that’s fine.  You should really think about prerequisites before you strike it.  There may be prerequisites you haven’t thought of that justify perhaps a High School equivalency.

Can we use the blueprint from the certification for the training as well?

Yes.  The blueprint is designed as a content outline for the test & the training.  Thereafter is where the split occurs between training and testing.

The training people don’t actually know what items are going to be on the test… the entire blueprint is a possibility.  So they train for the whole blueprint, and testing samples items from each domain.

Do we have to work together in the same physical location to develop all test questions?


Do you know if there is an ISO/IEC preference for using either Candidate or Examinee when referring to the test taker? I see it both ways and would like to standardize our documentation. Also, first letter capital Candidate or Examinee?

You use the terms in different contexts. A candidate is someone who has applied for certification. An examinee is someone who is taking a test or is about to take a test. So all examinees are candidates. But not all candidates are examinees (some candidates haven’t scheduled their test and some have completed their test).

Capitalize as you wish – just be consistent. Hope this helps.

From an auditor’s point of view, exactly what is the distinction between fair and impartial? I’m thinking impartial is that we don’t make any distinctions for who can take the exam and get certified. I assume fairness is that we have an exam that is fairly graded, etc.

You ask a great question about the difference between ‘fair’ and ‘impartial’, and your surmise is along the right lines.

The difference is the standard of reference.  The reference for ‘fair’ is what is culturally or commonly accepted.  The reference for ‘impartial’ is other people or organizations.

So if I pay you $500 for damages to your car that may be ‘fair’ in that it is what is commonly paid for that type of damage.

If I pay one person $500 and another person $5,000 for the same damage, that is not impartial, in that personal attributes are likely to play a part in the decision to pay differential fees.

Impartiality implies that all persons are treated the same regardless of affiliation, race, color or creed.

Fair implies that the treatment met some ethically or culturally determined standard, such as ADA requirements, test speededness, test reliability, item authenticity, blueprint conformance, etc.  Notice that to be fair you sometimes have to distinguish between people on some culturally or legally accepted principle.

Hope this helps…

How can I specify required training for our 17024 accreditation exam?

From Jennifer:

Hypothetical Scenario: Our organization has achieved ANSI 17024 accreditation on our technician exam. I can list on our website that training in the areas of overhead crane maintenance and inspection is required prior to taking my exam. However, I cannot say that the training must come from my training company, or dictate what training companies a participant must attend.

Is this a true statement?


Great question.  I like the way you posed it.  Let’s parse this.

  • I can list on our website that training in the areas of overhead crane maintenance and inspection is required prior to taking my exam.  TRUE
  • However, I cannot say that the training must come from my training company,  TRUE
  • or dictate what training companies a participant must attend.  QUALIFIED:  You can require that the alternate training meet certain content specifications (e.g., meet your blueprint) and require that trainers meet certain qualifications (e.g., that they are certified in a certain way, have a minimum amount of experience as an overhead crane operator, etc.)  You have to provide a way for alternative training providers to become approved by you.

What you cannot say (on your website or in your promo material) is that candidates who take your training will have an advantage over candidates who took training elsewhere. 

You can make a statement such as “80% of candidates who take our training become certified..”

How many questions are we to develop for the two exams?

For adequate reliability, you should have approximately 120 items or so in each form of the exam.  Note there should be 30-40 anchor items which appear on both forms.  In addition, there should be about 25 trial (unscored) items in each form to build up an item pool of parameterized items. 

I have reviewed the 17024 Standard and cannot find a requirement for a specific period of time following a Candidate failing an exam before retaking the same exam.

17024 contains no explicit limitation on the required delay between failing an exam and doing a retake.

However, your assessor will want to review your consideration of the number of alternative forms available, the likelihood of memorizing the items, and the possibility of practice effect on examination results.  A standard would be 6 weeks to 6 months.

I’m hoping you can provide some clarity on the ISO guidelines for certificates, specifically the unique ID. Is this ID unique to the certificate or the certificant? If it is unique to the certificant and the certificant holds multiple designations, can this same ID be on each certificate that they hold or do we need to augment it in some way? Does it have to be updated in any way when a new certificate is issued at recertification? If it is unique to the certificate, does this ID need to be updated each time a new certificate is issued at recertification? For the certification effective date, is this the date the certification was granted, the date of recertification, or is it best practice to include both?

I like one # for all certifications because it’s simpler… but then when someone wants to verify the certification, you have to make sure you filter by certification before you respond.

If you do multiple IDs for multiple certs, then the candidate has a raft of numbers to remember and document, and you have determine see which one to refer to when you correspond.  Not ideal.

Their ID, whichever way you do it, should be good for the life of the certification, meaning the original and all subsequent recerts.

The recert date is the date the certification was issued.  It is also the date on which the recertification was issued.  You may choose to include a field, “Certified since 2014”  or “Initially certified 2016” or whatever, but this is optional.  Most people don’t do it because how do you handle gaps in certification?

If a member of the Exam Development Committee wanted to use an item they wrote for the exam for another effort, are they allowed to do so? My first thought was no, but I thought I would check with you. Maybe if they revise the answer or question in some way?

Those items are the intellectual property of your certification body, and shouldn’t be used for any other purpose.
Exposure is the bane of item pools.  Items that become in any way ‘public’ should not be used on the exam.

Is a non-compete necessary for compliance to standard 5.2.3, or is an NDA on its own sufficient?

The non-compete is necessary.  Seems ISO/IEC wants to make sure that folks don’t leave you and apply the knowledge they got to a competitor.

Hard to enforce… but that’s what 17024 suggests.  Note that your lawyer may inform you that your state makes non-compete agreements unenforceable.

Is it possible to identify by email or a web meeting which topic areas for which each of the exam-writing committee members will be developing questions?

Sure.  Usually a facilitator (me or someone who is managing the process) will determine the number of items needed to be written in each domain, and people can volunteer to write items on specific topics within the domain.

Or are we limited to just the KSAs (Knowledge, Skill, Ability) that are provided by the scheme committee?

You are limited to the blueprint.  You are supposed to develop items independent of any training materials.  There is supposed to be a Chinese wall of no communication between training and testing.

Top management is thinking of creating/marketing a “new” credential to recognize people who are complete professionals, or Masters of [Content]. These people would be required to have the basic and intermediate credential (thus showing they know both the laws and how to manage or implement them) and an experience requirement of some sort. Would there be any barrier to us creating this new credential and would this affect the accreditation of the separate components used as prerequisites?

Not a problem.  Grouping them would not affect your other credentials.

If you want to create a ‘crown credential’, you could have a composite credential that requires:

  1. Experience
  2. Exam 1
  3. One of Exam 2 or Exam 3 or Exam 4
  4. A summary exam

This ‘crown credential’ would be able to be accredited under 17024.  It’s parallel to Oracle’s Certified Master program, or the Cisco Certified Network Engineer program or Citrix’ CNA (Certified Network Architect) program.

You could use any certification in step #2, and any other certifications as additional in step #3.

Whether you want to accredit the ‘crown credential’ or not is up to you.  If you don’t have an exam specific to the credential it would not be accreditable under 17024.

We are considering contacting a subset of certificants via phone if they are in danger of suspension to inquire why or if they intend to let their certification lapse, and if they don’t intend their certification to lapse, to help direct them to ways to complete their CPE requirement according to the CPE policy. Would we need to worry about this being any sort of threat to impartiality?

No threat to impartiality.  I view it as a very nice service to your certificants.

Just make sure that if they say ‘No’ you don’t continue with your reminders.

And make sure to include it in your impartiality threat analysis, with a clarification that you don’t ‘cherry pick’ candidates with certain scores or credentials for these notifications.

We just discovered a self-published “study guide” for Foundation available online at Amazon and Barnes & Noble. The author includes 150 sample questions. We’re ordering the book to be sure none of the sample questions are live items on our operational forms, but we wondered if you have any experience dealing with a situation like this.

Regarding the ‘self-study guide’, if the items appear on your exams and you have your exams copyrighted, that’s clearly copyright infringement and you should have your counsel shut down the book.  If the items are made up and the book states they appear on your exam, you can request a correction.  If your exams are not copyrighted then you don’t have a strong case to shut down the book… you might want to contact an Intellectual Property lawyer to see what your options are.

What are our obligations to our certificants if we decide to sunset a credential? Would the holders of a defunct certification effectively not hold that certification anymore (perhaps effective on their recertification date)? Or are we obligated to keep a recertification plan in place?

You are only obligated to maintain a method of verification for the credential while they are certified.

You should notify certificants when the credential will be no longer offered, and probably advise candidates that the credential will not be offered after a specific date.

As a courtesy, for two years after the last credential has been terminated, you may want to make available online a list of certificants and the last dates they were certified.  E.g., Jane Smith, ID 38747, – Held ABC credential from 2/2010 – 5/2017.  This would mitigate the decline in value resulting from the termination of the certification.

As far as I know, there are no legal guidelines for sunsetting a credential.

What are the working limitations for the exam-writing committee?

Very few.

What constitutes an assessment mechanism? I want to confirm what the JTA committee must do (and document) to fulfill 8.4.d below. The Job Task Analysis shall: …confirm the assessment mechanisms and examination content; 8.4 The certification body shall have documents to demonstrate that, in the development and review of the certification scheme, the following are included: d. the identification and alignment of the assessment mechanisms with the competence requirements;

Your ‘assessment mechanism’ is performance testing, as opposed to essay questions or multiple choice.

Other certification bodies might have an ‘assessment mechanism’ of multiple choice, essay questions, or structured interview, etc.

What is an appropriate ethics statement regarding disability notification?

The statement below would satisfy most conditions.

17024 Ethics Statement Regarding Disability
Should I become fully or partially disabled and unable to carry out the full responsibilities of a [Certification], I will immediately notify [CB] of the nature and extent of my incapacity.   I will advertise and contract to undertake only those capabilities as a [Certification] which I am able to competently fulfill.

Sample:  Should I become fully or partially disabled and unable to carry out the full responsibilities of a Certified Framing Carpenter, I will immediately notify The Carpentry Association of the nature and extent of my incapacity.   I will advertise and contract to undertake only those capabilities as a Certified Framing Carpenter which I am able to competently fulfill.

In your policies & procedures, you’ll need to include a process that confirms receipt of the above statement, and somewhere records and advertises the disability extent of the certificant.

What is the expected time period for the exam takers to complete the exam?

Usually 2 ½ to 3 hours, depending on item complexity.  If you go much more than 3 ½ hours you get into issues of bio breaks, how to handle people coming and going from the exam room, food (especially for diabetics), etc.  If you really need a long exam (5 hours or more) it’s best to break it up into two clearly delineated time sessions.

What math level is expected of the BOC graduates taking the exam and can the exam include calculations?

To be discussed.  The exam can certainly include calculations… the level of complexity is dictated by the committee’s view of who is taking the exam and the competence required and by the calculation support (calculators, spreadsheets) available to the candidate.

What percentage of continuing competency/education credits can be more general credits that cover multiple designations?

You want 50-60% of your CEUs to relate directly to your Blueprint.  The other 40% or so can be more generic.

What type of questions are to be created? Are the tests multiple-choice only?

You should discuss this first.  The answer to what type of items may be a function of the time available, budget, your delivery vendor and your imaginations.  It would be a good idea to have an initial meeting on this question alone.  You’ll want to know what types of items are available prior to assigning authors topics.

Would it be OK for a candidate taking an exam to audit the proctor administering the exam? Or do I need to hire someone that is not taking the exam?

I’m glad you recognize that there could be a conflict of interest here, especially if the candidate is unqualified or borderline.

You don’t need to hire reviewers, but if you’re going to use candidates to review proctors, I’d make sure that at least two candidates in the same administration were surveyed on the proctors’ performance.  Have them get a review sheet prior to the administration, not take notes during the administration, fill out the review sheet after the administration, and submit their reviews independently without interacting.  In fact, they should not even know there is another reviewer in the test administration area.