We’re revising our certification with some new items, a few revised items, and a large number of existing items with data. For the revised and existing items, should we give the Angoff judges the pValues of the items?
What Data Should Judges Get?
The answer is a qualified ‘no’.
Think of it this way.
Suppose you have a cutscore of 70 on your existing test. And you’d like to give your Angoff judges the pValues of revised & existing items. Why not?
The answer is because for an Angoff rating, you’re asking the judges to give their passing percent estimate of minimally qualified candidates.
But the pValue data from your test is from all candidates – including failing candidates and highly qualified candidates, neither of whom are minimally qualified.
If you want to give your Angoff judges helpful data, give them the item passing percents only for candidates who scored between 70 and 75 on the test. These are justifiably minimally qualified candidates.
What Should Judges Review?
The recommended Angoff procedure is for the judges to take the items themselves and reflect on the difficulty they can anticipate minimally qualified candidates will have in taking the items.
But there is no recommended process for presenting judges the items. The items should have choices in the order in which they are presented to candidates, with no indication of which choice is correct.
If the item choices are randomized in the test, then randomize the items as presented to judges.
Some item banking systems have the first choice as the item key, and it’s convenient to send items listed this way to the judges. But the choices should be randomized so the judges aren’t tipped off to the correct answer.
I agree with your potential threat. Usually the Certification Advisory Board will have a non-voting liaison member on the Board of Directors for communication purposes. But having someone from the BOD on your Certification Advisory Board could be intimidating.
I have one question that touches on the ISO standards surrounding the separation of training and certification. As you know, your health checks identify the relative difficulties of the exam blueprint domains. Would we be able to make that information available to the public or could it be perceived as helping candidates prepare for the exam?
We were thinking about something broad like: “Top three most difficult domains on the CIPP/US exam.” Obviously we don’t want to jeopardize our accreditation, but we thought it might be interesting information for our candidates to know.
Thank you for your insight!
Answer:
Great question, Doug.
You could put the relative difficulties on your website as studying information for all candidates. As long as the information was freely available to all candidates, there shouldn’t be a problem.
You may also want to point to the online info in your candidate handbook, so as it’s revised online you don’t have to revise the candidate handbook every time the relative difficulties change.
Nice idea!
It can’t be conducted by me… I’d be auditing much of my own work (a no-no). I can recommend several good external auditors, or you can have external staff conduct the audit. The problem with external staff is that 1) they typically don’t know much about certification and 2) they aren’t familiar with the 17024:12 standard.
The staff members can work on both training and testing for the following functions:
Registration
Payment
Back office
Scheduling
Appeals Registration & Scheduling (not hearings)
Refunds
Recertification verification
Credentials management
Staff members cannot work on all of the following. They must work on either training or testing, but not both:
Training Testing
Course Content Item Review & Selection
Course Development Item Writing
Classroom Teaching Proctoring Exams
Writing Course Quizzes Evaluating Item Parameters
Creating Examples Writing Item Scenarios
The general rule is that people involved in training cannot be exposed to the specific content of the test. This is to keep them from writing / designing / lecturing to the test, instead of to the general content to be mastered.
Not a good idea. The problem is that the CAB is responsible (ultimately) for the development of the cert, and may feel defensive of it. Better to have one CAB member, perhaps corporate counsel, and someone else clearly a disinterested party.
That responsibility can lie with any committee.
9.5.1 The certification body shall have a policy and (a) documented procedure(s) for suspension or withdrawal of the certification, or reduction of the scope of certification, which shall specify the subsequent actions by the certification body.
They can, if the same expertise is required for development of all exams. Usually, the Scheme Committee has oversight, and the people on the EDB are subject matter experts on the scope of one, or possibly two, exams.
Yes, assuming your delivery vendor can handle them. These are usually termed ‘exhibits’ in the parlance of test administration.
Yes. Good working process.
You can adapt items from other trainings, but that tends to break down the wall between training & testing because both items would evaluate the same ability. A test is a sample of all possible items. If the training and the test sample the same abilities it compromises the generalizability of the exam.
No.
Strike abilities… that’s fine. You should really think about prerequisites before you strike it. There may be prerequisites you haven’t thought of that justify perhaps a High School equivalency.
Yes. The blueprint is designed as a content outline for the test & the training. Thereafter is where the split occurs between training and testing.
The training people don’t actually know what items are going to be on the test… the entire blueprint is a possibility. So they train for the whole blueprint, and testing samples items from each domain.
No.
You use the terms in different contexts. A candidate is someone who has applied for certification. An examinee is someone who is taking a test or is about to take a test. So all examinees are candidates. But not all candidates are examinees (some candidates haven’t scheduled their test and some have completed their test).
Capitalize as you wish – just be consistent. Hope this helps.
You ask a great question about the difference between ‘fair’ and ‘impartial’, and your surmise is along the right lines.
The difference is the standard of reference. The reference for ‘fair’ is what is culturally or commonly accepted. The reference for ‘impartial’ is other people or organizations.
So if I pay you $500 for damages to your car that may be ‘fair’ in that it is what is commonly paid for that type of damage.
If I pay one person $500 and another person $5,000 for the same damage, that is not impartial, in that personal attributes are likely to play a part in the decision to pay differential fees.
Impartiality implies that all persons are treated the same regardless of affiliation, race, color or creed.
Fair implies that the treatment met some ethically or culturally determined standard, such as ADA requirements, test speededness, test reliability, item authenticity, blueprint conformance, etc. Notice that to be fair you sometimes have to distinguish between people on some culturally or legally accepted principle.
Hope this helps…
From Jennifer:
Hypothetical Scenario: Our organization has achieved ANSI 17024 accreditation on our technician exam. I can list on our website that training in the areas of overhead crane maintenance and inspection is required prior to taking my exam. However, I cannot say that the training must come from my training company, or dictate what training companies a participant must attend.
Is this a true statement?
Reply:
Great question. I like the way you posed it. Let’s parse this.
- I can list on our website that training in the areas of overhead crane maintenance and inspection is required prior to taking my exam. TRUE
- However, I cannot say that the training must come from my training company, TRUE
- or dictate what training companies a participant must attend. QUALIFIED: You can require that the alternate training meet certain content specifications (e.g., meet your blueprint) and require that trainers meet certain qualifications (e.g., that they are certified in a certain way, have a minimum amount of experience as an overhead crane operator, etc.) You have to provide a way for alternative training providers to become approved by you.
What you cannot say (on your website or in your promo material) is that candidates who take your training will have an advantage over candidates who took training elsewhere.
You can make a statement such as “80% of candidates who take our training become certified..”
For adequate reliability, you should have approximately 120 items or so in each form of the exam. Note there should be 30-40 anchor items which appear on both forms. In addition, there should be about 25 trial (unscored) items in each form to build up an item pool of parameterized items.
17024 contains no explicit limitation on the required delay between failing an exam and doing a retake.
However, your assessor will want to review your consideration of the number of alternative forms available, the likelihood of memorizing the items, and the possibility of practice effect on examination results. A standard would be 6 weeks to 6 months.
I like one # for all certifications because it’s simpler… but then when someone wants to verify the certification, you have to make sure you filter by certification before you respond.
If you do multiple IDs for multiple certs, then the candidate has a raft of numbers to remember and document, and you have determine see which one to refer to when you correspond. Not ideal.
Their ID, whichever way you do it, should be good for the life of the certification, meaning the original and all subsequent recerts.
The recert date is the date the certification was issued. It is also the date on which the recertification was issued. You may choose to include a field, “Certified since 2014” or “Initially certified 2016” or whatever, but this is optional. Most people don’t do it because how do you handle gaps in certification?
Those items are the intellectual property of your certification body, and shouldn’t be used for any other purpose.
Exposure is the bane of item pools. Items that become in any way ‘public’ should not be used on the exam.
The non-compete is necessary. Seems ISO/IEC wants to make sure that folks don’t leave you and apply the knowledge they got to a competitor.
Hard to enforce… but that’s what 17024 suggests. Note that your lawyer may inform you that your state makes non-compete agreements unenforceable.
Sure. Usually a facilitator (me or someone who is managing the process) will determine the number of items needed to be written in each domain, and people can volunteer to write items on specific topics within the domain.
You are limited to the blueprint. You are supposed to develop items independent of any training materials. There is supposed to be a Chinese wall of no communication between training and testing.
Not a problem. Grouping them would not affect your other credentials.
If you want to create a ‘crown credential’, you could have a composite credential that requires:
- Experience
- Exam 1
- One of Exam 2 or Exam 3 or Exam 4
- A summary exam
This ‘crown credential’ would be able to be accredited under 17024. It’s parallel to Oracle’s Certified Master program, or the Cisco Certified Network Engineer program or Citrix’ CNA (Certified Network Architect) program.
You could use any certification in step #2, and any other certifications as additional in step #3.
Whether you want to accredit the ‘crown credential’ or not is up to you. If you don’t have an exam specific to the credential it would not be accreditable under 17024.
No threat to impartiality. I view it as a very nice service to your certificants.
Just make sure that if they say ‘No’ you don’t continue with your reminders.
And make sure to include it in your impartiality threat analysis, with a clarification that you don’t ‘cherry pick’ candidates with certain scores or credentials for these notifications.
Regarding the ‘self-study guide’, if the items appear on your exams and you have your exams copyrighted, that’s clearly copyright infringement and you should have your counsel shut down the book. If the items are made up and the book states they appear on your exam, you can request a correction. If your exams are not copyrighted then you don’t have a strong case to shut down the book… you might want to contact an Intellectual Property lawyer to see what your options are.
You are only obligated to maintain a method of verification for the credential while they are certified.
You should notify certificants when the credential will be no longer offered, and probably advise candidates that the credential will not be offered after a specific date.
As a courtesy, for two years after the last credential has been terminated, you may want to make available online a list of certificants and the last dates they were certified. E.g., Jane Smith, ID 38747, – Held ABC credential from 2/2010 – 5/2017. This would mitigate the decline in value resulting from the termination of the certification.
As far as I know, there are no legal guidelines for sunsetting a credential.
Very few.
Your ‘assessment mechanism’ is performance testing, as opposed to essay questions or multiple choice.
Other certification bodies might have an ‘assessment mechanism’ of multiple choice, essay questions, or structured interview, etc.
The statement below would satisfy most conditions.
17024 Ethics Statement Regarding Disability
Should I become fully or partially disabled and unable to carry out the full responsibilities of a [Certification], I will immediately notify [CB] of the nature and extent of my incapacity. I will advertise and contract to undertake only those capabilities as a [Certification] which I am able to competently fulfill.
Sample: Should I become fully or partially disabled and unable to carry out the full responsibilities of a Certified Framing Carpenter, I will immediately notify The Carpentry Association of the nature and extent of my incapacity. I will advertise and contract to undertake only those capabilities as a Certified Framing Carpenter which I am able to competently fulfill.
In your policies & procedures, you’ll need to include a process that confirms receipt of the above statement, and somewhere records and advertises the disability extent of the certificant.
Usually 2 ½ to 3 hours, depending on item complexity. If you go much more than 3 ½ hours you get into issues of bio breaks, how to handle people coming and going from the exam room, food (especially for diabetics), etc. If you really need a long exam (5 hours or more) it’s best to break it up into two clearly delineated time sessions.
To be discussed. The exam can certainly include calculations… the level of complexity is dictated by the committee’s view of who is taking the exam and the competence required and by the calculation support (calculators, spreadsheets) available to the candidate.
You want 50-60% of your CEUs to relate directly to your Blueprint. The other 40% or so can be more generic.
You should discuss this first. The answer to what type of items may be a function of the time available, budget, your delivery vendor and your imaginations. It would be a good idea to have an initial meeting on this question alone. You’ll want to know what types of items are available prior to assigning authors topics.
I’m glad you recognize that there could be a conflict of interest here, especially if the candidate is unqualified or borderline.
You don’t need to hire reviewers, but if you’re going to use candidates to review proctors, I’d make sure that at least two candidates in the same administration were surveyed on the proctors’ performance. Have them get a review sheet prior to the administration, not take notes during the administration, fill out the review sheet after the administration, and submit their reviews independently without interacting. In fact, they should not even know there is another reviewer in the test administration area.