New Computerized Method of Pers Assessment Being Tested

[The Bread Guy]

At least that's what I'm picking out of the bureaucratese from this MERX posting:

.... Performance Appraisals (PA) are recognized as being the most important human resource practice after employee selection. The outcome of any PA has significant impact on an employee's career advancement and development opportunities, and in the same vein, on the organization as a whole. It can be logically understood that accurately assessing and developing employees leads to a stronger, better performing organization. Yet, despite the extensive and widespread use of this human management process, there are serious concerns about the soundness of performance ratings. Literature shows that performance appraisals often fail to deliver the accurate assessments they are intended to offer.

(....)

Developed by Borman et al (2001), the Computerized Adaptive Rating Scale (CARS) has shown promise in laboratory studies as a more reliable and valid method for evaluating performance in comparison to existing rating formats. Computerized Adaptive Rating Scales (CARS) combine the process of paired-comparison, where a respondent is asked to choose the most relevant or true statement out of two options, with Computerized Adaptive Testing (CAT) technology.

(....)

The aim of this project is to use this innovative technique to develop a performance evaluation protocol for the Canadian Forces Personnel Appraisal System (CFPAS), and hence, better calibrate rater judgment. The successful application of this method will be groundbreaking for the field of HR Management.  The advantages for the CF of implementing the tool are:  economies of millions of dollars annually, a significant saving of time and human resources, future leaders selected more accurately, perceptions of a fairer and more transparent system, higher employee satisfaction with overall career management and higher retention. Overall, this tool will contribute to a stronger CF, ensuring mission success.

This project aims at developing and applying a revolutionary method for rating CF personnel performance utilizing modern computer technology to address the limitations and weaknesses of the current CFPAS. The objectives of the project are:
- To develop more comprehensive definitions and descriptors of the assessment factors constituting the Leadership
- Development Framework at each rank.
- Generate and validate behaviours reflecting a wide range of effectiveness for each assessment factor.
- Populate and test the computerized adaptive rating scales system.
- Design and conduct a field experiment with CF personnel to test the reliability and overall effectiveness of the computerized adaptive paired-comparison rating scale system.
- Complete the project with final project report and research reports ....

Company currently getting the job: 
PDRI (Personnel Decisions Research Institutes)
100 S Ashley Drive, Suite 375
Tampa, Florida 33602
(More about PDRI at the company's site here)

Estimated Value:    $600-700,000.00 CDN

More on CARS in this paper from the winning company/institute.

If I read correctly, instead of coming up with narrative, you may eventually be able to answer a series of questions from the computer to help narrow in on specifics about the effectiveness (or not) of the person you're writing about.  My opinion:  since this sort of thing is "art" as much as a science, any computerized model with questions based on a series of qualities won't cover everything.  Can such assessments EVER be 100% objective, 0% subjective?  I don't think so.

 

[Infanteer]

Sweet - no more PERs!

I seriously doubt some computerized system of yes/no questions will supplant the Byzantine structure that is PER bubbles....

 

[Kat Stevens]

At least that's what I'm picking out of the bureaucratese from this MERX posting:
Company currently getting the job: 
PDRI (Personnel Decisions Research Institutes)
100 S Ashley Drive, Suite 375
Tampa, Florida 33602
(More about PDRI at the company's site here)

Estimated Value:    $600-700,000.00 CDN

More on CARS in this paper from the winning company/institute.

If I read correctly, instead of coming up with narrative, you may eventually be able to answer a series of questions from the computer to help narrow in on specifics about the effectiveness (or not) of the person you're writing about.  My opinion:  since this sort of thing is "art" as much as a science, any computerized model with questions based on a series of qualities won't cover everything.  Can such assessments EVER be 100% objective, 0% subjective?  I don't think so.

NO assessment system is objective.  Personal likes and dislikes will ALWAYS colour an assessment of someone, no matter how we like to pretend they don't.

 

[Infanteer]

NO assessment system is objective.  Personal likes and dislikes will ALWAYS colour an assessment of someone, no matter how we like to pretend they don't.

Yep - especially in a small military.  Good PER be damned if, come the boards, people are going "who?" or "oh....him."

Exams used to be used by the Army (I believe the Navy still uses them) and Exams were huge in earning spots to prestigious staff colleges and a shot at the General Staff for armies such as the Germans.  Does anyone feel that a proper system of leadership examinations could help to make leadership progression more objective?

 

[a_majoor]

Yes

 

[Franko]

Doesn't matter anyways. Your assessment is based solely on your Merritt board score, regardless of the narratives or PDRs.

Regards

 

[Kat Stevens]

Exactly, and merit boards are made up of people who are (supposedly) fighting for their guys positions on the list.  Often the guy with the loudest boss and the most friends in the room wins.

 

[Nfld Sapper]

Just put thost that are up for promotion in a steel cage and the last man/woman standing gets the promotion.....

;D

 

[vonGarvin]

Does anyone feel that a proper system of leadership examinations could help to make leadership progression more objective?

I wholeheartedly agree.  For the army, I believe that some sort of exam or other method should replace the automatic promotion to Captain.  Perhaps AJOSQ as a start?  I'm not sure what they used to use, but I am pretty sure that promotions to captain weren't always automatic.

 

[The Bread Guy]

To play the devil's advocate, any chance of someone being good with the books, thereby aceing any written test, but not able to deliver when given troops?  I've seen examples of folks who've passed all the PO's, and met the letter of the PO, but ended up poor bosses.

 

[PMedMoe]

To play the devil's advocate, any chance of someone being good with the books, thereby aceing any written test, but not able to deliver when given troops?  I've seen examples of folks who've passed all the PO's, and met the letter of the PO, but ended up poor bosses.

I agree.  Some people are good with the theory or academic part, but not so much with the practical part.

 

[vonGarvin]

That is indeed true; however, right now, there is nothing.

 

[George Wallace]

Exactly, and merit boards are made up of people who are (supposedly) fighting for their guys positions on the list.  Often the guy with the loudest boss and the most friends in the room wins.

So true.

 

[Lex Parsimoniae]

(I believe the Navy still uses them)

The Navy has recently eliminated command exams and replaced them with a course.  The first serial is running now.

 

[TangoTwoBravo]

Doesn't matter anyways. Your assessment is based solely on your Merritt board score, regardless of the narratives or PDRs.

Regards

Your PER score may well come from a unit-run merit board, but the PER narratives count for a lot when the real merit boards sit for each MOS in the fall.  Given that most of the files competing for the promotions will have similar scores (or else they wouldnt be there), the quality and detail in the narratives can make a difference.

As for exams, it would certainly create a market for promotion exam study guides.

 

[kincanucks]

Let the subordinates have input. Oh wait then there would only be one or two General officers.

 

[Kat Stevens]

That would be horrible, other than freeing up a big puddle of cash.  Think of the endless supply of subbies, though!

 

[Infanteer]

1.  Merit boards generally cover the "handling the troops" bit - having sat in at merit boards, they generally turn out as they should - the top, mid and bottom third end up where they need to end up.

2.  Once evaluations leave the unit, its anyones game - sub-unit/unit merit boards are easy as everyone is a known commodity.  T2B points to the narrative being important once you get to the MOI category.  Its unfortunate that promotions are made based off of the writing ability of one's superior.

3.  This is where I hinted that "who you know" comes into play.  I'd argue that it isn't always deliberate - it's human nature.  But when you have 2 identical MOIs staring you in the face and you know for certain that your ex-subordinate is a solid leader and the other guy across from the table agrees with you (having served with him at X).....

4.  360 evaluations definitely have a place, but I'm unsure of how to execute this.  Superiors generally rate off of "does this guy give me results" where as subordinates usually judge a guy by "is he good to work for" and "do I trust him to lead me competently".  Effective leaders aren't always liked - I had subordinates who gave me results and generally weren't liked by the troops.  Peer evaluation also has a place - if anything to gauge if the guy is a butt-snorkler/back-stabber.

5.  T2B spoke to exams and the industry that would build around them - this certainly was the case in the old British Army.  However, I still believe that exams are central to advancement within a profession and offer another way to objectively evaluate leaders.  Two leaders have the same PERs, but one can clearly identify staff procedures/doctrinal concepts/display critical thought better.  The kicker is executing the exam properly - lord knows we'd offer a stupid CTC exam with true/false questions and exhaustive lists of irrelevant nouns.

 

[vonGarvin]

The kicker is executing the exam properly - lord knows we'd offer a stupid CTC exam with true/false questions and exhaustive lists of irrelevant nouns.

:rofl:

Cap'n exam (draft):
1.  "Captain" comes from latin meaning "automatic promotion".  True or False?
2.  Captains have ten incentives.  True or False?
3.  Captain is the most cromulent rank in the CF.  True or False?
4.  Define the following nouns:
Grape
Sand
Bottle

 

[Kat Stevens]

Grape- something you load in a canon
Sand- something you use to build cool models out in the field
Bottle- what happens when you have too much or too little of the other two.

Do I get the promotion now?