{"id":3418,"date":"2018-05-03T20:53:46","date_gmt":"2018-05-03T20:53:46","guid":{"rendered":"http:\/\/second-icicle.flywheelsites.com\/?p=3418"},"modified":"2020-01-23T16:33:25","modified_gmt":"2020-01-23T16:33:25","slug":"validity-isnt-important","status":"publish","type":"post","link":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/","title":{"rendered":"Why Validity Isn\u2019t As Important As You May Think (When it Comes to 360 Assessments) \u2013 And 6 Best Practices Instead!"},"content":{"rendered":"<p><img fetchpriority=\"high\" decoding=\"async\" class=\"aligncenter size-full wp-image-3421\" src=\"https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2018\/05\/360-Assessments.jpg\" alt=\"360 Assessments\" width=\"800\" height=\"600\" srcset=\"https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2018\/05\/360-Assessments.jpg 800w, https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2018\/05\/360-Assessments-300x225.jpg 300w, https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2018\/05\/360-Assessments-768x576.jpg 768w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/><\/p>\n<p>We\u2019re occasionally asked about the statistical validity and reliability of the <a href=\"https:\/\/site.assessmentplus.com\/our-offerings\/assessment-services\/360-feedback\/\">360 Assessment process<\/a>, from the forethought behind <a href=\"https:\/\/site.assessmentplus.com\/our-offerings\/services\/survey-development\/\">drafting the items<\/a> to the analysis after <a href=\"https:\/\/site.assessmentplus.com\/our-offerings\/services\/interpretation-of-results\/\">receiving the data<\/a>. They\u2019re common questions when it comes to 360s \u2013 Are my results valid? What does my data mean in the grand scheme of things?<\/p>\n<p>&nbsp;<\/p>\n<p>Although such statistical measurements as validity and reliability are relevant, and even required, for other types of assessments that measure specific skills and\/or knowledge \u2013 say a U.S. History exam that should accurately determine your familiarity with the Civil War \u2013 they\u2019re not necessarily relevant for 360 Assessments. This is because, by design, a 360 assessment is a more subjective measurement of the perceptions and opinions of multiple people who view the world (and the behaviors of others) through different lenses.<\/p>\n<p>&nbsp;<\/p>\n<p>Throughout these assortments of tests that can be run to \u201cprove\u201d statistical validity or reliability, many don\u2019t quite pertain to 360 assessments. Here\u2019s why:<\/p>\n<p>&nbsp;<\/p>\n<ul style=\"margin-left: 30px; margin-top: 20px;\">\n<li><strong>Test-Retest Reliability<\/strong> \u2013 This analysis must be done over a short period of time to see if ratings remain stable. However, the whole point of a 360 assessment (and subsequent follow-up assessments is IMPROVEMENT! We don\u2019t want the ratings to remain the same \u2013 we want positive change in behaviors, perceptions, and performance.<\/li>\n<li><strong>Internal Consistency Reliability<\/strong> \u2013 In order for an assessment to prove this sort of reliability, items within a competency or category must rate consistently. However, it is actually quite likely that a rating for one area of communication, for example, will be much higher than a rating for a different area. The differences in ratings (by design) is exactly what we want to see to help the leader pin-point strengths and areas in which they can improve.<\/li>\n<li><strong>Inter-Rater Reliability<\/strong> \u2013 This test of reliability provides another prime example of a situation in which what some statisticians would deem \u201cinaccurate\u201d or \u201cmeaningless\u201d is actually something that we find treasured in the data! The differences in raters\u2019 perspectives are not treated as statistical errors, but as valuable information that makes the feedback even more reliable and provides a deeper meaning.<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>So, if these three tests ultimately fail when it comes to proving validity of 360 assessments, what can we rely on instead? Here are some of our best practices:<\/p>\n<p>&nbsp;<\/p>\n<ol style=\"margin-left: 30px; margin-top: 20px;\">\n<li style=\"list-style-type: none;\">\n<ol>\n<li style=\"margin-bottom: 10px;\"><strong>Ensure questions are psychometrically sound<\/strong> \u2013 Avoid posing any questions that could be ambiguous or vague. Measure one behavior in each question so that raters are not forced towards the middle of the scale. If a two-part question, the person being rated might excel in one of those areas, but need development in the other. The rater is left to \u201cride the fence\u201d on the rating, and makes the \u201caverage\u201d rating less relevant.<\/li>\n<li style=\"margin-bottom: 10px;\"><strong>Select larger groups of raters<\/strong> \u2013 Validity will be stronger with 15-20 raters vs. 3-5 raters.<\/li>\n<li style=\"margin-bottom: 10px;\"><strong>Write questions with \u201cFace validity\u201d in mind<\/strong> \u2013 When the questions and competencies appear to make sense for the job level and purpose of the assessment, \u201cface validity\u201d has been achieved.<\/li>\n<li style=\"margin-bottom: 10px;\"><strong>Identify the full range of behaviors\/skills<\/strong> believed to represent how your organization defines successful leadership.<\/li>\n<li style=\"margin-bottom: 10px;\"><strong>Customize Scale Anchors<\/strong> \u2013 While we recommend customizing everything for your <a href=\"https:\/\/site.assessmentplus.com\/5-ways-companies-can-make-customized-assessments\/\">particular organization,<\/a> the design of the scale \u2013 in particular \u2013 can limit ambiguity. Rather than using anchors of \u201cagreement\u201d or \u201cfrequency,\u201d consider using anchors that are clearer and linked to the employees\u2019 demonstrated behaviors.<\/li>\n<li style=\"margin-bottom: 10px;\"><strong>Ensure the items are actionable<\/strong> \u2013 Don\u2019t attempt to measure one\u2019s belief about why a leader demonstrates an action; merely measure whether the action is being demonstrated.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>We\u2019re occasionally asked about the statistical validity and reliability of the 360 Assessment process, from the forethought behind drafting the items to the analysis after receiving the data. They\u2019re common questions when it comes to 360s \u2013 Are my results valid? What does my data&#8230;<\/p>\n","protected":false},"author":10,"featured_media":3421,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Why Validity Isn\u2019t As Important As You May Think | Assessment+<\/title>\n<meta name=\"description\" content=\"Throughout assortments of tests that can be run to \u201cprove\u201d statistical validity or reliability, many don\u2019t quite pertain to 360 assessments. Here\u2019s why\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/site.assessmentplus.com\/validity-isnt-important\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Why Validity Isn\u2019t As Important As You May Think | Assessment+\" \/>\n<meta property=\"og:description\" content=\"Throughout assortments of tests that can be run to \u201cprove\u201d statistical validity or reliability, many don\u2019t quite pertain to 360 assessments. Here\u2019s why\" \/>\n<meta property=\"og:url\" content=\"https:\/\/site.assessmentplus.com\/validity-isnt-important\/\" \/>\n<meta property=\"og:site_name\" content=\"Assessment+\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/assessmentplusinc\" \/>\n<meta property=\"article:published_time\" content=\"2018-05-03T20:53:46+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-01-23T16:33:25+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2018\/05\/360-Assessments.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"800\" \/>\n\t<meta property=\"og:image:height\" content=\"600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Jennifer Taylor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@assessmentplus\" \/>\n<meta name=\"twitter:site\" content=\"@assessmentplus\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jennifer Taylor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Why Validity Isn\u2019t As Important As You May Think | Assessment+","description":"Throughout assortments of tests that can be run to \u201cprove\u201d statistical validity or reliability, many don\u2019t quite pertain to 360 assessments. Here\u2019s why","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/","og_locale":"en_US","og_type":"article","og_title":"Why Validity Isn\u2019t As Important As You May Think | Assessment+","og_description":"Throughout assortments of tests that can be run to \u201cprove\u201d statistical validity or reliability, many don\u2019t quite pertain to 360 assessments. Here\u2019s why","og_url":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/","og_site_name":"Assessment+","article_publisher":"https:\/\/www.facebook.com\/assessmentplusinc","article_published_time":"2018-05-03T20:53:46+00:00","article_modified_time":"2020-01-23T16:33:25+00:00","og_image":[{"width":800,"height":600,"url":"https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2018\/05\/360-Assessments.jpg","type":"image\/jpeg"}],"author":"Jennifer Taylor","twitter_card":"summary_large_image","twitter_creator":"@assessmentplus","twitter_site":"@assessmentplus","twitter_misc":{"Written by":"Jennifer Taylor","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/#article","isPartOf":{"@id":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/"},"author":{"name":"Jennifer Taylor","@id":"https:\/\/site.assessmentplus.com\/#\/schema\/person\/c21f96edb3605ea325a950e8652fca5a"},"headline":"Why Validity Isn\u2019t As Important As You May Think (When it Comes to 360 Assessments) \u2013 And 6 Best Practices Instead!","datePublished":"2018-05-03T20:53:46+00:00","dateModified":"2020-01-23T16:33:25+00:00","mainEntityOfPage":{"@id":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/"},"wordCount":600,"publisher":{"@id":"https:\/\/site.assessmentplus.com\/#organization"},"image":{"@id":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/#primaryimage"},"thumbnailUrl":"https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2018\/05\/360-Assessments.jpg","articleSection":["Info Worth Sharing"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/","url":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/","name":"Why Validity Isn\u2019t As Important As You May Think | Assessment+","isPartOf":{"@id":"https:\/\/site.assessmentplus.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/#primaryimage"},"image":{"@id":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/#primaryimage"},"thumbnailUrl":"https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2018\/05\/360-Assessments.jpg","datePublished":"2018-05-03T20:53:46+00:00","dateModified":"2020-01-23T16:33:25+00:00","description":"Throughout assortments of tests that can be run to \u201cprove\u201d statistical validity or reliability, many don\u2019t quite pertain to 360 assessments. Here\u2019s why","breadcrumb":{"@id":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/site.assessmentplus.com\/validity-isnt-important\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/#primaryimage","url":"https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2018\/05\/360-Assessments.jpg","contentUrl":"https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2018\/05\/360-Assessments.jpg","width":800,"height":600,"caption":"360 Assessments"},{"@type":"BreadcrumbList","@id":"https:\/\/site.assessmentplus.com\/validity-isnt-important\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/site.assessmentplus.com\/"},{"@type":"ListItem","position":2,"name":"Why Validity Isn\u2019t As Important As You May Think (When it Comes to 360 Assessments) \u2013 And 6 Best Practices Instead!"}]},{"@type":"WebSite","@id":"https:\/\/site.assessmentplus.com\/#website","url":"https:\/\/site.assessmentplus.com\/","name":"Assessment+","description":"","publisher":{"@id":"https:\/\/site.assessmentplus.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/site.assessmentplus.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/site.assessmentplus.com\/#organization","name":"Assessment+","url":"https:\/\/site.assessmentplus.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/site.assessmentplus.com\/#\/schema\/logo\/image\/","url":"https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2015\/06\/ASP_Logo_Color.png","contentUrl":"https:\/\/site.assessmentplus.com\/wp-content\/uploads\/2015\/06\/ASP_Logo_Color.png","width":763,"height":186,"caption":"Assessment+"},"image":{"@id":"https:\/\/site.assessmentplus.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/assessmentplusinc","https:\/\/twitter.com\/assessmentplus","https:\/\/www.instagram.com\/assessmentplus\/","https:\/\/www.linkedin.com\/company\/assessment-plus-inc-"]},{"@type":"Person","@id":"https:\/\/site.assessmentplus.com\/#\/schema\/person\/c21f96edb3605ea325a950e8652fca5a","name":"Jennifer Taylor","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/site.assessmentplus.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/391429117ba7c12d3b824574c3f83bc6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/391429117ba7c12d3b824574c3f83bc6?s=96&d=mm&r=g","caption":"Jennifer Taylor"},"url":"https:\/\/site.assessmentplus.com\/author\/jennifer\/"}]}},"_links":{"self":[{"href":"https:\/\/site.assessmentplus.com\/wp-json\/wp\/v2\/posts\/3418"}],"collection":[{"href":"https:\/\/site.assessmentplus.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/site.assessmentplus.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/site.assessmentplus.com\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/site.assessmentplus.com\/wp-json\/wp\/v2\/comments?post=3418"}],"version-history":[{"count":0,"href":"https:\/\/site.assessmentplus.com\/wp-json\/wp\/v2\/posts\/3418\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/site.assessmentplus.com\/wp-json\/wp\/v2\/media\/3421"}],"wp:attachment":[{"href":"https:\/\/site.assessmentplus.com\/wp-json\/wp\/v2\/media?parent=3418"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/site.assessmentplus.com\/wp-json\/wp\/v2\/categories?post=3418"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/site.assessmentplus.com\/wp-json\/wp\/v2\/tags?post=3418"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}