About the role

At Workera, our assessments follow a rigorous assembly process and our computerized adaptive test (CAT) is developed to orient learners throughout their careers. We're seeking a Psychometrician to handle tasks like psychometric analysis, competency modeling, item bank maintenance, scoring, and reporting. The role demands flexibility, initiative, and comfort in a fast-paced setting. You will collaborate closely with assessment, learning, and product teams to create valid assessments, enhance our platform, and ensure a superior user experience. You'll use your psychometrics expertise to influence UX design, integrate IRT models with deep knowledge tracing, and build our platform's psychometric infrastructure. This position offers the chance to lead internal projects and significantly contribute to improving our product by integrating engaging, formative assessment with the learning experience.

About your team

The Assessments team is composed of psychometricians, assessment program managers, and assessment developers who are subject matter experts in areas such as AI, data science, and software engineering. We own the development, administration, analyses, and maintenance of Workera’s catalog of assessments.

What You’ll Do:

Within 3 months, you will:

• Conduct common psychometric analyses, such as classical item analysis, IRT item calibration, and evaluating assessments for item fit, person fit, and dimensionality.
• Maintain technical documentation and dashboards related to assessment development and psychometric analyses for internal and external stakeholders.

Within 6 months, you will:

• Perform psychometric research, such as analyses to support and improve our CAT, and designing studies to establish validity evidence for our assessments.
• Support product teams by serving as an internal psychometric consultant to improve Workera’s assessment quality and user experience. You will balance assessment development and psychometric best practices with UX and product design considerations to support new product features, including the implementation of alternative item types and automated psychometric analyses.
• Manage item banks, including monitoring and evaluating item banks, performing gap analyses, developing crosswalks between assessment versions, creating item writing assignments, and monitoring item exposure rates and statistical anomalies.

Within 9 months, you will:

• Support client custom domain development, including providing guidance on assessment design, and performing statistical analyses, calibration, scoring, standard setting, and reporting.
• Lead projects to improve our assessments and product, including different measurement models, item types, and assessment design.

*Please note as we are a dynamic and quickly growing scale-up, things are always subject to change*

What You’ll Bring

• Command of assessment and measurement best practices. Knowledge of the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014) and/or NCCA standards is required.
• Comprehensive knowledge of psychometrics. Leverage experience in item response theory (IRT) and classical test theory (CTT) measurement models, including item analyses, calibration, scaling, and equating. Experience with computational psychometrics preferred
• Experience with the development and scoring of alternative item types. This includes matching, short answer, essay, coding, scenario, and/or situational judgment tasks
• Experience conducting validation studies. This includes establishing evidence of content, construct, and criterion-related validity
• Experience with computer adaptive testing (CAT). Experience with CAT and automated test assembly methods required
• Strong programming skills. Command of Python or R (particularly IRT packages) required. Experience with SQL and Git preferred.
• Strong communication and meeting facilitation skills. Experience guiding and facilitating meetings of SMEs, and ability to communicate technical information to non-technical audiences
• Experience working with product teams is a plus, not required.
• Experience in talent assessment is a plus, not required
• Master’s degree or PhD in I/O Psychology, Educational Measurement, Psychometrics, and/or related-fields.
• 2-3 years of applied psychometrics experience
• Proficiency in utilizing AI tools such as ChatGPT or equivalent digital assistants, applying them effectively within your work context.