Measuring the similarity of test questions that are equivalent but presented with different scenarios or examples

[erkanererkaner]

I want to see how similar a bunch of test questions (written in Turkish) are in terms of what they’re really testing, regardless of the situations or examples they use. It might sound a bit like checking how similar the sentences are, but it’s actually quite different. For example, these next questions should be seen as very similar or the same, even though the words in the sentences aren’t very much alike:

Question 1: You are given two balls, one made of rubber and one made of steel. You drop them simultaneously from the same height. Which one will hit the ground first, and why?

Question 2: In an experiment, you have two identical balls, one made of plastic and one made of glass. You release both at the same time from the identical height. Which ball will reach the ground first, and what physical principle explains the outcome?

So, I want to measure the similarity in terms of the similarity of the very specific skill measured by the questions. Is this possible?

[PaulBellow]

Welcome to the forum.

@erkanererkaner

So, I want to measure the similarity in terms of the similarity of the very specific skill measured by the questions. Is this possible?

Likely with a one-shot or two-shot… maybe even a zero-shot…

What have you tried so far?

[anon22939549]

This is called “factor analysis” or “item analysis.” And it’s related to Item Response Theory.

This is not something I would trust to an LLM if it is important to get correct results as it requires some moderately sophisticated statistical methods to perform.

Here are some resources;

• http://www.schreyerinstitute.psu.edu/pdf/GuideToItemAnalysis.pdf
• https://www.cs.princeton.edu/~bee/courses/scribe/lec_10_02_2013.pdf
• https://www.voxco.com/blog/factor-analysis-definition-sample-questions/

And here’s some instructions on how you might perform a factor analysis:

Python

• https://www.datacamp.com/tutorial/introduction-factor-analysis
• sklearn.decomposition.FactorAnalysis — scikit-learn 1.3.0 documentation
• factor-analyzer · PyPI
• Introduction to Factor Analysis in Python | by Avinash Navlani | Artificial Intelligence in Plain English

R

• https://www.youtube.com/watch?v=VCpVcXf_wOk
• https://cran.r-project.org/web/packages/psychTools/vignettes/factor.pdf
• Performing Analysis of a Factor in R Programming - factanal() Function - GeeksforGeeks
• https://techvidvan.com/tutorials/pca-and-factor-analysis-in-r/

That should be enough get you started.

It is possible the LLM in conjunction with access to a computing environment will be able to perform this task exceptionally well, but I don’t think any degree of prompting the API will help the model get there on its own.

[erkanererkaner]

@erkanererkaner

Question 1: You are given two balls, one made of rubber and one made of steel. You drop them simultaneously from the same height. Which one will hit the ground first, and why?

Question 2: In an experiment, you have two identical balls, one made of plastic and one made of glass. You release both at the same time from the identical height. Which ball will reach the ground first, and what physical principle explains the outcome?

I have tried sentence-transformers and I am also investigating the github of OpenAI → openai/openai-cookbook/blob/main/text_comparison_examples.md

I thought of summarizing/refining the questions to be able to keep the essence of the question while removing the parts forming the specific example. I guess I may benefit from fine-tuning for this case.