|
The problem is a teacher has to set an exam for a large number of students. Given the size of your class, one is forced to limit the number of questions given to each student on the mid-term exam to only five multiple choice questions. All test questions be pulled from a bank of approximately 400 “approved” test questions. To increase the robustness and ease of creation of your midterm exam you one can randomly and uniformly choose 5 questions from the question bank for each student. This means that each student’s exam consists of disjoint, intersecting, or identical sets of 5 questions. To assist with running such a large class, the university has provided you with last year’s student performance data . While it is OK not to use all the questions from this data, you must, at minimum, use 75% of them in order to achieve sufficient coverage of the curriculum. Which questions should you exclude from the exam generation algorithm such that the exam results will provide the most meaningful ranking of the students in the class? By meaningful one means that select more questions that are discriminating in terms of how the students score. The data set you are given by the department has one record per line, where each record consists of:
=====My solution==== Basically the intuition is that we dont want to give too many Questions that are scored too low or too high, because then the students may have scores that are bunched up too close together. Since since the question scores have a non uniform probability distribution the data is indicative of it. So I calculated the probability of a Question being right or wrong, by counting the frequency of it being correctly answered, and created a Cumulative Distribution of the probability scores of the questions. By doing an inverse sampling of this CDF one will get the values that are distributed along the maximum probability mass of the CDF curve and hence the underlying PDF curve also. Since I am assuming that the data from last year's test is fairly good at giving a meaningful ranking of the students, If there is a set of fairly large questions that are answered in a way that can discriminate(i.e. the distribution of these is not too heavily tailed or displays kurtosis) in the dataset then I expect to reproduce this distribution using the above solution. In the end to I sample this distribution repeatedly until I have 75% of the questions collected as required by the problem. === A Better Solution?=== Since the questions are supposedly belonging to a fixed number of topics, all topics, though some may be very hard and low scoring over all, should be covered in the exam. This suggests that a few number of latent factors may govern distribution of the scores of these Questions independent of the students. For example Special relativity might be very hard to score on as compared to linear algebra(say), but both topics must get there due in the exam. We should not leave out Relativity as it is important part of the curriculum. My earlier method chooses questions independently and would make this grave mistake. So the method to set tests needs to be improved. I dont know if this makes sense. But I feel that such a graphical model with latent factors and questions representing the topics, could possibly be sampled to get better results than my earlier solution. What do you think? How might i do this? |
How do you solve the problem of a test not having questions only for one topic?
Is that a question or are you summarizing the problem? Because that is the problem, viz: The questions that are in this one big exam but they are not from a single topic, but multiple, and all topics must receive fair coverage.