CMSC 601 Discussion Questions
February 5, 2009
You do not need to include written answers to all of these
questions in your paper summaries, but they may help you in organizing
your thoughts about the paper. You should come to class prepared to
comment on all of these questions---even if in some cases your answer
is, "I couldn't understand that from the paper." (In that case,
though, you should have some observations on why that
particular question is difficult to answer.)
Melville et al., "An expected utility-approach to active
feature-value acquisition," ICDM 2005
-
What are the main claims of the paper? Does the paper present theoretical
and/or empirical evidence for these claims?
-
Does the paper position itself thoroughly and accurately with respect to
related work?
-
Specific technical questions:
- Is a random baseline a reasonable choice for evaluating this
approach? Assuming that no other methods existed in the literature,
what would some reasonable alternative baselines be?
- How is the accuracy model (expected accuracy with and without
acquiring a particular feature value) estimated? What is the
computational cost of this approach (just an intuitive discussion, not
an actual complexity analysis)? How does the size of the training set
affect the performance of this approach? -- what problems would arise
if the training set was relatively small?
- The approach depends on the quality of the predictions of the
partially learned model in order to estimate expected utility. How
sensitive do you think this approach would be to the proportion of
missing values in the training set? (That is, how would its behavior
be affected as the number of missing values increased?)
- In the empirical results, all costs are equal -- are the results
still convincing, or do you think performance might change if the
costs were unequal for different feature values?
- Why did the authors decide to set the sample-size parameter to 10?
-
What are the key results of the paper?
-
What do you think are the most important/interesting next steps/future
work for this paper?
-
What references would you follow up on if you were to do further
reading on this topic?
-
What were the good and bad aspects of the presentation (organization, writing
quality, clarity) of this paper? Is the paper self-contained, or would
you need to do further background reading (or correspond with the authors)
to understand the details of the methods and results?
Ting, "A comparative study of cost-sensitive boosting
algorithms," ICML-00
-
What are the main claims of the paper? Does the paper present theoretical
and/or empirical evidence for these claims?
-
Does the paper position itself thoroughly and accurately with respect to
related work?
-
Specific technical questions:
- The methods described in the paper are all based on AdaBoost. Did
you think AdaBoost itself was clearly described? What about the
different variations? How could the paper be improved to make these
methods clearer to a general audience?
- Are the names of the variations useful for helping the reader to
remember them? What might be some better names?
- What kind of theoretical or intuitive justification did the author
give for the different variations that were proposed/presented? From
reading section 3, did you get a sense of which methods are likely to
be most successful, and why they should (or should not) work?
- Why does the minority class have a different "cost factor" F from
the majority class? Does it make sense that F is always greater than
1? Can you think of any applications for which this would not be the
case? Do the proposed methods depend on this assumption?
- It seems surprising that AdaCost performs worse than the other
methods, especially since it was previously published and shown to
perform better. Is Ting's explanation of this effect clear and
convincing? Are his new proposed AdaCost variations better than the
original, and/or than the CSB variations that Ting presents?
-
What is being evaluated by the experiments in Section 5? Are these results
convincing? Why or why not? Are the conclusions drawn by the authors supported
by the results?
-
What are the key results of the paper?
-
What do you think are the most important/interesting next steps/future
work for this paper?
-
What references would you follow up on if you were to do further
reading on this topic?
-
What were the good and bad aspects of the presentation (organization, writing
quality, clarity) of this paper? Is the paper self-contained, or would
you need to do further background reading (or correspond with the authors)
to understand the details of the methods and results?
Comparing the papers
- Both papers talk about "cost-sensitive learning." How are the
problems being addressed by the two papers similar? How are they
different?
-
The Melville et al. paper was published five years later than the Ting
paper.
Does the later paper represent a significant improvement or extension over
the earlier work?
-
The Ting paper explicitly positions itself as a "comparative study" of
different methods (including several developed by Ting and colleagues). In
contrast, the Melville et al. paper only compares the proposed
approach to a random baseline. Do you find one of the papers more
convincing than the other in its claims? Why or why not?
-
How much of the technical part of each paper did you find you needed
to understand in order to understand the main claims and contributions
of the paper? To what degree do you think the answer to this question
was a function of the general area of the paper (and your familiarity
with it), the nature of the particular method and results, and/or the
presentation (organization, writing style, quality) of the papers
themselves?
-
Did you find one of the papers easier to follow and understand than
the other? If so, what do you think are the factors that contributed
to your improved understanding?