Some insight was gained during the discussion of Type I and Type II errors, with respect to the TREC effort. By insisting on little chance of Type I error (e.g. alpha=0.05), thereby insisting that a difference be reported only when there's (say) a 1 in 20 chance that it happened by accident, we greatly increase the probability of a Type II error (e.g. beta = 0.6 or whatever) in which real differences between systems are not reported.
In TREC, systems do overlap in terms of performance, since they use the same corpus, the same queries, the same judgements, and sometimes the same search engines. Now might be a good time to re-examine the way in which systems are compared.
Terry Winograd's Keynote address was ok, but not inspirational. He seems to be caught up in the IETF's UR* discussions. He mentioned URA, URC, URI, URL, URM, and URN. That's six letters of the alphabet, so that leaves twenty acronyms waiting to be coined. Here's a few I thought of:
The session on Distributed IR and the Internet was probably the best of the conference. The paper by Viles and French on Dissemination of Content-Wide Information was especially relevant to MDDS.
The session on Text Summarization was also very good. McKeown's paper on generating summaries of news stories talked about using MUC templates to make up natural-sounding summaries.
I didn't attend the NLP session, but the article on searching gigabits of Chinese texts sounds important for MDDS.
I met Ken Church at the poster/demo session, which was very crowded but still manageable. Using graphical techniques to deal with gazillions of n-grams sounds reasonable.
There's danger in being unaware of, or ignoring, assumptions. Example: Recall is a problem, since every measure of recall is based on the assumption that there is only one set of relevant documents in a corpus for a given query.
Evaluation criteria is a huge area in every science -- research on evaluation itself. He notes that the famous network IR tools have not been evaluated. Tefko mentioned a study of Medline users, JAMA 93, in which a critical incident analysis noted seven instances when a Medline search saved human lives.
The highlight of the fusion strategies session was Ellen Voorhees' talk.
Banquest at Microsoft was very nice! Bill couldn't make it, though.
Nick Belkin asks, "What's fundamental?" His answer: IR is about people, in particular how and why they work with information.
Ed Fox says "Let's do something!" and points to his IR course home page at Virginia Tech.
David Lewis points out that problems in industry don't break down across clean academic lines. One has to keep up-to-date about what's out there, since it's (almost) always cheaper to buy rather than build.
Donna Harman says that we're training people for five years from now, so stress the fundamentals rather than the technology of the week. We should get people excited about using the library.
Many questions and comments from the audience. Someone noted that we need to pay more attention to research methods, perhaps taking cues from the social and behavioral sciences.