• Department of Computer Science and Engineering
• The Chinese University of Hong Kong
• Homepage: http://wiki.cse.cuhk.edu.hk/user/zhuyi
• Email: zhuyi@cse.cuhk.edu.hk

• Department of Computer Science and Engineering
• The Chinese University of Hong Kong
• Email: tobcli@gmail.com

The Dialog Research Center at Carnegie Mellon (DialRC) build a telephone-based spoken dialog system that provides bus schedule information for the City of Pittsburgh, PA (USA). The DialRC establish this challenge to call for participation with three fold: 1). build a system with the similar function; 2). build simulate user for these system and 3). build a machine judge to evaluate the user satisfication of the system.

Our goal is to evaluate the system based on user satisfication.

We analyze the dialogs with Natural Language Processing and statistical techniques to extract useful features from the dialogs, which is basically considering its question answer pairs. We could do regression with those features. Based on the user's query and system response, we can construct a state diagram to improve the regression step by divide the dialog by the state transition.

1. (done): Find the most popular path on the state diagram that the CMU's system works;
2. (done): Find out the states and construct the state diagram according to the dialog;
3. (done): Put part of the dialog onto the Amazon Mechanical Turk for labelling;
4. (done): Feature selection;
5. (on going): Find appropriate regression method;
6. (on going): Decide the weight on the state diagram;
7. (on going): Test experiment.

A_Graph-based_Semi-Supervised_Learning_for_Question-Answering

A_Graph-based_Semi-Supervised_Learning_for_Question-Answering paradise:a_framework_for_evaluating_spoken_dialogue_agents
Feature Selection for Evaluating Spoken Dialog System

Status Statistic Result

Status Transition Graph

Status Transition Graph (Add all the capital word in the System Response as the location)

For amazon's mechanical turk