汉语自动分词的研究现状与困难

汉语自动分词是信息提取、信息检索、机器翻译、文本分类、自动文摘、语音识别、文本语音转换、自然语言理解等中文信息处理领域的基础研究课题。尽管已被研究了二十多年，分词仍然是中文信息处理的瓶颈问题。基于对汉语自动分词研究的现状分析，构建了自动分词的形式化模型，论述了影响分词的诸多因素，分析了分词中存在的两个最大困难及其解决方法。最后指出了目前分词研究中尤其是在分词评测方面存在的问题以及未来的研究工作。

The State of the Art and Difficulties in Automatic Chinese Word Segmentation

Automatic Chinese word segmentation is a basic research issue on Chinese information processing tasks such as information extraction, information retrieval, machine translation, text classification, automatic text summarization, speech recognition, text-to-speech, natural language understanding, and so on. Though it has been investigated for more than twenty years, it is still a bottleneck for Chinese information processing. We give a detailed analysis of the state of the art in automatic Chinese word segmentation, build a formal model of word segmentation, discuss factors affecting word segmentation and the two great difficulties in word segmentation and their resolutions, and finally, point out the existing problems, especially those on the word segmentation evaluation, as well as the research problems to be resolved.