英文单词自动纠错 norvig

[hailiang-wang]

Description

norvig.com/spell-correct.html

Solution

第一步，构建一个很大的词表，统计出每个正确的词的频率

def P(word, N=sum(WORDS.values())): 
    "Probability of `word`."
    return float(WORDS[word]) / N

第二步，提供编辑距离为1和2的所有候选词的集合

def edits1(word):
    "All edits that are one edit away from `word`."
    letters    = 'abcdefghijklmnopqrstuvwxyz'
    splits     = [(word[:i], word[i:])    for i in range(len(word) + 1)]
    deletes    = [L + R[1:]               for L, R in splits if R]
    transposes = [L + R[1] + R[0] + R[2:] for L, R in splits if len(R)>1]
    replaces   = [L + c + R[1:]           for L, R in splits if R for c in letters]
    inserts    = [L + c + R               for L, R in splits for c in letters]
    return set(deletes + transposes + replaces + inserts)

def edits2(word):
    "All edits that are two edits away from `word`."
    return (e2 for e1 in edits1(word) for e2 in edits1(e1))

第三步，识别候选词集合中是有效单词的集合

def known(words):
    "The subset of `words` that appear in the dictionary of WORDS."
    return set(w for w in words if w in WORDS)

def candidates(word): 
    "Generate possible spelling corrections for word."
    return (known([word]) or known(edits1(word)) or known(edits2(word)) or [word])

第四步，计算候选词中概率最大的

def correction(word): 
    "Most probable spelling correction for word."
    return max(candidates(word), key=P)

[hailiang-wang]

Demo:
https://github.com/Samurais/pytudes/blob/master/py/spell.py