[ fixed #71 ] do not produce tokens of length 0

Changed `alex_scan_tkn` to return `AlexNone` instead of `AlexLastAcc` or
`AlexLastSkip` if length of newly processed input is still 0.

This fixes the problem that Alex would produce an infinite sequence of
tokens of length 0 when the DFA was stuck in a state accepting a
nullable token.

Now it will either produce EOF (if at end of input) or a lexer error (if
not at the end of input).  This is the intuitively expected behavior.

Author: andreasabel

templates/GenericTemplate.hs

@@ -152,7 +152,9 @@ alexScanUser user__ input__ IBOX(sc)
 alex_scan_tkn user__ orig_input len input__ s last_acc =
   input__ `seq` -- strict in the input
   let
-  new_acc = (check_accs (alex_accept `quickIndex` IBOX(s)))
+  new_acc = (if GTE(ILIT(0),len)
+             then AlexNone  -- issue #71, do not output tokens of length 0
+             else check_accs (alex_accept `quickIndex` IBOX(s)))
   in
   new_acc `seq`
   case alexGetByte input__ of

tests/issue_71.x

@@ -33,13 +33,7 @@ expected_result = [A,A,Whitespaces,A,A,Whitespaces,A]
 
 main :: IO ()
 main
-    -- Since the whitespaces token is nullable, Alex
-    -- will recognize an infinite number of those
-    -- at the end of file.  This behavior is problematic,
-    -- but we don't fix it here.
-    -- We just test here whether the expected result
-    -- is a prefix of the produced result.
-  | take (length expected_result) result == expected_result = do
+  | result == expected_result = do
       exitWith ExitSuccess
   | otherwise = do
       print $ take 20 result