Class PythonTokenizer

source code

Utility wrapper over python tokenizer.

Instance Methods

[hide private]

__init__(self, text)

source code

consume_till(self, delim)
Consumes tokens till colon.

source code

next(self)

source code

Method Details

[hide private]

consume_till(self, delim)

source code

Consumes tokens till colon.

>>> tok = PythonTokenizer('for i in range(10): hello $i')
>>> tok.consume_till(':')
>>> tok.text[:tok.index]
'for i in range(10):'
>>> tok.text[tok.index:]
' hello $i'