Python AST preserving whitespace and comments

Hi,

I'm working on Pythoscope[1], a unit test generator for Python and
stumbled into the following problem. I need a way to analyze and
modify Python AST tree, but without loosing source code formatting and
comments. Standard library ast module discards those, so I started
looking for other solutions. I found a library used by the 2to3
script[2], which does different kinds of source code refactorings and
preserves the formatting of the original file. Sadly AST generated by
this library is not compatible with the standard one, so I can't use a
nice interface of compiler.visitor[3]. It isn't also as well
documented. I kind of liked the clean and descriptive list of standard
AST nodes[4].

So here are my questions to the list. Is there any library built on
top of lib2to3 which makes traversing it easier? The pattern matcher
is nice, but not always feasible for the higher level view of the
source code.

Are there any tutorials/documentation about usage of lib2to3 which I'm
not aware of? I basically read through HACKING, README and a lot of
source code, but would appreciate some more examples and insights.

Is the lib2to3 library meant to be used outside of the 2to3 script or
rather not? Are there any projects that will incorporate its features
(enriched AST in particular) in a more "official" library-like
package? Will lib2to3 be put on PyPI anytime?

[1] http://pythoscope.org/
[2] http://svn.python.org/view/sandbox/trunk/2to3/
[3] http://docs.python.org/lib/module-compiler.visitor.html
[4] http://docs.python.org/lib/module-compiler.ast.html

Cheers,
mk

Michal Kwiatkowski wrote:
> Hi,
>
> I'm working on Pythoscope[1], a unit test generator for Python and
> stumbled into the following problem. I need a way to analyze and
> modify Python AST tree, but without loosing source code formatting and
> comments. Standard library ast module discards those, so I started
> looking for other solutions. I found a library used by the 2to3
> script[2], which does different kinds of source code refactorings and
> preserves the formatting of the original file. Sadly AST generated by
> this library is not compatible with the standard one, so I can't use a
> nice interface of compiler.visitor[3]. It isn't also as well
> documented. I kind of liked the clean and descriptive list of standard
> AST nodes[4].
>
> So here are my questions to the list. Is there any library built on
> top of lib2to3 which makes traversing it easier? The pattern matcher
> is nice, but not always feasible for the higher level view of the
> source code.
>
> Are there any tutorials/documentation about usage of lib2to3 which I'm
> not aware of? I basically read through HACKING, README and a lot of
> source code, but would appreciate some more examples and insights.
>
> Is the lib2to3 library meant to be used outside of the 2to3 script or
> rather not? Are there any projects that will incorporate its features
> (enriched AST in particular) in a more "official" library-like
> package? Will lib2to3 be put on PyPI anytime?
>
> [1] http://pythoscope.org/
> [2] http://svn.python.org/view/sandbox/trunk/2to3/
> [3] http://docs.python.org/lib/module-compiler.visitor.html
> [4] http://docs.python.org/lib/module-compiler.ast.html

You found the appropriate library, but I am unaware of anything that you
missed. I think others have had the idea that 2to3 might grow into
something more, but the core developers are currently focues on getting
2.6/3.0 out by the end of September.