Creating a Python Code Completer & More Abstract Syntax Tree Projects

The Real Python Podcast

How does a code completion tool work? What is an Abstract Syntax Tree, and how is it created in Python? How does an AST help you write programs and projects that inspect and modify your Python code? This week on the show, Meredydd Luff, co-founder of Anvil, shares his PyCon talk, “Building a Python Code Completer.”

Meredydd talks about his experience building a code completion engine for the Anvil platform. The conversation leads us to discuss how Python parses the code that you type. We examine tokenization, abstract syntax trees, and how parsing has changed in Python.

We cover additional projects that you can explore once you have a tool that inspects the Python code you’re writing. Join us as we dive into multiple rabbit holes of research and exploration.

Course Spotlight: Python Assignment Expressions and Using the Walrus Operator

In this course, you’ll learn about assignment expressions and the walrus operator. The biggest change in Python 3.8 was the inclusion of the := operator, which you can use to assign variables in the middle of expressions. You’ll see several examples of how to take advantage of this feature.

Topics:

• 00:00:00 – Introduction
• 00:02:11 – Building a Python Code Completer - Talk
• 00:08:48 – How is recursion being used?
• 00:10:28 – Abstract Syntax Trees, Parsers, and Tokens
• 00:14:58 – Sponsor: CData Software
• 00:15:39 – Is parsing into tokens a separate step?
• 00:21:45 – What makes the new PEG parser unique?
• 00:25:32 – Using the AST doesn’t have to be scary.
• 00:27:33 – More on the PEG parser
• 00:29:54 – Keeping scope and holding completions
• 00:39:43 – Video Course Spotlight
• 00:41:15 – Was this your first expirement in creating a code completer?
• 00:46:46 – How can adding type hints assist a completer?
• 00:50:53 – Projects you can build using AST
• 00:53:02 – Linter project
• 00:54:15 – Automatic code formatter
• 00:54:59 – Code testing and the pytest project
• 00:56:22 – Security tools
• 00:57:11 – Friendlier tracebacks
• 00:59:57 – Code completion vs GitHub Copilot
• 01:04:14 – Code completion in a text editor vs IDE
• 01:08:53 – What are you excited about in the world of Python?
• 01:09:54 – What do you want to learn next?
• 01:11:20 – How can people follow your work?
• 01:12:31 – Thanks and goodbye

Show Links:

• Talk - Meredydd Luff: Building a Python Code Completer - YouTube
• Anvil | Build Web Apps with Nothing but Python
• Episode #63: Create Web Applications Using Only Python With Anvil – The Real Python Podcast
• “Writing a PEG parser for fun and profit” - Guido van Rossum (North Bay Python 2019) - YouTube
• PEG Parsing Series Overview. My series of blog posts about PEG | by Guido van Rossum
• Abstract syntax tree - Wikipedia
• Parse tree - Wikipedia
• ast — Abstract Syntax Trees — Python 3.10.6 documentation
• Full Grammar specification — Python 3.10.6 documentation
• Python AST Explorer
• Examples of working with ASTs — Green Tree Snakes 1.0 documentation
• bellybutton: Custom Python linting through AST expressions
• black · PyPI
• pytest: helps you write better programs — pytest documentation
• friendly-traceback: Friendlier Python tracebacks.
• Raspberry Pi Pico – Raspberry Pi
• Raspberry Pi Pico W : ID 5526 : $6.00 : Adafruit Industries, Unique & fun DIY electronics and kits
• State management - Wikipedia
• Recoil
• Meredydd Luff (@meredydd) / Twitter

Level up your Python skills with our expert-led courses:

• Python Assignment Expressions and Using the Walrus Operator
• Testing Your Code With pytest
• Writing Beautiful Pythonic Code With PEP 8

Support the podcast & join our community of Pythonistas