I've made a new release of pyxie. Largely internal changes.

The focus of this release is the result of the independent intermediate node refactoring project I was undertaking. This was to transform the list oriented data structure for representing a "pure" intermediate form of code into a more stuctured/generalisable format.

The motivation behind this change was the realisation that implementation of end user functions (ie def) would be very difficult with this more structured approach.

The actual release notes (and source releases) are here:

• https://github.com/sparkslabs/pyxie/releases/tag/v0.1.25

Github repo:

• https://github.com/sparkslabs/pyxie

Package on PyPI:

• https://pypi.org/project/pyxie/

Also via PPA for Ubuntu:

• sudo add-apt-repository ppa:sparkslabs/packages
• sudo apt-get update
• apt-get install python-pyxie

What's new user facing

Before I dive into those changes, it's perhaps worth noting some of the other more minor changes, but are more noticeable from a user perspective.

• A collection of docs added in doc/
• A start on documenting the models used in Pyxie
• Licensing on pyxie's output. (Yes, it's what you want, but explicitly so)
• Some new examples focussed on specific aspects of language:
  • if
  • if-else
  • pass
  • print
  • while-break
  • while-continue
• The arduin profile has been extended to support the Adafruit_NeoPixel library
• neopixel example added

Worth noting:

• While there is an example "simplest_function" it is not expected to compile yet

Neopixel example

This is quite nice, and so since it's perhaps more interesting I'm including a snapshot here:

#include <Adafruit_NeoPixel.h>

pin = 6
number_of_pixels = 16
delayval = 500

pixels = Adafruit_NeoPixel(number_of_pixels, pin, NEO_GRB + NEO_KHZ800)

pixels.begin()

while True:
    for i in range(number_of_pixels):
        pixels.setPixelColor(i, pixels.Color(0,150,0))
        pixels.show()
        delay(delayval)

Note: This reuses the fact that #include is a comment in python. It might change at somepoint to be more like this...

from Adafruit_NeoPixel import *

...since that's logically closer, but for now this is a pragmatic approach that works.

Rationale behind changes

Before this change, pyxie used the following data structures, and code phases when parsing, analysing, transforming and performing code generation.

These were not ideal:

• Code phase: Parser - read in the source, parsed it, and created a data structure
  • Data structure: pynodes - these represent the concrete python syntax, and are generated during the code generation phase. These are python objects in a class hierarchy. Once created they are analysed and placed into a context to analyse datatypes.
• Code phase: Analysis. - Analyses the code, and decorates the existing pynodes with type information to understand the programs data and data types. (does not create a new data structure).
• Transform phase: walks the pynode CST, and generates an intermediate data structure intended to represent the program in the abstract independent of the language used. An independent intermediate form if you like.
  • Data structure: independent intermediate form - This is used to model the program in a "pure form" - which isn't constrained by the source language, and contains enough information for the target language output to be generated. That was the theory. In practice it was a nested list of list of lists ... Not ideal. More on this below.
• Code generation phase: This walked the independent intermediate form (the lists of lists), and created an output data structure representing the concrete final program.
• The third/final data structure is intended to represent the final output language. ie it is intended to be a concrete representation of the output language. This final data structure is then capable of creating the output. Again, forms a hierarchy and could be called "CppNodes" (Though they weren't before this change)

The problem here is that while the pynodes are currently well defined (to a large extent) and that the CppNodes are pretty well defined (even if they could be better), the independent intermediate form sucked because it was just nested lists. This meant in practice the code was fragile, difficult to change, and increasingly difficult to work with. In the early days of Pyxie this simplistic structure made sense. However as data analysis becomes more complex and tricky. This mirrors the fact that the same was true in the early days of the parsing side.

So the focus of this sub-project was to replace the intermediate form, and tighten up the differences and make clearer in the code base that we have PyNodes, iiNodes, and CppNodes, where:

• Pynodes - are the current structures, unchanged - note these are currently prefixed Py - PyDefStatement for example.
• CppNodes - are map to objects/hierarchy/etc that represents C++ programs, but made clearer (slightly). These are now prefixed Cpp rather than the previous C_ which some of the objects use.
• iiNodes - represent the independent, intermediate nodes. Since iiNodes need to be used in source code where there are either PyNodes + iiNodes or CppNodes + iiNodes, they have the prefix "ii" enable differentiation.

Since all 3 of these are actually models, the code for these has all moved to sit below pyxie.models.

At some point there will be a need to restructure the code more generally. Parsing, transforming to IINodes and code generation are actually all transforms, and should sit together. That was out of scope for this already relatively major internal change.

Changelog

I could include the changelog, but if you're curious about that, check out the release notes mentioned above.

Future

I'm not going to put a timeline on this - I've made that mistake in the past for pet projects but the following goals exist for future iterations:

• Code structure:
  • Consolidate the structure of CppNodes and make them "cleaner"
  • Similar for iiNodes
  • Shift type inference and variable detection from PyNodes to iiNodes
  • Change iiNodes to be more ECS based
  • Do the analysis/type inference in more of a visitor pattern (ala ECS)
• Types
  • Better implement core datatypes to allow transforms
  • Strings need implementing better
  • Aim for closest match representation of datatypes
  • Allow JSON type structures to exist
    • Implement lists
    • Implement dictionaries
  • Implement tuples
• Language/structural
  • def's with no arguments, and no local variables
  • def's with arguments
  • def's with local variables
  • classes
• Example system goals:
  • It should ideally be possible to run a perceptron style neural network on an Atmega 8A with the aim of making a robot that can follow a line based on said network.

The example system is deliberately ambitious/difficult - at least for a version of python compiled to C++ to run on such a small device.

Some parts of this will go quicker than others, and won't be done strictly in that order.

Most of it however relies upon better internal code structure. (When I started after all it wasn't clear that this would work or be a good approach, so used json objects throughout. As time goes on it becomes clearer that those ad-hoc structures are in the right sort of places, but could do with being more consistent to increase flexibility.

In order to keep releases interesting though, I'll try to add interesting examples at each stage :-)

As usual comments welcome - especially with regard to what examples you might find interesting.

Over the past several months during evenings and weekends I started work on a new python to C++ compiler -- called Pyxie (it's a play on words :). It's currently reached version 0.0.16, and is just beginning to be usable for vaguely fun things.

It's not intended to be a full implementation of python, but rather a "Little Python" that is simple enough to be compiled to C++. "Little Python" will be a well defined subset of python that can be compiled. The reason for this is to allow me to write code in python that compiles to run on embedded systems like Arduino, MSP430 and ARM mbed platforms.

This makes it different from Micropython - which runs a python interpreter on the microcontroller, and PyMite which effectively runs a simplified python VM. Both of those projects assume a device with more memory. Due to the constraints of these devices, the compiler is unlikely to ever support the entirety of the python language.

PLEASE NOTE It is not complete, and almost certainly won't do what you want, yet. If you're looking for a general python to C++ compiler - though not for arduino - take a look at ShedSkin. It is just beginning to be useful though for fun little things with an Arduino (or similar devices based on an Atmel Atmega 32U4), hence this post.

What job does it do?

Pyxie's job is to allow me to write python code, instead of C++ such that it runs on a microcontroller, and to do so efficiently. The reason I want this is to (eventually) use this with cubs and scouts.

• I quite like the idea of describing software in terms of the job it does. Since I came across the approach, I find it clarifies the purpose of a project dramatically.

Website

Pyxie has a site: http://www.sparkslabs.com/pyxie/

It's a work in progress and will become nicer and shinier as Pyxie itself gets nicer and shinier.

Release Schedule

There's no particular release schedule. I try not to have releases that are too large, or too far apart. What's planned to go into a release is done on a release by release basis. What actually has gone in is a reflection of what I've been up to in the evenings/weekends of any given week.

Name

The name is a play on words. Specifically, Python to C++ - can be py2cc or pycc. If you try pronouncing "pycc" it can be "pic", "py cc" or pyc-c". The final one leads to Pixie.

Target Functionality

The guiding principle is that Little Python should look and act like a version of python that was never written. (After all python 1.x, 2.0, 2.5, 3.x, 3.5 are all "python" but simply different iterations) You could almost say that Little Python should look like a simplified "ret-con" version of python.

At present a non-exhaustive high level list of things that are targetted are:

• Duck typing / lack of type declarations (but strong types)
• Whitespace for indentation
• Standard control structures (No else clauses on while/for :) )
• Standard built in types
• Lists and dictionaries
• Easy import/use of external functionality
• User functions (and associated scoping rules)
• Objects, Classes, Introspection, and limited __getattr__ / __setattr__
• Namespaces
• Exceptions
• PEP 255 style generators (ie original and simplest)

Quite a few of these may well be challenging in teeny tiny microcontroller environment, but they're worth targetting.

Status

So what DOES this do at the moment?

At present it supports a very simple subset of python:

• strings*, ints, boo leans, floats
• Variables - and typing via basic type inference
• while, for, if/elif/else
• Parenthised expressions
• Full expression support for ints
• For loops actually implement an iterator protocol under the hood
• The ability to pull in #include's using C++ preprocessor style directives - since they're automatically python comments

It also has 2 compilation profiles:

• Desktop/libc - that is for development and testing
• Arduino Leonardo - that includes things like DF Beetle/etc

This is a simple benchmark for a desktop:

countdown = 2147483647
print "COUNTING DOWN"
while countdown:
    countdown = countdown - 1

print "BLASTOFF"

(That "print" will become a python3 style print BTW - it's currently a python2 style statement while I'm bootstrapping the compiler!)

The arduino "blink" program for the leonardo profile:

led = 13

pinMode(led, OUTPUT)

while True:
  digitalWrite(led, HIGH)
  delay(1000)
  digitalWrite(led, LOW)
  delay(1000)

This is unlikely to ever be a completely general python to C++ compiler, especially in the context of a desktop machine. If you're after that, there are better options (shedskin and Cython spring to mind).

Where to get it -- Ubuntu 14.04LTS (trusty), 15.04 (vivid)

(This is the recommended mechanism)

I build packages for this in my PPA. You can install pyxie as follows -- add my PPA to your ubuntu, update and install the python-pyxie package.

sudo add-apt-repository ppa:sparkslabs/packages
sudo apt-get update
sudo apt-get install python-pyxie

This will automatically pull in the dependencies - PLY and guild

Where to get it -- PyPI

Simplest approach using PyPI:

sudo pip install pyxie

Alternatively go to the Pyxie page and download from there...

• http://pypi.python.org/pypi/pyxie/

... and then do the usual sudo python setup.py install dance.

Dependencies

Uses David Beazeley's PLY package. Either use your package manager, or install from pypi:

• http://pypi.python.org/pypi/ply

At some later point in time the batch compiler will make use of guild - my actor library. That can be found here:

• http://pypi.python.org/pypi/guild

Usage - Default Profile

Given the following python file : benchmark.pyxie

countdown = 2147483647
print "COUNTING DOWN"
while countdown:
    countdown = countdown - 1

print "BLASTOFF"

You compile this as follows:

pyxie compile benchmark.pyxie

That compiles the file using the standard/default profile, and results in a binary (on linux at least) that can be run like this:

./benchmark

Usage - Arduino Profile

Given the following python file : arduino-blink.pyxie

led = 13

pinMode(led, OUTPUT)

while True:
  digitalWrite(led, HIGH)
  delay(1000)
  digitalWrite(led, LOW)
  delay(1000)

You compile this as follows:

pyxie --profile arduino compile arduino-blink.pyxie

That compiles the file using the arduino leonardo profile, and results in a hexfile called this:

arduino-blink.hex

Depending on your actual leondaro type device you can use AVRdude to load the code onto the device. If you're using a bare Atmel 32U4, you can use dfu-programmer to load that hexfile onto your device.

Usage - Summary

More generally:

pyxie -- A little python compiler
Usage:

    pyxie -- show runtime arguments
    pyxie --test run-tests -- Run all tests
    pyxie --test parse-tests -- Just run parse tests
    pyxie --test compile-tests -- Just run compile tests
    pyxie --test parse filename -- Parses a given test given a certain filename
    pyxie parse filename -- parses the given filename, outputs result to console
    pyxie analyse filename -- parses and analyse the given filename, outputs result to console
    pyxie codegen filename -- parses, analyse and generate code for the given filename, outputs result to console. Does not attempt compiling
    pyxie [--profile arduino] compile path/to/filename.suffix -- compiles the given file to path/to/filename
    pyxie [--profile arduino] compile path/to/filename.suffix  path/to/other/filename -- compiles the given file to the destination filename

Open Source

Pyxie is Copyright © 2015 Michael Sparks, and Licensed under the Apache 2 license. It's not public on github at the moment because the internals are still changing somewhat between releases. Once this has stablised I'll open up the git repo.

The code is however on github, so if you'd like early access, let me know. Obviously you can do what you like with the code from pypi within the reasonable limits of the apache 2 license!

It's developed entirely using my own kit/etc too so nothing to do with work. (I mention this primarily because during the Micro:bit project I also built a scrappy python to C++ compiler. That had all sorts of limitations, but made me think "If I was to redo this from scratch, I'd...". This project is a result of that thinking as a result, but as I say nothing to do with work!)

Release History

This is to give some idea of progress it's good (though obviously not swift) progress, with an average of a few weeks between releases. (Mainly because dev depends on spare time:)

• 0.0.1 - (rolled into 0.0.2 - Initial structure)
• 0.0.2 - supports basic assignment
• 0.0.3 - Ability to print & work with a small number of variables
• 0.0.4 - Mixed literals in print statements
• 0.0.5 - Core lexical analysis now matches language spec, including blocks
• 0.0.6 - Character Literals, "plus" expressions, build/test improvements
• 0.0.7 - Structural, testing improvements, infix operators expressions (+ - * / ) for integers, precdence fixes
• 0.0.8 - Internally switch over to using node objects for structure - resulting in better parsing of expressions with variables and better type inference.
• 0.0.9 - Grammar changed to be left, not right recursive. (Fixes precedence in un-bracketed expressions) Added standalone compilation mode - outputs binaries from python code.
• 0.0.10 - Analysis phase to make type inference work better. Lots of related changes. Implementation of expression statements.
• 0.0.11 - Function calls; inclusion of custom C++ headers; empty statements; language spec updates
• 0.0.12 - While loops, break/continue, Website, comparison operators, simple benchmark test
• 0.0.13 - if/elif/else,conditionals/boolean/parenthesised expressions.
• 0.0.14 - For loops implemented. Added clib code, C++ generator implementation, FOR loop style test harness, parsing and basic analysis of of FOR loops using a range interator
• 0.0.15 - clib converted to py clib for adding to build directory
• 0.0.16 - Adds initial Arduino LEONARDO support, improved function call, release build scripts

Supporting this

Please do! There's many ways you can help. Get in touch.

Summary

Pyxie is at an very early stage. It is a simple little python to C++ compiler is at a stage where it can start to be useful for some limited arduino style trinkets.

Kicking the tires, patches and feedback welcome.