Discuss
  • Syntax is the Problem

Syntax is the Problem

# Michael Lewis (7 years ago)

Raul-Sebastian Mihăilă just made a post here about some mixin syntax. I didn't read it (sorry).

But, it got me thinking about a concept I've been thinking for years: syntax is the problem, and there's a better solution.

If you define syntax as a human <--> computer language (a human-readable

and computer-readable form of text), you necessarily need a very strictly defined syntax. One missing curly, and you're f'd.

Duh, we all know this. Hang onto your pants for a second, let's explore an alternative.

What if we edited scripts more directly in AST form (abstract syntax tree). Developers could implement their own UI to manipulate this AST. There are many, many benefits to this, I'll get to in a second.

First, let's remember what everyone in the JS community is doing right now: running their code through transpilers. (I refuse to do this, that's why I'm unemployed - I strongly dislike the concept). What is a transpiler? It's an unofficial middleman that interprets the syntax, converts it to an AST, performs transformations, and then returns the AST to syntax form.

Suddenly, scripting in AST form doesn't sound so crazy.

Clearly, you wouldn't be writing JSON. We would be using a GUI to create the AST. And this is the greatest benefit: moving away from syntax, and moving to a GUI.

Then, instead of babel plugins that provide alternative syntax, you could have plugins that provide alternative experiences.

What would designing a class look like, if using a GUI?

  1. Click [ create new class ]
  2. Click [ add new method ]

Or, click the [ extend ] button next to an existing class...

This is the future. I have to go eat breakfast, but I'd love to discuss this future if anyone is interested...

# Isiah Meadows (7 years ago)

You may be interested in this. It's actually not that uncommon in childrens' programming, animation, and other less technical needs for scripting. Also, consider Node-RED, which is also a visual platform for the Internet of Things (with a JS core, but you rarely need to actually dive into JS for most things).

Isiah Meadows me at isiahmeadows.com

Looking for web consulting? Or a new website? Send me an email and we can get started. www.isiahmeadows.com

# Alexander Jones (7 years ago)

This is projective editing. See for example www.jetbrains.com/mps

# Mike Samuel (7 years ago)

On Sun, Nov 5, 2017 at 8:22 AM, Michael Lewis <mike at lew42.com> wrote:

Raul-Sebastian Mihăilă just made a post here about some mixin syntax. I didn't read it (sorry).

But, it got me thinking about a concept I've been thinking for years: syntax is the problem, and there's a better solution.

If you define syntax as a human <--> computer language (a human-readable and computer-readable form of text), you necessarily need a very strictly defined syntax. One missing curly, and you're f'd.

Duh, we all know this. Hang onto your pants for a second, let's explore an alternative.

What if we edited scripts more directly in AST form (abstract syntax tree). Developers could implement their own UI to manipulate this AST. There are many, many benefits to this, I'll get to in a second.

Assuming you're bringing this to TC39 because you think the committee can contribute to this effort, there seem to be a few separable questions here:

  1. Would there be benefit to standardizing an AST?
  2. Would developers benefit from being able to manipulate ASTs?
  3. Should TC39 organize efforts to realign JS development around AST manipulation?

TC39 has already debated some AST proposals. You could add your use case to those. You really don't need them though. Babel didn't wait.

I'm unsure why manipulating ASTs is a benefit. It seems tools that manipulate ASTs can simply have a parser tacked on the front and a serializer tacked on the back and manipulate text files. This is what the refactoring tools in IDEs like Eclipse do.

Even if these tools are wildly popular, the syntax won't be going away though. Backwards-compatibility requires EcmaScript be embeddable in HTML, SVG and a number of other textual formats.

It seems to me, that TC39 need not be involved in the process of specing tools for manipulating ASTs and even if TC39 gives a full-throated endorsement to such tools, it would still have to spec syntax.

First, let's remember what everyone in the JS community is doing right

now: running their code through transpilers. (I refuse to do this, that's why I'm unemployed - I strongly dislike the concept). What is a transpiler? It's an unofficial middleman that interprets the syntax, converts it to an AST, performs transformations, and then returns the AST to syntax form.

Suddenly, scripting in AST form doesn't sound so crazy.

How does (developers would be better off authoring ASTs) follow from (tools like transpilers use ASTs internally)?

Clearly, you wouldn't be writing JSON. We would be using a GUI to create

the AST. And this is the greatest benefit: moving away from syntax, and moving to a GUI.

There's a whole literature on end-user programming that others have referenced. None of it has caught on with professional developers. How are the GUIs you propose different from past efforts?

Most Lisp syntaxes map directly to the AST with just a bit of syntactic sugar. There are also plenty of fine GUIs. You can have an awesome GUI for building an AST and still have syntax so that you can review code on long plane trips.

Then, instead of babel plugins that provide alternative syntax, you could have plugins that provide alternative experiences.

What would designing a class look like, if using a GUI?

  1. Click [ create new class ]
  2. Click [ add new method ]

I can do this in Eclipse. Most of the time I just type "class."

There seems little benefit to a GUI that replaces repetitive typing with repetitive clicking, and I'm not familiar enough with the terms you use to know why I would want to plug-in experiences.

Or, click the [ extend ] button next to an existing class...

This is the future.

This is an odd choice of tense.

I have to go eat breakfast, but I'd love to discuss this future if anyone is interested...

I hope you had an awesome breakfast! Maybe someday we'll all Click [ eat brekkie ].

# Michael Lewis (7 years ago)

The future of software development involves a meta experience. If I want to edit the name of a class property, and ensure that it updates everywhere, why isn't this possible? Right now, we have to find+replace. Could you ensure that all the users of your package also have their property names replaced? That would be tricky by today's standard. But what if that property has an underlying ID, and you're just changing it's label rather than its name/identifier.

When you require/import any package, and you begin using it - you may only be using a fraction of its capabilities. When you call a function, maybe that function has 3 different argument signatures. When the maintainer's deprecate some of the features/use cases, are you affected? It's hard to say unless you have very low level insight into what is actually in your scripts.

This insight is only available via the AST.

In the short term, I think you're right: babel is the realization of the AST. At least until we have better GUI scripting tools, better version control, better build tools, etc.

By the way, a GUI doesn't mean you can't use your keyboard, and doesn't even require a mouse. It just means a very visual, meta experience. It could be similar to coding w/ auto complete, only the auto completed things (vars, fns, classes, files, etc) would be real, tangible things, that you could drag and drop, if you wanted.

But, it's not about dragging and dropping. It's about the meta data. The ability to add encoded information associated with any of the things (apps, modules, files, classes, properties, arguments, variables, conditions, etc). One thing I've been working on a lot is an automated logger. Basically, create a console.group for every method call, showing args, grouping internal logs (nested method calls and direct log() calls), and showing the return value. This is easy with Proxies. What's not so easy, is configuring what you want logged, and when. Meta data and ASTs would help with this.

Instead of putting markdown in code comments (what a ridiculous hack? I mean, it's actually a pretty clever/easy/effective way to write formatted documentation nearby your code, but seriously, is *that *the best user experience we can imagine?), why not have a hotkey that activates a documentation/tests panel?

Babel is complex and shouldn't be necessary. Babel parses JavaScript into an AST, just to convert it back to JavaScript, then we usually minify and gzip before sending to client, then unzipping, parsing back to AST...

Why not serialize the AST directly to an optimized binary form? Checksum and ingest, no parsing needed? I've never written a line of compiled code, so I don't really know what I'm talking about here, but I think getting closer to the "metal" (processor) makes a lot of sense.

Also, shouldn't the AST parsing be the same algorithm? Why micromanage babel and the V8 internal AST parser? Seems like a lot of work.

I'm new around here - I have a lot to learn. Even so, it's clear to me that this system can be way better.

You asked what I meant by, "plugins for alternative experiences". Basically, just alternative workflows (hotkeys, auto complete algorithms/workflows, user interfaces, features, etc) for your GUI coding environment.