Monday, February 13, 2012

Lisp: First Impressions

I learned the basics of Lisp 10 years ago when I was the TA for ICS313: Programming Language Theory. This semester, I returned to Lisp after a decade away, again in the context of TAing for ICS313. After so long away, I basically had to start over from scratch again. Though it certainly goes much faster the second time around, it's still a lot like learning it the first time. So these are my "first" impressions.

We are learning Common Lisp. So far, I'm not very impressed.

First of all, I find that Common Lisp is bloated. There are usually about 6 different ways to do something. For example, want to compare two items? Consider =, eq, eql, equal, equalp... and that's just getting started. Want to print something? Consider print, princ, prin1, pprint, format, etc. The reason that so many options exist is because each is subtly different. I'm sure that, once mastered, all these options increase your programming power, since you can pick exactly the right tool for the job. And, in many cases, some of the options seem to be now-unused holdovers that persist from earlier days in Lisp's development. So eventually you learn which ones you can ignore. But it all makes Common Lisp rather tedious to learn.

My next stumbling block is the formatting. As evidenced by both my students' code and my own inclinations, it seems natural as a programmer to try to format the parentheses to make matching them up much easier. The parentheses are subtle, and it's easy to get one out of place. This can produce code that compiles or loads but then fails at runtime because the meaning is subtly different to what you intended. This well-discussed blog posting sums it up fairly well. In general, I agree with the author there. Parentheses in Lisp mark the start and end of various expressions in the same way that braces mark the start and end of blocks in C-like languages. In C-like languages, we have proponents of such extremes as the Allman style of indenting. These Allman proponents feel that such formatting is so essential to readability that every brace deserves its own line! Yet the Lisp community advocates the exact opposite: that no parenthesis should be clearly placed. Instead, they should all just be tucked away at the start or end of a line of code. Supposedly some day you get to the point were you can "see past the parentheses". But this seems to me like a convention that makes code unnecessarily hard to read.

I find most of the arguments for this "parenthesis stacking" format are weak at best. One of them that irks me is that your editor will help you. First of all, you're not always reading code in an editor. Secondly, I should have to move my cursor around or press keys or require fancy color highlighting to make quick sense of the code on the screen. It's called "readability", not "navigability". A third argument is that you can ignore the parentheses because the indenting should show you the structure. But the indenting is not what actually determines the code structure--the parenthesis do! So I need to be able to quickly spot when the parentheses are wrong even though the indenting is correct.

This formatting thing bugs me because it seems the problem comes from an asinine coding convention choice. And that's the one reason that has me formatting this way rather than the way that makes sense to me: because "that's how it is done in Lisp." Like the choice to drive on the right side of the road, it's hard to buck the community on such a choice as a late-comer! Yet, because it makes the code harder to work with, but with no good reasons that I can see, it feels a little like hazing: "This is the way we all had to deal with it when we learned, so you do too."

That brings me to the Lisp community in general. First of all, I don't care to worship at the altar of Emacs. As I mentioned above, I shouldn't have to have a special editor to write code. Don't get me wrong: an IDE is great at increasing productivity and I definitely want one. But the code should be both readable and writable without one. Then I'm free to choose the IDE that meets my own requirements. (On that note, after trying Emacs and LispWorks, I settled on Cusp. It's a little tricky to get up and running, and a bit quirky at first, but it works pretty well. The highlighting of parentheses structure is very helpful.)

Secondly, there's just this "Lisp is so much better!" vibe in the community. Now, obviously you've got to wave the flag for your favorite language. I have no problems there. But, as others have pointed out, if Lisp is so wonderful, how come we're not all using it? Just about every computer science major has to learn a Lisp dialect at some point, so it's not just an issue of exposure. Is it because it lacks good libraries for modern tasks? Is it because, while powerfully writable, its dynamic re-writability makes it hard for someone else to read or maintain? Is it because, while a pioneer of so many cool ideas, most of those ideas have now been imported from Lisp by other languages? Is it because Lisp has built up over 50 years of cruft, but each new Lisp project to simply and overhaul it fractures the small but fanatical Lisp community, leading to inter-dialect derision and flamewars? Hard to tell. What I can tell is that the fanatical belief in Lisp's supremacy over all other programming languages is a little hard to swallow.

My general conclusion is that Lisp is still worth learning for the history of it. However, I don't think I'll be taking it up as my day-to-day language. I'll be looking elsewhere for more modern implementations of Lisp's contributions that are useful to me.

Still, the code-as-data idea is largely still unique to Lisp. That would be fun to explore more, so finding a Lisp dialect that fits me better might still be rewarding. I've considered Scheme a bit, but I think either Clojure or newLisp would be even better. (I found newLisp because I thought: "Why doesn't someone clean up and simplify Lisp back to it's glorious essence? If I did that, I'd call it 'new Lisp'." I searched... and behold! Already done 20 years ago by someone else.) Both seem to have newer, friendlier, more open communities. Clojure has the advantage of the JVM and the entire Java API behind it. newLisp is targeting what I think is a great niche for Lisp: scripting. This is where powerful writability at the expense of readability and maintainability is a viable tradeoff.

These are my current impressions of Lisp. Perhaps they'll change with time. If so, I'll let you know!

21 comments:

  1. Re print, princ, prin1, pprint, format, these arise from Lisp's special ability to read, eval, and print objects. Learning the difference between them is easy, and hardly worth a complaint. But even if they were hard to learn, complaining wouldn't make sense. Imagine that a parent buys a car for her teenage son, but the son protests saying, "Aw come on! Now I have to learn to drive!" Also, instead of learning to ignore, you should just learn.

    Re formatting, it's odd to me that anyone would write Lisp code without paredit-mode or some IDE equivalent of it. Editing code in a structured manner, where commands operate on s-expressions as opposed to lines or characters, is a wonderful thing. As a bonus, the parens are balanced for you (http://i.imgur.com/1l3QI.gif). The Lisp community has probably failed to communicate this to newcomers.

    Re "if Lisp is so wonderful...", a common introduction is http://www.paulgraham.com/avg.html . Unfortunately, the real value of Lisp is only revealed after some experience with Lisp. But since by definition newcomers have no experience with Lisp, they don't see the value in learning it.

    ReplyDelete
    Replies
    1. Thanks for the comment.

      Re: all the printing functions--yes, they're not that hard to learn. But they do need to be learned, and the names aren't very pretty. During that learning process, it's annoying that there are six different non-orthogonal versions of each construct, rather than one or two versions, as in most other languages. print and format together seems to cover most needs, but you still need to learn all others to make sense of other people's lisp code.

      To use your car analogy, it's like the teenager's new car comes with an automatic transmission, a manual stick shift, a steering wheel, and a tiller, showing the full evolution of car controls.

      My problem with parenthesis matching is not so much while writing the code, but when I'm editing it or reading it for bugs later. As I mentioned, Cusp's highlighting does help a lot with the writing process.

      Delete
    2. May I ask why you deleted my second comment?

      Delete
    3. I didn't delete your post. Blogger bug, perhaps? I know I usually can't post comments from Firefox, so this system's not perfect. I did get an email notification of your post, though. You said:

      I'm a little baffled by your take on the print functions.

      The most commonly used one is print. You might want to pretty print, which is pprint. prin1 and princ are for lower-level printing; they output computer-readable and human-readable forms, respectively.

      In another category there is format, which is a convenient DSL for printing data.

      That is all you really need to know. When I look at your post and comment above, I don't understand the hubbub.

      The nice thing about CL is that the hyperspec gives you a concise explanation of every function. There's little need to google for context, examples, and corner cases, which is often necessary in less-precisely-specified languages. In Emacs, C-c C-d h opens the hyperspec web page for the function below the cursor. You don't need to memorize details before getting started.

      You may have misunderstood the car analogy. Some teenagers don't have cars -- those are the languages without a designed-in system for reading, evaling, and printing arbitrary objects, both as code and as data. Lisp has a car.

      You may have misunderstood my paredit point as well. When I spoke of writing Lisp code, I was referring to both writing and editing. Paredit lets you cut and paste arbitrary s-expressions with ease. There are operations to split and join sexps, to nest and unnest them, and more. It's the most efficient way to write and edit code ever conceived, if you ask me. Parentheses are always matched when you use paredit. Editing operations are paren-wise balanced.

      Delete
    4. This is a fair amount of hubbub. :) It's not a big deal at all once you've learned them. I feel the same way about basic HTML markup behaviors that are duplicated in slightly different "simpler" markups by various wikis, forums, etc. Once you've learned something, it's hard to see what everyone else's problem is, but, if you want to know what those first impressions were, see above. And it's not just the print fns, it's nearly everything in Lisp: checking equality, setting variables, etc.

      Thanks for the HyperSpec tip; I've been using that. My IDE also reminds me of the structure of the form with a popup--no triple-key-press required. Both are useful once you know that a particular function or form exists and just need a referesheer on its details.

      Okay, if the language feature is the car, rather than the whole language, then I guess I'm complaining about the state of the garage, insurance forms, parking fees, and trips to the DMV: all the other overhead you need to get through before you get to the as-yet-unexperienced thrill of driving. I'll stick with it for a while yet--apparently driving is a lot faster and more exciting than taking the bus, though both still get you from point A to B.

      Delete
    5. Sorry, I still don't understand. You emphasized "once you've learned them", but how long does it really take to learn the following:

      "The most commonly used one is print. You might want to pretty print, which is pprint. prin1 and princ are for lower-level printing; they output computer-readable and human-readable forms, respectively."

      Just pass the object to print. Done. Optionally pass the stream as the second argument. They all take the same argument(s). Why does this require a protracted learning period? The hyperspec has a simple explanation as well, in which all are conveniently given on the same page.

      Delete
    6. It doesn't take that long to learn them. I'd say I've probably spent more time writing about learning them now than I did actually learning them. :)

      But there are a lot of them. Their names are not very meaningful (prin1 and princ? Why those names?), and so they are harder to keep separate in my head. The most obvious function name--print--is the one I end up using the least because I usually want to print human-readable forms. But what's better to use in that situation? princ or format? What if I want to print a human-readable string followed by a newline? Would princ followed by terpri better than just using format?

      So my point is that this high function count in lisp is a contrast to most other languages that usually only have a couple different print functions. And the rest of Common Lisp is the same way: innumerable ways to compare objects (=, eq, eql, equal, equalp, string=, string-equal, etc), 2 different ways to create a global variable (defvar, defparameter), four+ conditional statements, and so on.

      As mentioned in my original post, this apparently increases your programming power once you learn the subtle differences between them all and know when you should use one vs the other. That's cool. But, with about four times the number of function choices for each thing you want to do, it takes longer to go through the docs, read them all, and internalize it. And then there's a slightly higher cognitive load afterwards: If I have three different ways to print a prompt to a user, I have to decide which one makes the most sense to use in the current situation.

      Delete
    7. You seem to finally agree that the difference between print, pprint, prin1, and princ is easy to understand, but then you demur, saying there are "a lot of them". I just cannot see how these 4 functions, which have dead-simple meanings and take dead-simple argument(s), could pose any problem at all for even the newest newcomer.

      I don't understand why you say that print "is the one I end up using the least because I usually want to print human-readable forms." But print outputs human-readable forms.

      My communication efforts have utterly failed with regard to the car analogy. You are back to comparing car-owning teenagers with no-car teenagers. Of course a car-owning teenager has more to learn, if he wishes. It's not a fair comparison.

      Your hangups about the print functions suggest that you misunderstand their purpose. They are for writing objects to a stream, sort of like Java's Serializable.writeObject but with the elegance and power of s-expr code/data equivalence along with more options such as human-readable output.

      The P in REPL means "print" in the sense of the Lisp function "print". So the print functions would be used for making a REPL or for writing an object database, for example.

      I think the answer to the hubbub is this: just use format. format is like an advanced printf. It's a DSL for outputting data.

      Delete
    8. > I just cannot see how these 4 functions... could pose any problem at all for even the newest newcomer.

      I certainly do envy your ability to read documentation once and immediately understand, internalize, and remember it!

      > But print outputs human-readable forms.

      It "produces output suitable for input to read". By human-readable, I mean that I don't usually want the escape characters that read requires, such as quotes around all of my strings.

      > I think the answer to the hubbub is this: just use format.

      This seems like good advice. I notice that some people do seem to favor format for most of their print needs.

      Delete
  2. A friend said:
    I can't say I agree with his position, but posted it here anyhow because I hoped it would provoke discussion. The crux of his problem is that he is unwilling to use Emacs--one can't just hop into a Formula One car and expect to drive it the same way as a '98 Dodge Neon.

    Because he is handicapping himself by his choice of tools, he is forced to memorize more than is needed (if there's room in his head for C-c C-d C-h the rest will come with time).

    It may be /unfortunate/ that to be maximally effective in Common Lisp you must learn Emacs, but that's just the way things are in life. I'm interested in kayaking. Maybe I don't care to learn to swim, and sure, I could learn to kayak without getting a few swimming lessons in first (the Eskimos did)... It's just not generally recommended, and I'm pretty sure it'll slow down my learning.

    I could proceed to iterate over my many points of dispute (such as the author's hubris in labeling coding conventions for a language he is barely acquainted with "asinine"), but will let it rest instead. None of this is really a big deal. It is one man's opinion, and he is entitled to it. Meanwhile, I will continue to use Common Lisp (that is until something better comes around), and love it, despite its flaws.

    ReplyDelete
  3. "The parentheses are subtle, and it's easy to get one out of place."

    It's true. I'm still searching for the programming language where it's hard to get a character out of place.

    "This can produce code that compiles or loads but then fails at runtime because the meaning is subtly different to what you intended."

    Yes, sadly. I prefer to use programming languages which can only compile to bug-free code.

    ReplyDelete
    Replies
    1. Touche.

      Yes, it is possible to get a brace or semicolon out of place in other languages and also to have that mistake get past the compiler to cause a runtime error. But I find the situation seems a little worse in Lisp because 1) so many of the syntax-delimiting characters are the same and stacked up next to each other and 2) there's a formatting standard in those other languages that, if followed, makes it easier to spot those mistakes.

      I'm learning that there's not such a coding standard in lisp because there's an editor standard instead: Emacs.

      Delete
    2. Yep... emacs is your friend.

      Delete
    3. "I prefer to use programming languages which can only compile to bug-free code."

      Yes, we are having more fun with Haskell! :)

      Delete
  4. It is perfectly possible to edit lisp code with VIM, if you are so inclined. Just don't expect to get more support from it then you would get from an unconfigured VIM for editing Java code.
    Or use nano, if you want to.
    Editing Common Lisp programs using Emacs can superficially be compared to editing Java programs using Eclipse or IntelliJ. The support is incredible.

    Regarding the parentheses: It is true that you will stop seeing them. And getting them right with an editor that supports highlighting is not hard. Not to mention the kind of support you get from paredit cmode.

    ReplyDelete
  5. When I started learning Common Lisp a few years ago I too had some reservations about a language that forced an editor upon you.

    Since I was learning it as a hobby though I had the extra time to put into emacs. I found two killer features in the emacs / slime combination:

    1. Tab reparses the code and indents to where the parens indicate. So rather than me indenting the code to where I though it should be based on functionality in my head I'm pressing tab and double checking the indentation I got from my editor against my expectations. It's a bit of an inversion of control but I found it to be a pretty powerful way to work.

    2. Built in key bindings to search the online (or local copy) of the CLHS.

    Finally, having had a couple of TA's who clearly hated their current assignment I'd implore you to try really hard to get enthusiastic about the subject you're teaching, for the sake of your students at least

    ReplyDelete
    Replies
    1. Thanks for the tips! I've been wondering what the eventual payoff for emacs is supposed to be.

      * Eclipse's Cusp will do the indenting using Ctrl+Tab by default. I changed the binding to Tab and will try to use this more often. It is a strange way to work at first.

      * Cusp will popup a short content assist reminder on the basic form of under the cursor if you hover over it or hit Ctrl+Space. Alt-H (one key-pair rather than a sequence of 3) is the binding to the full online HyperSpec for the form currently under the cursor.

      I love my TA job, and I enjoy Programming Language Theory. One of the reasons I'm exploring these options is so that, when my students come in moaning about lisp and emacs and how they would have already solved this assignment in their old language/IDE, I can sing some praises about the eventual payoff. I'm still looking for concrete personal experiences of those payouts--I'm sure I'll hit them soon. :) I have more confidence of this for lisp than I do for emacs, though. We're getting to lambda fns and closures this week.

      Delete
  6. If you don't like common lisp, there are plenty of lisps to choose from. You might try Racket or another scheme based language.

    As for the editor, I would hardly attempt to write any code without modern auto-indenting. From C to Python to Perl: it's a must-have for anyone who writes code.

    Finally, try not to think of lisp as a language. (I know, I know) Lisp is more of a Proto-language. When you compile or interpret other languages you must create an abstract syntax tree or AST. In Lisps, you work with it directly, conveniently skipping half the process. This does have real benefits to the programmer, such as a built in EVAL loop, or the ability to add language features as you go.

    I love lisp, it's the most fun I can have on a computer. (Really!) But I do have a close second which might strike you as an anti-lisp: Forth. If you have time, I highly recommend reading the (one or two files) Jones Forth source. Brilliantly illuminating.

    Also, I somewhat agree with you on the number of commands. It was explained to me, however, that common lisp is not for beginners. You can take a small subset of CL and they can easily learn it, but common lisp is made primarily for master programmers. It's similar to a master carpenter's shop. I do not intuitively understand every tool in his shop. He, however has taken the years of practice to become proficient with them. The beginner hacks at his problem with a few tools and gets his job done, but it is rough and requires much effort. A master is done in minutes by using a more powerful tool.

    Oh, and I also recommend "In the Beginning Was the Command Line" and "The Art of Unix Programming". Both are freely available on-line.

    ReplyDelete
    Replies
    1. Thanks for the comments. As I mentioned, I'm thinking about checking out newLisp next. Yeah, I'm sure the multitude of CL commands increases your expressive power once you learn them all. I'll add Forth to my long list of languages to check out... probably after Scala and Haskell. :)

      Delete
  7. Hi Zach, this is Bill from UH ICS. Yes, I agree. It's bloated. A nightmare for production projects. And it's usually thought of as a late-binding philosophy language, I understand, and it's usually interpreted in that mode.

    ReplyDelete
    Replies
    1. Thanks for the comment. :) I've decided to check out Clojure as my next lisp dialect to see if that suits me better. However, Scala's currently first on my "next language to learn" list, so Clojure might have to wait a little while. :)

      Delete