Arc Forumnew | comments | leaders | submitlogin
Is Arc good for big projects?
8 points by carbon 5927 days ago | 20 comments
Paul Graham writes:

"Over the years my appreciation for lists has increased. In exploratory programming, the fact that it's unclear what a list represents is an advantage, because you yourself are unclear about what type of program you're trying to write."

I wonder how Lisp programmers can live with such unclear code in real-life projects. Ok, maybe Paul is smart enough to remember what each list mean, but what about working in a team? You can criticize strong types or using also records besides the list, but they add self-describeness to programs, which is very important in practice. Or maybe Arc is only intended to be used in very small one-man projects?



5 points by nostrademons 5927 days ago | link

Hahahaha no. Lisp-1 + unhygienic macros + no module system = a world of pain when projects grow beyond a few thousand lines, even ignoring the lack of user-defined data types. The lack of structured data is less of a problem than it might seem, because you can write accessor functions that abstract away whether it's stored in a list or hashtable and the precise keys, and just use those. The issue is namespace-pollution from all those accessors.

I'd kinda like to see Arc grow some sort of module system - even a very simple dictionary-based system like Python's or JavaScript can go a long way towards making larger programs tractable. Come to think of it, I think you could emulate JavaScript's system with closures and hashtables. Just wrap every module in a closure, which binds a hashtable to the module name and inserts each function into the hash table:

  ([do
   (set My-Module-Name (table))
   (= (My-Module-Name 'foo) (fn (arg1 arg2) ...))
   (= (My-Module-Name 'bar) (fn (arg) ...))
   (= (My-Module-Name 'baz) (fn () ...))
  ] 'ignored)
A little macrology could make this a bit more elegant:

  (module My-Module-Name
     (private my-private-method (arg1 arg2) ...)
     (public my-public-method (arg1 arg2) ...)
     (public my-other-method (arg1) ...))
The macro code would look something like this:

  (mac module (name . spec)
    (with (car-is (fn (sym) [is sym (car _)])
           public-forms (keep (car-is 'public) spec)
           private-forms (keep (car-is 'private) spec)
           make-public [`(= (,name ,'(cadr _)) (fn ,@(cddr _))]
           make-private [`(,(cadr _) (fn ,@(cddr _)))])
      `([(do
          (set ,name (table))
          (with ,(mappend make-private private-forms)
             ,@(map make-public public-forms)))] 'ignored))
Now I kinda want to install MzScheme and try it out...I've been working off the tarball source only. :-)

-----

4 points by chandler 5926 days ago | link

As far as I can tell, having a purely closure-based module system won't actually solve the lisp-1 redefinition problem; to use your module system, some manner of compile-time token capture is necessary. Essentially, the problem is:

  (= my-module
     (let obj (table)
       (= (obj 'rem-xs) (fn (s) (rem #\x s)))
       ...
       obj))

  ((my-module 'rem-xs) "abcxdefxghi") ;; => "abcdefghi"
Now, suppose I redefine "rem" at the toplevel to be like the BASIC rem, or a remark function that ignores its parameters:

  (def rem args
    nil)

  ((my-module 'rem-xs) "abcxdefxghi") ;; => nil
I think Arc uses a lexical scope, so this problem shouldn't show up in the following example:

  (aload 'my-module)

  (let rem (fn args nil)
    ((my-module 'rem-xs) "abcxdefxghi"))

  ;; => "abcdefghi"
However, you would need to be careful when attempting some kind of dynamic import:

  (let rem (fn args nil)
    (aload 'my-module)
    ((my-module 'rem-xs) "abdxdefxghi"))

  ;; => nil
I think the former (though not latter) problem can be mitigated by shadowing all functions/globals you want to stay invariate, as in:

  (= my-module
     (with (rem rem
            table table
            fn fn
            obj (table))
       (= (obj 'rem-xs) (fn (s) (rem #\x s)))
       ...
       obj))

  ((my-module 'rem-xs) "abcxdefxghi") ;; => "abcdefghi"
However, as you can imagine, this will get unwieldy fairly quickly (like you, I've downloaded the arc0.tar file, but not actually bothered to downgrade my copy of plt-372; i.e., I'm not sure how canonical the provided arc snippets are).

Of course the ignored caveat is this: so many bloggers are up in arms over this point, it's presupposed that being able to change core library functions is a defacto terrible thing--at work, where we have a massive C/C++ codebase running on some unmaintained, sourceless, legacy APIs, being able to (for example) add/fix methods in some of the base classes would be a significant time-saver.

I think Arc as-is makes it a bit too easy to shoot yourself in the foot; however, I'm firmly in the camp that being able to update core functionality should be allowable.

Also, for better or worse, look at emacs: a massive, mature system that contains no module/namespace system, no closures, and, due to its dynamic scope, would fail each of the above examples. I'm not saying I hope Arc emulates these (lack of) features, however, it's still proof that they're not necessary for large, real-world (whatever that means) projects.

-----

3 points by cpfr 5925 days ago | link

The big problem is def and =s update a global symbol table. They are not like Scheme and define. If def obeyed local scoping rules and could be shadowed, the problem is shadowing has to be done explicitly. That is why this is painful and unwieldy. Ultimately, this needs to be fixed or Arc will be needlessly crippled.

-----

2 points by jdvolz 5927 days ago | link

What kind of syntax should we use to denote the module? Something like:

(module::function args)

This idea comes from Ruby, though I am not sure it necessarily plays well with the function composition operator (func1:func2). I wouldn't want to use -> or "." because of the concept collisions those would create. Could we just use something that doesn't have a mainstream meaning, like (module#function) or (module>>function)?

-----

1 point by nostrademons 5927 days ago | link

module::function looks good to me. Someone else suggested allowing user-defined syntax characters; this would be a good use-case. It only has to expand into (module 'function), so the implementation is basically trivial.

-----

2 points by cpfr 5926 days ago | link

Why not module:function? Abuse the property that : stands for composition. A more lispy solution of (module function) might also work, and its only an extra character. Though if I were going for an Arc solution, it should probably be (bin fn) or (md fn).

-----

1 point by randallsquared 5924 days ago | link

In case you hadn't noticed, there's a sketch of a module system in the arc-wiki repo that does exactly this, now.

It doesn't solve the problems mentioned in this thread, of course.

-----

1 point by greatness 5927 days ago | link

This seems like it'd be closer to classes/objects than modules.

-----

2 points by nostrademons 5927 days ago | link

In JavaScript they're the same thing - objects do quintuple-duty as hashtables, records, objects, classes, and modules. Arc is very similar to JavaScript in terms of the primitive data types available - they're both Scheme-derived languages.

The reason I'd term call this modules is that they don't have the implicit-receiver that classes/objects do, nor do they have constructors. Once you define a module, that's it, and they live in the global namespace. You could change things around pretty easily so that instead of immediately executing the closure containing the module's namespace, you define it as a constructor function. And you could probably then come up with some macro to expand to a message send, which passes the object as the first argument to any method defined in it. I think you'd be stuck with explicit-self though, like in JavaScript or Python, instead of the implicit `this` of Java.

-----

3 points by mst 5926 days ago | link

... and scopes.

Javascript's scope chaining is one of my most and least favourite things about the language. But it's a damn clever approach once you get it.

-----

13 points by pg 5927 days ago | link

Did Viaweb count as a real-life project? If so, the answer is that the usual Lisp programming style works well.

Nothing comes for free. There's a cost as well as a benefit to all the bureaucratic stuff that "enterprise" languages and programming styles clutter up your code with: your program gets harder to change. While Viaweb's competitors were following all sorts of virtuous best practices, we were leaving them in the dust in terms of features.

-----

6 points by zachbeane 5927 days ago | link

If Viaweb is anything like the code from Arc, ANSI Common Lisp, or On Lisp, it's hardly the usual Lisp style.

-----

7 points by pg 5926 days ago | link

I meant programming style in the sense of using exploratory programming techniques, not in the sense of using particular textual conventions or language features.

I think you're also confusing Lisp with Common Lisp. In a lot of ways (the long names, CLOS, the loop macro) CL is kind of unLispy. The coding style used in those two books (and Viaweb) is closer to the conceptual core of Lisp.

-----

7 points by pg 5926 days ago | link

Exploratory programming is definitely something that happens more in small groups than large ones. You could use it in large projects if they were organized so that each bit of the program had a single "owner." But in the kind of large project where lots of code is touched by multiple authors, I don't think you could do exploratory programming. In fact, you can barely do non-exploratory programming.

-----

8 points by akkartik 5927 days ago | link

Arc is probably not good for large projects.

However, large projects are a bad idea anyway.

Arc or any lisp will keep your project from getting large for longer.

-----

1 point by gugamilare 5927 days ago | link

It is not really true. Lisp has been used for many large projects. Should I mention "Smigo" (from "The Lord of the Rings") and he former game company "Naughty Dog"?

-----

1 point by akkartik 5927 days ago | link

Sorry, s/lisp/arc

It's an easy mistake :)

-----

2 points by tjr 5926 days ago | link

Not yet, if for no other reason, then because it seems too volatile. I wouldn't want to take the time to write a big project in Arc right now, when future releases of Arc will be incompatible with my code in possibly entirely unforeseeable ways.

A module system would be a big plus, too, though I (and many other people) have successfully managed rather large C projects sans a real module system.

-----

2 points by jdvolz 5927 days ago | link

I would think that Arc is not yet stable enough to use for large projects. I think that the missing module system is the major concern. I am not so concerned about unstructured data because it can be mitigated with access methods and because if you are properly creating your DSL it should be pretty obvious what is going on. Also, how big of a project would you have to have before you needed enough Arc to make it confusing? The language spec is < 5,000 lines of code.

-----

1 point by carbon 5926 days ago | link

Ok, I can see that access methods is a solution of the problem of self-describeness of a program. Could you tell me if I am right doing it like this:

(get-title book)

(set-title book "New title")

(Book (title "Blabla") (authors ("Tom" "John")) (year 2008) )

Is it the way an expirienced Lisp programmer would do it?

-----