Arc Forumnew | comments | leaders | submitlogin
Does Arc need modules? Maybe not.
9 points by cchooper 6117 days ago | 31 comments
What benefits do modules give us? I can think of three:

1. They hide internal definitions that the user shouldn't see.

2. They prevent the namespace from getting polluted.

3. They allow code written by different users to be integrated without names clashing.

1 can be achieved by using a with form to hold all your private stuff. 2 can be achieved by prefixing your variables (and you can use a local non-prefixed version in your with form to make this more convenient). 3 is a bit more tricky, but will work if you can supply a prefix dynamically. A few macros will make all this nice and convenient. Is anything else required?

Maybe I'm being a little optimistic about how well this will work, but I'm sure there'll be some way of doing this correctly. I think it's more Arc-y to address the problems that modules solve directly, rather than importing an onion from someone else's varnish.



4 points by shiro 6117 days ago | link

Yes, you can do all those stuff in existing primitives. But maybe a little convenience macros will help, like auto-prefixing macros. And another thin macro that expands to 'with', with some bookkeeping features such as specifying the prefix in one place? Then let's call that macro 'module' and voila! You have modules.

Really, it is possible to build module system if you have enough primitives (what's 'enough' depends on what model you use). The point of modules is not whether it is possible or not. The point is that whether everybody agrees on a certain convention or not.

-----

3 points by cchooper 6117 days ago | link

I originally considered writing a macro that did just that, and even called it module, but then I realised I was packing a load of disparate features into one macro just so it would look like modules in other languages.

I don't think the point is conventions (at least not yet). I think the point is that modules may be an onion, and we don't want any onions.

-----

3 points by shiro 6117 days ago | link

If the module system is an onion, it is an onion with a kernel, which is really a convention.

For example, see SLIB, the portable Scheme library http://www-swiss.ai.mit.edu/~jaffer/SLIB.html . It is designed to work with R5RS Scheme and before, thus it can't use any module systems. Instead, it uses a naming convention, which works pretty well. The problem is that, to make it work, all programmer should agree and follow the convention. The so-called module systems make it easier to follow the rules by making the existence of the convention invisible (automatically adding prefixes effectively, e.g.) or preventing programmers from doing things differently.

In the discussion of R6RS Scheme, where a module system is finally adopted, some people argued that you can roll your own modules system out of given primitives, and it had proven in the production code. But if you use your own module system (or convention) and I use my own, it will be difficult to mix two libraries. Whatever it is, we have to agree on one way. I think that's really the point.

-----

1 point by cchooper 6117 days ago | link

I agree that module systems are conventions. What I don't agree with is that we should uncritically adopt them in Arc. There may be a better way. What that is, I don't know.

-----

4 points by shiro 6117 days ago | link

Then the important question is this: When you say better, better in what sense? Beyond a certain point, people have different criteria and your better and my better start to diverge.

In Scheme, traditionally such matter has been settled by "agreed to disagree"---instead of choosing one, the spec just leave it unspecified. But R6RS had a goal to allow portable libraries, so it had to choose one; for modules, it is better to have lesser but one module system, rather than having multiple ("better" in various regards) or than having none.

In Arc, ultimately it's up to pg.

[Edit: To be more PC, R6RS module is 'less agreed'. I do understand it is better in its design criteria.]

-----

1 point by cchooper 6117 days ago | link

Good question. I'm sort of inspired by PG's thoughts about object-oriented programming, that it would be better if they offered the features a la carte rather that packaging them up into objects.

-----

4 points by lojic 6115 days ago | link

This may be related to #2, but I find modules/classes, very helpful in finding functionality. Someone just asked on IRC if there is a function to determine if a file exists. In Ruby, I would expect to find such a function in the File class, and sure enough, there is a File.exist? function.

This can be done in other ways such as partitioning functionality into files (if the code is the spec/doc), or organizing documentation in certain ways, but it's been my experience that a module/class system makes finding functions easier.

-----

5 points by pau 6117 days ago | link

At some point the amount of code in arc-wiki.git will cross a threshold, and a solution will be needed. If we wait until then, the way to solve it might be more obvious.

I also wanted to mention a fourth benefit that I think modules can offer, which is:

4. They allow some form of management of dependencies between pieces of code.

This is optional, but I would integrate it in the discussion about what problems modules solve...

-----

1 point by cchooper 6117 days ago | link

A list of such problems would be another good starting point.

-----

2 points by almkglor 6117 days ago | link

A (with ...) form is just a module by another name. Heck, ECMAscript "modules" are done exactly like Arc "modules" would be made:

  MyStuff = (function(){
    var modulevar;
    function innerfunction(){...}
    function exportedfn(){...}
    return {exportedfn: exportedfn}})();
which is practically the same code as:

  (= MyModule
    (let (modulevar innerfunction exportedfn) nil
         (def innerfunction () ...)
         (def exportedfn () ...)
         (fill-table (table) 'exportedfn exportedfn)))
However, suppose we instead had a macro, named 'module, which supported the following syntax instead:

  (module MyModule
    (module-var modulevar)
    (public exportedfn)
    (def innerfunction () ...)
    (def exportedfn () ...))
The above strikes me as being shorter and having much fewer parentheses, as well as eliminating quite a bit of boilerplate.

-----

4 points by drcode 6117 days ago | link

I agree with your sentiment. I also agree with pgs sentiment that modules are a cognitive barrier when dealing with code... that doesn't mean it can do without one, but I think it needs to be thought through carefully...

-----

2 points by pau 6117 days ago | link

> pgs sentiment that modules are a cognitive barrier

Can you give pointers to PGs essays or books about this?

-----

1 point by drcode 6117 days ago | link

I think it's in the ansi common lisp book- unfortunately, my copy became too dog eared (the paperback is unfortunately a bit flimsy) and I threw it out, otherwise I'd let you know for sure. I think he was complaining about the CL module system being to hard to understand...

-----

6 points by pau 6117 days ago | link

Found it, I think!

"The kind of modularity provided by packages is actually a bit odd. We have modules not of objects, but of names. Every package that uses 'common-lisp' has access to the name 'cons', because 'common-lisp' includes a function with that name. But in consequence a variable called 'cons' would also be visible in every package that used 'common-lisp'. If packages are confusing, this is the main reason why; they're not based on objects, but on their names".

You know, I now remember having read this... ;)

-----

5 points by cchooper 6117 days ago | link

I like the interesting footnote, very relevant to this discussion:

> So perhaps packages will turn out to be a reasonable way of providing modularity. It is prima facie evidence on their side that they resemble the techniques that programmers naturally use in the absence of a formal module system.

-----

5 points by Jesin 6117 days ago | link

He seems not to be saying we shouldn't have modules, but instead that we shouldn't use this kind of module.

-----

5 points by CatDancer 6117 days ago | link

Can you go into detail a bit more on #3? How would I supply a dynamic prefix?

-----

1 point by cchooper 6117 days ago | link

Something like this would be a starting point:

  (mac def-with-prefix (prefix name args . body)
    `(eval (list 'def
            (read (+ (coerce ,prefix 'string)
                     "-"
                     (coerce ',name 'string)))
            ,args
            ,@body)))
called like this:

  (= prefix 'blub)
  (def-with-prefix prefix foo (x) (+ 1 x))
produces this:

  (def blub-foo (x) (+ 1 x)))
Note this is quite clunky because I couldn't find a better way to bind a dynamically determined variable in Arc. Set only takes a symbol as a first argument.

-----

1 point by cooldude127 6117 days ago | link

what is the point of the eval thing? isn't that what macros are supposed to do by themselves?

-----

2 points by cchooper 6117 days ago | link

You mean like this?

  (mac def-with-prefix (prefix name args . body)
    `(def
        (read (+ (coerce ,prefix 'string)
                 "-"
                 (coerce ',name 'string)))
        ,args
        ,@body))
It doesn't work, because

  (def-with-prefix 'blub 'foo (x) (+ 1 x))
expands to

  (def (read (+ ...blah...blah...)) (x) (+ 1 x))
and def will only take a literal symbol as the first argument. That's also the problem with set I mentioned before.

-----

2 points by shiro 6117 days ago | link

How about

    `(def ,(read (+ (coerce prefix 'string) "-" (coerce name 'string))) ,args ,@body)

-----

1 point by cchooper 6117 days ago | link

Doesn't work I'm afraid:

  (= pre 'blub)

  (your-def-with-prefix pre foo (x) (+ 1 x))
  => #<procedure: pre-foo>

  (my-def-with-prefix pre foo (x) (+ 1 x))
  => #<procedure: blub-foo>
Without being able to take the prefix from the value of a variable, the whole macro is pointless.

-----

1 point by eds 6117 days ago | link

Then why don't you just eval prefix?

  `(def ,(read (+ (coerce (eval prefix) 'string) "-" (coerce name 'string))) ,args ,@body)
It's still much less clunky than expanding the whole def under an eval.

-----

1 point by cchooper 6117 days ago | link

Still has a bug:

  (let pre2 'blub (your-def-with-prefix pre2 foo (x) (+ 1 x)))
  => Error: "reference to undefined identifier: _pre2"

-----

2 points by cooldude127 6117 days ago | link

that's because the macro gets expanded (and thus eval is called on pre2) before the let form is evaluated.

-----

1 point by eds 6117 days ago | link

Yeah, I see the problem with that.

cchooper, did you actually ever run your version? I can't make your orginal version work.

  arc> (def-with-prefix pre foo (x) (+ 1 x))
  Error: "reference to undefined identifier: _x"
And looking at the macro expansion, I'm not at all surprised:

  arc> (macex '(def-with-prefix pre foo (x) (+ 1 x)))
  (eval (list (quote def) ; my indentation
              (read (+ (coerce pre (quote string)) "-" (coerce (quote foo) (quote string))))
              (x) (+ 1 x)))
args and body would need to be quoted for this to work properly. And then it admittedly does what it is supposed to, but it becomes even uglier.

  (mac def-with-prefix (prefix name args . body)
    `(eval (list 'def
            (read (+ (coerce ,prefix 'string)
                     "-"
                     (coerce ',name 'string)))
            ',args
            ,@(map [list 'quote _] body))))

-----

1 point by cchooper 6116 days ago | link

Hmmm... it worked last night. I must have made a mistake when I copied it into the comment. Thanks for catching it.

Here's a slightly less clunky version:

  (mac def-with-prefix (prefix name args . body)
    `(eval (join (list 'def 
                  (sym (+ (coerce ,prefix 'string)
                          "-"
                          (coerce ',name 'string)))
                  ',args)
            ',body)))
I also tried doing it with nested quasiquotes but that just looked worse.

-----

1 point by shiro 6117 days ago | link

Ah, I got it.

-----

1 point by cooldude127 6117 days ago | link

yes, like that. there is no reason i can see for a macro to have to call eval except in the MOST EXTREME cases.

-----

2 points by cchooper 6117 days ago | link

Unfortunately, Arc makes this an extreme case. There's just no way to get the exact required behaviour in Arc without using eval.

In Arc:

  arc> (= foo 's)
  s
  arc> (set foo 4)
  4
  arc> foo
  4
  arc> s
  Error: "reference to undefined identifier: _s"
In Common Lisp:

  CL-USER> (setf foo 's)
  S
  CL-USER> (set foo 4)
  4
  CL-USER> foo
  S
  CL-USER> s
  4

-----

1 point by cchooper 6116 days ago | link

I've posted a follow-up to this article here:

http://arclanguage.org/item?id=3315

It expresses what I wanted to say rather better.

-----