1. Silicon Valley.

2. I'm specifically interested in making Arc (or Lisp in general) a good option for doing everything that you'll ever need to do. I think making games is a good thing to aim for: it's worth doing in itself (I like games, and there's one old no-longer-being-developed game I'd really like to implement myself), and it implies handling graphics, audio, and keyboard/mouse input, plus respectable execution speed. Seems either one could focus on the Racket libraries (I've found them pretty good but somewhat buggy and slow), learn to use the Racket FFI, or use Arc as a skin over a completely new platform or language. I'd be pleased to hear of progress in these domains. Any other topics are good too, though.

3. I could talk about various things I've done in Arc, and the ad hoc libraries I've constructed in the process. In the category of math, I'll quickly mention primality testing, modular exponentiation and inversion, prime factorization, polynomial arithmetic and interpolation, multiple-variable polynomial arithmetic, and integer nth roots of arbitrarily large real and complex integers.

1. Create a special form that rebinds 'assign (which is the assignment operator that '= and pretty much everything else expand into--except not things like 'scar or the equivalent (= (car x) ...)), so that instead of affecting global variables, all '= and 'def forms evaluated within this special form will modify values within a hash table. This special form can return the hash table. You'd put (load "file.arc") inside that special form.

This will probably have to be done by the Arc compiler (or something that duplicates its functionality), because global references to preexisting variables will need to be compiled normally, while global references to variables within the module will need to be compiled into references to the hash-table (or perhaps to variables in a local environment that spans the whole module), and references to local variables will need to be compiled normally. I think this will be somewhat hard to implement, but if done, then it could do exactly what hasenj asks for.

2. A hacky idea that seems it'll work... very well, in fact. It has several caveats, which I'll get into later. Here are the pieces of the idea.

A. Scheme has a built-in (namespace-mapped-symbols) function. It returns a list of all variables that have values in the current namespace.

B. We can use this to extract a list of all Arc variables.

  (def arc-namespace ()
    (map [sym:cut string._ 1]
         (keep [is string._.0 #\_]
               ($.namespace-mapped-symbols))))

Details. First, note that 'eval doesn't respect local variables, and I think it kinda has to be that way, so the way we should load the file is like this:

  (eval `(with (newvar nil newvar2 nil modvar nil ...)
            ,@(readfile "stuff.arc")
            (let gtb (table)
              (= gtb!newvar newvar
                 gtb!newvar2 newvar2
                 ...)
              gtb)))

Less importantly, note: a) I don't think Arc has something like (let u 'ach (setf (symbol-value u) desired-value)). This is easily patchable with something like (let u 'ach (($ namespace-set-variable-value!) ($.ac-global-name u) desired-value)). You can also do (eval `(= ,u ',desired-value)) but that sucks in a few ways and it really should be put into the language. (Note that $.[name-containing-bang!] breaks due to ssyntax, so you need to put it in parentheses form.)

b) Currently, hash tables can't contain nil; with table tb, (= tb!x nil) deletes the value associated with x. This is worked around in the definition of 'defmemo by creating a "nilcache" table. Not sure if that's the best way to handle it. For the moment, I'd probably use an assoc-list--and note that 'alref doesn't allow an optional "fail-value" argument, so I'd use 'assoc and deal with it.

c) I don't think Arc has a way to undefine variables. Again, this is easily patchable: (($ namespace-undefine-variable!) ($.ac-global-name var)).

...And here is an implementation, and it in action.

http://pastebin.com/qJRepGcM

(I modified it from the above to respect macro definitions within the file. Interestingly, this leads me to load a file three times: once to extract changed variables, once to get macros, once to evaluate all definitions. I've heard of "load, compile, and run". Maybe this is like that.)

We see that creating the module in a local environment containing all the variables makes each one independent. You can just extract 'ppr, and its lexical environment will contain all its dependencies; you can throw away the rest of the hash table, and your namespace is totally unpolluted; additionally, there is no cost of doing hash table lookups (in fact, they're lexical-variable lookups) or of figuring out how to make the compiler optimize them out.

Also, take note of this: Suppose a module contains a huge amount of stuff and all you want is one little function that depends on just a couple of things. If we were, say, going to save the Arc process into a self-contained executable and distribute it, we wouldn't want to stick it with 40 MB of mostly unnecessary libraries (I'm looking at you, SBCL). Well, guess what happens if you use this local-environment method and extract only the function you want: All the unnecessary crap will have nothing referencing it and will get garbage-collected. This is so good. I believe it is the perfect solution. (Returning a hash-table and extracting the desired functions and throwing away the table also has this effect.)

Caveats. First, there are fundamental issues with this design. All toplevel expressions will be evaluated more than once (three times currently; two is easily possible, but not one without compiler power); if you want the module to do any multiple-evaluation-sensitive initialization, put that inside a function named 'initialize or similar, and the person loading the module should call (initialize) afterwards. Also, your module shouldn't generate new names when loaded multiple times. And it shouldn't contain any functions (including 'initialize) that do global assignment to variables that don't exist yet. So if you want it to do (= counter* 0) within a function, you should do a global (= counter* 'nonce). Finally, don't even think about using macros from a module (though the definitions in the module can themselves contain macros). Macros will be put into the table, and you can extract their functions with 'rep and put them into your own macros, but if their expansions are supposed to reference things from the module, that seems impossible without at least code-walking power.

Second, there are some issues that can be fixed with some surface modifications of the design. If you want to load a module and then modify anything inside it (global functions or variables), that currently will have no effect unless the module provides specialized functions to do so (the functions depend on what's in the lexical environment of the module, which you can't touch by modifying the hash table). However, it is trivial to make 'load-module insert 'get-var and 'set-var functions into the module that will get and set the lexical variables. A user of the module will have to use (ns!set-var 'x val) instead of (= ns!x val).

Issues with my implementation: It uses 'is to determine what variables were changed. If, say, your code goes (= thing* nil) and the module goes (= thing* nil), my implementation won't notice the difference and won't localize it. Likewise for (= counter* 0) and (= name* "Bob"). This could be worked around by, say, using something like "defvar" to define global variables (not to be confused with the defvar from Common Lisp, which creates special variables with dynamic scope), where 'defvar would record everything it does in a table that 'affected-vars could access. (For helpfulness, 'load-module could give warnings when attempting to load a file containing toplevel '= things.) Or: We could modify the body of '= to contain a clause like

  (if we-are-loading-a-module*
      `(do (= (things-changed* ',var) t)
           ,(proceed-as-usual))
      (proceed-as-usual))

Also, btw, it doesn't reset the 'sig function signature table. Easily fixable.

With all these issues in mind, here is my evaluation of my implementation:

1. If your module just defines functions (possibly using macros to define them) and possibly hash tables, then it is flawless and requires no boilerplate.

2. If your module creates global variables with initial values that 'is could think are identical to those already created (integers, nil, strings, symbols), there's a small chance that bad things will happen (if your code and the module's code happen to initialize the same global variable to the same value). This could be fixed in a few different ways.

3. If you want to load a module and then modify and use its global variables, then, with a fix I have described above, you can do that with a little bit of boilerplate.

4. If your module does weird stuff like modifying different variables the second time it's loaded, or performing structural modifications (= (car ...) ...) on existing global variables or the universe (writing to files), this will die badly, although in non-weird cases it could be fixed with the minor boilerplate of putting stuff in an 'initialize function.

5. If you want to export macros... bad. Sorry. Note, however, that a certain class of macros can be covered by creating functions that accept thunks, and then the person using the module can write a macro on top of it. (See 'on-err as an example; imported from Scheme.)

  arc> (eval `(+ 1 2))
  3
  arc> (eval `(,+ 1 2))
  Error: "Bad object in expression #<procedure:+>"
  arc> (eval `(',+ 1 2))
  3
  arc> (eval `((obj a 1) 'a))
  1
  arc> (eval `(,(obj a 1) 'a))
  Error: "Bad object in expression #hash((a . 1))"
  arc> (eval `(',(obj a 1) 'a))
  1

  ; compromises in this implementation: 
  ; no objs in code
  --arc.arc

It works in Jarc without quoting.

  Jarc> (eval `(,+ 1 2))
  3
  Jarc> (eval `(,(obj a 1) 'a))
  1

  export PATH=/Applications/Racket\ v5.0.2/bin:$PATH

As aw says, don't use the "-m" option. Also, use the "-i" option, for "interactive"; without it, when you ctrl-C an Arc session, the whole process will abort, but with it, you'll get a MzScheme prompt, at which point you can use (tl) to return to Arc.

I also highly recommend that you install a tool called rlwrap. It does paren-matching and lets you use arrow keys (or ctrl-N, ctrl-P, ctrl-F, ctrl-B for down, up, right, left) to move around and access previous commands. Here's the command that I suggest you use to start Arc from the arc3.1 folder:

  rlwrap -c -r -l ~/Documents/ARC_LOG -q "\"" -C arc mzscheme -i -f as.scm

For extra credit, add the following to the file ~/.bash_profile:

  alias arc='cd [path-to-arc]/arc3.1; rlwrap -c -r -l ~/Documents/ARC_LOG -q "\"" -C arc mzscheme -i -f as.scm'

  $ racket -il readline -f as.scm
  Welcome to Racket v5.0.2.
  Use (quit) to quit, (tl) to return here after an interrupt.
  arc> 
  (+ 1 2)
  3
  arc> 
  "bad"
  "bad"
  arc> 
  (quit)

But the paren matching is definitely possible -- I have "set blink-matching-paren on" in my ~/.inputrc file, and never had any problems with it.

  arc> (sort < '(2 3 1))
  Error: "reference to undefined identifier: _copylist"

  * (coerce () 'cons) ;SBCL

  debugger invoked on a SIMPLE-TYPE-ERROR:
    The length requested (0) does not match the type restriction in CONS.

First, here's a list of my recommendations: (is nil 'nil) -> t [that might be hard; have to compile (quote nil) into ()], (type nil) -> 'sym [easy], make nil get printed as "nil" [that might be hard, though I wouldn't demand perfection immediately], and [as mentioned above] (coerce nil 'cons) -> error [easy]. In other words, make 'nil operate exactly as it does in official Arc.

I will now talk in great detail about the semantics, implementation, and usage of nil.

nil is defined to be the empty list. In Scheme, that's all nil is good for (they call it null, actually, and write it as ()). I'm going to reject this system with some brief comments about why it sucks: it's conceptually wasteful (you have to invent a whole new category of object, instead of using a symbol), and it makes things horribly and unnecessarily verbose. I imagine most users of Arc feel the same way. Related humor: http://pastebin.com/raw.php?i=TQNh668k

In Common Lisp and Arc, nil is represented as the symbol 'nil. But here's an idea: why not make nil be a cons cell whose car and cdr point to itself? That seems like a very elegant way to terminate lists; then you don't need to invent anything new or give special status to a certain symbol. Also, it makes (cdr nil) = nil, which I find nice and useful. I think there is historical precedent for this--if nothing else, it's how I implemented linked lists in C recently.

There's a problem, though. Cons cells are used for things other than flat lists, like trees or argument lists. Given the other uses of nil, as boolean false and as the symbol 'nil (which I like very much and take as a given), you are likely to have nil as an element of one of these structures. But if you're treating nil as an element of a structure built of cons cells, then you really don't want (acons nil) to be true.

In Common Lisp and Arc, (cdr nil) = nil, and (acons nil) [or (consp nil) in CL] is false. This suggests that nil is some sort of trick object that acts like a narcissistic cons cell when you ask for its cdr, but an atom (specifically, a symbol) when you ask for its type. This brings to mind two questions: 1) What is nil, and 2) Do we want a trick object like that in our language?

To answer (1), it really doesn't matter what nil is, but rather what interface the programmer sees. The programmer sees that nil prints to "nil", it is treated as false in conditionals, it terminates lists, (car nil) = (cdr nil) = nil, (acons nil) is false, it is equal (by the "is" comparison) to the symbol 'nil, and (type nil) -> 'sym. (I think that's everything.) If all these things are true, then nil could be internally represented as the string "ACHTUNG"--or, in your case, as the Scheme empty list ()--and the programmer wouldn't know it and wouldn't care.

I answer (2) with a definite yes. Here's a train of thought: When you deal with trees, nil is the symbol 'nil and isn't a cons. When you deal with lists, nil really should be a self-directed cons cell. The cdr operation is implemented so that it special-cases, saying "If this is nil, then return nil, otherwise extract the cdr of this cons cell and follow the pointer".

But when your program is compiled into perfect machine code (by type inference or help from the programmer), in the list case, it will represent nil as the self-referencing cons cell, and it will generate a raw unchecked cdr that just says "Extract the cdr of this cons cell and follow the pointer". In the tree case, it will represent nil as a symbol, and it will still generate a raw cdr, because the cdr will only be used on cons cells; and the function 'acons won't need to special-case "Check if this 'cons cell' is nil, and if it is, return false anyway".

nil is an abstraction over a couple of different internal representations of things that English speakers could call "nil", in the same way that Lisp integers are abstractions over the distinct "fixnum" and "bignum" representations of mathematical integers. It's little more of a trick for cdr to dispatch on nil when it might be represented as a symbol, than for + and * to dispatch on numbers which might or might not be (and whose sum and product might or might not be) fixnums.

So nil isn't a trick object: when you're working with lists, it is the solid, reliable terminator of lists, and in other contexts, it is the solid, reliable symbol 'nil. I recently wrote some code to handle polynomials, and nil was the same as the polynomial 0. I also played with streams a long while back, and I used nil as the empty stream.

This model of nil breaks down only when you try to deal simultaneously with two or more categories of object, which would each have their own kind of nil. More specifically, when you perform an operation that could return either of two or more kinds of nil. For example, (read) may return nil because it read the symbol 'nil or because it encountered an EOF character. These cases are relatively rare, and they're not hard to deal with when they do come up. And when you do deal with them, you probably write the same code you would have written had you been forced to do it from the start.

The read functions generally carry an optional "eof" argument, which you can bind to whatever sentinel value you like (such as a gensym, or the fail* global variable). Hash table lookups carry an optional "if not found" argument. I think these things model a good way to deal with nil: provide it as the default, with an easy way to make it more specific.

(At the moment, hash tables can't hold the value nil anyway, as (= ht.x nil) wipes the entry for the key x in the table ht. The optional thing is probably more for convenience, as in (++ (count-tb x 0)).)

Hmm, it sounds like that's what you had before waterhouse's input. :-p

In Lathe's generic/inheritance utilities (orc/orc.arc and orc/oiter.arc), I base the inheritance graph on a new type function, 'otype, mainly so that I can have a different type for nil. Because of the inheritance system, I also get to have a list type that both the nil type and the cons type inherit from, so that I can extend every list case in a single branch.

You might just implement it for a couple of built-in functions to make them nicer (e.g. say that + or > called with one argument should probably curry rather than returning immediately) and not make it a general feature of the language, but I think that would be bad. For one thing, it wouldn't be nicer if your code contained, say, (apply + xs) when xs will have anywhere from 0-10 elements: nasty casework even if you are aware of this "feature".

Or you might say, "Well, we can certainly say that if a function f requires at least two arguments, then, when f is given only one argument, it should be curried." I think that's a bad idea too. It seems like a pretty rare case in the context of currying, and it makes things more complicated--now you have to figure out how many arguments f requires, instead of how many you intend to supply.

If you want implicit currying, I think you have to make it happen everywhere in the language, which means throwing out optional and rest parameters. (Interestingly, it does seem like it would work fine with keyword arguments.) Take a look at Haskell.

I prefer the way things are currently done. I don't find it a problem to write [> _ 4] instead of (<= 4) or something. I am annoyed whenever I have to write (fn args (apply f x args)), but if that becomes a problem, I'll write a partial macro to express it as (partial f x). ...Actually, it can just be a function.

  (def partial (f x)
    (fn args (apply f x args)))

  arc> (map (partial + 2) '(1 2 3 4))
  (3 4 5 6)
  arc> (map (partial (partial + 2) 3) '(1 2 3 4))
  (6 7 8 9)

Very interesting observation! XD It's inspiring....

Well, CSS has the slickest keyword argument API I've ever seen. The declaration "border: 2px solid black;" conveys the same information as a whole lot of other declarations like "border-top-width: 2px;", "border-bottom-style: solid;", etc. What if the "border" keyword were definable as a library?

  ; Haskellish syntax with liberties taken
  
  border width style color f = f -: border-width width
                                 -: border-style style
                                 -: border-color color
  
  border-width width f = f -: border-top-width width
                           -: border-bottom-width width
                           -: border-left-width width
                           -: border-right-width width
  
  Type Border-Top-Width  ; singleton type
  
  border-top-width value f = f -: keyword Border-Top-Width value

The "keyword" function is axiomatic to the language. It takes a key value, a value to associate with it, and a function, and it returns the result of currying that function with that keyword information, overwriting the previous association for that key if necessary.

As for defining a function that does something with the keyword arguments it gets this way... I'm out of steam for now. What do you think?

  (fn (a b 'opt (c 1) d e) ...) --> c,d,e optional args
  (fn (a b (quote opt) (c 1) d e) ...) --> same thing
  (fn ((a b)) ...) --> destructuring

  (a b 'opt c 1 d nil e)

  (a b 'opt c.1 d e)

  (a b ? c 1 d nil e nil)

...egad. It turns out that it is not just legal, but it works fine, to use "quote" as a variable name just like any other, in Arc and Common Lisp and Racket.

  > (let ((quote 2)) quote) ;Racket
  2
  > (define quote 10)
  > quote
  10

  > (define quote 10)
  > '(1 2 3)
  procedure application: expected procedure, given: 1; arguments were: 2 3

Well, what do you think of forbidding people to rebind quote (locally or otherwise)? I think it's acceptable. quote is a fundamental part of Lisp. If it is rebound, then either that will screw up quoted things, or the Lisp parser will handle (quote blah) forms specially, in which case rebinding quote to a function and attempting to call it will fail (you'll just quote the arguments). In other words, either this will fail to return 12:

  arc> (let quote [+ _ 2] (quote 10))
  12

  arc> (let quote [+ _ 2] '(1 2 3))
  Error: "application: bad syntax in: (1 2 3 . nil)"

So, if using "quote" as a parameter is officially unsupported, then this officially makes room for "quote" to be used as a special marker in parameter lists. When a program that parses parameter lists encounters (quote blah), it should stop and say "Aha, this is not a legal parameter. What now?" And at this point we can give it whatever desired features in a nice, modular way.

In official Arc, we would have, say, "If blah is 'opt or 'o or 'optional, then proceed to interpret optional arguments." Then akkartik might add, "If blah is 'key, then proceed to interpret keyword arguments", and aw might add "If blah is 'as, then proceed to interpret coerced arguments", and these would be totally compatible extensions to Arc, as long as they didn't choose the same name.

I do think this is the way to go.

[1]A "Lisp-2 function/variable namespace separation" buff might say at this point, "Aha! See, with the namespace separation, this isn't a problem; you can use quote as a variable all you like and it creates no problems." Retort: "I might just as well want to locally create a quote function with flet, and then you have a problem. (And if your language doesn't let you locally bind functions, then it sucks.)" Example case:

  (flet ((quote (x)
           (format t "~S~%That's what she said!" x)))
    ...)

  (each name '(html title head body div span p h1 h2 h3 h4 i em strong b)
     (eval `(def-tag ,name)))

  (= tag-list '(html title head body))
  (apply-mac-to-list def-tag tag-list)

  (mac apply-mac (mac-name args)
      (prn:string "applying " mac-name " to " args)
      `(each arg ,args
          (eval `(,',mac-name ,arg))))

I got there by replacing 'eval' with 'prn', and then I kept playing with commas and quotes until it printed the right thing, then I put eval back in.

So it seems to work but I have no idea why.

EDIT:

I'm not sure if that sort of thing is common in lisp/arc? Am I approaching macros in a totally wrong way?

I presume I'm thinking in python, where a lot of meta programming techniques constitute of taking strings and using them to get fields that would normally be accessed using the symbol directly.

  getattr(obj, 'field') # same as obj.field

  def apply_to_fields(obj, field_name_list, fn):
    for field in field_name_list:
        new_value = fn(getattr(obj, field))
        setattr(obj, field, new_value)

  http://www.bookshelf.jp/texi/onlisp/onlisp_17.html#SEC109
    The full book is there, but if you want it as a pdf or
  something, see Paul Graham's web site:
  http://paulgraham.com/onlisptext.html

  arc> (macex1 '(let x 1 (+ x 2)))
  (with (x 1) (+ x 2))
  arc> (macex1 that)
  ((fn (x) (+ x 2)) 1)

  (def apply-mac (mac-name args)
    (each arg args
      (eval `(,mac-name ,arg))))
  ;Usage
  (= tag-list '(html title head body))
  (apply-mac 'def-tag tag-list)

     `(each arg ,args
         (eval `(,',mac-name ,arg))))
  ;Rewrite as:
     (w/uniq garg
        `(each ,garg ,args
            (eval `(,',mac-name ,,garg))))

Regarding general practices in Lisp (you ask about how you're approaching macros)... I can say a few things. First, know that eval is slow (it performs full syntactic analysis), and you most likely should not be using it at runtime. Second, using eval is almost always unnecessary; you could use a macro or something else, and if you can't see how, then that's probably due to insufficient macro-fu. Exceptions are when the code you want to generate and evaluate actually depends on stuff that happens at runtime (e.g. your program reads Arc expressions, evaluates them, and prints the results).

In this case, this code just sets up a bunch of definitions; it's run once and it takes an imperceptible amount of time anyway. Performance isn't an issue. I would probably write it as the one-liner "(each x tag-list (eval `(def-tag ,x)))" when I was sitting at the REPL just wanting to get code working; then I'd probably write up the 'def-tags macro in my actual code file. Here's my preferred version (paraphrasing aw):

  (mac def-tags tags
    `(do ,@(map (fn (name) `(def-tag ,name)) tags)))

Finally, you ask whether this sort of thing is common. The thing you said before that was "It seems to work but I have no idea why", so I'll assume that's what "that sort of thing" refers to. My answer: Generally, no. Sometimes I'll get something working by brute force search, as you seem to have done; and sometimes I'll get something working by pasting it in from someone else's code; but in either case, I will think about it and verify that it is correct and should work, and usually I'll come away satisfied in my understanding.

Note:

  (let name 'BOB
     `(meh `(ach ,',name)))
  -->
  (meh `(ach ,'BOB))
  -->
  (meh (ach BOB))

> I see you've encountered the strange ,',name idiom. I congratulate you for writing it yourself

cool :D

  ,',X

See http://repository.readscheme.org/ftp/papers/pepm99/bawden.pd... for a description: