> There's something to be said for minimalism like that. Not only does it make the initial development easier, but I imagine it's easier to do mashups and derivative works too.
If I may kick off a tangent, this is the part of "Worse is better" that tends to be forgotten/deemphasized in Pitman's formulation. C and Unix succeeded because they focused on keeping the implementation simple and accessible for many years. (They eventually forgot that lesson, of course, and have been coasting on the initial momentum for a very long time.)
Indeed. And Richard actually makes that point, that the "initial virus" has to be good and simple, and that having won it will have much more pressure to improve until it gets to 90% of "good". Unfortunately, in the process it conditions users to accept worse, and the patching process probably doesn't result in a simple end result.
In fact, reading the story about the "PC loser-ing problem", I realized that I was so conditioned by the Unix solution that I had never even _considered_ the former as a possibility. I do sometimes wonder how many amazingly good ideas we've lost, that would now actually be much simpler than the stack we have, but we're just used to it.
I think the concept could be better generalized by rephrasing it as "cheaper is better" though. Technically it's not "worse", it just has a different set of values. Obviously, users value it more, or they wouldn't adopt it.
I see it as closely related to ideas like "compatibility is key", "customer is king", and "money is power", each of which builds on the following.
Customers adopt products that have the best cost-benefit ratio. It doesn't matter if the fancy "good" solution is 10% better (from 90% to 100%) if it also costs 2x as much. Maintenance of the ideal solution may actually be cheaper, but it's really hard to estimate maintenance in advance, especially in design fields like software development.
Once the "cheap" solution is adopted, future adoption and upgrades are even cheaper compared to switching to the "good" solution, because the user is already invested, and has built a network of integrations that would be very hard to replicate.
The network effect and basic epidemiology probably provide good explanations for the rapid victory of "cheap" solutions—they spread faster because they are easier to "get", and that amplifies the infection rate to new nodes. Anyone can understand why to adopt something cheap. It takes a lot of effort to learn and understand the technical advantages of a superior system. Given the work involved in properly evaluating competing options to discover technically superior solutions, I think it's safe to assume that the percentage of potential customers that just pick the cheapest one that works, or that is already adopted by the largest number of other users, will always be higher than those who actually compare all the options to pick a better solution.
So "worse" solutions actually are "better", because they're cheaper to adopt. This is especially visible when you look at history and see how many times the systems focusing on backwards compatibility won out over those that merely tried to be "new." Compatibility reduces the cost of adoption. It's that simple.
Does that mean that we're doomed to a "race to the bottom"? I don't think so. In fact, I think with some care new solutions can be designed that are sufficiently better/faster/cheaper that they do disrupt the existing ecosystem. It happens all the time. We see Facebook beating Myspace, all the various chat programs killing XMPP, Slack starting to eat IRC, etc. Most of those did it by making adoption easier for new users. The secret is that a new system doesn't have to replace the existing system, just be easy to adopt. Lots of people use multiple chat programs at the same time. The Lean Startup book was written by an entrepreneur working on a chat system, who initially thought that to make adoption easy he had to integrate with existing systems. What they learned was that people didn't mind adding it to their list of chat systems, and actually liked the ability to meet new networks of friends.
I've been very intrigued recently by a lot of early internet protocols, like IRC, SMTP, NNTP, etc. which are very clean and simple. So easy to use that you can literally connect to an SMTP server via telnet and send an email by hand with just a few simple text commands. I've seen people mention gopher a few times recently (the core doesn't change very fast, but people like to implement custom clients), and even HTTP is pretty simple. I think there's a lot to be said for simple, text-based protocols, because they're easy to understand and implement something that connects to them. I almost think a good test for how complicated an interface is, is how easy it would be to implement in arc, which has very little library support for most of these things. It turns out to be quite easy to build an IRC bot with arc.
It is interesting to me that arc may not be very widely adopted, but it is probably one of the few programming languages that has almost as many implementations as it has community members. If we made it just a little bit easier to pick up and start using (particularly in production), the community would probably grow a little faster.
I think there's a lot of opportunity now and in the near future for reintroducing simple foundations, perhaps slightly extended, but mostly made more accessible for new users. Our technology stack has gotten so tall and complicated in the name of shortcuts and simplicity, that a lot of efficiency can be gained by cutting out a few layers. Once people start targeting certain abstraction boundaries, like WASM + WASI, it should be pretty easy to replace everything under that boundary with a much simpler system. A lot of the disadvantages of "good" systems, like microkernels vs monolithic ones, are now so completely outweighed by the rest of the environment that it should be pretty straightforward to build an OS with much better security much closer to the metal than what we have now with 2+ layers of VM sandboxing.
But you should elaborate on your last 2 paragraphs. I'm not sure I buy either that Arc adoption can pick up or that the mainstream tech stack will ever cut out layers.
My synthesis of "Worse is better" for myself (with Mu and SubX):
a) I don't think of evolution as "bad". Building something incompatible is indeed maladaptive. I'm clear-eyed about that.
b) Mu doesn't try to come up with the perfect architecture that doesn't need to evolve. Instead it tries to identify and eliminate every source of friction for future rewrites.
c) My goal isn't to go mainstream. I'd be happy to just have some minor Arc-level adoption. I think it's better to have a small number of people who actually understand the goal (an implementation that's friendly to outsiders) than to have a lot of adoption that causes Mu to forget its roots. My real goal is to build something that outlasts the mainstream stack (the way mammals outlasted the dinosaurs). That doesn't feel as difficult. It's clear the mainstream has a lot of baggage bogging it down. It'll eventually run out of steam. But probably not in my lifetime.
Anyways, I hope in a year or so to give Mu an Arc-like high-level language. It won't improve Arc's adoption, but hopefully it will help promulgate the spirit of this forum: to keep the implementation transparent, and to be friendly to newcomers without burning ourselves out.
As of Python 3.7 at least, it comes with pip package manager and IDLE ide already installed. Yeah, you can use other tools, but it comes with the bare minimum all right there in the installer -- everything "just works" in a way I've never gotten with a lisp to date.
Although someone recently suggested I try Portacle, which is common lisp bundled with emacs and quicklisp already set up. That's definitely got my interest.
I still wish it was as "elegant" as arc, but maybe I'll find out it's complicated for good reason. We'll see.
Bundling a package manager for Anarki would be a little like being all dressed up with nowhere to go. There just aren't any libraries outside of the repo, to my knowledge. The community tends to add code directly to the repo since anyone can easily get commit access. That seems more elegant than a package manager, at least for this stage of Anarki's life.
I suppose that's fair. I guess I'm just angsty because arc clearly delivers on the promised elegance of lisp better than common lisp does, but arc isn't yet at a point where I can just press play and start using it to automate everything. Someday I hope to get to a point where I can contribute to anarki myself and make it into the power tool I know it ought to be, but I'm still at the stage where build tools scare me and everything seems needlessly complicated.
I'm not following "..just press play and start using it to automate everything." Is there a specific task you're trying to use Arc for?
It's definitely true that Arc has lots of deficiencies when it comes to tooling. We're not going to fix them all at once. So if you have suggestions for the most important three (or dozen!) tools you'd like it to provide, I'd love to see them. More specific tool suggestions are superior here to general ones. "Package manager" means wildly different things and has wildly different responsibilities in different contexts and languages and platforms. So I'd particularly like to see suggestions that talk about a specific situation that applies to you. "In ___ context, If I run ___, Arc should ___."
>"Package manager" means wildly different things and has wildly different responsibilities in different contexts and languages and platforms
I disagree. Most package managers for most languages share the same core functions and responsibilities: import and export code (likely from or to a repo), organize code into "packages" or "applications" according to some hierarchy, describe dependencies for packages using metadata and manage those dependencies when importing or updating.
The "special sauce" between them is in the way packages are defined and the metadata, and the way dependencies are handled, and those are details the Arc community could argue about until the heat death of the universe. But ~90% of what new people will want from Arc is to import News as an application or import libraries into their own local version of News.
Hmm, we've had this conversation a few times, and I think I may be unfairly holding others back. To reiterate the obvious, you should absolutely feel free to create a package manager in Anarki. The way you put it suggests it may not even be that much work for a rudimentary tool. Why not try it? Let a thousand flowers bloom :)
>The way you put it suggests it may not even be that much work for a rudimentary tool.
From what little I know, it would be an exercise in suffering [0,1] and that's not even getting into arguments over how versioning should work. But I still want to do it... eventually.
I kind of suspect it would be pointless without some changes to the language itself which I don't know how to implement myself yet, and I don't want to just wait for other people to do the work. Basically, I don't think we can have useful packages until we can do something about the possibility of packages being able to globally redefine things and create unintended havoc. Namespaces would be nice too.
I am hoping to get something like plugins working in the forum, though. I've already combined the repo and what was the plugin manager and I've been trying to get hooks to work through it so that it would be possible to 'plug' scripts in or out of those hooks during runtime. It might also be useful as a way to add updates by hotpatching the existing code (which is really the only valid use I can even think of for being able to redefine things in Arc.)
Maybe, if that ever gets anywhere, something like a package manager can emerge from it.
I'm actually somewhat intrigued with the idea of using guix as a language package manager. It has really nice facilities for setting up "environments" that contain a set of packages, so you can use it for bundling together all of the deps needed to build a particular application. As a bonus, it uses scheme as its configuration language :)
It wouldn't solve arc side of actually loading packages, which probably still needs work, but it should work pretty well at managing them.
This seems like egregious hype mongering. Two examples:
a) "Pure functions are easily parallelizable since they encourage immutable data structures which reduce the side-effects that make code hard to run on multiple processors. This is how Bitcoin will reach its infinite scalability."
Pretty abrupt transition there from a sentence that had me nodding along to one that gives me whiplash. Wait, wha? Pure functions are used in lots of places outside Bitcoin. None of them brags infinite scalability. So perhaps you need more than pure functions?
b) "@TensorFlow uses the functional programming paradigm of lazy evaluation. A tensor flow graph exists separately to the computation of that graph.
"Bitcoin can and will be used to create true artificial intelligence."
I don't think I need to say anything about the last sentence. For the rest, the author needs to make up their mind whether the paradigm is functional programming or lazy evaluation. Lazy eval is just a mechanism. Tensorflow uses it in the part of the system that is stateless. Other parts of ML are incredibly (and incredibly subtly) stateful.
My homebrew blog server has a part with lazy evaluation. It's a few lines of code. It's not going to give me infinite scalability anytime soon.
 Since many ML systems don't explain their models, there's a state dependency from any results they provide to all training data they've ever seen.
I'm not following the two of you on precisely what this anti-feature is. Assigning to local variables using '=' does not create new global variables. Can you share a code sample showing what you're talking about?
Sorry I just saw this. Can you elaborate? What is an example of a variable that C and Python define correctly but has a worse equivalent in Arc? Or what's a variable in Arc that bit you by being globally scoped?
In particular I'm surprised that you mentioned C. If we're worse than C in this respect I'd love to understand that better.
He wants it to be a little more like scheme wherein:
(define v 10)
creates v as a local variable. In Arc, and most lisps, you need to do:
(defun f ()
(let ((v 10))
Additionally, Arc has the wrinkle (i think, i haven't used Arc recently) where if you do:
(defun f ()
(= v 10)
you create a global variable and assign 10 to it, instead of the python default which creates v in local scope. Basically, he's trying to use imperative programming and doesn't want to declare local variables using let.
I see. Yes, it can be a little sharp-edged for new-comers from other languages that `=` auto-defines globals. It's not immediately obvious that we don't use it much. Let me think about how to improve that.
Yeah, as a newcomer I can attest that I did not know that. V here being global is really counter-intuitive:
(defun f ()
(= v 10)
I would have expected (and would prefer) the default to be to bind to whatever the current scope is, and to have global (file level, then application level) scope be opt-in rather than opt-out. We can't assume that it's more likely new users will be familiar with lisp idioms.
In your example, the variables in scope inside `f` are exactly the same as the variables in scope outside of it. You're implying the "current scope" has changed, but if the set of variables in scope hasn't changed, then what other part has?
I think you were expecting a concept that Arc doesn't have. Adding an unnecessary concept to Arc would make the language more complicated for newcomers who weren't expecting it, right?
Of course, if users do consistently come in with the same intuition, it's pointless to design against it. Often we've gotta design for complex realities rather than simple principles. :) If you think it's a case like that, I can sympathize, and maybe someday I'll see it.
Sometimes I feel like general-purpose plain text programming language is such a specific topic that it leads to only one possible language design. Feeling that way is probably the only way I'll design a single language at all, rather than designing a lot of half-languages and never finishing any of them!
In this case, I actually _don't_ feel like there's _no_ potential to alternative notions of lexical scope or variable assignment, but I think Arc's at a sweet spot, and I've some extensive reasoning as to why....
"We can't assume that it's more likely new users will be familiar with lisp idioms."
If not some other language's idioms, where did you get the idea of there being a "current scope"?
It's true that many popular languages have features where they infer a variable declaration at some notion of "current scope" around innermost point (Python) or outermost point (CoffeeScript, Ruby, MATLAB) where a variable is assigned. Newcomers to Arc from to those languages might expect this. (I think uu must be bringing in Python experience.)
I think some languages (R, Kernel, maybe some Scheme interpreters) represent the lexical scope as a run time data structure, and variable assignments can add new variables to the local scope that were previously looked up from an outer scope. They have lexical scope, but arguably not static scope.
I also want to mention PHP, which is off doing its own thing where there's hardly any implicit inheritance between lexical scopes at all. Every variable lookup or assignment is restricted to the current function unless there's an explicit `use` or `global` declaration to imply otherwise. I kind of admire PHP's willingness to make the interaction between scopes explicit like this; it means PHP could evolve to have different parts of the code written in different languages, with explicit marshalling of values between all of them.
So let's look at Arc as its own language.
Thanks to Arc's lexically scoped `fn`, it's basically an extension of the lambda calculus, and it has easy access to all known lambda calculus techniques for Turing-complete computation. This means Arc programmers basically don't have to use assignment at all unless they want to.
In Python, those lambda calculus techniques are possible to use in theory, but every nontrivial lambda must be named and pulled out onto its own line, giving us something a lot like `goto` label spaghetti.
In PHP, every nontrivial lambda must have a `use` declaration to pull in all the variables it captures. This can get to be particularly verbose, eventually to the point where it might be easier to pass around explicit context objects.
Even using lambda calculus techniques a little bit in Python or PHP means we start to have trouble with mutable variables. Lambda calculus uses functions for control flow, but using functions in Python or PHP means creating new scopes, which means we can't easily assign to outer variables from inside our conditionals and loops. Most uses of mutable variables involve some kind of conditional or loop (or variable capture for its own sake), since that's what makes them anything more than a sequence of variables that happen to share the same name. So the more we use lambda-calculus-style conditionals and loops, the less we effectively have access to mutable variables in the programs we're writing.
In both Python and PHP, it just takes a little more boilerplate to work around this: We give up on mutable variables altogether and simulate them with immutable variables that refer to mutable objects. (There's also `use (&$foo)` in PHP and `nonlocal foo` in Python, if you prefer not to give up on mutable variables, but they amount to far more boilerplate.)
The standard boilerplate for these things in Arc is pretty much less than zero, thanks to macros. An Arc programmer can write a custom conditional operator as a higher-order function, and then when they're tired of putting the conditional branches in `fn` every time they use it, they can write a macro that generates the `fn` automatically.
Since Scheme and Common Lisp were already well-worn combinations of lexically scoped `lambda`, mutation (`set!`/`setq`), and macros, all of this could pretty much be predicted when Arc was designed.
Nevertheless (or maybe out of having different goals than I'm expressing here), Paul Graham and co. tried out automatic local variables anyway. It was implemented for an early, unreleased version of Arc. Then they pulled this feature out because they realized they kept introducing or removing lexical contours by mistake and breaking parts of their code. I bet this is because they were implementing some of their control flow macros in terms of `fn`.
Could it be possible to follow through on their experiment without recognizing all the same mistakes and pulling the plug again? Yes, I bet it is.
But I think Arc's local variable scoping rules and variable assignment behavior are exactly what they need to be:
- Implicit inheritance of lexical scope to enable lambda calculus techniques (unlike PHP).
- The easy ability to mutate variables in outer scopes so mutation can work together with lambda-calculus-style control flow (unlike Python, R, and Kernel).
This still leaves the CoffeeScript/Ruby/MATLAB approach on the table, where only the outermost assignment is treated as a declaration. I don't particularly like this approach, and that's because I prefer for the outermost level to be relatively seamless with the rest of the language. That way it's easier to break parts of the language off into optional libraries when it turns out they're not as helpful as expected. Arc's top level already isn't seamless with the rest of the language, but I think this would be a step in the wrong direction.
In summary: If users come in with incorrect ideas about Arc's variable assignment behavior based on their experience with other languages, I think that's most likely a place where other languages could learn something from Arc rather than the other way around. The higher-order techniques of lambda calculus are a sweet spot in language design, and Arc's system for local variable scope is well tailored to that. The Arc designers originally did try automatic local variables. They found them to be unnecessarily complex to work with, and I agree.
 http://www.paulgraham.com/arcll1.html "Here is a big difference between Arc and previous Lisps: local variables can be created implicitly by assigning them a value. If you do an assignment to a variable that doesn't already exist, you thereby create a lexical variable that lasts for the rest of the block. (Yes, we know this will make the code hard to compile, but we're going to try.)"
 http://paulgraham.com/arclessons.html "In Arc we were planning to let users declare local variables implicitly, just by assigning values to them. This turns out not to work, and the problem comes from an unforeseen quarter: macros. [...] In a language with implicit local variables and macros, you're always tripping over unexpected lexical contours. You don't want to create new lexical contours without announcing it. [...] It seemed to us a bad idea to have a feature so fragile that its own implementors couldn't use it properly. So no more implicit local variables."
 In Racket, the `racket/splicing` module (https://docs.racket-lang.org/reference/splicing.html) has a few rough edges, but it's a good example of how the choice of whether a macro changes the "current scope" can be controlled deliberately, even in a language with lambdas and macros. I didn't bring up Racket or Scheme's notion of "current scope" with all the other examples because it doesn't interact with variable assignment, but I think even that notion is a kind of ill-conceived complexity that I'm glad Arc doesn't have. It's handy to have local syntax that roughly resembles the top level to aid in refactoring, but on the one hand the resemblance isn't required to be perfect (and isn't perfect in Racket), and on the other hand the Scheme top level isn't very easy to make modular, so it's not even a good thing to resemble.
Oh, right, here's another reason I don't like the CoffeeScript/Ruby/MATLAB approach where a variable is declared automatically at the outermost point where it's assigned: I like to be able to declare local variables that shadow variables from outer scopes. When `=` declares non-shadowing variables only (since the rest of the time it acts as an assignment rather than a declaration), shadowing is more cumbersome to do.
Setting local variables should be simpler (or equivalent to) setting global variables. It's so much easier to do (= v 10) than (let ((v 10) ...)) despite "let" / "with" arguably being preferable to "=" in most cases.
I have been thinking about a modified version of LET that takes a sequence of expressions (like begin) that are either assignments or not assignments. Then it would collect all consecutive assignments together into a letrec. For example:
(= a 1)
(= b 2)
(= c 3)
(= x 4)
(= y 5)
Sure, but losing data you don't want to lose because you reloaded a source code file does seem like more of an architectural than language issue. It would be a code smell in any other language.
My comment was slightly facetious but the more I think about it the more I'm wondering whether something like redis or php's apc wouldn't be a good idea - and not just as a lib file but integrated into Racket's processes for dealing with arc data directly.
It could serve both as a global data store and a basis for namespacing code in the future (see my other rambling comment about namespaces), since a "namespace" could just be a table key internally.
Otherwise hitching the code to a third-party db, as a requirement, would really limit what could be done with the language and would create all kinds of problems. You would be locked into the db platform as a hardened limitation. You would inherit the complexity of external forces (i.e. what if some other app deletes or messes with the db). What about securing the access/ports on the db.. etc..
It's always possible, but I think you would have to implement something internal where you can properly isolate and fully support all platforms the language does.
Seems likes namespaces would solve these problems the right way.
Yes. Currently, the options we have for stateful data are file I/O, which doesn't work perfectly, or tables that can lose their state if the file they're in gets reloaded. I'm suggesting something like Redis or APC, but implemented in Arc at the language level, to separate that state from the source code.
I was also thinking (in vague, "sketch on the back of a coffee-stained napkin" detail) that it could also be used to flag variables for mutability and for namespacing. In that if you added "x" from foo.arc it would automatically be namespaced by filename and accessible elsewhere as "foo!x",so it wouldn't conflict with "x" in bar.arc.
>Otherwise hitching the code to a third-party db, as a requirement, would really limit what could be done with the language and would create all kinds of problems.
Yeah, but to be fair, Arc is already hitched to Racket, which appears to support SQL and SQLite, so maybe third party wrappers for that wouldn't be a bad idea as well... sometime in the future when things are organized enough that we can have a robust third party ecosystem.
Languages that have decent bindings to a database also have global variables that still have uses, and that can be lost when you restart the server or do other sorts of loading manipulations. There's a category of state that you want coupled to the state of the codebase.
Yes, you can definitely try to make these different categories of state less error-prone by architectural changes. But I don't think other languages do this well either. Mainstream languages, at least. I know there's research on transparent persistence where every global mutation is automatically persisted, and that's interesting. But I'm not aware of obvious and mature tooling ideas here that one can just bolt on to Arc.
All that said, database bindings would certainly be useful to bolt on to Arc.