Arc Forumnew | comments | leaders | submitlogin
Is it taking longer to run the Anarki tests than it used to?
2 points by akkartik 1269 days ago | 11 comments
I just tried to run the unit tests after a long time.

The good news: the tests all pass!

But things seem slower than I remember.

When I run (load "tests.arc") at the REPL, it takes a minute to load all the suites (until the line "type: 15 tests, 0 nested suites"). Then it takes another 1:15s to see the next "arc>" prompt, with no visible progress for most of that time.

Do others have the same experience? I'm trying this on a 15-inch 2019 Macbook Pro with Racket 8.0.

Have things always been this slow? Am I just spoiled after programming on bare-metal machine code for 2 years? :)



2 points by zck 1269 days ago | link

Hrm. The tests definitely were faster. They weren't as fast as I'd wanted, but certainly not this slow. It just took three minutes for me to load that file. It's slightly complicated by that file loading _and_ running the tests, but again, it wasn't that slow before. For reference, I'm on Ubuntu running Racket 8.1.

One thing that definitely has changed is that now, Anarki requires Racket >=7.7. I just had to update my Racket from 7.2, which worked the last time I ran Anarki, Perhaps this is related?

-----

2 points by akkartik 1268 days ago | link

I also see some failures in unit-test.arc:

    (require 'lib/unit-test.arc/unit-test.arc)
    (load "lib/unit-test.arc/tests.arc")
    (test-and-error-on-failure)

    unit-test-tests.suite-creation.add-tests-to-suite.tests-are-wrapped-to-create-test-result-template failed: (isa result (quote table)) should be nil but instead was nil
    unit-test-tests.remove-thing.remove-nothing-from-single failed: (remove-thing (quote (not-found)) single-suite) should be (obj (single #s(ar-tagged tem (suite #hash((full-suite-name . suite-with-no-full-name) (nested-suites . #hash()) (suite-name . suite-with-no-name) (tests . #hash((nested . #s(ar-tagged tem (test #hash((suite-name . test-with-no-suite-name) (test-fn . #<procedure: g5264>) (test-name . test-with-no-name)) #hash())))))) #hash())))) but instead was (obj (single #s(ar-tagged tem (suite #hash((full-suite-name . suite-with-no-full-name) (nested-suites . #hash()) (suite-name . suite-with-no-name) (tests . #hash((nested . #s(ar-tagged tem (test #hash((suite-name . test-with-no-suite-name) (test-fn . #<procedure: g5264>) (test-name . test-with-no-name)) #hash())))))) #hash()))))
    unit-test-tests.remove-thing.remove-nested-test failed: (remove-thing (quote (single double one)) two-nested-tests) should be (obj (single #s(ar-tagged tem (suite #hash((full-suite-name . suite-with-no-full-name) (nested-suites . #hash((double . #s(ar-tagged tem (suite #hash((full-suite-name . suite-with-no-full-name) (nested-suites . #hash()) (suite-name . suite-with-no-name) (tests . #hash((two . #s(ar-tagged tem (test #hash((suite-name . test-with-no-suite-name) (test-fn . #<procedure: g5264>) (test-name . test-with-no-name)) #hash())))))) #hash()))))) (suite-name . suite-with-no-name) (tests . #hash())) #hash())))) but instead was (obj (single #s(ar-tagged tem (suite #hash((full-suite-name . suite-with-no-full-name) (nested-suites . #hash((double . #s(ar-tagged tem (suite #hash((full-suite-name . suite-with-no-full-name) (nested-suites . #hash()) (suite-name . suite-with-no-name) (tests . #hash((two . #s(ar-tagged tem (test #hash((suite-name . test-with-no-suite-name) (test-fn . #<procedure: g5264>) (test-name . test-with-no-name)) #hash())))))) #hash()))))) (suite-name . suite-with-no-name) (tests . #hash())) #hash()))))
    unit-test-tests.remove-thing.remove-one-of-two-suites failed: (remove-thing (quote (first)) two-suites) should be (obj (second #s(ar-tagged tem (suite #hash((full-suite-name . suite-with-no-full-name) (nested-suites . #hash()) (suite-name . suite-with-no-name) (tests . #hash((nested . #s(ar-tagged tem (test #hash((suite-name . test-with-no-suite-name) (test-fn . #<procedure: g5264>) (test-name . test-with-no-name)) #hash())))))) #hash())))) but instead was (obj (second #s(ar-tagged tem (suite #hash((full-suite-name . suite-with-no-full-name) (nested-suites . #hash()) (suite-name . suite-with-no-name) (tests . #hash((nested . #s(ar-tagged tem (test #hash((suite-name . test-with-no-suite-name) (test-fn . #<procedure: g5264>) (test-name . test-with-no-name)) #hash())))))) #hash()))))

-----

3 points by zck 1267 days ago | link

Updated unit-test.arc to the latest. PR at https://github.com/arclanguage/anarki/pull/205

-----

2 points by zck 1267 days ago | link

Hrm, this should have been failing for a while. It comes down to functions never being able to be `iso` to each other.

  arc> (iso (fn ()) (fn ()))
  'nil
I suspect this might've changed in the new Racket? I hope this hasn't snuck through for so long.

Anyway, working on a fix now.

-----

3 points by rocketnia 1266 days ago | link

At one point, years ago, I relied on every lambda returning a newly allocated object. I think I brought it up as a bug in Rainbow that the [] expressions in my Arc code were generating the same lambda-lifted result instead of creating unique objects each time.

But I think this was never something I could rely on in the original, Racket-based Arc either. The Racket docs don't specify whether a lambda expression returns a new object or reuses an existing one, and in fact they explicitly allow for the possibility of reusing one:

"Similarly, evaluation of a lambda form typically generates a new procedure object, but it may re-use a procedure object previously generated by the same source lambda form."

That sentence has been in the reference since the earliest versions I can find online:

The latest docs, currently for 8.1 (2021): https://docs.racket-lang.org/reference/Equality.html#%28part...

5.0 (2010): https://download.racket-lang.org/docs/5.0/html/reference/eva...

4.0.0.1 (2008): https://web.archive.org/web/20080620030058/http://docs.plt-s...

3.99.0.13 (2008): https://web.archive.org/web/20080229164003/http://docs.plt-s...

I'm not sure if the bug you're referring to had to do with making the same mistake I was making back then, or if you're talking about making the opposite assumption (expecting two procedures to be equal), but it's probably best not to expect stability either way.

-----

2 points by zck 1266 days ago | link

I was doing a deep-`iso` comparison I called `same`, because in Arc, hash tables are not the same. In Arc3.1:

  arc> (iso (obj) (obj))
  nil
  arc> (iso (obj 1 2) (obj 1 2))
  nil
So I made a function that iterated over the key/value pairs and compared them. In some unit tests, I used that on a hash table that contained functions. Yesterday, I refactored the unit tests to instead make sure the keys were right.

Interestingly, it seems that Anarki can compare hash tables:

  arc> (iso (obj) (obj))
  't
  
  arc> (iso (obj 1 2) (obj 1 2))
  't
Anyway, this should fix the unit test tests in the Anarki repo.

-----

2 points by zck 1267 days ago | link

Oh jeez, let me take a look.

-----

2 points by akkartik 1267 days ago | link

Not on you. I know we forked it as well a while ago.

-----

1 point by krapp 1267 days ago | link

>should be nil but instead was nil

I'm sorry, what?

-----

1 point by zck 1267 days ago | link

It's printing out the two values it's comparing. Not sure yet why they're not the same.

It makes more sense when you see an error message like "should be 3 but instead was 2".

-----

2 points by zck 1266 days ago | link

I just ran the unit-test.arc unit tests on both Arc3.1 and Anarki master, and they ran in virtually the same time. Arc3.1 took 49 seconds, and Anarki took 51 seconds.

Certainly would be worth trying to optimize. I'm sure there are a lot of improvements to be had, both in unit-test.arc and Anarki.

-----