Experimental results

1. Execution time per Bench and Test Suite (Don’t figure in the paper)

We highlight in red the points that have been described in details in the paper

Time Color (seconds)      
JAVA JS CPP CS PHP std_dev
0,833 0,608 0,402 0,696 1,821 0,553103517
0,7 0,541 0,374 0,615 1,145 0,288826072
0,843 0,547 0,452 0,626 1,312 0,342674043
1,192 0,811 0,68 1,212 6,573 2,514878387
1,184 0,734 0,619 0,772 4,205 1,525593557
28,584 16,668 48,314 45,973 127,71 43,49731344
0,907 0,676 0,474 0,673 1,755 0,503539969
0,893 0,665 0,469 0,625 1,091 0,24419828
0,883 0,582 0,4 0,552 0,899 0,219172763
0,72 0,61 0,44 0,567 0,614 0,101124676
0,692 0,547 0,412 0,572 0,854 0,166381489
0,788 0,635 0,516 0,619 1,216 0,275511706
0,793 0,551 0,422 0,59 1,254 0,325856564
1,006 0,744 0,482 0,875 4,949 1,875919428
0,909 0,636 0,454 0,732 2,273 0,729938833
0,712 0,554 0,39 0,529 0,649 0,123177514
0,835 0,593 0,465 0,62 1,252 0,309013754
0,713 0,526 0,417 0,638 1,282 0,336134943
8,611 4,516 10,73 14,988 279,27 120,6095134
Time Core (seconds)      
JAVA JS CPP CS PHP std_dev
0,723 0,578 0,385 0,483 1,266 0,346856599
0,571 0,474 0,398 0,473 0,427 0,065668105
0,716 0,536 0,413 0,55 1,173 0,29717049
416,845 665,777 699,11 1161,288 61777,212 27299,89286
1,214 1,113 0,752 1,513 14,819 6,119898177
2,43 2,151 1,182 2,877 51,103 21,89678831
0,662 0,575 0,438 0,479 1,445 0,414613917
0,817 0,516 0,431 0,548 1,105 0,276431366
1,013 0,974 0,477 0,857 2,507 0,779223203
1,204 0,8 1,447 1,397 5,234 1,816620847
1,795 4,269 1,573 5,714 407,327 180,677793
2,056 4,707 1,596 5,363 417,138 185,0227926
5,908 6,262 11,044 14,141 297,206 128,783975
0,881 0,703 0,46 0,629 2,23 0,714602897
0,718 0,481 0,425 0,466 0,556 0,115692264
0,695 0,572 0,41 0,528 1,978 0,646157721
0,894 0,575 0,404 0,59 1,034 0,257162983
0,588 0,496 0,357 0,509 0,792 0,15956284
1,197 1,181 0,487 0,714 2,045 0,598056185
0,962 0,731 0,721 0,973 13,796 5,79234178
0,676 0,585 0,373 0,492 1,563 0,474768049
0,963 0,926 0,543 0,844 6,927 2,736563411
0,671 0,602 0,431 0,533 5,341 2,140301334
1,255 0,678 0,708 1,034 9,386 3,794212066
0,715 0,549 0,368 0,58 0,871 0,188489522
0,676 0,486 0,37 0,539 0,664 0,127910125
1,388 0,978 0,917 2,846 13,911 5,590600013
1,057 0,742 0,579 0,71 4,578 1,711120013
0,694 0,515 0,407 0,55 1,069 0,257257653
0,822 0,601 0,432 0,509 1,528 0,443802096
0,911 0,641 0,478 0,737 4,497 1,708970801
1,957 0,745 0,703 1,269 6,463 2,421428256
1,343 1,308 1,076 1,627 19,872 8,290741625
1,121 0,94 0,758 1,1 12,707 5,246615071
0,878 0,692 0,501 0,63 3,568 1,300788299
0,837 0,675 0,528 0,752 3,201 1,125130348
1,217 0,861 0,63 0,797 5,361 2,01703054
0,883 0,614 0,446 0,806 1,133 0,262174179
0,562 0,481 0,39 0,475 0,818 0,164194092
1,714 1,041 1,049 1,79 19,532 8,117293681
0,776 0,696 0,479 0,717 12,848 5,448668984
0,655 0,513 0,366 0,583 0,566 0,108047675
0,82 0,571 0,469 0,589 3,763 1,414894272
0,838 0,663 0,499 0,754 4,156 1,555799312
0,571 0,505 0,395 0,455 0,669 0,10591506
0,979 0,698 0,507 0,809 3,053 1,044887649
0,707 0,538 0,423 0,519 1,01 0,231041771
0,945 0,702 0,529 0,768 3,724 1,344546875
1,263 0,714 0,556 1,784 5,085 1,855698063
1,239 0,664 0,484 1,638 3,687 1,283773461
0,739 0,603 0,434 0,487 1,139 0,281893242
Time Hxmath (seconds)      
JAVA JS CPP CS PHP std_dev
3,387 2,04 1,42 4,013 73,446 31,64873033
1,094 1,115 0,725 0,914 10,505 4,270666962
2,973 1,535 1,386 3,131 70,078 30,34134256
0,663 0,545 0,397 0,561 1,067 0,253453743
0,671 0,589 0,439 0,6 1,452 0,401306242
0,942 0,776 0,505 0,668 2,634 0,869427973
Time Format (seconds)      
JAVA JS CPP CS PHP std_dev
0,718 0,601 0,378 0,542 1,117 0,277720543
2,102 2,313 1,811 6,076 148,238 64,94199757
5,031 5,402 7,672 12,383 220,756 95,36140842
0,91 0,635 0,446 0,886 4,019 1,48800494
Time Promise (seconds)      
JAVA JS CPP CS PHP std_dev
0,636 0,721 0,399 0,482 1,132 0,285440887
1,722 2,09 1,127 1,712 31,186 13,20769983
1,226 1,527 0,544 0,792 3,585 1,207494803
Time Culture (seconde)      
JAVA JS CPP CS PHP std_dev
0,552 0,472 0,376 0,439 0,714 0,130210599
0,58 0,464 0,374 0,437 0,617 0,101332621
0,628 0,512 0,403 0,474 0,726 0,128334719
0,826 0,577 0,628 0,655 3,797 1,400889467
Time Math (seconde)      
JAVA JS CPP CS PHP std_dev
12,511 3,008 16,297 14,143 1448,9 642,8494602
2,284 1,228 1,678 2,179 65,158 28,31881081
0,988 0,752 0,931 0,943 55,459 24,39812735

 

2. Memory usage per Bench and Test Suite (Don’t figure in the paper)

We highlight in red the points that have been described in details in the paper

Memory Color (MBytes)      
JAVA JS CPP CS PHP std_dev
0,856064 1,851392 1,560576 3,19488 24,571904 10,18989368
1,089536 0,958464 1,37216 0,856064 3,657728 1,173761669
1,716224 2,527232 1,449984 3,05152 0,811008 0,886140988
0,843776 1,937408 1,241088 3,035136 69,578752 30,33892048
0,663552 2,048 1,216512 3,01056 72,78592 31,78744857
1362,55488 900,702208 2275,49184 1283,309568 758,788096 593,0467585
1,8432 2,646016 2,473984 13,340672 30,031872 12,14154875
0,569344 1,88416 3,473408 2,912256 1,028096 1,225195011
1,753088 0,847872 1,39264 5,681152 1,605632 1,9452365
0,692224 1,220608 0,16384 3,489792 1,159168 1,271946834
1,994752 0,7168 0,610304 0,704512 1,134592 0,57485722
1,486848 1,024 2,924544 3,33824 0,958464 1,110406638
0,946176 1,470464 1,687552 0,516096 1,093632 0,457699364
1,626112 2,154496 1,630208 3,301376 104,804352 45,9009635
1,835008 2,523136 1,18784 3,059712 34,394112 14,43662455
1,7408 3,842048 1,527808 0,811008 1,757184 1,133062733
0,53248 2,29376 1,82272 1,056768 0,88064 0,721701949
1,060864 0,708608 2,912256 0,425984 1,20832 0,971061311
819,924992 253,014016 923,992064 327,606272 2189,86496 777,3228414
Memory Core (MBytes)      
JAVA JS CPP CS PHP std_dev
0,864256 4,07552 1,10592 0,835584 0,974848 1,403980107
0,970752 0,946176 1,388544 3,70688 1,20832 1,167344191
0,557056 1,634304 1,011712 0,83968 2,011136 0,596791557
768,221184 303,099904 618,422272 235,753472 1237,151744 403,1545547
0,827392 1,368064 1,163264 31,268864 97,800192 41,94968891
277,331968 79,310848 1,10592 77,312 502,255616 203,5496957
0,651264 0,978944 2,51904 0,96256 45,285376 19,69419082
0,868352 2,535424 2,125824 0,856064 1,093632 0,781415467
1,089536 1,232896 2,12992 3,223552 69,894144 30,41136245
1,097728 1,732608 53,223424 33,992704 140,3904 57,18758274
186,056704 61,689856 80,769024 44,179456 185,536512 68,91570787
170,545152 64,897024 98,807808 54,525952 228,401152 74,19385015
696,160256 77,754368 168,251392 65,06496 358,858752 263,9915712
0,909312 2,654208 1,08544 3,719168 46,497792 19,89266752
1,441792 0,774144 0,790528 0,96256 0,712704 0,297384833
0,720896 1,093632 0,811008 0,872448 64,131072 28,28953505
1,613824 0,69632 1,466368 3,698688 1,142784 1,158861845
0,548864 0,917504 0,966656 3,063808 1,110016 0,99579817
1,851392 1,503232 1,011712 3,301376 47,427584 20,37085786
2,2528 1,31072 0,974848 3,19488 288,653312 128,2280065
1,073152 0,090112 1,318912 3,260416 55,88992 24,37980449
0,69632 1,880064 1,216512 3,268608 172,220416 76,23590611
1,179648 0,98304 0,950272 3,198976 43,597824 18,81533836
1,089536 1,380352 1,028096 3,2768 162,701312 72,01071858
0,811008 0,88064 1,355776 0,966656 1,08544 0,213876318
0,847872 1,073152 0,929792 6,180864 1,417216 2,297321852
193,077248 1,499136 1,335296 42,450944 202,559488 101,5293322
0,626688 1,76128 1,253376 3,19488 99,340288 43,67228965
0,950272 1,036288 0,774144 2,965504 1,212416 0,896029374
1,736704 0,905216 1,839104 3,162112 10,71104 4,017711885
0,851968 1,630208 1,048576 3,186688 118,714368 52,34766327
299,470848 2,51904 1,10592 6,209536 169,619456 134,7459487
1,396736 0,864256 0,765952 41,51296 191,758336 82,65969108
0,851968 0,999424 1,265664 3,325952 201,89184 89,57402224
1,716224 0,565248 1,503232 3,690496 33,36192 14,1300038
1,544192 2,244608 1,482752 2,961408 74,510336 32,40713983
1,880064 1,368064 1,41312 3,82976 52,871168 22,7176544
0,688128 2,527232 1,47456 6,18496 1,384448 2,187861443
0,856064 0,7168 1,3312 1,093632 1,26976 0,263554605
179,85536 1,241088 1,118208 55,27552 289,841152 126,2916781
1,61792 0,7168 1,482752 3,19488 71,069696 31,0124077
1,224704 1,794048 1,191936 3,375104 1,236992 0,934397166
0,946176 0,847872 2,084864 3,272704 114,372608 50,35905441
1,388544 2,502656 1,081344 3,063808 30,035968 12,55992267
1,622016 2,461696 1,482752 3,309568 1,00352 0,912225272
0,954368 1,007616 1,236992 0,974848 62,046208 27,28149549
1,47456 3,137536 0,950272 3,457024 1,59744 1,104635841
2,00704 3,162112 1,101824 3,219456 36,745216 15,39699601
1,486848 2,404352 0,868352 24,80128 87,015424 37,0051634
0,8192 0,790528 2,555904 27,377664 53,440512 23,28520965
2,138112 0,569344 3,305472 3,538944 1,257472 1,280507476
Memory Hxmath (MBytes)      
JAVA JS CPP CS PHP std_dev
335,50336 104,009728 296,431616 156,409856 1192,98048 444,1830293
1,175552 1,37216 2,2528 3,321856 348,16 154,7961038
389,726208 111,681536 273,12128 136,486912 1146,052608 425,6505535
0,770048 2,023424 2,51904 3,063808 1,110016 0,955735576
0,958464 1,499136 1,400832 0,8192 40,718336 17,68914629
0,704512 2,80576 1,204224 1,093632 104,59136 46,13230432
Memory Format (MBytes)      
JAVA JS CPP CS PHP std_dev
0,671744 2,420736 1,122304 1,081344 0,585728 0,735510909
168,538112 58,028032 73,740288 51,224576 302,198784 106,8708471
685,903872 92,348416 204,857344 69,16096 385,118208 255,3629001
1,069056 2,945024 1,470464 2,985984 20,803584 8,400726047
Memory Promise (MBytes)      
JAVA JS CPP CS PHP std_dev
1,232896 1,015808 0,458752 0,876544 0,659456 0,301736504
128,294912 55,291904 0,598016 34,496512 134,201344 58,76203811
1,875968 38,64576 1,736704 3,190784 39,022592 20,0366956
Memory Culture (MBytes)      
JAVA JS CPP CS PHP std_dev
4,046848 1,503232 1,35168 0,950272 1,122304 1,276574108
1,155072 0,712704 1,634304 11,20256 1,110016 4,506159814
1,724416 1,593344 2,02752 0,55296 1,888256 0,584928304
1,90464 0,704512 1,744896 0,692224 36,323328 15,69030929
Memory Math (MBytes)      
JAVA JS CPP CS PHP std_dev
831,442944 493,666304 1492,97152 806,330368 3088,154624 1041,528633
226,902016 2,535424 296,484864 127,430656 626,83136 234,9340201
1,212416 0,933888 1,613824 3,371008 630,382592 281,1199797

2. Further statistics (Don’t figure in the paper)

2.1. x= benchmark, y= average memory usage on all test suites

Sans titre.png

2.2. x= language, y= average memory usage on all benchs

Capture d’écran 2016-07-01 à 16.17.44.png

2.3. x= language, y= average execution time on all benchs

Capture d’écran 2016-07-01 à 16.19.21.png