exit code of "se test"

[Raphael Mack-2]

Hello,

when executing "se test" I get output of failed tests (c compilation did
not succeed), which means, that some tests did not work:

==========================
In directory "/home/r/eiffel/eiffel-libraries/trunk/eiffel-gsl/test/".
test_complex011.o: In function `r25make':
test_complex011.c:(.text+0x96ef): undefined reference to `se_cmpT54'
collect2: ld gab 1 als Ende-Status zurück
se test (eiffeltest): Error while running:
   se c -all_check test_complex01.e -o test_complex01.exe
(Return status was 1)
se test (eiffeltest): Unable to compile "test_complex01.e".
Command used:
se c -all_check test_complex01.e -o test_complex01.exe
In directory "/home/r/eiffel/eiffel-libraries/trunk/eiffel-gsl/test/".
==========================

Now, the call to "se test" returns 0. Is there a good reason why? - else I
suggest to return 1 if at least one test failed.

Thanks,
Rapha

[Raphael Mack-2]

Hello,

PLEASE comment on this. Would it be possible to get a patch commited, if
I'd prepare one? What do other users (are there any) think?

Thanks,
Rapha
Am Sonntag, den 19.10.2008, 11:01 +0200 schrieb Raphael Mack:

> Hello,
>
> when executing "se test" I get output of failed tests (c compilation did
> not succeed), which means, that some tests did not work:
>
> ==========================
> In directory "/home/r/eiffel/eiffel-libraries/trunk/eiffel-gsl/test/".
> test_complex011.o: In function `r25make':
> test_complex011.c:(.text+0x96ef): undefined reference to `se_cmpT54'
> collect2: ld gab 1 als Ende-Status zurück
> se test (eiffeltest): Error while running:
>    se c -all_check test_complex01.e -o test_complex01.exe
> (Return status was 1)
> se test (eiffeltest): Unable to compile "test_complex01.e".
> Command used:
> se c -all_check test_complex01.e -o test_complex01.exe
> In directory "/home/r/eiffel/eiffel-libraries/trunk/eiffel-gsl/test/".
> ==========================
>
> Now, the call to "se test" returns 0. Is there a good reason why? - else I
> suggest to return 1 if at least one test failed.
>
> Thanks,
> Rapha

[Philippe Ribet]

Raphael Mack wrote:

>Hello,
>
>PLEASE comment on this. Would it be possible to get a patch commited, if
>I'd prepare one? What do other users (are there any) think?
>
>Thanks,
>Rapha
>Am Sonntag, den 19.10.2008, 11:01 +0200 schrieb Raphael Mack:
>  
>
>>Hello,
>>
>>when executing "se test" I get output of failed tests (c compilation did
>>not succeed), which means, that some tests did not work:
>>
>>==========================
>>In directory "/home/r/eiffel/eiffel-libraries/trunk/eiffel-gsl/test/".
>>test_complex011.o: In function `r25make':
>>test_complex011.c:(.text+0x96ef): undefined reference to `se_cmpT54'
>>collect2: ld gab 1 als Ende-Status zurück
>>se test (eiffeltest): Error while running:
>>   se c -all_check test_complex01.e -o test_complex01.exe
>>(Return status was 1)
>>se test (eiffeltest): Unable to compile "test_complex01.e".
>>Command used:
>>se c -all_check test_complex01.e -o test_complex01.exe
>>In directory "/home/r/eiffel/eiffel-libraries/trunk/eiffel-gsl/test/".
>>==========================
>>
>>Now, the call to "se test" returns 0. Is there a good reason why? - else I
>>suggest to return 1 if at least one test failed.
>>
>>Thanks,
>>Rapha
>>    
>>
>
>
>  
>

You're right, return code should reflect the test status. I can commit
the patch.

Best regards,

--

             Philippe Ribet


SmartEiffel:
one methodoology, one language,
highest quality kept secret.
Visit http://smarteiffel.loria.fr

[Raphael Mack-2]

Hi,

Am Montag, den 24.11.2008, 22:33 +0100 schrieb Philippe Ribet:
> You're right, return code should reflect the test status. I can commit
> the patch.

Thanks. This is encouraging.
Here is a patch against r9294, but actually this does not completely
satisfy me:

We have simple test cases that essentially look like:

class TEST_FEATURE1
inherit EIFFELTEST_TOOLS
creation make
feature
        make is
                do
                        assert(1 = 2)
                end
end -- class TEST_FEATURE1

This is very simple to create and I like it. But compiling and running
this also does has an exit code of 0.

Now I see three solutions:
1. adapt the test:
make is
        local
                exit_code: INTEGER
        do
                assert(1 = 2)
                if 1 /= 2 then
                        exit_code := 1
                end
                die_with_code(exit_code)
        end

2. adapt EIFFELTEST_TOOLS:
call die_with_code(1) at the end of label_assert

3. adapt EIFFELTEST_TOOLS and the test
- add a failed_counter (similar to assert_counter) to EIFFELTEST_TOOLS
  which is incremented for each failing assertion and a feature
  exit_code: INTEGER which returns 1 if failed_counter > 0
- adapt the test to
make is
        do
                assert(1 = 2)
                die_with_code(exit_code)
        end

All versions have pros and cons. I'd prefer 2 or 3.

Cheers,
Rapha

[exit_code.patch]

---

Index: tools/commands/eiffeltest.e
===================================================================
--- tools/commands/eiffeltest.e (Revision 9294)
+++ tools/commands/eiffeltest.e (Arbeitskopie)
@@ -858,6 +858,7 @@
  echo.w_put_string(once "%NIn directory %"")
  echo.w_put_string(directory_path)
  echo.w_put_string(once "%".%N")
+ final_die_with_code_result := exit_failure_code
  else
  running_of(test_file, exe_name, options)
 
@@ -918,6 +919,7 @@
  log(once "No error/warning message for %"")
  log(bad_file)
  log(once "%".%N")
+ final_die_with_code_result := exit_failure_code
 
  elseif not file_tools.file_exists(msg) then
  echo.w_put_string("se test (eiffeltest): In directory %"")
@@ -929,6 +931,7 @@
  echo.w_put_string("%".%NIf so, then, change the name of this file as %"")
  echo.w_put_string(msg)
  echo.w_put_string("%".%NThis will register this message as correct.%N")
+ final_die_with_code_result := exit_failure_code
  else
  error_message_comparator.do_compare(msg, new)
  if error_message_comparator.error_flag then
@@ -941,6 +944,7 @@
  echo.w_put_string("%" and %"")
  echo.w_put_string(msg)
  echo.w_put_string("%" do differ too much.%NPlease check manually.%N")
+ final_die_with_code_result := exit_failure_code
  else
  log_line := "Removing %"" + new + "%"."
  dummy := excluded_execution_of(log_line, agent file_tools.delete(new))
@@ -1262,6 +1266,7 @@
  log(once "Abnormal exit status of %"")
  log(log_line)
  log(once "%".%N")
+ final_die_with_code_result := exit_failure_code
  end
  -- Now one second of pause to allow the user to kill the main "se test" process.
  -- Actually, this is not a pause... I do not like to use the cpu for nothing,
@@ -1382,6 +1387,7 @@
  echo.w_put_string(once "%".%N(Time limit was set to ")
  echo.w_put_integer(very_long_run_time.force_to_integer_32)
  echo.w_put_string(once "%".)%N")
+ final_die_with_code_result := exit_failure_code
  end
 
  if file_tools.file_exists(once "profile.se") then

---