Quantcast

Running test who have two labels?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Running test who have two labels?

Eric Noulard
Hi there,

I'm playing with ctest LABELS and I wanted to know whether if it is possible
to run the set of tests which have 2 labels ?

I manage to have all tests which have **either** L1 or L2:

ctest -L "L1|L2" 

but how can I write a proper command line for both L1 and L2 ?

apparently

ctest -L "L1" -L "L2" 

only takes into account the last -L argument.

--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Running test who have two labels?

Nils Gladitz-2
On 03/24/2017 11:50 AM, Eric Noulard wrote:

> Hi there,
>
> I'm playing with ctest LABELS and I wanted to know whether if it is
> possible
> to run the set of tests which have 2 labels ?
>
> I manage to have all tests which have **either** L1 or L2:
>
> ctest -L "L1|L2"
>
> but how can I write a proper command line for both L1 and L2 ?
>
> apparently
>
> ctest -L "L1" -L "L2"
>
> only takes into account the last -L argument.
>

I don't think it is possible currently.
To a degree you might be able to work around it by creating additional
labels that combine the existing ones e.g. "L1", "L2", "L1-L2".

I was pondering trying to implement test filtering based on the
condition evaluator CMake uses for if()/while() at some point
(cmConditionEvaluator.cxx).
e.g. --run-tests-if "L1 IN_LIST TEST_LABELS AND L2 IN_LIST TEST_LABELS"

Nils
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Running test who have two labels?

Eric Noulard


2017-03-24 12:30 GMT+01:00 Nils Gladitz <[hidden email]>:
On 03/24/2017 11:50 AM, Eric Noulard wrote:

Hi there,

I'm playing with ctest LABELS and I wanted to know whether if it is possible
to run the set of tests which have 2 labels ?

I manage to have all tests which have **either** L1 or L2:

ctest -L "L1|L2"

but how can I write a proper command line for both L1 and L2 ?

apparently

ctest -L "L1" -L "L2"

only takes into account the last -L argument.


I don't think it is possible currently.
To a degree you might be able to work around it by creating additional labels that combine the existing ones e.g. "L1", "L2", "L1-L2".

Yes of course.
I do test name mangling but I was hoping for more "classifying" feature with labels and was expecting a simple
OR and AND operation with labels. This is not the case.

 

I was pondering trying to implement test filtering based on the condition evaluator CMake uses for if()/while() at some point (cmConditionEvaluator.cxx).
e.g. --run-tests-if "L1 IN_LIST TEST_LABELS AND L2 IN_LIST TEST_LABELS"

It could have been nice but may be overkill my needs.
A repetable ANDing -L or -LE will do the job.

That said since test filtering is regex-based I simply rediscovered that ANDing or negating regex is not easy.

--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Running test who have two labels?

CMake mailing list
Here's a bash script wrapper you could use with existing ctest. Save
it out to a file named ctest-two-labels.sh and then call it with bash
on Mac or Linux:


label1=$1
label2=$2

if [ -z "$label1" ]; then
    echo "script takes two label arguments as input, missing arg 1"
    exit 1
fi

if [ -z "$label2" ]; then
    echo "script takes two label arguments as input, missing arg 2"
    exit 2
fi

tmpfile=./ctest-two-labels-sh-tmp.txt
testlistFile=./ctest-two-labels-sh-testlist.txt
testnumsFile=./ctest-two-labels-sh-testnums.txt

ctest -L "^($label1)$" -N > "$tmpfile"
ctest -L "^($label2)$" -N >> "$tmpfile"

cat "$tmpfile" | grep "  Test #" | sort | uniq -d > "$testlistFile"

cat "$testlistFile" | awk -F "#" '{ print $2; }' | awk -F ":" '{ print
$1; }' > "$testnumsFile"

testnums=$(cat "$testnumsFile" | paste -s -d, -)

#echo Tests with both labels $label1 and $label2:
#cat "$testlistFile"
#
#echo Just the test numbers:
#cat "$testnumsFile"
#
#echo The test numbers, assembled into a ctest -I string to run just
those numbered tests:
#echo $testnums

echo Running command line:
echo ""
echo "  ctest -I \"0,0,0,$testnums\" -N"
echo ""
echo Run it without the -N to actually execute the tests...
echo ""

ctest -I "0,0,0,$testnums" -N



It's a "back of the envelope / proof of concept" script. Polish it up
and make it nice if the approach seems reasonable to you.


HTH,
David C.





On Fri, Mar 24, 2017 at 8:08 AM, Eric Noulard <[hidden email]> wrote:

>
>
> 2017-03-24 12:30 GMT+01:00 Nils Gladitz <[hidden email]>:
>>
>> On 03/24/2017 11:50 AM, Eric Noulard wrote:
>>
>>> Hi there,
>>>
>>> I'm playing with ctest LABELS and I wanted to know whether if it is
>>> possible
>>> to run the set of tests which have 2 labels ?
>>>
>>> I manage to have all tests which have **either** L1 or L2:
>>>
>>> ctest -L "L1|L2"
>>>
>>> but how can I write a proper command line for both L1 and L2 ?
>>>
>>> apparently
>>>
>>> ctest -L "L1" -L "L2"
>>>
>>> only takes into account the last -L argument.
>>>
>>
>> I don't think it is possible currently.
>> To a degree you might be able to work around it by creating additional
>> labels that combine the existing ones e.g. "L1", "L2", "L1-L2".
>
>
> Yes of course.
> I do test name mangling but I was hoping for more "classifying" feature with
> labels and was expecting a simple
> OR and AND operation with labels. This is not the case.
>
>
>>
>>
>> I was pondering trying to implement test filtering based on the condition
>> evaluator CMake uses for if()/while() at some point
>> (cmConditionEvaluator.cxx).
>> e.g. --run-tests-if "L1 IN_LIST TEST_LABELS AND L2 IN_LIST TEST_LABELS"
>
>
> It could have been nice but may be overkill my needs.
> A repetable ANDing -L or -LE will do the job.
>
> That said since test filtering is regex-based I simply rediscovered that
> ANDing or negating regex is not easy.
>
> --
> Eric
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Running test who have two labels?

Eric Noulard


2017-03-24 21:17 GMT+01:00 David Cole <[hidden email]>:
Here's a bash script wrapper you could use with existing ctest. Save
it out to a file named ctest-two-labels.sh and then call it with bash
on Mac or Linux:


label1=$1
label2=$2

if [ -z "$label1" ]; then
    echo "script takes two label arguments as input, missing arg 1"
    exit 1
fi

if [ -z "$label2" ]; then
    echo "script takes two label arguments as input, missing arg 2"
    exit 2
fi

tmpfile=./ctest-two-labels-sh-tmp.txt
testlistFile=./ctest-two-labels-sh-testlist.txt
testnumsFile=./ctest-two-labels-sh-testnums.txt

ctest -L "^($label1)$" -N > "$tmpfile"
ctest -L "^($label2)$" -N >> "$tmpfile"

cat "$tmpfile" | grep "  Test #" | sort | uniq -d > "$testlistFile"

Strange enough I got the very same idea to dump and "uniqify" the list I want.
 

cat "$testlistFile" | awk -F "#" '{ print $2; }' | awk -F ":" '{ print
$1; }' > "$testnumsFile"

testnums=$(cat "$testnumsFile" | paste -s -d, -)

#echo Tests with both labels $label1 and $label2:
#cat "$testlistFile"
#
#echo Just the test numbers:
#cat "$testnumsFile"
#
#echo The test numbers, assembled into a ctest -I string to run just
those numbered tests:
#echo $testnums

echo Running command line:
echo ""
echo "  ctest -I \"0,0,0,$testnums\" -N"
echo ""
echo Run it without the -N to actually execute the tests...
echo ""

ctest -I "0,0,0,$testnums" -N

It's a "back of the envelope / proof of concept" script. Polish it up
and make it nice if the approach seems reasonable to you.

The approach is reasonable. I was simply afraid that dump + filtering of tests and/or labels
wouldn't be efficient with 600++ tests. That said it's working. Thank you very much for the proof of concept.
I guess that may be dumping whole tests names (with ctest -N ) and labels (with --print-labels) would make it
simple for me to craft a script to easily select tests subset with a ctest wrapper.

It would even be simpler if ctest could dump all tests with associated labels.
I'll think about it.


--
Eric

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
Loading...