Handling real-time output from execute_process

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Handling real-time output from execute_process

Person Withhats
Hey there!

I'm running a script that downloads some files and untars them. This takes a while but it outputs progress. Unfortunately I can't get the output to show up in real-time with CMake GUI or CLI.

Current workaround to see output live, loses it in log though......
    execute_process(COMMAND "cmd" /c "start /wait download_sdks.exe"
    WORKING_DIRECTORY "${SCRIPT_DIR}"
    OUTPUT_VARIABLE OUT
    RESULT_VARIABLE RES)

Exit code is always 0 no matter what I do or how I get this set up. I want to see the output as it's running (preferably in same cmake window) and for the GUI stop-button to properly stop the script instead of hanging until script ends by itself.

Right now I can get the output at the end but after 1.5 hours of untarring/etc. I'd be worried if it wasn't just frozen......

This is for windows 10 ofc and the .exe file is PyInstalled. Works fine double-clicking (launches terminal/etc.) but Cmake....ugh......

tl;dr script runs and works fine as usual but I can't see output while it's running instead of after.

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Handling real-time output from execute_process

frodak17


On Fri, Dec 21, 2018 at 8:30 PM Person Withhats <[hidden email]> wrote:
Hey there!

I'm running a script that downloads some files and untars them. This takes a while but it outputs progress. Unfortunately I can't get the output to show up in real-time with CMake GUI or CLI.

Current workaround to see output live, loses it in log though......
    execute_process(COMMAND "cmd" /c "start /wait download_sdks.exe"
    WORKING_DIRECTORY "${SCRIPT_DIR}"
    OUTPUT_VARIABLE OUT
    RESULT_VARIABLE RES)

Exit code is always 0 no matter what I do or how I get this set up. I want to see the output as it's running (preferably in same cmake window) and for the GUI stop-button to properly stop the script instead of hanging until script ends by itself.

Right now I can get the output at the end but after 1.5 hours of untarring/etc. I'd be worried if it wasn't just frozen......

This is for windows 10 ofc and the .exe file is PyInstalled. Works fine double-clicking (launches terminal/etc.) but Cmake....ugh......

tl;dr script runs and works fine as usual but I can't see output while it's running instead of after.

How about use OUTPUT_FILE instead. Then while CMake is running you can do a tail -f on the output file in a different window or just check the contents in notepad. I don’t know what happens if you don’t redirect standard output from CMake maybe it shows in terminal. 


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Handling real-time output from execute_process

Person Withhats
1) I want output in real time on a Windows system(no tail), and piping output to a file then reading it is a very poor solution. 'just check the contents in note pad'....
2) This is a general-purpose solution and not something I'm making just for myself, hacky fixes are not an option.

On Fri, Dec 21, 2018 at 7:26 PM frodak17 <[hidden email]> wrote:


On Fri, Dec 21, 2018 at 8:30 PM Person Withhats <[hidden email]> wrote:
Hey there!

I'm running a script that downloads some files and untars them. This takes a while but it outputs progress. Unfortunately I can't get the output to show up in real-time with CMake GUI or CLI.

Current workaround to see output live, loses it in log though......
    execute_process(COMMAND "cmd" /c "start /wait download_sdks.exe"
    WORKING_DIRECTORY "${SCRIPT_DIR}"
    OUTPUT_VARIABLE OUT
    RESULT_VARIABLE RES)

Exit code is always 0 no matter what I do or how I get this set up. I want to see the output as it's running (preferably in same cmake window) and for the GUI stop-button to properly stop the script instead of hanging until script ends by itself.

Right now I can get the output at the end but after 1.5 hours of untarring/etc. I'd be worried if it wasn't just frozen......

This is for windows 10 ofc and the .exe file is PyInstalled. Works fine double-clicking (launches terminal/etc.) but Cmake....ugh......

tl;dr script runs and works fine as usual but I can't see output while it's running instead of after.

How about use OUTPUT_FILE instead. Then while CMake is running you can do a tail -f on the output file in a different window or just check the contents in notepad. I don’t know what happens if you don’t redirect standard output from CMake maybe it shows in terminal. 


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Handling real-time output from execute_process

frodak17


On Fri, Dec 21, 2018 at 11:16 PM Person Withhats <[hidden email]> wrote:
1) I want output in real time on a Windows system(no tail), and piping output to a file then reading it is a very poor solution. 'just check the contents in note pad'....
2) This is a general-purpose solution and not something I'm making just for myself, hacky fixes are not an option.


By default the command runs in the same console as cmake.
So if it is a typical executable that prints to standard output it will be shown with the rest of the cmake output.
Also the executable output can be captured to a file or variable instead of being shown.
What you can't do is capture it in a variable or file and display it at the same time.

When you use '"cmd" /c "start /wait download_sdks.exe"' this spawns a new console window that cmake knows nothing about.
This breaks the normal usage of execute_process() because "start /wait" creates an independent console to run download_sdks.exe.
cmake has no influence over this second console.  So while you can see its outputs you can't capture it in any way through cmake.

You get a return value of 0 in this case because the errorlevel from the second console isn't passed back correctly to the first console and then back to cmake.
A quick google search shows that this would work: COMMAND "cmd" /c "start /wait  download_sdks.exe & exit ^!errorlevel^!" assuming that download_sdks.exe actually returns any error codes.

You don't say why download_sdks.exe can't be run as a normal executable.
Nor do you say why you aren't using "start /wait /b" to keep everything in the same console as cmake is running in.
You said you want to see the output from the command but are clearly trying to capture the output in a variable instead but don't mention why.

I guess it's hacky but the tool download_sdks.exe seems to be broken in terms of console output (as it has to be run in a cmd.exe shell and not just as a standalone executable) and you have to compensate for that.
That leaves you with redirecting the output to a file.  This saves it so your script can read this file contents and do whatever you originally intended by capturing OUT and you can monitor as the file gets updated.

Of course Windows has options for monitoring a file, powershell comes with "Get-Content mylogfile.log -Wait".  I would have thought most developers have Git for Windows installed and it does provide tail.

Now if download_sdks.exe can be fixed to print to standard out without all these cmd.exe console redirections than you can simply spawn powershell, run  download_sdks.exe , use a Tee-Object to log the data to a file and still see its output in the original cmake console. CMake can then read the log back in and do whatever processing you originally intended.

Now the GUI does seem to lag behind console output when hitting the configure button (at least when I tested it because I never used it before).  In this case you may want to spawn a separate console to see the output instead of waiting for the GUI to display it.  I'm not sure it's reasonable to expect the GUI stop button to kill a process it doesn't know about.  You just close the console window and the GUI recognizes that execute_process() was interrupted.  Not sure why you wouldn't consider that as acceptable.  So you would still need to capture output to a file to process it.

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Handling real-time output from execute_process

Person Withhats
Oops, forgot to add mailing list.

Also, here's some sample CMake output that shows some issues.
1) Carrige return doesn't replace previous line of output == tons of %'s
2) Perhaps cuz I'm forcing no buffer in CMake it causes the below extra 2 new lines on each line of output?? Should be condensed at least...

What I'm using to unbuffer:
class Unbuffered(object):
   def __init__(self, stream):
       self.stream = stream
   def write(self, data):
       self.stream.write(data)
       self.stream.flush()
   def writelines(self, datas):
       self.stream.writelines(datas)
       self.stream.flush()
   def __getattr__(self, attr):
       return getattr(self.stream, attr)

sys.stdout = Unbuffered(sys.stdout)
sys.stdout.buffer = Unbuffered(sys.stdout.buffer)



Downloading/Extracting SDKs...



Download URL: "XXXXXX"



Downloading 779738578 Bytes



65536 [0.01%]



131072 [0.02%]



196608 [0.03%]



262144 [0.03%]




On Fri, Dec 21, 2018 at 11:56 PM frodak17 <[hidden email]> wrote:


On Fri, Dec 21, 2018 at 11:16 PM Person Withhats <[hidden email]> wrote:
1) I want output in real time on a Windows system(no tail), and piping output to a file then reading it is a very poor solution. 'just check the contents in note pad'....
2) This is a general-purpose solution and not something I'm making just for myself, hacky fixes are not an option.


By default the command runs in the same console as cmake.
So if it is a typical executable that prints to standard output it will be shown with the rest of the cmake output.
Also the executable output can be captured to a file or variable instead of being shown.
What you can't do is capture it in a variable or file and display it at the same time.

When you use '"cmd" /c "start /wait download_sdks.exe"' this spawns a new console window that cmake knows nothing about.
This breaks the normal usage of execute_process() because "start /wait" creates an independent console to run download_sdks.exe.
cmake has no influence over this second console.  So while you can see its outputs you can't capture it in any way through cmake.

You get a return value of 0 in this case because the errorlevel from the second console isn't passed back correctly to the first console and then back to cmake.
A quick google search shows that this would work: COMMAND "cmd" /c "start /wait  download_sdks.exe & exit ^!errorlevel^!" assuming that download_sdks.exe actually returns any error codes.

You don't say why download_sdks.exe can't be run as a normal executable.
Nor do you say why you aren't using "start /wait /b" to keep everything in the same console as cmake is running in.
You said you want to see the output from the command but are clearly trying to capture the output in a variable instead but don't mention why.

I guess it's hacky but the tool download_sdks.exe seems to be broken in terms of console output (as it has to be run in a cmd.exe shell and not just as a standalone executable) and you have to compensate for that.
That leaves you with redirecting the output to a file.  This saves it so your script can read this file contents and do whatever you originally intended by capturing OUT and you can monitor as the file gets updated.

Of course Windows has options for monitoring a file, powershell comes with "Get-Content mylogfile.log -Wait".  I would have thought most developers have Git for Windows installed and it does provide tail.

Now if download_sdks.exe can be fixed to print to standard out without all these cmd.exe console redirections than you can simply spawn powershell, run  download_sdks.exe , use a Tee-Object to log the data to a file and still see its output in the original cmake console. CMake can then read the log back in and do whatever processing you originally intended.

Now the GUI does seem to lag behind console output when hitting the configure button (at least when I tested it because I never used it before).  In this case you may want to spawn a separate console to see the output instead of waiting for the GUI to display it.  I'm not sure it's reasonable to expect the GUI stop button to kill a process it doesn't know about.  You just close the console window and the GUI recognizes that execute_process() was interrupted.  Not sure why you wouldn't consider that as acceptable.  So you would still need to capture output to a file to process it.

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Handling real-time output from execute_process

Person Withhats
Cmake GUI output*

On Sat, Dec 22, 2018 at 11:03 AM Person Withhats <[hidden email]> wrote:
Oops, forgot to add mailing list.

Also, here's some sample CMake output that shows some issues.
1) Carrige return doesn't replace previous line of output == tons of %'s
2) Perhaps cuz I'm forcing no buffer in CMake it causes the below extra 2 new lines on each line of output?? Should be condensed at least...

What I'm using to unbuffer:
class Unbuffered(object):
   def __init__(self, stream):
       self.stream = stream
   def write(self, data):
       self.stream.write(data)
       self.stream.flush()
   def writelines(self, datas):
       self.stream.writelines(datas)
       self.stream.flush()
   def __getattr__(self, attr):
       return getattr(self.stream, attr)

sys.stdout = Unbuffered(sys.stdout)
sys.stdout.buffer = Unbuffered(sys.stdout.buffer)



Downloading/Extracting SDKs...



Download URL: "XXXXXX"



Downloading 779738578 Bytes



65536 [0.01%]



131072 [0.02%]



196608 [0.03%]



262144 [0.03%]




On Fri, Dec 21, 2018 at 11:56 PM frodak17 <[hidden email]> wrote:


On Fri, Dec 21, 2018 at 11:16 PM Person Withhats <[hidden email]> wrote:
1) I want output in real time on a Windows system(no tail), and piping output to a file then reading it is a very poor solution. 'just check the contents in note pad'....
2) This is a general-purpose solution and not something I'm making just for myself, hacky fixes are not an option.


By default the command runs in the same console as cmake.
So if it is a typical executable that prints to standard output it will be shown with the rest of the cmake output.
Also the executable output can be captured to a file or variable instead of being shown.
What you can't do is capture it in a variable or file and display it at the same time.

When you use '"cmd" /c "start /wait download_sdks.exe"' this spawns a new console window that cmake knows nothing about.
This breaks the normal usage of execute_process() because "start /wait" creates an independent console to run download_sdks.exe.
cmake has no influence over this second console.  So while you can see its outputs you can't capture it in any way through cmake.

You get a return value of 0 in this case because the errorlevel from the second console isn't passed back correctly to the first console and then back to cmake.
A quick google search shows that this would work: COMMAND "cmd" /c "start /wait  download_sdks.exe & exit ^!errorlevel^!" assuming that download_sdks.exe actually returns any error codes.

You don't say why download_sdks.exe can't be run as a normal executable.
Nor do you say why you aren't using "start /wait /b" to keep everything in the same console as cmake is running in.
You said you want to see the output from the command but are clearly trying to capture the output in a variable instead but don't mention why.

I guess it's hacky but the tool download_sdks.exe seems to be broken in terms of console output (as it has to be run in a cmd.exe shell and not just as a standalone executable) and you have to compensate for that.
That leaves you with redirecting the output to a file.  This saves it so your script can read this file contents and do whatever you originally intended by capturing OUT and you can monitor as the file gets updated.

Of course Windows has options for monitoring a file, powershell comes with "Get-Content mylogfile.log -Wait".  I would have thought most developers have Git for Windows installed and it does provide tail.

Now if download_sdks.exe can be fixed to print to standard out without all these cmd.exe console redirections than you can simply spawn powershell, run  download_sdks.exe , use a Tee-Object to log the data to a file and still see its output in the original cmake console. CMake can then read the log back in and do whatever processing you originally intended.

Now the GUI does seem to lag behind console output when hitting the configure button (at least when I tested it because I never used it before).  In this case you may want to spawn a separate console to see the output instead of waiting for the GUI to display it.  I'm not sure it's reasonable to expect the GUI stop button to kill a process it doesn't know about.  You just close the console window and the GUI recognizes that execute_process() was interrupted.  Not sure why you wouldn't consider that as acceptable.  So you would still need to capture output to a file to process it.

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake
Reply | Threaded
Open this post in threaded view
|

Re: Handling real-time output from execute_process

Person Withhats
In reply to this post by Person Withhats
Fixed extra spaces, was due to buffer (had to strip line endings). I'mma make a new thread this mess is my fault.

On Sat, Dec 22, 2018 at 11:03 AM Person Withhats <[hidden email]> wrote:
Oops, forgot to add mailing list.

Also, here's some sample CMake output that shows some issues.
1) Carrige return doesn't replace previous line of output == tons of %'s
2) Perhaps cuz I'm forcing no buffer in CMake it causes the below extra 2 new lines on each line of output?? Should be condensed at least...

What I'm using to unbuffer:
class Unbuffered(object):
   def __init__(self, stream):
       self.stream = stream
   def write(self, data):
       self.stream.write(data)
       self.stream.flush()
   def writelines(self, datas):
       self.stream.writelines(datas)
       self.stream.flush()
   def __getattr__(self, attr):
       return getattr(self.stream, attr)

sys.stdout = Unbuffered(sys.stdout)
sys.stdout.buffer = Unbuffered(sys.stdout.buffer)



Downloading/Extracting SDKs...



Download URL: "XXXXXX"



Downloading 779738578 Bytes



65536 [0.01%]



131072 [0.02%]



196608 [0.03%]



262144 [0.03%]




On Fri, Dec 21, 2018 at 11:56 PM frodak17 <[hidden email]> wrote:


On Fri, Dec 21, 2018 at 11:16 PM Person Withhats <[hidden email]> wrote:
1) I want output in real time on a Windows system(no tail), and piping output to a file then reading it is a very poor solution. 'just check the contents in note pad'....
2) This is a general-purpose solution and not something I'm making just for myself, hacky fixes are not an option.


By default the command runs in the same console as cmake.
So if it is a typical executable that prints to standard output it will be shown with the rest of the cmake output.
Also the executable output can be captured to a file or variable instead of being shown.
What you can't do is capture it in a variable or file and display it at the same time.

When you use '"cmd" /c "start /wait download_sdks.exe"' this spawns a new console window that cmake knows nothing about.
This breaks the normal usage of execute_process() because "start /wait" creates an independent console to run download_sdks.exe.
cmake has no influence over this second console.  So while you can see its outputs you can't capture it in any way through cmake.

You get a return value of 0 in this case because the errorlevel from the second console isn't passed back correctly to the first console and then back to cmake.
A quick google search shows that this would work: COMMAND "cmd" /c "start /wait  download_sdks.exe & exit ^!errorlevel^!" assuming that download_sdks.exe actually returns any error codes.

You don't say why download_sdks.exe can't be run as a normal executable.
Nor do you say why you aren't using "start /wait /b" to keep everything in the same console as cmake is running in.
You said you want to see the output from the command but are clearly trying to capture the output in a variable instead but don't mention why.

I guess it's hacky but the tool download_sdks.exe seems to be broken in terms of console output (as it has to be run in a cmd.exe shell and not just as a standalone executable) and you have to compensate for that.
That leaves you with redirecting the output to a file.  This saves it so your script can read this file contents and do whatever you originally intended by capturing OUT and you can monitor as the file gets updated.

Of course Windows has options for monitoring a file, powershell comes with "Get-Content mylogfile.log -Wait".  I would have thought most developers have Git for Windows installed and it does provide tail.

Now if download_sdks.exe can be fixed to print to standard out without all these cmd.exe console redirections than you can simply spawn powershell, run  download_sdks.exe , use a Tee-Object to log the data to a file and still see its output in the original cmake console. CMake can then read the log back in and do whatever processing you originally intended.

Now the GUI does seem to lag behind console output when hitting the configure button (at least when I tested it because I never used it before).  In this case you may want to spawn a separate console to see the output instead of waiting for the GUI to display it.  I'm not sure it's reasonable to expect the GUI stop button to kill a process it doesn't know about.  You just close the console window and the GUI recognizes that execute_process() was interrupted.  Not sure why you wouldn't consider that as acceptable.  So you would still need to capture output to a file to process it.

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake