Solved: Invoke a cygwin script from an asp.net web application and stream the output to the browser in real time

For back-end system monitoring like log analysis, I frequently use cygwin scripts to process data.  These tools are often useful for diagnosing issues or checking whether a problem could be specific to one server in a pool.  I’ve wanted to train more of my colleagues on using these scripts to do the same type of analysis, but it requires logging in to a server via RDP and knowing how to execute Cygwin scripts and interpret their output.

It occurred to me that if I just wrote an ASP.NET page that would execute the Cygwin process and parse the output (or output files), I could expose the scripts to the larger audience without the need for so much training or system access.  Since I had already worked out how to invoke a Cygwin script from a .bat file, I figured this should be easy, right?

Well, not so much, actually.  As it turns out, it is pretty difficult to invoke a .bat file from an ASP.NET application and be able to gather standard input/output/error logs.  I won’t go into the details here, but google on this a bit and you will see what I mean.  After a bunch of experimentation, I finally figured out the right set of parameters to kick of a Cygwin shell script by invoking bash.exe directly.


Process p = new Process();
 
string cygwinDir = @"c:cygwinbin";
string cygwinScript = @"/cygdrive/g/foo.sh"
p.StartInfo.FileName = cygwinDir + "bash.exe";
p.StartInfo.Arguments = "--login -c "" + cygwinScript + """;
p.StartInfo.WorkingDirectory = cygwinDir;
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.RedirectStandardError = true;
p.StartInfo.EnvironmentVariables["HOME"] = "C:";  // may not be necessary, depending on environment

The key things here are to invoke the “bash.exe” command directly, with the working directory in the Cygwin folder.  You then pass it the “–login” argument to initialize the environment and the path (in Cygwin directory style) to the script you want to execute.  The very last command creates an environment variable to specify where the HOME directory should be.  Before I added it, cygwin kept giving me messages about copying skeleton files (i.e. .bashsrc, etc)  and then having errors trying to access a non-accessible drive.  By setting the HOME variable, I was able to give it a place to find/copy these files, but you might not need it.

Okay, so far, so good… I am able to invoke the script.  But what about the output?

Some of my scripts can take a long time to run (10 or 15 minutes!) as they read and process data files in other locations, so I wanted to make sure that I was giving the user the feedback of seeing the scripts output as it executes.  This meant that as it generated standard output and standard error data, I wanted to grab it, write it out to the html response, and flush the buffer so that it showed up in the browser in real time.  I also wanted to make sure that the standard output and standard error data was displaying in the correct order, so that if there were any script issues, it was clear what errors were occurring and at what point.

This turned out to be much trickier.  Initially I was fumbling around with trying to process the standard output and standard error streams directly, but they kept showing up in the incorrect order.  Fortunately, one of my colleagues had faced a similar issue in a difference scenario and provided me with the solution.  It’s possible to set up an event handler function whenever some standard output or standard error data is generated, and these can then write the data to the response and flush.  I also used some simple CSS to color the standard error data red.

This way, as the script progresses, the output will appear in the user’s browser in real time, giving them a sense of progress and the opportunity to see errors.  Here’s the code:

 
    protected void Page_Load(object sender, EventArgs e)
    {
        Response.ContentType = "text/html";
        Response.Clear();
        Response.BufferOutput = true;
        Response.Write("<html><head><style>.stderr { color: red; }</style></head><body>Script Execution Beginning...<hr/><br/>");
        Response.Flush();
 
        // Insert code from the previous sample here to set up the Cygwin process invocation
        // ...
 
        p.OutputDataReceived += new DataReceivedEventHandler(OutputDataReceived);
        p.ErrorDataReceived += new DataReceivedEventHandler(ErrorDataReceived);
 
        p.Start();
 
        p.BeginErrorReadLine();
        p.BeginOutputReadLine();
 
        p.Refresh();
 
        p.WaitForExit();
        Response.Write("<hr/>Script Execution Complete</body></html>");
        Response.End();
    }
 
    //-------------------------------------------------------------------------------------------------
    void ErrorDataReceived(object sender, DataReceivedEventArgs e)
    {
        if (e != null && e.Data != null)
            WriteLogData(e.Data, true);
    }
    //-------------------------------------------------------------------------------------------------
    void OutputDataReceived(object sender, DataReceivedEventArgs e)
    {
        if (e != null && e.Data != null)
            WriteLogData(e.Data, false);
    }
 
    void WriteLogData(string data, bool isError)
    {
        Response.Write(string.Format("<div class="{0}">{1}</div>n",
            (isError) ? "stderr" : "stdout", HttpUtility.HtmlEncode(data)));
        Response.Flush();
    }

To test the code, I created a simple shell script that iterates five times, writing a message out and then sleeping for two seconds between each iteration.  If the script is working, I should see the page start to render the script output but pause every two seconds as each line is written out and then appears in the web browser.  I also intentionally put an error into the script so that it would write some data to standard error, and I could verify that it appeared at the right place and that my syntax highlighting worked.

Here is the script:


#!/bin/sh
echo "beginning execution of bash script $0"
echo "Invoke a missing command to generate stderr info"
notarealcommand
 
for i in {1..5}
do
   echo "Sleeping for two seconds, iteration $i  of 5"
   sleep 2
done
echo "all finished"

And it worked like a charm.  I invoked the page, and I saw the error appear, and then a new line of output would appear every two seconds, just like I would see if I were invoking the script directly in a command shell:

Image001

And when it was complete:

Image002

This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.

3 Responses to Solved: Invoke a cygwin script from an asp.net web application and stream the output to the browser in real time

  1. leRoy says:

    Wow, create script/post, I have a similar problem…
    I know this comment is a bit late but has a easier way come up ?
    or can I have source code on how you did this ?

    Thanks in advance

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s