Rory Tulk's Blog

Migrating

2010-02-26T09:19:00.000-08:00

Hi all.

I'm migrating my blog from here over to http://rorytulk.wordpress.com, soon to become www.rorytulk.com. Please adjust accordingly.

Software Testing Techniques, an Empirical Approach

2010-02-07T13:21:00.001-08:00

Proper software testing regimes are a cornerstone of effective software engineering. Progress has been made to teach students sound testing techniques, but improvement can still be made. My master's research supervisor and I conducted a study designed to empirically determine the difference in ability between student and professional software testers, and elicit from the experts behaviours or techniques which may be used to enhance undergraduate curriculum.

Our experimental setup consisted of in-lab observational sessions where subjects wrote thorough suites of JUnit tests for sample software we'd created. Subjects were drawn from the University of Toronto’s undergraduate computer science student body and professional developers from the Greater Toronto Area. The test code and video logs created during these sessions were examined for trends present in the student and professional groups.

Our intuition going into the study was that professionals would find more defects with their test suites, an advantage stemming from some metric such as number of tests written, lines per test, code coverage per test, etc. Analysis of these metrics did not confirm these hypotheses, however. Students and professionals performed equally well in terms of number of bugs found. However, student code contained more defects and, more importantly, the types of bugs found differed strikingly between the two groups.

Bugs in the sample code were broken down into two categories: stateless and stateful. A stateless bug is uncovered by inputting invalid values into a method invocation, and the method returns invalid results or throws an exception. A stateful bug occurs when a method call corrupts the object's state, and so subsequent calls perform incorrectly. Students found a mix of stateless and stateful bugs in the code, with a strong majority being stateless. The professionals sampled found strictly stateful defects. There are several possible explanations for this effect, although no evidence to support one over the others is immediately apparent.

The full text can be found here.

Left-Fold for Bash

2010-02-01T07:07:00.000-08:00

I'd like to share a recent bash programming experience I've had. It began while processing the reams of data generated in my M.Sc. research study. I was producing long lists of frequency data in text files, and had the need to sum up all the lines in these tables. This is of course a trivial problem in many languages, and I had a wide array of options available to me:

I could write another ant task to do the summing (this required more work that I was willing to invest, as ant isn't really suited for computational tasks)
I could write a python script that took the contents of the specified file and returned the sum. I didn't really like this approach because it involved yet another file in my build process, invoked from the ant task. I always sort of thought that if you were forced to use , you were performing a task beyond the scope of your tool.
I could skip the generation of the table and go straight to the sum. A few of the tables were created with XSLT, so this was a valid option. However, my XSLT programming ability is very much trial-and-error based, so I thought this might take some time. Also, some of the other tables, created with grep would not be affected.
Write it in shell. I liked this idea. I really liked the feel of being able to just pipe something to 'sum' and have the sum returned. So this is what I chose.

My first version looked like this:

#!/bin/bash

sum=0
while read line
do
sum=$(($sum + $line))
done

echo $sum
exit 0

It did the trick quite well. I had suggestions from office mates for the following alternatives:
Using python (courtesy Aran Donohue):

python -c "import sys;print sum(float(x) for x in sys.stdin.read().split())"

Using tr (courtesy Zak Kincaid):

cat numbers | tr '\n' '+'|head --bytes='-1'|bc
(Note that this version doesn't quite work. bc throws a syntax error. not exactly sure why.)

Using my original design, I realized that if I abstracted out the operator, I could use this script to perform any 2-operand function I wished on the list, essentially creating a basic left fold:

#!/bin/bash
op=""
if [ "$1" == "" ]; then
op="+"
else
op=$1
fi

sum=0
while read line
do
sum=$(($sum$op$line))
done

echo $sum
exit 0

I've done a little bit of error checking, to see if the parameter supplied is blank, and if so replace it with + by default.

I've only seen this work for '+' and '-'. If I use '*', it breaks because it replaces the wildcard/multiplication character with the listing of the current directory.

Unifying Visual theme

2009-11-03T09:46:00.000-08:00

A discussion this morning over coffee has prompted me to again consider creating a personal web presence, more than just a facebook or twitter page. A public, hosted site with data about my work and research. Inevitably, whenever I begin to consider this, I always get hung up on the visual theme for said space. In a perfect world, I would create a site in which the visuals closely match my personality, in a cool-hip sort of way, be cleanly unified across all the pages, and indeed could even integrate well into my local operating system's visuals (this doesn't serve any particular pupose, the idea of total uniformity just seems sort of cool to me). Anyway, looking at hosting and gnome-look.org for inspiration right now. Probalby won't lead me anywhere :P

Multi-User SVN on a Single Debian Account

2009-10-27T09:00:00.000-07:00

Just finished setting up a multi-user subversion repository, based on the instructions found here. The catch here is that the SVN server is running on a single server-side user account, and the multi-user support is done by multiplexing on the incoming ssh rsa key. Every key gets its own artificial user name, so we can track who has been doing what. The process was fairly straight forward, just backup your .ssh directory in case you bork something, like I did:P

Algonquin Park Map

2009-09-16T14:12:00.000-07:00

After doing some looking around for an electronic map of the canoe routes in Algonquin Provincial Park, I was only able to come up with the following :

http://www.cs.cmu.edu/~crpalmer/algonquin/map.html

It's pretty old, and the site hosting it looks like it may be somewhat unreliable, so I duplicated the file and made it available here for posterity:

http://www.cs.toronto.edu/~rtulk/alg2-1.pdf

In Praise of the TS&CC

2009-07-03T09:58:00.000-07:00

This past weekend, while enjoying a leisurely walk along the waterfront between High Park and Ontario Place, I came across the Toronto Sail & Canoe Club (TS&CC). I headed back there, by bike, last night, in hopes of getting some more information about the club. Within minutes of arrival I was on the crew list, and had spoken to a few skippers who were looking for crew. I ended up on a Beneteau First 235, which is a nice little racer. After 45 minutes or so of reconfiguring the race course, we were off! Just in time for a the storm front to get close enough to steal away all our wind! The race was uneventful, but relaxing, and I got to meet some new people, so it was an overall positive experience. I think I'll be going back next week for more of the same.

Oh, did I also mention that a Crew membership at the TS&CC is extremely affordable? Well, it is!

Creating Defects

2009-06-29T10:56:00.000-07:00

I'm in the process of inserting defects into three pieces of software I've written - for the purpose of creating testing samples for the study. This process is a lot more painful that I would have anticipated. I expect it is due to being trained for so long at removing defects, not deliberately creating them. Every time I break something, I realize the cool test case that will fail because of it, and then I feel bad. Also, trying to create non-obvious errors, or defects that are a little more human than those created by my java mutator, is challenging. For example, consider a class constructor like the following:

public MyAccount(int account, int initialBalance) { m_accountNum=account; m_balance=initialBalance; }
The mutator would do something like this
public MyAccount(int account, int initialBalance) { m_accountNum=account; m_balance=++initialBalance; }
following its set of one-line operator rules. I think I a more 'human' error would be something like this:

public MyAccount(int account, int initialBalance) { m_accountNum=initialBalance; m_balance=account; }
or this:
public MyAccount(int account, int initialBalance) { m_accountNum=account; }
All still valid java programs, but definitely erroneous, and certainly slips that an overworked developer could make. What sets them apart from some of the mutation bugs is that many of the mutation rules require more effort on the part of the developer, rather than less (ie. a ++ at the end of a variable manipulation command is more likely to be omitted than included by accident).

Think Aloud and Coding Analysis

2009-06-19T10:56:00.000-07:00

Chris pointed me at a paper by Mayrhauser and Vans "Identification of Dynamic Comprehension Processes During Large Scale Maintenance" that seems fairly relevant, in that they are using methods that align with mine so far. They've used a Think Aloud process and recorded participant actions while performing a maintenance change request. The activity took approx. 2 hours per subject (11 subjects. I think I can do better). Video and audio recordings were transcribed and coded. The authors posit that a) coding should be based on categories defined a priori (before the video/audio is recorded), and that b) Think Aloud does not work out of phase with the change action (thinking aloud after doing the task). This concerns me as a) I don't have a set of codes yet (I could certainly come up with some rather quickly, but they would be without significant justification), and b) I kind of liked the idea of the post-task interview.
These concerns aside, the data analysis in this paper is excellent. The authors code all the transcripts, and derive a set of patterns that the participants take while performing the tasks. These are formulated as finite state machines, in which each state represents a code. This, to me, validates their choice of codes. This may be a good model to follow for at least part of my analysis procedure.

Nielsen's Heuristics for Software Testing

2009-06-17T15:25:00.000-07:00

I had a quick conversation today with Dustin, of DGP and MSR fame, and he asked me if there was anything similar to Nielsen's heuristics for usability that might be used when looking for errors in code. There wasn't anything that jumped to my mind, but that certainly doesn't mean that nothing in fact exists. However, the first 10 hits for "software testing heuristics", "nielsen heuristics code", "nielsen heuristics software testing errors" didn't contain what I was looking for, either. It makes me think that the imagined output from my research study could be a valid contribution to knowledge. I think the list would probably contain things like:

always try negative numbered parameters
always try null values
how well does the .equals() method work?
add-remove-add to/from the collection, is it the same semantically?
always check date-based roll-overs

Simple Web Services are Too Hard to Find

2009-06-15T08:33:00.000-07:00

So I'm building some software to use as a System Under Test for my thesis experiment, and found several seminal examples in Paul Jorgensen's "Software Testing". Since I had already implemented the Simple ATM problem (well, a variant of it, without any GUI), and the Triangle problem, I decided that the Currency Exchange problem might fit well in between these two, in terms of complexity and size of code. Basically, the program takes in a value and source currency, and converts it to a destination currency of your choice (4 options in Jorgensen's text). This seemed a bit outdated to me, as it relies on the programmer hard-coding the exchange rates by hand. Boo-urns, I say! So, I figured I'd use some benevolent, free web service to pull down live exchange rates and use them in my program. This should be simple, I imagine, because a) this is the type of service presented in every tutorial on making web services, and b) it provides exactly the type of functionality for which web services are suited: some mysterious online entity that has a little glob of information that I want to use in my application. Google searching for such a web service turned up a disappointing lack of results. There are several online currency converters, of which I'm sure the reader is aware, but none of those are particularly machine friendly - I would have to hack up and throw away a bunch of HTML to get a single number out of the page. I encountered one service that was designed to be machine readable, but it used the bloated WS* XML web service stack, requiring me to auto-generate hundreds of lines of code, all so that I can read a single number (this is not an option for me, because I don't want my subjects attempting to write tests for a bunch of machine created JAX-WS goop). If that wasn't enough, I also had to apply for a Trial API Key which would allow me to access the single number I needed for a period of two weeks, after which I would need to purchase a commercial API license. Grrrrrrr! A subsequent search of "REST currency exchange web service" turned up bupkiss. Why is this so hard? There should be dozens of services like this. Maybe I'm just not looking in the right place.

Idea for a Meta-Study

2009-06-04T08:28:00.000-07:00

Idea for a meta-study: lots of papers have been published in which a controlled experiment is performed to examine the potential benefits of TDD. These are mostly all of the flavor: have control group implement some spec, using code-first-test-last, have experimental group implement same spec using TDD, measure time to complete and defect count of both groups. It seems (at least in my experience), that there are two possible outcomes from this, either the results are inconclusive, or they tend slightly towards the author's own feelings on the subject, either for or against TDD. I think an interesting, and probably easy to conduct, meta-study would be to pull down copies of all papers that are performing studies like these and see what the trend is, as well as analyzing things like geographic region of the authors or other relevant, although maybe not immediately obvious, correlations.

Iteratvie Thesis Development

2009-06-03T08:23:00.000-07:00

Yesterday I decided to try a new way of organizing my time. In 'the real world', summer time always correlated with reduced productivity of a development team, in part due to developers taking vacation time, but also supervisors being absent and leaving the team with a lack of direction. I have proactively given myself this direction by dividing my summer into 3 iterations, which coincide with the three remaining months of the season. In each iteration, there are four phases, in which I will examine and refine the methodology, data acquisition, and analysis of this thesis I'm pursuing, and in the 4th phase I will write up my results. I figure that by doing three iterations in this way, the pilot study this summer should give me a really good idea on how to run a successful study in the fall, as well as a head start on some of the write-up.

A User Study for Mutation-based Testing Analysis

2009-06-03T07:57:00.001-07:00

I recently read some material by Andreas Zeller in which he discusses the merits of using mutation testing as a method of verifying the quality of a test suite for a piece of software. These methods are meant to expose tests which perform an action, and assume that it was performed properly so long as an error is not thrown by the system under test - they do not verify that the action resulted in the desired program state. Code mutation (either on source or directly on the bytecode), when combined with accurate code coverage data, can identifiy these deficient tests by mutating the code they cover and observing tests that do not fail.

I believe that there is value in using this data, if it can be presented in the appropriate manner. If a developer has to spend an hour to generate the mutation report and cross-reference it with code coverage, then the investment likely outweighs the benefit. However, if I can arrive at my desk at 9am, and have in my inbox a build report, test report with coverage, and a mutation report for every branch, all of which are properly hyperlinked to each other and backed up on a central storage, then I would certainly use it. The problem here seems to be that this degree of automation and integration is hard to set up, and often times delicate when in place. It seems that some sort of standard platform for use by build engineers for integrating all of their reports, packaging operations, tests, etc, is called for. However, I have yet to see anything more sophisticated than Ant or Perl in widespread use by engineers. Maybe I just have an unrepresentative sample, though.

Testing Heuristics and Static Analysis at ICSE 09

2009-05-18T22:38:00.000-07:00

I had a very critical talk with two grad students from Queen's yesterday, whom I unfortunately can only identify as Jackie and Brahm. They had a difficult time accepting the immediate usefulness/significance of the study I was proposing, and that's fine, as they came up with a few useful suggestions that I can leverage, primarily to keep a narrow focus and always be aware of the result I'm trying to generate. Two interesting opinions they shared with me are below:

Are the good testers simply those who are most familiar with the SUT (software under test)? This seems entirely plausable (in my own work experience, I was a vastly better tester of sotware that I had written, or at least software written by someone on my team). If so, then the correlation between programming experience and software testing would be much weaker if we are inventing the SUT for our subjects to test. However, I believe to have some empirical evidence to the contrary, in the form of a man called Scott Tindal, who is one of the most skilled QA engineers I've ever had the pleasure of working with. Scott was of the caliber that he could almost single-handedly test every product in the OpenText/Hummingbird line. An interesting configuration of our study may be to experiment with developers testing their own code versus devlopers testing new code that they've never seen before.

Will the heuristics extracted with this study be the same as those already in use by static analysis tools like FindBugs?
It appears that determining a correlation between the techniques that experienced testers use and the heuristics employed by static analysis tools such as FindBugs could be an interesting topic. If there is no correlation between these two, may it be possible to integrate these into SA suites for any significant improvement? If there is a correlation, why haven't more testers adopted static analysis to aid in their efforts? Is it a matter of usability of SA tools? Is it that the formalism used to express the SA heuristics isn't understood by everyday testers, and so they aren't aware of what SA has to offer? I plan to talk to Nathaniel Ayewah, University of Maryland, who is working on the FindBugs team, about this further.

ICSE 09 - Day Three

2009-05-18T20:58:00.000-07:00

Andreas Zeller, professor at Saarland University in Germany, apart from being on of the nicest people I've met so far this week, also had some very positive feedback on my research idea. He agreed with it's principle, and that research in this area would be valid. One of his first questions after I finished presenting my idea was about incentive for participation. I thought he was asking about how we would convince students and pros to give up their time to participate in our study, but this wasn't the case. he recounted results found at Saarland when teaching students to test their code. Traditional methods obviously didn't work effectively, so they switched tracks. Student assignments were assigned with a spec and a suite of unit tests that the students could use to validate their work as they progressed. Once nightly a second set of tests were run by the instructor, the source of which was kept secret, and the students were informed in the morning of the number of tests they passed and failed. To grade the assignment, a third set of (secret) tests were run. Students begin with a mark of 100%. The first failing test reduces their mark to 80%. The second, to 70%. The third to 60%. The grading continues in a similarly harsh fashion. The first assignment in the course is almost universally failed, as students drastically underestimate how thorough they need to be with their tests. Subsequent assignments are much better, and the students focus strongly on testing, and indeed collaborate by sharing test cases and testing each others assignments to eliminate as many errors as possible before the submission date. Once the students have the proper motivation to test, they eagerly consume any instruction in effective testing techniques. Andreas found that simple instruction in JUnit and maybe TDD was all that was required, and the rest the students figured out for themselves. This type of self-directed learning is encouraging, but the whole situation makes me think that these students may be working harder, but not smarter, to test their software. It may be possible that by providing instruction not only in the simple operation of an XUnit framework, but also in things like effective test case selection, they may be able to reach similar test coverage or defect discovery rates, while expending less frantic effort.
Thought: drop the first assignment from the course's final mark, as it is not used for evaluation as much as it is a learning experience (of course, don't tell the students this, otherwise the important motivational lesson will be lost).
In addition to this, Andreas, just as other folks I've spoken to so far, emphasized the need to keep my study narrowly focused on a single issue, as well as controlling the situation in a way that enables precise measurements to be made both before and after implementing the curriculum changes we hope to elicit from the study, to accurately determine the improvements (if any exist). I am beginning to see a loose correlation between a researcher's viewpoint regarding empiricism in SE research (positivist on the right and constructivist on the left) and their amount of emphasis on narrowness of focus. Often times, those people who advocate exploratory qualitative studies also recommend wider bands of observation while conducting these studies. This is likely an over generalization, and I apologize in advance to anyone who may disagree with this.

ICSE 09 - Day Two

2009-05-18T20:14:00.000-07:00

Day Three of ICSE 09 was another great success! It began with a rousing keynote by Tom Ball of Microsoft Research fame. He described the early beginnings of version control and the emergence of mining techniques for these version systems, and continued the narrative through to the latest version of visual studio and the tool Microsoft calls CRANE, which suggests to developers area of the code which should be examined given that they are changing some other area. Pretty cool. I got to talk to him one-on-one about it over lunch.

On the previously blogged march through stanley park, I had a talk with Tom Ostrand from AT&T Labs in New Jersey. One thing I wanted to talk about (besides pitching my research idea) was leveraging the version history and reports generated from a test suite to predict bugs in target code using MSR techniques to augment the existing tools, such as mining deltas and bug trackers. This topic came up in the morning's MSR session, but aparently Tom missed it (making me look much smarter than can be measured in reality). He seemed interested in the possibility, but of course there are barriers to getting it going. Primarily, assuming that the dev team for the software we're analyzing is writing tests and putting them into the vcs, the error and coverage reports almost certainly aren't, making it difficult to to versioned history analysis over them. I thought that maybe, given the source code of the target software and the test code (both of which at some version), the MSR tool could build the code and execute the tests to create the needed reports. This will likely be extremely difficult to get going in the field, however, and make MSR mining, which is already an extremely expensive, long running process, even slower.

Farewell to the Borkenstocks

2009-05-17T22:40:00.000-07:00

The time has finally come to retire the Canadian Tire brand cork bottomed sandals, so lovingly referred to as the 'Borkenstocks'. After a season in the closet and a plane ride to Vancouver, the sandals got their first taste of summer today, as they accompanied me on a walk through Stanley Park. The results of this were as follows: blisters and uncomfortableness.

Feedback on my Research Proposal

2009-05-17T16:21:00.000-07:00

Over the last 24 hours I've had the chance to talk to a few individuals about my research idea. It seems to go as follows: I give my 30 second pitch, the marks stares at me blankly. Then, I elaborate. The mark gets it. They tell me first a problem they see, and then what they'd like to see out of the results. In a couple of cases, the mark got particularly excited about the outcome. I've summarized below.

Chris Bird
Chris has a strong background in empirical software engineering, with particular emphasis on qualitative exploratory studies. As such, the quantitative analysis aspects of my pitch fell on deaf ears, but he was extremely interested in the in-lab observation sessions I was proposing. He felt that 5 or 6 actionable recommendations that might come out of observing professionals would be invaluable. He suggested that I find an older Microsoft Research study in which they trained new hires by having them monitor a screen shared by a senior developer for some amount of time (probably a few days), and had the new hire ask questions with the senior at a later date, by reviewing the video logs. I like this idea because it allows us to elicit the information from the developer directly, instead of trying to infer it ourselves, but since we're not bothering them during the initial testing session, we wouldn't be affecting their performance. This isn't without problems, though. Primarily, there is the risk that the subject isn't always sure why they do the things they do, and so are likely to invent reasons, or invent foresight where none necessarily exists. Interesting idea, though. Also, should we interview students in the same way? On one hand, they students likely don't have any special insights that we can leverage (assuming they are less effective at testing than pros). On the other hand, it may illuminate any areas of misconception or misunderstanding which we could address in future curriculum changes.

Jim Cordy
Jim is a professor at Queens, and was my instructor in my 4th year compilers course. After he heard my pitch, he had a warning about an affect he had seen in his industrial work, and it comes from a generational difference in the training of developers. Developers who were trained more than 15 or 20 years ago had a delay between changing the source code and the results of program execution on the order of hours; new developers are used to delays on the order of minutes. Also, the current state of the art in debugging utilizes interactive debuggers, which were either unavailable or unreliable in earlier days. This has lead to 'old-school' programmers to a) rely heavily on source code inspection and b) insert enormous amounts of instrumentation (debugging statements) when running the program becomes required. In comparison, new generation developers often use smaller amounts of instrumentation, relying on quick turn around times to find the cause of errors. In Jim's experience, the old school programmers were orders of magnitude more effective (in terms of bugs found or solved per hour) than younger programmers. If this is in fact the case, it should be an effect we can see if we recruit subjects trained during this era of computer science research.

ICSE 09 - Day One

2009-05-16T16:39:00.001-07:00

We're wrapping up the first day of talks here at ICSE 2009. I've talked my way into the Mining Software Repositories (MSR) workshop. Here's a quick breakdown of some noteworthy points:

Keynote: Dr. Michael McAllister, Director of Academic Research Centers for SAP Business Objects
An hour and a half long talk in which he sells BI to the masses. Spoke a lot about integrating data silos, and providing an integrated, unified view of the data to business level decision makers. Also interesting anicdotes on how BI helped cure SARS. Forgot what OLAP stands for. Kind of concerning. This talk made me (after spending 2+ years working for a BI company) want a running example of what BI is and how it is used in the context of an expanding organization - from the point before any computational logistics assistance is required, and progressing forward to a Wallmart sized operation. Most examples I've seen start with a huge complex organization, complete with established silos, and then installs things like supply chain management, repository abstraction, customer relations management, document management, etc.

Mining GIT Repositories - presented the difficulties in mining data out of GIT (or DVCS systems in general) as opposed to traditional centralized systems like svn. Noteworthy items include high degree of branching and the lack of a 'mainline' of development.

Universal VCS - by looking for identical files in the repos of different projects, a single unified version control view is established for nearly all available software. Developed by creating a spider program which crawled the repos of numerous projects, downloading metadata and inferring links where appropriate.

Map Reduce - blah blah blah use idle PCs for quick pluggable clustering to chuck away on map reduce problems. Look @ Google MapReduce and Hardoop.

Alitheei Core - A software engineering research platform. Plugin framework for performing operations on heterogeneous repositories. Can define a new Metric by implementing an interface, and then evaluate the metric against all repositories in the framework. Look @ SQO OSS.

Many of these previous talks created tools for mining repositories, but with no greater purpose than that. When asked about this ('mining for the sake of mining'), none of the authors seemed to have a problem. The conclusion of the discussion was that this lack of purpose was a problem, and that the professional community should be surveyed to find out what needs they have for mining repos.

Research extensions/ideas:
The third MSR session today focused heavily on defect prediction. After showing off 3 or 4 methods of mining vcs systems to predict buggy code that improved prediction probability by 4% or 5%, the discussion boiled down to this, "What do managers/developers want in these reports to help them do their jobs?" Obviously, the room full of academics didn't have a definitive answer. One gentleman asked the question I had written down, which was "has anyone used the history and coverage of a software's test suite in combination with data from the VCS as a defect predictor (in theory, heavily tested areas are less likely to contain bugs)?" I found this particularly interesting. Also, I began to wonder, if we had one of these defect prediction reports, does it improve a developer's ability to find bugs, and if so, to what extent? Would it be measurable in the same way as I intend to measure testing ability with students and professionals?

Stay tuned for more info (and pictures of beautiful vancouver)!

AeroPress and Number Theory

2009-04-20T12:28:00.000-07:00

A warning to all who may use the coffee grinder in the SE lounge to make fine-grained coffee for use in an AeroPress: shaking the coffee grinder will blow the circuit breaker! It seems some of us don't realize that by applying rotating the axis of a spinning body, orthogonal to the plane of rotation, we are in fact applying a force against the angular momentum of said body. If this rotating device is powered by an electric motor, this causes the motor to draw more current to maintain its current speed. In short, don't shake the coffee grinder.

Also, I picked up the Annotated Turing again over the weekend, and read the first two chapters on number theory. I found this to be absolutely fascinating! Now, if you're already well versed in number theory, then these overviews may be redundant for you, but I thoroughly enjoyed it. Looking forward to what else is in this book.

Things on my Todo list

2009-04-19T09:54:00.000-07:00

My current list of things I want/need to do:

Reading:
Petzold, "The Annotated Turing"
Homer, "The Odyssey"
Huth and Ryan, "Logic in Computer Science"
Tennant, "Specifying Software"
"Software Architecture, A Primer"
Gorton, "Essential Software Architecture"
Dickens, "Oliver Twist"

Writing:
2125 end of term summary paper
2130 empirical study paper
conceptual model of REST process & motivation
ERB paperwork

Coding:
Instrumented JUnit and PyUnit
Diplomacy relationship analyzer
Qualitative Coding Application
iPhone app for navigation and aviation
Custom Braid mod - or just re-implement the engine with Java 2D and OpenGL

Misc:
Taxes
Health Insurance Claim
Personal/professional website, portfolio, and cards. Would like these to have a unifying visual theme.
Bribe people on craigslist for sold out Johathan Coulton tickets
Plan Carmen's bachelor party, 3 camping trips, and a houseboat rental scheme.
Paperwork for windsurfing class.
Compare & choose sailing clubs for the summer
Finish Bluenose

Obviously, this list is waayyy to long to be reasonable. I think you can see where the priorities should go, however (finishing ERB paperwork > reading Oliver Twist :P ).

REST in Django

2009-03-31T04:55:00.001-07:00

So I'm now officially a Google Summer of Code mentor for the Django open source group! w00t! Now all I have to do is pull off all those cool ideas I came up with earlier, as well as have a group of open source developers all agree that they are as cool as I think they are (which could be easier said than done).

While brushing teeth this morning, was thinking "I wonder what sort of empirical study I could pull out of this situation? What sort of research is there to be done in the field of REST? What can I learn, for academic purposes, not my own, from this experience?"

Ideas? I'm going to mull it over at the gym.

Google and Space and Time

2009-03-24T07:34:00.001-07:00

Has Google finally solved that whole space-time-bendy problem? Observe:

Reading Last Week

2009-03-22T22:21:00.000-07:00

Seaman: Qualitative Methods

Description of ways in which qualitative methods can be used in conjunction with a positivist stance. Tools include observational studies and interviews.
Interesting tradeoff: amount of data collected in an interview vs. amount of interviewee's time used vs. amount of direction in interview. Often, the importance/implication of the data collected isn't known for a long time after the interview.
When conducting an interview, stress that it is not an evaluation. There are no 'right' or 'wrong' answers.

Cohen: Statistical Power Analysis

Examines the importance of analyzing and reporting the power of a statistical relation in empirical research.
Author proposes that a sound target power be 0.80, as it produces feasible sample sizes for given values of alpha and ES.
Not a very understandable piece of literature, at least from my perspective.

Rosenthan and MiMatteo: Meta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews

A good introduction to meta-analysis (that is, analyzing the results of many studies/experiments to determine h0, instead of directly testing subjects to prove h0).
Interesting points about making sure the studies in your meta-analysis are independent (if meta-analyzing multiple studies from the same research group, subjects and or data may be reused, and so the results may be overlapping).
Also interesting discussion of inherent bias in meta-analysis, arising in the form in which the experimentalists choose to include/exclude studies from their sample space.

Card: Ender's Game

Enjoyable novel about a young boy who is called upon to train as a military commander to protect the earth from the threat of alien invasion.
Good character development, although the author lays on the bloodlust and homo-eroticism a bit thick.
As a consequence, the sci-fi aspects of the story seemed secondary to the character plot, even tacked on in some places.
Generally, I liked it, but probably wouldn't invest the time to read the 5 or 6 sequels

Modeling Topic

2009-03-22T08:39:00.000-07:00

After a couple months of soul searching, the students enrolled in John Mylopoulos' Conceptual Modeling course are narrowing down ideas on what to model. The requirements of the project (to 'model something') left it fairly open, but perhaps too open to be immediately tractable. However, Michalis, Alicia, and I are closing the gap. I'm leaning toward modeling certain aspects of the REST web service framework I've been working on for the last little while in CSC 2125.

Obviously, system descriptions and class diagrams are uninteresting, as John pointed out earlier in the term, but modeling the requirements of what such a system should do are not. I'm looking into Tropos, but not really sure if it applies. However, we can probably do a goal model illustrating the motivation and principles of REST, contrasted with those of ws*. Also, a use case diagram describing the interface to a general ROA service could be created. Finally, we could model certain pieces of important logic, such as a delayed get, using a description logic syntax.

Reading Last Week

2009-03-22T07:25:00.000-07:00

Sim et. al: Using Benchmarking to Advance Research: A Challenge to Software Engineering

Argues the merits of creating benchmarks in software engineering as an exercise to strengthen the community and promote advancement, using the reverse engineering community as an example.

Lau: Towards a framework for action research in information systems studies

Proposes a framework with which action research efforts can be categorized and evaluated.
Describes Action Research as an iterative process, in which a researcher introduces a small change, observes its effect, and uses it as input to the next small change.
Reminded me of Agile. I wonder if there are any other lessons from Agile that we can apply to Action Research?

Taipale and Smolander: Improving Software Testing by Observing Practice

Case study conducted to shake out some ways of improving software testing, where it is deemed to be lacking. Methods used include subject interviews and grounded theory.
Authors found that testing practices were most strongly correlated to business processes.
Thought this could lend some insight into how to observe testers at work (ie. as they write tests). No such luck, though. All recommendations for improving testing had to do with business process alterations/improvements, not hands on testing stuff.

Also, while glancing at my bookshelf, I came across a couple of old undergrad texts that I would like to glance through. By looking at the spines, I don't think I've ever opened them:

Logic in Computer Science by Huth and Ryan. This was the text for my computational logic course in 3rd year. The course notes and instructor were good enough without having to read this, but my propositional logic has become so rusty, I think I need this as a refresher.

Specifying Software by Tennant. Text for a formal methods course. Turing machines, model checking & verification, etc.

Also, my Amazon shipment arrived a couple days ago, bringing with it a copy of The Annotated Turing, by Charles Petzold, and O'Reily's Programming Erlang (this one is for Aran, but when we're both done with our purchases, we'll likely swap).

Testing Tools

2009-03-19T07:13:00.000-07:00

UTest
Implements a reverse test oracle (submit tests to black box piece of code). Unsure of level of functionality. Also has an eclipse plugin. Mentions sandboxing of code being run. Candidate for virtualization efforts I've been looking at.

WebCat
An online grading system developed at Virginia Tech, in which students submit assignments and have the instructor's test suite run against it. Use of this system was found to encourage test-first development practices among students, as well as early assignment submission (thanks to a hint system). Impossible to install, however, unless you are Stephen Edwards, and even then only on alternating weeks.

Marmoset
System for snapshot collection and automated testing. Using Marmoset, researchers can easily gather detailed information about students development patterns, as an Eclipse plugin checks in all code changes to a central version control repository, which can be mined. Also, Marmoset provides automatic test feedback to students, which they can use during development of an assignment, the goal of which is to improve their experience while learning to program. It is unclear whether or not these tests are also used for [semi]automatic grading.

JUnit
Although I haven't found any evidence of it yet, I'm pretty sure some combination of junit and the java remote debugger can be used to create a quick and cheap reverse test oracle. More digging required.

ORM-REST Code Sprint - Day 2

2009-03-17T11:19:00.000-07:00

5:00 pm, day two of the code sprint is almost wrapped up. Not as much amazing progress today as I would have liked (I was minus one team member for some reason). That aside, here's what we've got:

Mohammad blocked out and implemented the pseudocode for the URL reverser described in our blog here. I took a further look at it, and filled in the magic that inverts the django url list and gives us a url from a view name and primary key value. Yay! Plugged it into the xml serializer, and voilla! Rest-like xml representation, with hyperlinks! The only thing missing from this bit is the 'http://hostname:port' part. From past experience, I've found this to be trickier than you might think (gets hairy if you've got one web server feeding into another, or a proxy/load balancer in the way). I think we'll try just using relative URIs for now.

After a couple more feature points are implemented, this thing needs a huge refactoring pass to clean it up and encapsulate it. Also, it uses some classes from the Django Rest Interface, but this library has some pathological faults that I want to not include in the tool. Yay for open licensing.

Also, discussions with Aran produced some new feature proposals. Lots of useful, tiny, easy-to-implement things that will improve the overall RESTability of the library. Further yay!

Automatic URL localization

Automatic anything localization

Model introspection and url pattern creation

Computed resources instead of data resources - make http interface automatic

ORM-REST Code Sprint - Day 1

2009-03-16T13:46:00.000-07:00

At quarter to 5:00, the first day of the ORM REST code sprint is winding down. Mo' and I hacked from 2:00, and this is what we've got to show:

Rory finished one direction of proper xml serialization of Django models. That is, given a [list of] model instance[s], we get either a nice xml document (unlike the object name="" pk="" garbage we had before), or a concise list of objects with names and placeholder URIs, which will be changed to live urls when Mo' gets his piece working.

Mohammad synchronized with the svn, set up a django development environment, and familiarized himself with the code I had written. Following this, he began work on the reverse URL mapper. Given a model classname and a primary key value, he's pulling a live instance from the django ORM, and using it to query the Django URL dispatcher. This gives us the regular expression which will match URLs to access the specified object. Now he's got to turn the whole thing on its head! Good luck Mo! You can do it!

So, if you're keeping score, we are 1/2 + 1/2 = 1 feature point finished, out of 4. Might just make it by end of term :)

CSC2125 Live

2009-03-16T10:15:00.000-07:00

My first live-ish blog. Hope it turns out well.

Class in the GSU pub

This class we did a quick series of elevator pitches by each group, as a way to practice their presentation skills. The first few groups were pretty good. Subsequent groups had to take two or three tries at the pitch.

Mohammad and I both had to present. I got to stand on a chair because I'm too short and Greg likes to pick on me:)

The bar will not make Irish coffees.

The last week of classes is in three weeks. The demo day is supposed to be the monday after that, but that is easter. This demo day will be moved further into the future, either some other time that week, or the following monday.

Greg's token plot twist: do another lap of elevator pitches, but this time for thesis work instead of 2125 project. Grad students have to explain their topic, undergrads need to come up with a thesis idea on the spot. 2 minutes prep time.

My topic: Studying the effects of integrating unit testing into standard CS undergrad programs. Questions: can be determine a measure for unit test effectiveness, can we track testing improvement over course of career, and is RTO useful?

I got cut off after my 20 seconds :( Only had like 3 words left.

Everyone's job outside of class is to come up with a thesis topic for nick. The winner gets ice cream.

I'm willing to bet we're going to go around again to try to pear things down. I don't know how to modify my speech, though.

Yup. This time I was too short.

Now we're talking about consulting fees. Greg charges $150/hr plus expenses. Clearly, the undergrads have no idea how much to charge. They've never seen the Entrepeneurship 101 talk about this. You should discount a yearly salary for an employee doing the same job you're shipping out, and normalize it over the length of the project.

If you get some consulting work on the side, U of T Legal Services can look over any contract you may have. It's free, but not speedy (couple of weeks). See Jason Betcham.

Combined degree CS + Law = name your price. Too bad law is dreadfully dull.

Next round of interrogation: what special skills do you have that cannot be easily picked up by other CS grads? I don't think I have one. Maybe experience in ECM?
Speaking another language is a big one!
Also, having a well connected professional network is important.

Last question: if you don't have something special, what are you doing to fix that?

Go get Garth Gibson's PHD thesis: how do raids work? Really good communication.

Pay attention to email for info on last class.

Zak is the worst person! The worst!

2009-03-13T09:20:00.000-07:00

Muller and Pfahl: Simulation Methods

Chapter describing the way in which simulation can be used to project the outcome of a software project.
Most readers found this method to be too clunky, or simply inappropriate for software development estimation. The counter example of embedded or safety critical systems seemed to sway a few minds, however.
Interesting discussion about whether this actually qualifies as an empirical method. Also, everyone seemed to agree that what the Hadley Center is doing is valid science, even though it is simulation.

Atkins, et al: Using version control data to evaluate the impact of software tools

Paper evaluating possibly the worst version control system ever! At a more meta level, it was an example of how you can run an empirical study who's sole input is data mined from a past project (similar to what Samira did for her master's).
Nick mentioned that, despite its archaic premise, a versioned editor like this one would have been helpful at EA.
Discussion ensued as to whether this type of validation was actually required for this tool. It seems almost anything would be better than the existing 'version control'. In fact, there are some in the field who feel that expert intuition is ultimately more useful than empirical experimentation.

Sharp & Robinson: An Ethnographic Study of XP Practice

Ethnographic study of an extremely well-oiled XP team in england
Study found that, in this case, XP was the style best suited for maximal performance of the team
Threats to validity include not spending enough time (one iteration?) with the subjects

Kitchenham & Pfleeger: Personal Opinion Surveys

Chapter describing the process of creating and administering personal opinion serveys (questionnaires and the like)
Primary message is: making a questionnaire isn't easy! There's lots of confounding effects/sources of bias to worry about.
Interesting discussion ensued concerning the reuse of standard instruments from psychology, and whether or not SE should have similar standard instruments.

Cherubini et al: Let's go to the whiteboard: how and why software developers use drawings

Interesting case study conducted by Microsoft Research to see how their developers use graphical representations of code
Researchers were able to categorize their uses into Understanding, Design, and Communication, and the amount of investment into Transient, Reiterated, Rendered, and Archival.
Pretty good

Flyvbjerg: Five Misunderstandings about Case Study Research

This paper attempts to disprove several common misconceptions about case study research, primarily things like "case study results cannot be generalized to a larger population", "case studies cannot be used to test hypotheses".
A fairly good piece of advocacy. It certainly makes me feel better about considering a case study as a direction for my research.

Edwards: Using software testing to move students from trial-and-error to reflection-in-action and related papers

Details findings of the WebCAT system - an online assignment submission and automatic grader created at the Virginia Tech.
Edwards found that the system was useful and well received by both instructors and students. The primary objective, encouraging students to do test-first development, was achieved.
Interesting effects of introducing hints into the automatic test cases to discourage last minute submissions.

Juristo et al: Reviewing 25 Years of Testing Technique Experiments

A taxonomy/summary of the various means of divining test cases that have been invented over the last quarter century.
Focuses mainly on machine-derived cases (random input-output samples, etc), doesn't focus too much on human-created unit tests, unfortunately.

RESTful Efforts in Django

2009-03-12T14:04:00.000-07:00

My CSC2125 team appear to have concrete direction in what we're going to build for end of term. We've been building prototype REST web services, relying on ORM data, for the last few weeks, most recently in Django. Since Django is so closely integrated into the community here, we're going to use that as our target platform.

It seems that both myself and Bill Conrad started with the Django REST Interface, a Google SoC project from 2007. This interface takes your Django Models and throws a quick and dirty xml interface ontop of them. Functional yes, but not really complete (or even good REST). However, since this interface has some problems, they're just aching for me to fix them!

First, the xml/json/yaml returned from the interface is the standard Django serialzation format, which isn't very pretty for REST purposes. It can be cleaned up.

Most importantly, I think, is that the inter-object relations are expressed as sets of primary keys, instead of URLs to related objects. This flyes in the face of REST. It will require something akin to reversing the urls.py map to get a URL from a Model and primary key. Non-trivial, interesting, and crucial to the correctness of the resluting service.

A few other nice-to-haves include implicit delayed GET, algorithmic(query) resources, and dynamic representations.

I spoke with Bill today about his work on the Basie REST API. He seemed convinced that he solved most of what I'm after already, with the notable exception of reverse URL mapping. The REST blog post on the Basie Blog mentions the following features:

Generic Models - looks like Django Models will be turned into REST resources automagically, a la the Django REST Interface. Also, the Basie team had need of a deep synchonization of objects, and so added this to their REST API. While I'm sure this fulfulls their requirements, according to Robert Brewer via Greg's Blog this is a RESTful no no.

Intuitive Data Access - discusses url structure. There's mention in this section of an algorithmic (Bill calls them filtered) resources. If this is done, then that's one item off my list!

AJAX Friendly - not really interested in this, but good on ya.

The impression I get after going through this (and I'll admit, I haven't looked at the Basie codebase yet, but its next on my list), is that there has been effort in the area I want to pursue in 2125, but not to the extent or in the specific details I intended. Yay! Project still holds validity!

Code Monkies and the Recipe for Happiness

2009-02-18T12:10:00.000-08:00

Hanging out in my office today. Being moderately productive, but in an inexplicably good mood. The apparent recipe for happiness is as follows:

Purchase a ridiculously expensive bagel
Coffee
Install shiny new operating system on laptop
Meet with supervisor
Coffee
Create UI mockups for hypothetical testing tool
Listen to 'Code Monkey' by Jonathan Coulton a few hundred times.
Coffee

I like this song. It gives me this mental image of developers as knuckle-dragging primates. Especially the way Coulton removes all articles from the lyrics of the song. Ex. "Code monkey get up, get coffee." And after all, if you can't laugh at yourself, at whom can you laugh? (sentences also can't end in prepositions). A couple excerpts:

"Rob says Code Monkey very diligent, but his output stink. Code Monkey's code not functional or elegant. What do Code Monkey think? Code Monkey think maybe manager want to write god damn login page himself? Code Monkey not say it out loud. Code Monkey not crazy, just proud!"

"Much rather wake up, eat coffee cake. Take bath. Take nap. This job fulfilling in creative ways. What a load of crap."

Saxaphone

2009-02-16T09:19:00.000-08:00

This is for the numerous saxophone players I know:

http://www.youtube.com/watch?v=RXyC7S-4LR8

Security

2009-02-10T13:17:00.000-08:00

After a conversation with Aran about the U of T library site possibly being vulnerable to sql injection attacks, we came up with the idea that a really really cool name for a book on internet security would be

"); DROP TABLE Books;

Not only is it funny, but is very existence would serve to test the security measures of countless online bookstores and libraries across the globe. Most valuable internet security book you'll never have to read.

A Long Train of Thought with One (1) Cool Idea and Several Tangents

2009-02-05T12:26:00.000-08:00

So I've been complaining for months now that I'd like an operating system, or just a desktop manager, that was similar to the kind of thing you've seen in movies like Swordfish and Hackers. Some eclectic, that looks really frigging cool, and allows me to do the kinds of operations that I want to do, quickly and easily (Note to you UX experts out there, I am well aware of the fact that interfaces that look cool aren't usable. I'm not going for market appeal, this is a totally custom job). It seemed obvious that the only way to get this was to build it myself. So, I started small, thinking "What kind of operations/features would I want on a small, portable device, like a Netbook or EEE PC?" (Note: I decided I wanted an EEE PC when I saw Richard Stallman speak earlier this week. It was the only thing I really liked about his talk.) A simple window manager (with really flashy graphics), file system navigator, browser, and IDE. That's pretty much it. Oh, and the whole thing should be built on top of some flavor of Unix, so that I can still use 3rd party apps, etc. Ambitious project, I know, but I'm just fantasizing here. Also, while I was daydreaming about this, I decided to restructure my personal computing setup, but that's another story.

Anyway, I figured I'd start with the IDE, since I had always wanted to make one. This specific idea came up over the summer while I was at work. I really liked the coding features in IntelliJ, but wished it fit better with our continuous integration infrastructure. I came to the conclusion that the best IDE would have the most notable coding features from IntelliJ, but be transparent enough to allow you to plug in whatever tools you might already be using: SVN/Perforce, any JDK, any Ant, etc. Before you start saying "But Rory, Eclipse does ..." or "But Rory, NetBeans blah blah blah...", one of the strongest motivating factors for this idea was that it sounded like a lot of fun, regardless of wheter the requirements have been fulfilled by something else. Also, these heavyweight IDEs are just that, too heavy. For me, their feature set can be at times too full, and having a single application that consumes > 1GB of memory seems a bit silly, when all it really has to do is edit text files and invoke some commands from the system prompt. Also, keep in mind that I enlisted (is that the right word?) in grad school to do just this: redesign developer tools.

I mentioned this to Zak, to which he replied "It sounds like you just need to learn to use Emacs properly." That also doesn't sound like fun. I mentioned this to Aran, to which he replied "Oh my god, me too!" Yay! Someone else wants to make and IDE. Except, Aran's IDE is a Javascript IDE. Written in Javascript. That runs in a browser.

My first instinct was that this idea sounds rather boring. Then I thought more carefully about it. Google has browser based versions of all the other applications you could use on a daily basis: mail, word processor, spreadsheet, image editor, etc. Why not a browser based IDE? Are the UI controls available in a browser less expressive than those on a native client? Probably not.

Now, I had originally thought of this IDE as running in the browser, operating on local files, etc, but what if it were more closely integrated with the web? What if it were a component of an online software portal? It would automatically know which source code repository you're using. It could automatically update documentation (wiki's). It would have strong integration with the portal's bug tracker. Imagine it, you're favorite portal would have a "Code" tab in addition to "Wiki", "Docs", "Browse", "Mail", etc, at the top of the page, and when you clicked it, everything was configured and ready to go.

What can be get if we utilize the cloud for some of the processing, instead of relying on the browser to be the engine of this fantastic IDE? First off, my IDE no longer consumes > 1 GB of memory. What else? Rendering of the controls would still be done on the client, but can the cloud be used for more interesting problems, like static analysis? Anything that could benefit from a bit of parallelization is a good candidate for migration.

Could this include running unit tests? Should the compiled code be run on the server, or in the browser? Security concerns say that it should be run on the client, but if it were run on the server it could possibly be done faster, and run in multiple browsers in multiple environments instead of just the ones installed on the client. To protect security, we could run the build and execute the code in a virtual machine, like a SnowFlock VM for example. Now, as for unit tests, executing these tasks are extremely parallelizable. We could fork a vm for every test, run them all at once. Huzzah!

I may have to rewrite this in a more concise form :P

Also, Aran mentions Heroku and AppJet, which are similar to this.

BitWhat?

2009-02-04T06:56:00.000-08:00

I've noticed that a lot of the tech blogs I've been reading have titles that include creative uses of the word 'bit'. Ex. The Third Bit, BitWorking, Bitgistics, etc. Now, don't get me wrong, the authors and opinions presented are insightful, but how often does a python programmer, for example, worry about bits? Shouldn't these catchy names use higher level concepts, like "Objection", "DataDemon", or "Quine 'n Cheese". Bleh, just my opinion?

Reading Week

2009-02-03T12:28:00.002-08:00

While thinking about which papers/books I should consume over reading week, I came to the conclusion that what I would really rather do is finish up the planking on my scratch-built Bluenose. This was a project that I started a little over a year ago, and boxed when I moved to Toronto. My plan was to finish it before christmas, what with all the spare time I'd have as a lazy grad student. Turns out, things are a bit different than I originally estimated, and the Bluenose has stayed in the closet for the last 4 or 5 months.

The hull was carved from a single 6"x6"x48" Basswood timber, with additional pieces carved/shaped/cut from mixes of pine, balsa, and basswood. The false-underdeck is pretty much done, just needs a bit of sanding along the rails, then I get to start planking the deck. I estimate I'll spend most of the first day figuring out the scales & sizes I chose for everything. In hindsight, I wish I had written some of this stuff down, but I think I'll manage.

RESTful Questions

2009-02-03T12:28:00.001-08:00

While working on my 2125 project, my partner and I created a quick little RESTful web service using CherryPy, and SQLAlchemy for persistence. SQLAlchemy worked wonderfully. CherryPy did a great job of making data-driven web pages, and the MethodDispatcher made it easy to invoke certain methods within a class when an http request comes in, based on the http method. This seemed almost ideal for REST, but some clunkiness in the design prevents it from being really what we're after.

What are we after, exactly? We're trying to find ways in which we can avoid duplication of effort when using both Object Relational Mappers and RESTful Web Services. In their book "RESTful Web Services", Richardson and Ruby hit on the point that the process of translating objects into REST resources is very similar to the process of translating the same objects into tables in a relational database. So, if a web service is storing objects in a database and exposing them via a rest api, we would be doing the same sort of mapping procedure twice.

My partner (in crime?) and I had a chat with Greg about this, and came up with some questions to investigate. Below are the questions and their answers:
How do REST APIs represent foreign-key relationships (ie. object aggregation)? Specifically, are references to the other objects stored/returned, or the entire object on each request?

It is common REST practice to return hyperlinks to other objects/resources that are aggregated by the given resource. This would require an additional http request for each referenced resource.

Can we uniquely identify object instances (REST resources) based on some identifier?

Yes. The resource's URI is its identifier. Every resource has one that identifies it. However, it is possible for one resource to have many URIs that point to it (ex. /releases/2_05 and /releases/latest could be the same thing).

Can we cache REST objects on the client side, based on their identifier (whatever that may happen to be)?

Yes. It would be silly not to. However, the multi-identifier problem stated in the last answer might make this less efficient.

If we assume that the meat of the service is some object graph (probably a DAG), can we reconstruct the graph on the client side, out of stubs instead of actual objects, given identifiers and caching?

I think so.

Thats all for now. Check here or on the project wiki for more information coming soon!

What I've Read This Week

2009-02-03T12:16:00.000-08:00

Singer, J.; Vinson, N.G., "Ethical issues in empirical studies of software engineering," Software Engineering, IEEE Transactions on , vol.28, no.12, pp. 1171-1180, Dec 2002
URL: http://www.ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=1158289&isnumber=25950

This paper presents a handful of ethical dilemmas that researchers who conduct empirical studies can get themselves into, along with advice on getting out or avoiding the situation all together.

What kings of studies could be create which contain no human subjects, but in which individuals can be identified (ie. from their source code)?
When can an employee's participation in an empirical study threaten their employment?
Is it possible to conduct a field study in which management doesn't know which of their employees are participating?
Should remuneration rates be adjusted to compete with a standard software engineer's salary?
Are raffles or draws valid replacements for remuneration? Does the exclusivity of the compensation (ie. only one subject wins the iPod) affect the data collected by the study? Will subjects 'try harder' in the task assigned if they think they may win a prize? Can prizes affect working relationships/situations after the researcher has left?
Does ACM Article 1.7 eliminate deceptive studies?
Regarding written concent/participation forms, does having a large number of anticipated uses of the data detract from a studies credability, and thereby make subjects less likely to participate?

John P. A. Ioannidis, "Why Most Published Research Findings Are False"
URL: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1182327

This paper describes a detailed statistical method (proof?) illustrating evidence that the majority of research papers published in this day and age go on to be refuted in the near future.

What is the 'power' the authors are referring to?
Is corollary 5 (corporations sponsoring research supress findings that they deem unfavorable for business reasons) just plain evil or misleading?
Null fields sound interesting. How do I tell if I'm stuck in a null field?
How do we determine R for a given field?

M. Jørgensen, and D. I. K. Sjøberg (2004) "Generalization and Theory Building in Software Engineering Research"
URL: http://simula.no/research/engineering/publications/SE.5.Joergensen.2004.c

Null hypotheses are a tell tale of (sometimes misused) statistical hypotheses testing. Should we as readers be concerned when we see clearly stated null hypotheses?
In their recommendations, the authors suggest that purely exploratory studies hold little or no value, given that vast amounts of knowledge concerning software engineering has been accumulated in other, older fields such as psychology. Although I agree that cross-disciplinary research is useful for SE, and many old ideas can be successfully applied in SE, I'm not sure I agree that there is no use in exploratory studies.
Proper definition of populations and subject sampling is important
It is difficult to transfer the results in one population to another. The most common example of this is performing a study on CS grad/undergrad students and expecting it to transfer to professionals. Is there any way we as CS grad students can perform studies that will be relevant to professionals, then?

Still working my way though RESTful Web Services. Just wrapped up the author's definition of ROA (resource oriented architecture). Very interesting. Hopefully this answers some questions brought up by my 2125 project.

Also on the stack are this paper about the Snowflock VM System and A Software Architecture Primer.

And, if there's time, I'll try to finish Ender's Game.

Calling all Build Engineers

2009-01-22T15:37:00.000-08:00

During my recent CSC301 lecture, I was surprised to find out how few students were aware that they could actually get a job as a full-time build engineer. It makes me think that there's probably a study to be done in this area.

To what extent are build processes and project maintenance discussed in CS education?
What is the market value of a professional build engineer?
How to build engineers perform their job? Is there a way it can be improved?

The last point is something that causes me mild concern. It seems that, in some cases, the methods used to set-up and execute project wide builds can be semi-structured voodoo.

Safe Server-Side Unit Testing

2009-01-22T14:22:00.000-08:00

I like build systems :) My first experience with integrating a vcs, bug tracker, and ant was a very fulfilling experience, and it only got better when we added things like EMMA to give developers a feel of how their project was progressing. So, you can understand why my ears perked up when, during a conversation about the SVN setup in Dr. Project/Basie, Greg mentioned that they had tried to incorporate a continuous integration routine into Dr. Project, but failed, citing complexities and difficulty with the administration. Now, being the cinical, cold-hearted person that I am, my first thought was, "You clearly need better administrators", but then I remembered trying to do something similar with VMWare, and how rediculously hard it was to get it working, and once it was, keeping it there was almost impossible, so I held my tongue.

The basic premise here is to have the server which runs the Dr. Project/Basie installation also manage a system of virtual machines. When code is checked into the SVN repository for a given project, a virtual machine is spawned. Inside this VM, we download a copy of latest revision from the SVN, build it, run the unit tests, generate the reports, publish them, then kill the VM. Obviously, we can't do the build and test by just forking a process, without the VM, because that would allow the project groups to run arbitrary code on the Dr. Project web server, which is just about the biggest security hole I can think of. So, the goal here is to utilize the virtual machines to completely isolate the code from the web server, so that the tests are run in a completely safe environment, and at the same time providing benefits like strictly reproducible execution environments (every unit test starts from the same vm snapshot).

To accomplish these goals, we're looking at using the SnowFlock system. All vm's start from a master image, clones are quick to create (~100msecs), we can instantiate many, many clones at the same time, and the whole thing is wrapped up in a nice little Python API.

It will be interesting to see if this works for Dr. Project/Basie's needs, and if it does, I'd like to see if it could be extended to do cluster testing for larger distributed systems projects. The ease and speed of creating a new clone vm means that for each test, a small cluster of machines could be created, the test run, and torn down. I'm not sure if a tool like this exists already, it sounds like a fairly straightforward idea, but should be fun to investigate either way.

ORM Mapping for Web Service Definition

2009-01-22T14:11:00.001-08:00

This post is an experiment with the Blogger/Google Docs interoperability functionality. I'm not terribly impressed with the quality of the translation between doc and blog post. If you're as disgusted with the layout as I am, feel free to read the google doc here. This is a document describing an idea Greg proposed to me and another student in his CSC2125 class. In once sentence, the problem is : Can we use the object mapping definition from an Object-Relational Mapping tool to describe objects/resources in a RESTful web API, and in so doing leverage some of the benefits ORMs have lent to persistance, reduce redundancy, and generally make people happier? Unfortunately, the more I look into it, the more I think the answer is 'no'. However, we're not quite finished the investigation yet.

ORM Mapping for Web Service Descriptors

Traditional Situation

Figure 1 illustrates the traditional deployment situation for a client/server application, in which the client communicates with the server via an exposed web service, and the server persists data into a database using an object-relational mapper. The server-side process consists of a layered architecture, in which the business logic interfaces with the database via an ORM mapping layer. The application logic stores its data in the form of objects (hence the ORM), and a mapping is defined by the application programmer between the class definitions for these objects and a relational database schema.

The client side application code performs some useful operation with the data or service exposed by the server-side business logic. To access this information or service, the client application utilizes stub objects. These stubs expose the same interface as the live objects on the server (possibly a subset of methods for security/feasibility reasons), but the implementation of the object lives on the server; client side methods all contain logic for making calls to the server, and returning the response as if the method were implemented on the client. These stub objects are created automatically at build time by a tool which is able to read a description of the web service, and interpret into source code which can be compiled and used by the application. In traditional web services, this description is a WSDL file. <what is this for a REST web service??>

Figure 1: Traditional Client-Server Web Service Deployment, with ORM Mapping Definition and a Web Service Descriptor

Problem

The problem with this deployment is that there are redundencies in the way the shared objects and web service is defined, which could be streamlined to the benefit of both server-side programmers and clients who wish to interface with the server. In the event that the class definition for one of the shared objects changes (ex. adding a new public member), the server-side application programmer must update both the ORM Mapping Definition file/logic, as well as the Web Service Descriptor, and the client-side programmer may be required to at least rebuild their application, to update the stubs (this is addresses in Versioning).

Single Mapping Definition

It has been proposed that the ORM Mapping Definition and Web Service Descriptor can be combined into one artifact, as the two separate documents both essentially describe how to serialize instances of a given class. With hopefully only a small amount of modification, an ORM Mapping Definition could be used to serve both these purposes. Also, if the ORM side of the interface to this artifact is properly preserved, it is hoped that it can still be used for many of the other functions the ORM layer uses it for, like relational database schema migration/exporting.

Note: it may be interesting to investigate this further, as the ORM Mapping Definition serializes an object's state, but a Web Service Descriptor would likely only describe an object's behavior/interface!

Figure 2: Client-Server Web Service Deployment, with single Shared Object Descriptor

Versioning

Following from the automatic schema migration/updating, questions are raised about how similar functionality could be used to span the client-server gap, not just the server-database gap. Obviously, client-side stub classes can't be updated completely automatically, as this would require rebuilding the application. However, the server could expose several concurrent versions of the same service, multiplexed based on a version field in incoming requests. As part of the schema update process, in addition to modifying the database, the ORM layer (or some other piece of code) could generate the infrastructure required to support backward-compatible calls to the web service API.

My First Lecture

2009-01-22T13:35:00.000-08:00

So, I've been neglecting this old blog for the last few weeks, so I figure it's high time I let my captive audience in on what I've been doing at grad school.

Once again, I'm TAing CSC301. This term, in addition to my standard duties of marking, critiquing assignment questions, and coming up with exam problems, Greg asked each of his TAs to give a lecture to the class. My topic was unit testing with javascript. My first reaction was "Yay, I already know about unit testing, this will be a breeze", but then I remembered that my knowledge of javascript extended to image rollovers, and no further. So, I spent about 10 or 15 hours over the next week or so learning proper OO javascript, as well as how to use (or not use) JsUnit and some of its competitors, JsCoverage, Selenium, and trying to beat CruiseControl into state that fits together with these tools (no such luck, unfortunately).

The lecture went pretty much as I expected. I had prepared a loose agenda of items, a timeline for discussion and demonstration, and a few canned questions designed to get the undergrads to turn their brains on - all of which I forgot as soon as Greg introduced me. Luckily, I had my laptop and a big desk to hide behind. After a few moments of awkwardness, things got back on track, however.

Lessons learned:

leave your pen at your desk (aparently I click-click-click it as a nervous twitch).
always bring a glass of water to a lecture, so that a) you can keep your throat lubricated and b) you can take short pauses to think/fabricate answers to questions without looking like you're thinking/fabricating

All in all, I think it was a pleasant experience. I was worried everything I said went past the class and out the back door of the room, but I received several questions and comments from a couple of the project teams, asking about setting up JsUnit to test their code! Hooray! I hope they can get it working.

Relaxing Christmas Break

2008-12-29T07:13:00.001-08:00

My relaxing Christmas break is coming to an end, and not a moment too soon. Just to recap:

Spent the first few days adjusting to the noise level in my house, which I had apparantly forgotten about
Had 5 (five) wisdom teeth removed. Important note: always get your pain medication before the Novocaine wears off. If the wait time at the pharmacy is longer than an hour, get the doctor to call in the order ahead of time. Most unpleasant two hours ever.
Acquired 1 (one) head cold from either family members at Christmas eve party, or mobs on boxing day. Just in case an aching, pounding pain in your jaw bone isn't enough, here's two more aching, pounding pains: one in your head, and one in your throat!
Sub-par productivity levels, which I'm blaming on the codine.
Modem feels the need to reset/crash about once every 5 minutes

Anyway, it will be nice to be back in Toronto again.

Migration Woes

2008-12-21T13:39:00.000-08:00

Spent some quality procrastination time this afternoon migrating all of my disparate google services to be owned by my shiny new gmail account. Simply switching ownership account wasn't an option, so I had to create a new calendar, reader, and blog with my gmail account, export everything into xml files, then import them into the new services. Everything looks pretty much seamless, with the exception of this blog, which now has two authors, both named Rory Tulk. Oh well, can't win 'em all, I guess.

Power Laws in Software

2008-11-07T10:39:00.000-08:00

A good paper explaining Power Laws, and the 80-20 observation. Some of the correlations between the distribution and actual data are a bit loose, but the authors provide error measures (r squared) for all datasets presented, and in some cases make mention of some poor fits. Overall, very helpful.

Couple of tangent thoughts while reading this:

static dependancy analysis with Java reflection sounds like fun :)

"... only a few reusable components can be reused profitably [Glass 1998]" look into this further. Does this correlate with the previous paper on code reuse?

Jogging Over a Distance – Supporting a “Jogging Together” Experience Although Being Apart

2008-11-07T09:07:00.000-08:00

http://www.cs.toronto.edu/~khai/classes/csc2526-fall2008/readings/09-mueller.pdf

The authors have created a simple audio communication system for use by joggers. As reported in a survey they conducted, joggers enjoy the company of others while jogging, for conversation and encouragement. However, finding a jogging parter of similar ability and availability is not always possible. To this end, the Jogging Over a Distance system links two joggers who are not physically collocated. Each jogger is equipped with a cell phone, wireless headset, and small computing unit. Joggers hear the conversation of their partner though the headset, augmented to seem localized in space around them based on their partner's speed (ie. If their parter is faster, they will sound farther ahead).

Jogging Over a Distance insightfully a market of users which seem to be in a very receptive position for this device. As the study points out, 54% of surveyed participants said they engage in conversation while jogging. Also, the proposed design retains all the mentioned benefits of conversation while jogging (socializing, motivation, fun, encouragement) while lessening the barriers presented by running in pairs (differing pace, differing geographic location). In addition to this, the 3D audio positioning is an extremely creative, novel way to indicate to the user the relative speed of their jogging parter. This clearly seems to be influenced by research done in the area of ambient/passive displays.

I feel there are several areas in which this system could be improved, or concerns addressed. The device's form factor seems overly cumbersome, especially given the wide availability of mobile phones with adequate computing power for managing both calls and audio positioning. This was pointed out by the authors as an issue which will be addressed, but I wonder why it wasn't a hard requirement since design inception. Also, in relation to the audio positioning, I was surprised to read that off the shelf quadraphonic headphones were unable to give the users proper discrimination between sounds in front and behind them. The solution proposed, to have the forward axis tilted to a 1:30-7:30 orientation, seems undesirable. It would be interesting if the authors could study this orientation with respect to the route runners take, to see if it leads them on a series of gradual right-hand turns (as one runner follows the virtual runner in front of them). Also, if the audio location features were omitted, how is this technology different than a simple cell phone and head set? In this case, the merit of the authors' work is not the technology, but the sociological study they can perform with it.

This brings me to my next line of thought, which is concerned with the marketability of this 'device'. If a jogger typically jogs for an hour, once a week, and uses the Jogging Over a Distance to effectively make a one hour phone call to someone who is too far away to run physically with them, will this not present a large associated cost for the jogger? For the purposes of research, this is not an issue, but if it were to be a consumer product, this would have to be solved.

The Familiar Stranger: Anxiety, Comfort, and Play in Public Places

2008-10-27T07:42:00.000-07:00

http://www.cs.toronto.edu/~khai/classes/csc2526-fall2008/readings/08-paulos.pdf

The authors present the results of a study conducted to examine the results of two studies conducted to determine the extent to which the Familiar Stranger concept still exists in modern urban scenarios, and whether or not a ubiquitous social networking solution could leverage this effect in a useful way. The first study accurately recreates Stanley Milgram's 1972 Familiar Stranger study, in which photos of a light-rail station at morning rush hour are distributed to individuals at the station, and familiar strangers are determined by labeling the people in the photo. The second study consisted of an urban walking tour, in which participants indicated their level of familiarity and comfort in certain locations based on four dimensions: number of familiar people in the area, degree of familiarity with those people, have familiar people visited this place before, and do the people currently here visit the same places I do? Using these results, the authors propose Jabberwocky, a device used to tag familiar items, locations, and individuals. In this system, Bluetooth connected devices (base stations, cell phones, and iMotes) communicate to provide a measure of familiarity to the user about their current location.

I found this paper to be particularly informative from a purely sociological standpoint. Although the concepts presented cannot be wholly attributed to the authors (Milgram's study was novel in 1972, but not in 2008), they are none the less insightful. One noteworthy design constraint imposed by the authors is that any ubiquitous device which functions based on familiar strangers must not encourage explicit interaction with said strangers. The authors argue that the existence of familiar strangers is an indicator of a healthy urban community, and not a negative or anti-social aspect. Further observations on people's behavior, such as frequently checking one's cell phone in unfamiliar settings, adds a depth of insight to the paper.

The majority of the limitations found with this study are attributed to the technical merits of the device proposed. The Jabberwoky platform seems to be applicable to most of the situations applied in the paper, but using the Motes to tag static objects or locations seems a bit infeasible. For example, leaving these devices attached to public structures brings into question the cost of the device and the possibility of theft. Perhaps a less physically obtrusive solution for tagging familiar locations would be to submit the GPS coordinates of the location to a central server, which can be queried at the same time and with the same frequency of the Bluetooth polling.

Augmenting the Social Space of an Academic Conference

2008-10-27T07:41:00.000-07:00

http://www.cs.toronto.edu/~khai/classes/csc2526-fall2008/readings/08-mccarthy.pdf

The authors present the results of deploying two proactive displays into an academic conference setting: AutoSpeakerID (ASID) and Ticket2Talk (T2T). Both these systems leverage RFID tags that are physically installed into conference attendees' badges, which are paired with an profile containing personal affiliation information and a photograph. The ASID display consists of a RFID reader embedded in a microphone stand and an accompanying large display. In this way, when an attendee approaches the microphone to ask a question, their information is rendered on the display, providing context for their question. The T2T system is of similar configuration, in that it has a display which renders an attendee's profile when they come into proximate context with the display. However, T2T is installed at refreshment stations to promote personal interactions between attendees.

The novel element presented in this paper, as the authors point out, is the close focus on the evaluation of these devices. Each system is examined thoroughly by the researchers, gathering qualitative observational and questionnaire data. These results were used to gauge the systems' performance in the areas of Enhancing the Feeling of Community, Mesh with Established Practices, and Privacy Concerns. Some unexpected, yet somewhat beneficial results were produced when users attempted to 'game' the system buy providing falsified, comical profiles (ie. The Bill Gates profile).

Although the authors focused on qualitative analysis of their displays, I think further investigation is required before coming to definitive decisions on the systems' utility. Although this is clearly a difficult domain to measure, it is generally proposed that user surveys/questionnaires can skew results. For example, the results of the survey for the Ticket2Talk system reported 41% positive feedback and 3% negative feedback, with 66% of the attendees unaccounted for. If we take into consideration a variation of Self-Selecting Respondents, we could propose that the participants who found the system very useful were motivated to fill out the questionnaire, and those who strongly disliked the system were motivated to distance themselves from the system, it would not be a stretch to propose that the majority of the 66% unaccounted attendees had a negative view of the system, and so the results of the questionnaire are invalid.

A Taxonomy of Ambient Information Systems: Four Patterns of Design

2008-10-27T07:39:00.000-07:00

http://www.cs.toronto.edu/~khai/classes/csc2526-fall2008/readings/07-a-pousman.pdf

This paper presents the current state of the art in ambient or peripheral information displays. The authors propose four dimensions on which the currently available ambient systems can be measured: information capacity, notification level, representation fidelity, and aesthetic emphasis. Information capacity measures the number of information sources a device can display. Notification level indicates the degree to which the system will interrupt the user, or demand their attention. Representation fidelity measures the level of abstraction in the data representation. Finally, aesthetic emphasis measures how important aesthetics are to the device's designers. Based on these four dimensions, the authors propose four patterns of design in this domain: Symbolic Sculptural Displays, Multiple Information Consolidators, Information Monitor Displays, and High Throughput Textual Displays.

The authors do a very nice job of describing the current areas of research and development in ambient and peripheral displays. The exercise of classifying current projects on their four selected dimensions is quite insightful, and serves greatly to provide organization and structure to the field.

This paper is lacking in a tangible contribution to knowledge, however. In the paper's introduction, it mentions at least one other existing method of categorizing ambient and peripheral displays. I can see no measure that indicates that the new classification system which is proposed here has any advantage over existing methods. Also, these patterns of design have been used to classify existing projects, but how could they be used to facilitate the creation of new products, in much the same way that OOP design patters are used? Finally, it could be suggested that the four patterns proposed are insufficient to categorize all possible ambient displays, since it is incapable of being applicable to all devices in the sample used in the paper. Perhaps there are more than four patterns that can be extracted from the four dimensions of classification proposed by the authors, and perhaps there are even anti-patterns to be found within these four dimensions, that would result in poor ambient displays.

Heuristic Evaluation of Ambient Displays

2008-10-27T07:37:00.000-07:00

http://www.cs.toronto.edu/~khai/classes/csc2526-fall2008/readings/07-mankoff.pdf

In this paper, the authors present a method for evaluating ambient displays using a technique similar to Nielsen's Heuristic Evaluation. A set of heuristics are created, by modifying the Nielsen heuristics, specifically tuned for ambient displays, hereafter referred to as the Ambient Heuristics. Two evaluation groups are then formed to apply heuristic evaluation to two novel ambient displays, both developed by the authors: busMobile and daylight display. BusMobile is a simple ambient display which shows the locations of campus buses, relative to the building in which the display is placed. Daylight display uses a lamp to convey the brightness level outside to users in a lab with no windows. The use of Ambient Heuristics showed an increased ability to find severe usability issues over the Nielsen heuristics.

The authors have presented an insightful tool for evaluating the usability issues of ambient displays. This revised set of heuristics provide a cheap, effective way for researchers to evaluate the usability of their ambient display products.

I feel that this study would be more valid if the Ambient Heuristics were applied to more than just the two displays created by the authors. There are a number of existing products/projects that could have been used as samples in this study. This would both fulfill one avenue of future work, and reduce cost on the authors because they would not have to develop their own displays simply for the purpose of testing the evaluation scheme (that is, unless the display devices were existing projects). In addition to this, it would be interesting to see if the margin of difference between the number of issues found with Ambient Heuristics and the Nielsen Heuristics scales to larger sets of data (ie. More than 30 possible issues). If the gap were to widen, it would imply the Ambient Heuristics are at an advantage for finding issues in this domain. However, it seems then counter intuitive that the Nielsen Heuristics would be capable of finding issues that the Ambient Heuristics are not. Perhaps the heuristics need to be adjusted so that they find a superset of all issues found with the Nielsen set.

Hardcore SE Papers

2008-10-16T09:11:00.000-07:00

Software Libraries and Their Reuse: Entropy, Kolmogorov Complexity and Zipf's Law
http://www.cs.toronto.edu/~gvwilson/reading/veldhuizen-libraries-reuse.pdf

This paper presents an interesting argument, stating that the entropy of a given problem domain can be measured, and in so doing we can predict the amount of library reuse that is appropriate, or indeed possible, for programs in that domain. Also, a handy proof states that the only complete library for a given domain is of infinite size, effectively securing jobs for library writers in the future.

Unfolding Abstract Datatypes
http://www.cs.toronto.edu/~gvwilson/reading/gibbons-unfolding-adt.pdf

This paper should have been called "Unfolding Abstract Datatypes in Functional Programming", so that I had a valid warning before I started reading it, and by reading I mean staring at the pages looking for something I understood. Here's what I got out of it: in functional languages, ADTs are possible, but less commonly implemented than in OOP languages. The primary reason for this is because most user-defined types expose their data structures so that you can do pattern matching on them. The paper argues that this is bad, and proper information hiding can be obtained without breaking matching ability. Also, ADTs represent codata (whatever that is).

Asking and Answering Questions during a Programming Change Task

2008-10-14T13:57:00.000-07:00

http://www.cs.toronto.edu/~gvwilson/reading/sillito-questions-program-change.pdf

This paper presents a study conducted by getting programmers (students and professionals) to work while thinking out loud, then categorizing the questions they ask themselves (and their debugger). These fall into 44 distinct categories, under 4 main groups. Following this, each of the categories is analyzed to see if existing tools are able to directly answer the question proposed.

I found my mind wandering around while I was 'reading' this paper. Not that the subject matter is uninteresting, far from it. I found myself coming up with ideas for new tools all throughout this read, which eventually started to detract from the primary text itself. Anyway, the following are some thoughts:

There is mention of how programmers divide their workspace to show them, for example, code and executing program, or two code files, etc, by using emacs screen splitting, multiple windows, or multiple monitors. I wonder what the results would be in a study where we a) measure a programmer's productivity with one monitor, then b) add a second monitor (I'm pretty sure this has been done before), allow them to get used to it (productivity should plateau), then remove the second monitor. I predict that productivity will drop below that measured in a) for a while, then gradually come back to a nominal level.

Of the two groups studies (students and professionals), both had a single category of question (of 44 possible) that was asked vastly more than all others. For students, this was "Where is this method called or type referenced?". For professionals, this was "What will be (or has been) the direct impact of this change?". A couple of things come out of this.

First off, students seem more concerned with direct program behavior or structure, while professionals are concerned with impacts of code change. This seems like a much more organization-oriented behavior. I'm having trouble expressing my exact idea here, so I'll come back to it. Bottom line, is that professionals are less hack and slash than students.

Secondly, there are tools for addressing the students' question. Why aren't they using them? The tools for the developers' questions, however, are lacking. Can we make them better?

Exemplar-driven documentation. There is discussion in this paper about finding examples of the type of operation one is trying to create or modify within the subject code base, and using that as a template for the new feature/modification. I wonder if this could be applied not only to the target code base, but to any code base (or indeed every code base). Lets say, for example, I want to implement a convolution matrix to do a gaussian blur over a java BufferedImage. Imagine I had a search engine that would search the code of a vast number of open source code bases, with some natural language query, and returned code snippets of convolution matrices over BufferedImages. Useful? I dunno, just had to write it down before I forgot it.

This leads into another idea that popped up. A few months ago, when I had a job and spare time, I was playing around with the jMonkeyEngine, which is a handy little open source scene graph for java, based on JOGL. Its documentation is in the form of a Wiki, which unfortunately has a bunch of holes in it. However, I found that downloading the source trunk and looking at the extensive unit tests was a much better learning tool. I simply loaded the unit test hierarchy into the IDE, looked for a test for the feature I wanted to use, ran it to see it work, then looked at the test code, which by definition is short and concise. I propose a study where we take two groups of developers and one large API, and task them with implementing a given application off of this API. One group will have standard documentation, and one will have a complete set of unit tests. Let them go and check the results. If the unit tests turn out to be better, this would be a huge boost for the motivation for TDD.

Two more quick ideas, and then I'm done. This one relates back to finding usages of methods/classes, as was one of the prime questions asked by students in the paper's study. Using a 'Find Usages' feature in an IDE can solve this, but it is not the most efficient when looking for loose relationships between two or more elements. What if I wanted a tool that was "Find Usages of these TWO methods" or three or four or etc. Basically, find the class,method, block, or statement which uses all of the given input elements. I think this would be handy.

Lastly, the paper used ArgoUML as its code base for the student tests. The authors had the students fix bugs submitted via the ArgoUML tracker. I wonder if there's a market for shopping out bug fixing time to ethnographic research subjects?

Conceptual Modeling Extraveganza

2008-10-13T16:51:00.000-07:00

Three papers concerned with conceptual modeling:

First off is Ross' seminal 1977 paper Structured Analysis (SA): A Language for Communicating Ideas. I don't think there's much I can say about this paper that hasn't been said already, so I'll keep it short. This paper presents the argument that "SA is the best thing since sliced bread", and continues to illustrate this point they present pretty much the entire meta-model for SA, and go though, in great detail, all the primitive constructs in the SA vocabulary.

Prof. Mylopoulos wrote an interesting opinion article Desert Island Column: A Trip to Carthea praising the previous paper's insight. One thing that Prof. Mylopoulos brings up is that "the world consists of more than things and happenings", which seems to be something that Ross argues strongly against in his paper. One of the cornerstones to the Ross paper was that anything worth talking about consists of 6 or fewer things and happenings.

Lastly, I took a look at Jennifer's paper Reflective Analysis of the syntax and Semantics of the i* Framework. Now, to be honest, I would have gotten a lot more out of this if I was more familiar with the i* syntax, but the idea of reflective analysis that this paper presents could be applied to any modeling tool. A study conducted by Horkoff et al looked at assignments and research papers in the community, and recorded the most frequent deviations from the U of T i* syntax, and proposed that these deviations were made due to non-optimal design choices in the languages syntax.
A couple of useful results came out of this investigation. First, the authors propose modifications to the i* syntax to address these common mistakes, and some conclusions are drawn about how users are learning i* (ie. not enough focus on areas where syntax mistakes occur). I wonder if a similar study has been done for SA or UML, and if so what their conclusions were (ie. UML activity diagrams are never used?)

Also, I was surprised to hear so many positive opinions about SADT. In my undergrad, it was touched on briefly in a Requirements Engineering class as something that was 'old, and not used by anyone anymore'.

Evaluating Effectiveness Efficiency of TDD

2008-10-07T15:18:00.000-07:00

http://www.cs.toronto.edu/~gvwilson/reading/gupta-effectiveness-tdd.pdf

Kind of a hokey paper that describes an ethnographic study done in India comparing up-front design-code process to test driven development. Related work shows that a) tdd is more efficient, b) tdd is less efficient, and c) there is no difference, so it seems clear that there is still debate over this issue. This paper presents evidence (not vast amounts, just more ammunition for the debate) in support of TDD, ultimately coming to the conclusion that developers will probably prefer a modified version of TDD, in which more design is done up front, but still using the test-code-refactor waltz.

12 Steps to Better Code

2008-10-07T13:40:00.000-07:00

I read this one before it showed up on the reading list :) Very good article. If more dev shops followed Joel's 12, our job would be much more enjoyable. Restated here for completeness, every successful software team should pass the following 12 tests:

The Joel Test
1. Do you use source control?
2. Can you make a build in one step?
3. Do you make daily builds?
4. Do you have a bug database?
5. Do you fix bugs before writing new code?
6. Do you have an up-to-date schedule?
7. Do you have a spec?
8. Do programmers have quiet working
conditions?
9. Do you use the best tools money can buy?
10. Do you have testers?
11. Do new candidates write code during their
interview?
12. Do you do hallway usability testing?

If you answered 'no' to more than 2 of these, you're not doing things properly. Pokes holes into some common patterns seen in software houses these days, like the low-walled cubicle bullpens and pushing off bugs to the end of the iteration. I think this is one that every developer and manager should commit to memory, but I certainly wouldn't call it science.

Learning TDD by Counting Lines

2008-10-06T08:31:00.000-07:00

IEEE Software May/June 2007

Interesting little paper describing Nokia Networks migration from waterfall to agile, and the tools used by the group training Nokia's developers on how to use test driven development. The exercise in question talks about the first step the new developers took in creating a program to count non-commented lines in a source file, using a TDD approach. Start off with the low-hanging fruit (a test with an input program of one line), and scale it up until hitting a code-wall. The third movement in TDD, refactoring, is apparently often overlooked by new TDD developers (I'm guilty of this, too). This includes not only refactoring commonalities in the test cases, but in the production code and even refactoring the design of the production code. Emergent design, in this case, means more than just making the most logical step at each point and hoping that the best design will result.

Anyway positive results on a non-statistically significant (12) number of development teams.

Replication in Research

2008-10-05T16:35:00.000-07:00

Just read a couple papers concerned with replicability in research. That is, authors of academic papers publishing, along with the body of their paper, the data and code used to generate their figures and come to their conclusions. The primary moral is this: not enough structure is in place to make authors prove that their methods actually work, but the technology to distribute the materials required to double-check their findings is widely available, just not widely used.

I wonder if the previous article (size confounding metrics) has been replicated?

The Confounding Effect of Class Size on the Validity of Object-Oriented Metrics

2008-10-04T18:13:00.000-07:00

A very good paper, with significant results. Research into metric code analysis on object-oriented programs shows that the most significant metric that can be used to infer the fragility of a class is the classes size (LOC), and that most other existing metrics are simply layers of indirection on top of class size (ie. large classes have higher coupling). I'm not sure the actual impact this paper has had on the community, but I would think it should be fairly revolutionary.

One thing that sort of rubbed me the wrong way was in the paper's introductory phases, they discussed procedural vs. object programming, and implied (with some references) that OO programs are in general harder to maintain. I wonder about this statement.

SenseCam: A Retrospective Memory Aid

2008-09-30T09:16:00.001-07:00

http://www.cs.toronto.edu/~khai/classes/csc2526-fall2008/readings/04-hodges.pdf

The authors of this paper illustrate the findings of the initial clinical trial of SenseCam, a small form factor camera which is worn around the neck and records still images both at regular intervals and when the device's sensors are stimulated. This creates a pictorial record of events that happen in the wearer's immediate proximity. The motivation for this record is to aid in rehabilitating memory loss sufferers. The trial presented in this paper has shown a marked improvement in the cognitive ability of the test subject. However, further clinic trials are required before a definitive statement about its merits can be made.

I think that one of the most significant insights the creators of SenseCam had was that the success of the device relied critically on a small, compact form factor. Earlier incarnations, which utilized mobile PCs carried in a backpack, would be too unusable to have a net benefit for a patient. Ease of use is of vital importance. Another important point in the discussion of the clinical results is that the authors make the distinction between the patient remembering the events recorded by the SenseCam instead of remembering seeing the pictures it recorded in previous sessions. Although the patient claims to be remembering the actual events, I believe the experimental method could be altered to assert this claim more concretely.

The SenseCam represents a simple enough product, and it seems obvious that it could benefit a patient suffering from a memory dysfunction. However, before it can claim to out perform other methods, I believe further clinical study, under more controlled circumstances, needs to be carried out (it should be noted that the authors freely admit this, it is just being restated here for completeness). Primarily, more patients need to be examined; a single case study is not sufficient. In addition to increasing the number of subjects, the number of 'important events' recalled by each subject should be increased as well, preferably to some statistically significant level.

In addition to a small sample size, there is a strong possibility that the results for the single given sample, recorded by Mr. B, may have been skewed, perhaps even unintentionally, due to the nature of Mr. B's relationship with the subject. A more pure result would be obtained by using an impartial third party to administer the tests to Mrs. B.

Designing Capture Applications to Support the Education of Children with Autism

2008-09-30T09:16:00.000-07:00

http://www.cs.toronto.edu/~khai/classes/csc2526-fall2008/readings/04-a-hayes.pdf

The authors present three prototype devices for assisting caregivers dealing with children with autism (CWA). The first, called Waldon Monitor (WM), consists of a wearable video camera and Tablet PC for observing and recording the behavior of CWA. Secondly, the Arabis system replaces traditional pencil and paper based recording with a Tablet PC, and is used by caregivers to record a subject's performance in one-on-one testing and diagnosis. Finally, CareLog is a distributed, ubiquitous system for recording data about a child from any wireless accessible device, including a cell phone, PDA, or PC. A therapist can record and access data using any of these devices, which is stored on a mobile storage unit that is co-located with the subject.

The most insightful findings that these studies show is the importance of properly planning for complicated, multi-user interaction with their devices. The proposed systems seem trivial (video recording unit, software for tabulating test scores, etc), until the use cases are presented, which span multiple users at different times, with very different goals for the data, potentially at different stages in the recurring care cycle. That is, a therapist may use CareLog to record data about a CWA in an intermediate phase of the care cycle, and an analyst will use the collected data, aggregated with past results, to make a diagnosis and set goals in the early stages of the next iteration of the cycle.

One area in which the proposed devices can be improved is in CareLog's portable storage unit. Although this is a novel approach, I believe that the same functionality could be achieved with less cost if the data were stored on a remote, web-accessible server, instead of in a device that needs to be carried around the by subject. This way the physical hardware cost is reduced, and the subject can't loose or destroy their own data. Also, using an existing commercial product to perform the data analysis required for these prototypes could reduce the upfront cost, making them easier to adopt by caregivers and therapists.

Ideas

2008-09-29T10:54:00.000-07:00

I'm taking this opportunity to write down some ideas I had during today's talk, so that I don't forget them :)

Non-standard programming interfaces - what could we do if we wrote programs in a WYSIWYG editor? Or an auto cad type editor? Or any domain application? What would that be like? Also, could you bootstrap such a system (ie. implement the non-standard language using a non-standard language)? Intuitively not, but maybe that's why we haven't figured it out yet.

Program diff - diff a program not as a text file, but as a syntatically (and semantically?) correct piece of code.

Program representation - how can we represent a program other than a bunch of lines of text? Does this apply to the previous point?

BRAINSTORM!!!
Gdankin problems!! = design patters. According to Greg's definition (going to double check this against wikipedia in a minute), a Gdankin problem is one that results in the same solution when solved by independent domain experts. Now, as I understand it, when the gang of four first wrote Design Patterns, they analyzed large amounts of code, created by programmers in different organizations, independently of each other, and noticed that certain problems resulted in similar structures in the code. Therefore, I propose that the Design Patterns are the common solutions to a set of fundamental Gdankin problems.

The CHAOS Boondoggle

2008-09-23T19:21:00.000-07:00

So the Standish CHAOS report was released in 1994, estimating that the majority of software projects go over-budget by 189%. The presentation of the report was questionable at best, and the data contained in it seemed inconsistent, with little more than hand-waving to back it up.

Enter the second paper, Jorgensen and Molokken, actively calling out Standish on the numbers it presents. Good for you, Jorgensen and Molokken!

In the interview, the interview does a pretty weak job of grilling Standish about his numbers, and they both leave having answered very little.

I'm pretty sure Standish blundered this number, and covered its tracks by a) not disclosing research methods and b) significantly altering the following year's CHAOS report.

But this is all just my opinion. I've been wrong in the past.

Why Line for Java Names

2008-09-23T08:50:00.000-07:00

WTF-J = Whyline Toolkit For Java

WTMF-J = Whyline Toolkit (Multi-threaded) For Java

Automatic Bug Triage Using Execution Trace and Ownership Vector Space

2008-09-22T08:35:00.000-07:00

Came up with this idea by smashing together two papers from Greg's reading list. Using this for my NSERC & OGS applications, so please don't steal it :)

Previously presented topics on automatic bug triage (directing bug reports to the appropriate member of the development team) showed very bright prospects, but lacked enough accuracy to make them a usable product [1]. This entry point for bug information is also an excellent location to apply any number of other filtering heuristics desired, such as the duplicate detection algorithm proposed by [2]. The method proposed in [2] requires, in addition to a natural language description of the problem, and execution trace that can be used to more quantitatively measure two, or more, bug's similarities. I propose that the same execution trace could be used to assist in the triage functionality described by [1]. Since the vector space created by Wang et. al. to measure bug similarity is based on function calls, a similar approach could be used, requiring the same input execution trace, to determine bug ownership. A master vector space could be created at build time from all possible called functions in a source code repository, and assigning ownership of these functions to developers based on either activity in a source revision system, or some static assignment. This would create volumes of ownership within the function vector space. In theory, a bug report, assuming it is not a duplicate, should be assigned to the developer in whose volume the bug's vector terminates. This may also have interesting and relevant applications to visualizing ownership of code, for management purposes.

[1] D. Cubranic and G. C. Murphy, "Automatic bug triage using text categorization," in Proceedings of the Sixteenth International Conference on Software Engineering & Knowledge Engineering, F. Maurer and G. Ruhe, Eds., June 2004, pp. 92-97. [Online]. Available: http://www.cs.ubc.ca/labs/spl/papers/2004/seke04-bugzilla.pdf

[2] X. Wang, L. Zhang, T. Xie, J. Anvik, and J. Sun, “An approach to detecting duplicate bug reports using natural language and execution information,” in Proceedings of the Thirtyth International conference on Software engineering

Lazy Delete for Email

2008-09-22T07:39:00.000-07:00

So I'm going through my inbox this morning (procrastinating on my nserc application, btw), and I notice I have a lot of messages that take the form "Don't forget about on ". After reading, I don't want to delete this message, because having it in my inbox serves as a nice reminder about whatever it is I'm not supposed to forget about. However, after a week my inbox is now clogged with messages about events that have passed. What I would like to be able to do is, when initially reading the message, click a button next to the Delete button, lets call it Delete On ... and then I can specify a date, so that the message will be deleted once it has become invalid.

Maybe run some kind of computational linguistic method to read the message first and propose a deletion date?

Context Aware Communications

2008-09-20T13:19:00.000-07:00

This paper presents research activities into the field of Context-Aware Communication, which is considered as a subset of context-aware computing applications. The authors structure their presentation into five main categories: routing, addressing, messaging, caller awareness, and screening. Routing involves directing communication (phone calls, text messages) to physical devices in close proximity to the callee, and has been successfully implemented by combining Xerox PARC's Etherphone and Olivetti's ActiveBadge system. Addressing uses context information (“is this user in the building?”) to dynamically adjust traditional email mailing lists. Messaging is similar to context-aware call routing, but will instead route text messages to any proximal device capable of displaying text information. Caller awareness provides callers with information on their contact's context, so that they can actively choose not to call at inappropriate times. Screening is an approach that works in contrast to caller awareness; it filters out incoming calls based on the callee's context.

The authors presented several insightful technologies which were eventually adopted by modern day ubiquitous systems. The first of note was customizable phone ringers, depending on context. In the example, these ringers were used to distinguish callee, even though they have been successfully applied to determining caller in current applications. MIT's Active Messenger bears a striking resemblance to modern cell phone SMS capabilities. Also, AwareNex's context feature has been replicated in countless instant messaging systems. A possible extension of this idea is to incorporate automatic context sensing, via ActiveBadges or some other comparable technology, into current applications which would benefit from context information, such as an instant messaging system. Thereby, instead of manually setting one's IM status to 'on the phone', all one would have to do is simply pick up the phone. Not sitting at your workstation would change your IM status to 'away'.

The primary limiting factor in the applications presented in this paper is the technology that was available at the time it was written. Although the devices developed successfully demonstrated the concepts intended, further effort needs to be made to make these devices more marketable before they will be widely adopted, and form the ubiquitous network required for the proposed applications. It is entirely possible, however, that these advances have been made between the time this paper was published and present day.

Also, the authors made mention of certain situations where the context-aware communication applications would, for example, hold or screen incoming traffic because the callee is at the movies or eating dinner. This was only speculated at, because at the time of publishing the context sensing network wasn't pervasive enough to determine a user's location outside of the office (with the exception of GPS, but that won't work indoors). I propose that this limitation should be included in future systems which can determine a users' context outside of the workplace, to add a measure of privacy to the system.

Context Aware Computing Applications

2008-09-20T13:18:00.000-07:00

This paper presents an interesting summary of current (1994) activities in the field of context-aware applications. This consists primarily of applications developed for the workplace which leverage the user's physical location, as well as the locations of coworkers and resources within said workplace. The authors present four areas or application features that rely on context, implemented with Xerox PARC tabs and boards: proximate selection, automatic contextual reconfiguration, contextual information and commands, and context-triggered actions. Proximal selection is a UI technique that visually makes objects closer to the user's physical location easier to select. Automatic contextual reconfiguration refers to a process through which ubiquitous devices (boards, tabs, etc) can be accessed by a user simply by being in their immediate vicinity. Contextual information and commands can be used to display default appropriate information, or alter the standard set of commands (or parameters to these commands) based on the user's location. Lastly, context-triggered actions represent actions (in this paper, unix shell commands) that are executed by context events. That is, when a predefined context state occurs.

The application features that Schilit et. al. present in this paper have proven to be insightful in that they have found their way into many pieces of modern ubiquitous devices. The proximal selection UI outlined bears a striking resemblance to fisheye menus, most notable found in Apple's OSX. Automatic contextual reconfiguration performs a similar function to modern BlueTooth networks, with the exception that BlueTooth doesn't rely on a centralized network to control all devices within a building.

One issue that I believe the authors may have overlooked, especially in respect to proximal selection and contextual reconfiguration is that of permissions. In the example of a user printing a document, the closes printer may not be the optimal choice if it is a restricted resource. Aside from this, the only element of context that the authors seemed to use was the location of individuals and resources. Other environmental variables could be utilized to augment the function of a ubiquitous device to better suit the needs of it's user. For example, by monitoring the ambient noise level around the user, a phone could decide change its ringer volume to match, or simply vibrate if the room is too noisy for the device to compete.

Debugging Reinvented: Asking and Answering Why and Why Not Questions about Program Behavior

2008-09-18T08:11:00.000-07:00

http://www.cs.toronto.edu/~gvwilson/reading/ko-debugging-reinvented.pdf

This paper is an interesting follow up to an earlier paper I read about Why Lines. If you're not familiar, a Why Line is a debugging tool that instruments a piece of code, allows you to execute the code (for a brief time, ~ a minute), then use a custom UI to ask questions about the output (ie. Why is this circle red?). Experimental results (on an admittedly small test size) showed dramatic improvement in debugging time.

There was a paragraph in this paper that I feel the authors glossed over. It concerned translating user submitted bug reports into Why Line questions. My applying some computational linguistics techniques, I bet it would be possible to automatically generate one or many Why Line questions from a bug report. Combine that with the previously presented technique on automatic bug triage, and you have a system which will (in theory) automatically assign bugs to particular developers, and present them with a Why Line question that will allow them to quickly assess the problem.

Also, I'm pretty sure there's something that can be gained in an organization by archiving/databasing their Why Line traces, but I'm not sure what yet.

Fantastic Contraption

2008-09-17T13:40:00.000-07:00

This is the sole reason it took me all afternoon to read Jorge's paper:

http://fantasticcontraption.com/

Anchoring and Adjustment in Software Estimation

2008-09-17T13:06:00.000-07:00

http://www.cs.toronto.edu/~jaranda/pubs/AnchoringAdjustment.pdf

This paper presented Jorge's results about anchoring and adjustment when estimating software project time consumption. Looking back at my notes on this paper, everything seems to be presented fairly well, all the numerical results seem sound. Couple of quick things:

The Anchoring and Adjustment phenomenon is clearly observable when the problem can be expressed as a number in a range. Can it be applied to other classes of estimations?

Why is the description of COCOMO so verbose? It don't see how it contributes to the paper, other than to demonstrate that current software estimation techniques don't work.

What is a null hypothesis?

Now, I want to take this opportunity to discuss an estimation technique that a former colleague of mine once described to me:

Estimating the time required to complete a software project is inherently a random activity, and it is generally accepted that estimating smaller, individual tasks within a project is more accurate than trying to estimate the project as a whole.

Lets divide our hypothetical project P into n subprojects, P0 ... Pn-1. For each of these, instead of guessing a completion time, we provide a low ball and high ball estimate (ie. P0 will definitely take more than a week, but less than 4).

We take the sum of all the low ball estimates, and the sum of all the high ball estimates, and we get two figures, one for the earliest possible completion time for P and one for the latest. While this may be sufficient for some, one further refinement is to apply a Gaussian distribution between these two figures, that way you could say with some degree of reasoning, that project P has an 80% chance of finishing in X weeks.

Ferenc, I apologize if I've gotten any of this wrong.

Google Integration and Tools

2008-09-16T18:20:00.000-07:00

Couple of interesting google tools that I have come across in the last couple of days, probably old hat to most of you.

Google Reader (http://www.google.com/reader). This is an online, customizable RSS/Atom aggregator. Pretty handy for keeping track of slashdot, zdnet, ieee, and, for example, 15 graduate student blogs (like this one).

This last one doesn't exist yet, but I want my facebook events to be synchronized onto my google calendar. Sounds like a rainy friday afternoon project.

A Field Guide to Computational Biology

2008-09-16T10:45:00.000-07:00

http://www.cs.toronto.edu/~lilien/CSC2431F08/readings/CompBioGuide.pdf

This magazine article presents the opinion that computational biology will without a doubt be the cause of future miraculous advances in biology, disease, and gene research. Also emphasizes that traditional biologists will have to get used to using higher level mathematics than they are used to, and much more interdisciplinary interaction with others.

Good, not great.

Can a Biologist Fix a Radio?

2008-09-16T10:09:00.000-07:00

http://www.cs.toronto.edu/~lilien/CSC2431F08/readings/CanBiologistRadio.pdf

Explores the idea of having a formal language for expressing biological processes and structure, using the analogy of trying to fix a radio. An engineer would use a 'formal language' to describe the internal structure of the radio (amplifier, 10k ohm resistor, etc), and deduce the problem that way. A biologist would likely spend years of comparative research on other working radios, classifying parts based on phenotype, etc. This system quickly becomes too complicated for any one person to understand, primarily due to conflicting definitions of parts originating from different researchers. The author proposes that formal methods and language for biology will make cellular analysis much easier and vastly different, in the same way PowerPoint revolutionized slide-based presentations, and that biologists much catch up or be left by the way side.

NaviCam

2008-09-15T13:33:00.000-07:00

In Ubicomp today, while discussing a paper previously presented in this blog (see The Human Experience), we got to see a video of Sony's NaviCam system in action (see link http://www.sonycsl.co.jp/person/rekimoto/navi.html). Seeing this system actually working was pretty cool, and a handful of applications extending this functionality immediately jumped to mind, all of which could be potential thesis topics, or business plans.

Combining the augmented reality capabilities of the camera & display setup with a wearable, glasses-based display could allow application developers (like me :P) to create real-time navigation software, meta-information pop-ups, and all kinds of cool stuff!

One immediate thought that jumped to mind as a detractor was the image I had seen of some geek in the wearable computing field with a webserver in a backpack that he lugged around everywhere. No consumer would buy that, but then I thought that if this backpack could be shrunk down to the size of an iPhone, which it almost certainly could, then this would be a viable market opportunity.

One target audience for such applications would be the military. Personnel in the field could have information on way points, location of friendly/hostile persons, radar & network coverage, etc, overlayed on top of real world vision, eliminating the need for secondary maps & gps devices. And, the military's level of network connectivity is legendary, so gaining access to this information is essentially a solved problem.

Iunno, just an idea. Sounds like it would be fun to tune around in one of these sets, seeing maps and stuff overlayed on regular vision.

Gregory D. Abowd et. al. The Human Experience

2008-09-14T20:23:00.000-07:00

Gregory D. Abowd et. al. present a follow-up to Weiser's The Computer for the 21st Centry. In this paper, the authors closely examine some of the ideas proposed by Weiser, and explore the changes in traditional design and development patterns needed to adopt ubiquitous applications. This is loosely broken down into three categories: defining physical interaction models to and from ubiquitous computing devices, discovering ubiquitous computing application features, and evolving the methods for evaluating human experiences with ubicomp. The physical interaction problem examines new ways to gather input from a user, beyond simple keyboard/mouse combinations. This includes gesture-based and implicit input. Also, non-standard output methods are explored, different from the traditional video display (ex. Ambient output). Utilizing these non-standard IO methods to create a 'killer app' is the next challenge the authors discuss. Applications which use the user's context (location, identity, etc) as input for providing useful features, as well as relative changes in context, are discussed. The use of changes in context lead to the problem of continuous input, whereby applications must respond to constant subtle input from users over extended, possibly infinite, time frames. This is in sharp contrast to current application mentalities, which are meant for discrete usage sessions (ie. Word processor). Lastly, the authors propose that traditional HCI evaluation techniques will be at a disadvantage when used with ubiquitous computing applications, and so they introduce three new cognition models: Activity Theory, Situated Activity, and Distributed Cognition.

This paper provided several new examples of ubiquitous computing devices and applications, and served to 'pin down' some of the specific details that Weiser left for further research. The showcase of new devices and technologies clearly illustrates the path of development between the time the Weiser paper was published (1991) and 'present' (2002). One noteworthy insight presented by Abowd et. al. is that of the physical means of interaction with ubiquitous devices, drawing particular attention to 'implicit input'. It seems apparent that the future of the embodied virtuality will not be interfaced with a keyboard and mouse, and the devices presented in The Human Experience demonstrate subtle input and output methods (ex. Network traffic monitor) which clearly show success.

Although this paper provided excellent insight into the concepts previously proposed by Weiser, I feel that it didn't introduce as much original work as could be possible. It can be argued that this was the purpose of the paper, in which case it has succeeded. However, I feel it may have contributed more value to the scientific community had it contained more unique ideas. Apart from this, the paper presented a discussion of using ubiquitous computing applications to perform one of the fundamental activities humans perform on a daily basis: capture and access of data. That is, the recording of information presented by a colleague and summarizing it for later retrieval. It is my opinion (and this clearly is not, nor should it be, shared by all) that automating a fundamental process such as this will contribute to a strong dependence on said application, reducing an individual's ability to be self reliant. In addition, it is not beyond the realm of possibility that regularly exercising the intellectual system by absorbing and recording information in this way is beneficial, and removing the need for this exercise could have negative impacts on cognitive ability. This, however, just just my opinion. This represents an area of further research, which should be pursued with as much importance as the technical developments in the field of ubicomp (that is, the implications of ubiquitous devices).

Mark Weiser's The Computer for the 21st Centry

2008-09-14T20:22:00.000-07:00

The paper by Mark Weiser presents the current (as of time of publishing, 1991) efforts of Xerox PARC in the field of ubiquitous computing devices and applications, and follows this with speculation/projection of possible future scenarios. At time of publishing, these devices were divided into three classifications: tabs, pads, and boards. A tab is similar in size and function to a post-it note, except that it contains a dynamic display. A pad can be thought of as a scrap piece of paper, but again with a dynamic display and stylus-based interface. A board functions like a dynamic white board, or any large display. The intermixed use of these three classes of devices, the author argues, will form the basis of the future of ubiquitous computing in the “embodied virtuality”. Extrapolation on trends in technology evolution suggests that it would be capable to implement Weiser's embodied virtuality in the not-too-distant future.

I found the discussion of 'current' technology and research into ubiquitous computing devices quite interesting. I was previously unaware of the efforts of Xerox PARC toward these ends. The details of these devices operation, at the time, were a major innovations and greatly enhanced the field of computing. Also, the author presents a very insightful discussion on how reading became a ubiquitous technology, and how this can be used to define computing (ie. Anywhere you currently read, you could potentially compute as well). This idea, coming from the father of ubicomp, is of enormous significance to the scientific community. The paper mentions that this idea of ubiquity, in the same way as reading, means a drastic change in not only application features, but methods for measuring terse human actions which will eventually define the features of ubiquitous applications.

While the majority of Weiser's paper was interesting and beneficial to the scientific community, there were some holes which needed to be filled in before ubiquitous computing can fully take hold. The issue of privacy and security is one which I feel requires further investigation. The proposal to have relatively loose security seems like a bad idea to me. For example, in a current system, it is impossible to 'break in', until someone finds a way never thought of by the developers of the system. This makes Weiser's argument that someone can 'break in', but it is impossible to be unnoticed, invalid. This presents a possible area of further research: how to guarantee that an unwanted intruder can leave 'fingerprints' which can readily be discovered. Also, if embodied virtuality were to be implemented, I believe much more emphasis should be placed on an individual's privacy than is proposed in The Computer for the 21st Century. Simply having a system that knows where an individual is located at all times represents a serious invasion of privacy. I would propose a system where an individual can disconnect themselves from the ubiquitous network, should they so desire, and possible 'black-out' zones, in which no ubiquitous devices will operate.

A Reference Architecture for Web Servers

2008-09-14T20:15:00.000-07:00

Don't let this paper's name fool you, this one is (at least in my opinion) presenting a process for semi-automatically generating reference architectures for any established domain, not just web servers. Having said that, a lot of really interesting, useful information is presented about three popular servers: Apache, AOL, and Jigsaw. Not particularly deep, however. The methods for automatically generating the architecture were glossed over, as the authors were using an existing tool. Next, they simply refined the architecture until it fit the 3 example architectures. I think it could use more meat. Still, a very interesting (and dare I say,entertaining) read, especially if you're familiar with the web domain.

Oh, and there were no 'results', per se. Wonder if they should have some.

Applying Complexity Metrics to Measure Programmer Productivity in HPC

2008-09-14T19:57:00.000-07:00

This paper is based around a fairly ambitious task: to measure programmer productivity. As far as I understand it, measuring productivity of a worker outside of an assembly line scenario has been an open topic for as long as there have been assembly lines. Despite that, the authors discuss the set of tools they have used to instrument programmer's workstations, and compare and contrast differing methods of performing the same task; from a command line interface and from a GUI. Much to everyone's surprise, the GUI was quicker and easier to use!! (insert sarcasm) The two methods are measured using a metric devised by the authors, which it seems to me suffers from the same problems as the paper on Measuring Configuration Complexity I read earlier last week; the outcome depends wholly on the heuristics used to define and quantize a single step in work (or configuration), which is still a very open research problem. Also, one of the fundamental assumptions used to build the paper's heuristics, that more steps equals more complexity, is disproved by example in the paper's final movement! Overall, a valiant attempt, but I don't think I would endorse this one.

Agile Software Testing in a Large Scale Project

2008-09-14T10:21:00.000-07:00

A fantastic article! This paper presents the findings of a team of developers employed by the Israeli Air Force. This team began the project with very strong focus on Agile processes, particularly TDD and short iteration lengths (at least these are the ones that stuck with me). This team has, self admittedly (at time of publish), one of the most complete sets of data recording an Agile project from inception to delivery. The results show a project which appears to have been run rather smoothly; the up-front testing influenced the amount of defects (compared to what previous result, though?)

More opinions, to be continued ...

In Praise of Tweaking

2008-09-14T09:22:00.000-07:00

This paper presents the results of the second annual Matlab tweaking competition. In this contest, users are tasked with solving an algorithmic problem while minimizing a 'score' created by composing various metrics about the algorithm's performance, such as speed and resources consumed. All submitted algorithms are immediately available to all other contestants, and so a new solution can (and is encouraged) to be created by 'tweaking' someone else's work. In this way, programmers are encouraged to adopt a greater community-centered process.

Although the idea presented was intriguing, this is a magazine article and contributes little to the scientific community.

Storytest Driven Development

2008-09-14T09:16:00.000-07:00

Started off promising, but didn't really deliver. Who writes these story tests? I don't think a customer alone will go though the effort of formalizing the requirements using FitLibrary, but the developer shouldn't be writing them alone, either, because he shouldn't be deciding the requirements of the system. Became harder to follow as it progressed (lots of flipping between figures and paragraphs of text). Also, where are the results? The idea of story tests is presented well enough ,but its not novel, and there is not indication in this paper of who actually uses this approach. Maybe I'm being too picky or it is too late at night. Maybe read this one again later.

Automatic Bug Triage Using Text Categorization

2008-09-14T09:15:00.000-07:00

An extremely fascinating topic concerned with improving development process, not only in open-source projects, but in any project with a reasonably sized team and automatic bug tracking. A real money saver! A well presented paper (even though I got sort of lost in the details of the Bayesian algorithm). Results were presented clearly, even if they weren't as accurate as the authors would have liked. No effort is made to disguise the inaccuracy of the method. I think this idea is worth looking at, just need to swap out the Bayesian algorithm for one that actually works.

Presentations by Programmers for Programmers

2008-09-14T09:13:00.000-07:00

Sounds like a very useful tool (gives the ability to queue up a live IDE session, with the UI tuned for a presentation, along side other presentation materials such as PowerPoint slides). I'm a bit skeptical as to whether the up-front cost of setting up this queue is worth the added visual appeal during the presentation (would have to be more of a formal programmer-to-programmer presentation, like in a conference). Combined some existing products to create this. However, where are the results? There is no study or empirical evidence to support that this product is actually useful. I would classify this as a report on a new product, not necessarily a research paper.

An Approach to Benchmarking Configuration Complexity

2008-09-14T09:06:00.000-07:00

An interesting idea; this paper attempts to quantitatively measure the effort required to properly configure a new piece of software. In their example, they use a web container. This could provide a way to measure, and presumably, compare the configuration effort required between different products. The problem I found with this method was that the fundamental step in the process, determining the atomic configuration actions and assigning probability of failure to them, is an open research problem, and so the results reported in this paper are speculation based on hand-tuned data. Seems like they tried to slip this one past me.
Also, really bad graphs and mislabeled figures.

Inaugural Post

2008-09-14T08:30:00.000-07:00

So this is the first post on my shiny new blog. I tried blogging once before, but it was pretty much just an exercise in setting up a persistent web container on my own hardware, and seeing how long it took for me to get bored of it. Two weeks. Now, seeing as I have neither the time nor the inclination to run dedicated web hardware, I'm using blogger.

What will follow will be a series of short posts containing my opinions of papers I'm currently reading. Enjoy :)