Research Evolution in Software Development

Research Evolution in Software Development

Introduction

Software development research has changed significantly over the past 20 years.  While in the late 1980s researchers were concerned with user acceptance of computers in general (Forrest, Stegelin &  Novak 1986), in the past few years researchers have been more concerned with why people accept or reject certain software products, and not others (Khanfar,  et al. 2008; Garrity et al. 2007).  This literature review will explore the changes in research methods by taking a sampling of research articles from the late 1980’s, and articles from recent years. 

This literature review will examine 27 peer reviewed articles relating to software development.  Sixteen of these articles were written in the past 5 years and eleven were written in the late 1980’s.  Using these articles, I will examine the changes over time to the software research process.

By reviewing the differences in software development research I will get a picture of where software development research is going in the future.  In this review I will look at information such as sample size, methodology, statistical analysis being preformed, and other factors such as use of students in research.  This paper will show the trends in software development research and how they affect researchers.

Literature Review

Research methods

Although the basic way people research has not changed over the past 20 years, the tools available to researchers and the methods that they tend to use has.  In my study of 27 articles I found that 54% of articles written before 1990 were qualitative, while only 6% of current articles written in the past five years used a qualitative method. 

Quantitative methods are being used more frequently in recent years.  This trend may reflect the ease in which data is easily computer tabulated or may be due to a preference by researchers for hard data.  When Ali Montazemi researched user satisfaction (1988), he chose to interview people.  In these seminal types of interviews he found information that quantitative research could have missed.  Montazemi, for example, he found that 20% of people would have preferred a query system that gave them “what-if” scenarios.  Today, this type of research today would be a separate topic of decision support systems.  In contrast, when Garrity et al. did his research on user satisfaction on websites, he chose to have users use a quantitative study.  With modern tools Garrity et al was able to do a more extensive statistical analysis using statistical techniques such as average variance extracted, sum square, and f-test (p. 27).  This type of analysis speaks to the growing maturity of software development research. 

Modern researchers have more tools to analyze data.  Qualitative data lends itself more readily to this type of analysis.  Modern tools make it possible to analyze more data. There are tools that exist now that did not exist 20 years ago.  There are also more categories of research than there were 20 years ago.  In the research field of software development researchers are looking at project management, testing, security, websites and usability to name a few. 

Sample Populations

Comparing research between the two periods shows that surveys today can have far larger populations.  Data collected from the research on types of samples from twenty years ago shown in Table 1 shows a sample size that is far smaller than the sample size of data taken from fore recent times shown in in table 2.  The average sample size for research taken 20 years ago is 113.  The average sample size for research taken today is 7150.  This is primarily due to the availability of data.  Most of the data that was collected 20 years ago was obtained through face to face surveys.  One exception to this was a museum experiment.  In this experiment, museums were equipped with a Hypertext display that allowed users to review information about the museum.  The data collection was limited when one of the two monitors that they were using broke (Shneiderman et al, p. 49).  The price of monitors has come down in recent years and this problem would most likely be fixable in today’s environment.

Today researchers have access to more tools that give them a much greater ability to review data. Crowston et al, for example, was able to collect data on over 100,000 open source projects (2008).  This type of data collection would be unheard of 20 years ago.  Large scale systems like the Internet were not in place then and the ability to harvest vast quantities of data was nonexistent.

            Now researchers have new techniques for sending surveys.  Phone and mail surveys can be expensive in both time and money.  Modern surveys can be done cheaper and more efficiently.  When one group of researchers, for example, wanted to have a large sample size they sent emails out to 10,000 people, and 3276 people responded (Tam et al. p. 280).  First class postage as of the time of writing this is 42 cents.  Sending this type of survey through the mail would cost $4,200.  This type of expense is out of the range of many research projects.
            When looking only at direct surveys of people without the help of modern data collection methods there is virtually no difference between the two time periods.  The data collected in the 1980’s had, on average, surveyed 123 people while the data collected in the past few years had on average 124.  Interviewing and surveying people clearly takes longer than more modern methods.  While these interview may be necessary in qualitative research the amount of data collected can be quite small compared to data mining through already existing information systems or email surveys.

Statistical Analysis

 Literature from the two periods did not show a change in the way that statistics are done.  The articles contained a wide variety of methods.  Earlier research involved more averages.  This can be attributed however, to the number of qualitative studies.  Qualitative lend themselves more readily to averages. 

One thing that is apparent in the studies is that modern research employs more statistics.

With the advent of research software, researchers are able to take the same data and quickly run multiple statistical analyses to help determine the best analysis.  Table 3 shows the statistical methods used in research in the 1980’s.  In table 3 there is only one test preformed on each set of data.  In more recent studies, as seen in table 4, one can see that researchers performed more kinds of statistical analysis.  In the case of Banker et al the group performed three tests to analyze their data (2006).

The analysis showed a propensity toward using multiple statistical analyses in research in more recent research.  This may be due to the more detailed work on the subject.  Modern research delves into subject such as usability of websites.  Research from 20 years ago delved into broader questions such as the acceptance of computers into workplaces.

Software products such as SPSS make it possible to take a dataset and mine it for correlation.  Garriey et al (2007) did this when they looked at three different statistics to come to their conclusion. The trend now is to do a more in-depth analysis.  Researchers have the ability to gather vast amount of data and perform complex statistical analysis on them.   Researchers 20 years ago did not have this option.  For them research often had to be done by hand.  This would naturally lead to more work on the part of the researchers.  Performing statistical analysis by hand can be a time consuming process.  While today performing a complex statistical analysis could take hours in the past it could take days.  Statistical analysis without the benefit of statistical software can be wrought with mathematical mistakes.  Human error plays a larger factor when surveys have been hand coded then that data has been hand analyzed.  

Response Rates

Response rates between the two time periods vary.  In the 1980's researchers used more college students who were forced to take the surveys (Jarvenpaa et al, 1988; Dos Santos et al, 1988; Kirs et al. 1989). They also relied on people volunteering (Jarvenpaa et al, 1988) more often.  These types of candidates often do not represent the average person from the population.  When a general inquiry is made for volunteers or participation is compulsory there is no response rate.  We see only two response rates in the earlier surveys.

There were two response rates out of our 10 research articles.  Of these the response rates were 36% and 47%.  These rates are far lower than the response rates in more recent studies.  The response rates for data collected for people is 81%.  Even email response rates in our survey are 32% which is closer to standards 20 years ago than today's standards.

Response rates differ depending on the type of medium that people are using to survey.  An email to CEOs may have a lesser response rate than someone standing at a shopping mall with a clip board.  The more people feel connected to the researcher the more they are willing to participate in the research.

In articles of the past 20 years there is a tendency away from giving money to participants the way Luzi et al, did in their study on study on performance (1984).  Giving money to people can be a great motivating factor.  This motivation may affect research.  For example, in the Luzi et al article the money given away to top performers may make them act in a way that they would not at work.  In this way they made the respondents compete.  This type of competition could introduce factors that the researchers may not want; the study would be meaningless to any situation other than one where incentives are given out.  This type of research can introduce skewed results.  Some people may work better under the pressure that money and competition brings while others may get confused by the pressure.  It appears that in modern research this practice is less likely. 

Research Themes

Seven out of the ten articles from the 1980's dealt with computer usability issues.  Jarvenpaa et al dealt with how groups interact with computers (1988), Dos Santos et al dealt with user interface issues (1988), Montazemi (1988), DeLone (1988), and Williams dealt with user satisfaction.  As software development gets more mature researchers delves into new areas.  What was once a new topic such as user satisfaction has been replaced by more mature themes such as personalization (Tam et al, 2005). 

Topics such as information overload have been explored and solutions such as drill down menus and customization are common place.  In the 1980's these were still topics that needed discussion on the foundation level(Dos Santos et al, 1988).These are all ideas that have evolved as the technology evolved. 

Today new topics in software development are emerging.  Issues on new software products such as software that allows people to share knowledge and work together are still in their infancy (Taylor, 2004).  These issues will evolve as technology evolves.  Collaboration software such as Google Document which allows people to work on the same document at the same time is still in their infancy.  Wiki's are only a few years old.  As the ability for people to work together and share information grows the need for research to help people use that technology to reach their goals will also increase.

Four times in this review of recent materials there are articles on software errors.  Bugs in software always been a major issue with software development, but now researchers have better tools to help the programmer deal with these bugs.  In the article A Replicated Survey of IT Software Project Failures (2008) there is a meta-analysis of various reports of why software fails.  This body of information did not exist a few years ago.  Now with the ability to collaborate researchers have access to a vast hither unto unimagined body of knowledge. 

Crowston et al take this idea one step further in their article Bug Fixing Practices within Free/Libre Open Source Software Development Teams.  This article takes 100,000 records from open source projects and analyzed the way software bugs are dealt with. 

Technology has also given us the ability to do things that did not exist a few years ago.  For example Van Pham and Gopal et al write about outsourcing.  This relatively new practice has undergone much research over the past few years.  It is a complicated issue.  Off shoring software development may risk handing over company secrets to people who do not work for the company (Van Pham, 2006) but in doing so companies can create wealth by reducing the cost of software development.  This is a topic that researchers have looked into in great detail over the past few years.  As companies face economic difficulties, many of them face the inevitably that they must reduce cost to stay in business.  The debate over outsourcing and what to outsource is a topic that will plaque researchers for years to come.  There is no one right answer for every company.  In the end as a practitioner one must use the available knowledge to make a decision.  In research every decision answer is either proven or disproved, as a practitioner decisions often have many sides to them.  This discussion of off shoring is one of those issues.

Another topic in modern research is inner-organizational software development (Robey et al, 2008).  Gone are the days of standalone software.  Today organizations want software to be able to communicate.  When software crashes the help desk needs to know.  When there is potential fraud detected in one system the other need to be aware of it.  Also the ability to transfer information from one system to another is crucial in such systems as decision support systems.  In these types of systems, information is gathered from various sources within the organization and given to decision makers who can view the larger picture of the organization.

In software development inner-organizational software can help companies collaborate and use their time more efficiently.  Inner-organizational software is developed with common interfaces such as XML or Application interfaces.  Software packages that companies buy can also have ways to interface with them programmatically.  Interoperability is standard practice for software being developed today.  

These ways of working with software are becoming wide spread.  In their article Theoretical Foundations of Empirical Research on Inter-organizational Systems: Assessing Past Contributions and Guiding Future Directions, Robey et al use seminal research to show the foundations of inter-organizational systems (2008).  The authors note that it took several years for the idea off inter-organizational research begun to spread. In research developing the seminal research is important but there may be a lag between the research itself and the practical application of that research.

Management of software development is an issue that has been the issue of a good deal of research. Todays software products are more complicated and expensive.  Researchers today are looking for ways of minimizing risk.  As well as the ability to overcome things witch might lead the project to failure.

Researchers are also looking at the way software is build. Sugumaran et al's article on software time lines (2008) delves into ways that programmers can manage expectations and develop time lines that will help lead to the success of the project.  This type of software reflects what is going on in the industry with new project management methodologies such as Last Planner and Scrum.

Personalization is another issue researchers research today. Personalization is an emerging technology.  Software development of personalization is a complex task.  Tam et al (2005) in their article Web Personalization as a Persuasion Strategy use seminal research in the fields of psychology to argue their case.  The easy availability of research in other fields such as psychology help made new inroads in research.  In their article they talk about how familiar landmarks can help the user process the page more quickly. Users catch more messages from the web page when there is some sort of familiarity.  With the advent of technologies that allow for more customized user interfaces such as hypertext markup researchers are concerned with how the interface makes people feel.  Twenty years ago a button on a program would most likely be the one that came standard with the operating system.  Today with web pages the button can have gradients, borders, and have movement. Research is showing us this type of customization can really affect use of the site. 

This type of cross-disciplinary research affects the way people make software products.  Articles involving biasness, marketing and psychology are just some of the disciplines that researchers are looking at when they research software development.  Today software like video games can have advertisement built into them.  This type collaboration between businesses was unheard of twenty years ago. 

Authors

There are more authors for each article in more recent articles.  As technology allows people to work collaboratively there is a greater collaboration among researchers.  In the review of older articles there is a 60% collaboration rate. In more recent articles that number has climbed up to 93%.  Today researchers have the ability to send drafts via email as well as chat on a free service.  Twenty years ago the only options that a person had if they did not live in the same town was to use the mail.  This would have made collaboration prohibitively difficult.   

Subjects

In many of the older articles there is a greater usage of students as subjects.  These students are often compulsorily made to answer questions.  Fifty percent of the articles from the 1980's used students for their research.  While only 12% of more recent article do.  Chart 3 shows the difference in the two different groups.

The disfavor of the use of students could be for a number of reasons.  The first of which is that students are often coerced into doing the surveys.  They are often a requirement for classes. This type of coercion can produce poor results in the studies because students may not truly represent the larger population.  In Jarvenpaa et al's research they used students to test the usefulness of group collaboration software.  The flaw in this study is that these students are not the typical user of group software and thereby have no context for witch to use it.  If the researchers had used business people they may have had a real world context for using the software.  This could have drastically changed the outcome of the research.

When there is no coercion researchers in the past often use bribes to encourage the student to attend. These bribes can be as detrimental.  Students may behave differently due to a bribe.  This can taint the results.

Analysis

Software development research is going in many directions.  Software development is a relatively new field which is always changing.  Research on software development evolves as software evolves. 

The availability of data, and new ways to analyze data, is causing a steep increase in the amount of quantitative research being done.  Today’s researcher can gather data from various sources.  They can also access enormous amounts of articles on the subject.  They have the ability to collaborate like never before.  Researchers today have cross disciplinary information readily available leaning to new ways of looking at software development.

Today when developing software, people use a broad range of information brought about by research.  Software developers of today look at usability and customizability.  They look at the psychology of getting people to use their products and the marketability of their products.

Conclusion

The future of software engineering research is bright.  There are new devices that require an entirely new approach.  Small devices like cell phone are starting to get a good deal of use.  In the future researchers will look at such devices and look at all of the aspects that they did with software and web development.  Researchers will look at why people use these devices.  They will look at ways of building quality software that provide the user with a sound user experience.  These types of devices lend themselves to more research because for the first time location plays a part of the equation.  New devices are location sensitive.  Giving users contextual information depending on the place they are in.

This paper discusses how research has changed over the past 20 years.  Although research methods have not changed the ability to gather large amounts of data and then analyze them has.  There are also more people willing to participate in research.

Researchers are more willing to use different methods to gather data such as email.  There are also trends away from using students in research.

Managing projects has become an issue in software development.  With large projects come large costs.  Software research is looking into ways of managing risk.  Software research is also looking into ways of determining the time and cost of software projects.  There are also more options available to software developers than ever before.  Today’s programmers can use geographic data from a service.  They can interface with small devices such as phone or have access to unprecedented processing power and storage through web services and cloud computing.  Today’s software developers have access to tools such as knowledge bases and research to aid in there professionalism. 

The research of the past has aided with things such as group interactions and user interactions.  The research of today takes those seminal studies and builds upon them to take us toward the future of research.

   

References

Armstrong, D. J., Nelson, H. J.,  Nelson, K. M., & Narayanan V. K. (2008). Building the IT workforce of the future: the demand for more complex, abstract, and strategic knowledge. Information Resources Management Journal, 21(2), 63-79.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Banker, R. D., Bardhan, I., & Asdemir, O. (2006). Understanding the impact of collaboration software on product design and development. Information Systems Research, 17(4), 352-373,440.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Brockhoff, & Klaus.  (1984). Forecasting quality and information. Journal of Forecasting, 3(4), 417.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Capra, E., Francalanci, C., & Merlo, F. (2008). An empirical study on the relationship between software design quality, development effort and governance in open source projects. IEEE Transactions on Software Engineering, 34(6), 765-782.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Crowston, K., & Scozzi, B. (2008). Bug fixing practices within free/libre open source software development teams. Journal of Database Management, 19(2), 1-30.  Retrieved March 1, 2009, from ABI/INFORM Global database.

DeLone, & William H.  (1988). Determinants of success for computer usage in small business. MIS Quarterly, 12(1), 51.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Dos Santos, Brian L.,  Bariff, & Martin L. (1988). A study of user interface aids for model-oriented decision. Management Science, 34(4), 461.  Retrieved February 28, 2009, from ABI/INFORM Global database.

Doll, William J.,  Torkzadeh, & Gholamreza. (1989). A discrepancy model of end-user computing involvement. Management Science, 35(10), 1151.  Retrieved March 1, 2009, from ABI/INFORM Global database.

El Emam K.,  & Koru A. (2008). A replicated survey of it software project failures. IEEE Software, 25(5), 84-90.  Retrieved February 17, 2009, from ABI/INFORM Global database.

Garrity, E. J., O'Donnell, J. B., & Kim, Y. J., & Sanders, G. L.. (2007). An extrinsic and intrinsic motivation-based model for measuring consumer shopping oriented web site success. Journal of Electronic Commerce in Organizations, 5(4), 18-38.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Gopal, A., Sivaramakrishnan, K., Krishnan, M. S., & Mukhopadhyay, T. (2003). Contracts in offshore software development: an empirical analysis. Management Science, 49(12), 1671-1683.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Jarvenpaa, Sirkka L.,  Rao, V. Srinivasan,  Huber, & George P. (1988). Computer support for meetings of groups working on unstruct. MIS Quarterly, 12(4), 645.  Retrieved February 28, 2009, from ABI/INFORM Global database.

Khanfar, K., Elzamly, A., Al-Ahmad, W., El-Qawasmeh, E., Alsamara, K., & Abuleil S. (2008). Managing software project risks with the chi-square  technique. International Management Review, 4(2), 18-29,77.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Kirs, Peeter J.,  Sanders, Lawrence G.,  Cerveny, Robert P.,  & Robey, Daniel. (1989). An experimental validation of the Gorry and Scott Morton fr. MIS Quarterly, 13(2), 183.  Retrieved February 28, 2009, from ABI/INFORM Global database.

Luzi, A. D.,  & Mackenzie, K. D. (1982). An experimental study of performance information systems. Management Science (pre-1986), 28(3), 243.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Lucas H. C.  (1981). An experimental investigation of the use of computer-based graphics in decision making. Management Science, (pre-1986), 27(7), 757.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Montazemi, & Ali Reza.  (1988). Factors affecting information satisfaction in the context o. MIS Quarterly, 12(2), 239.  Retrieved February 28, 2009, from ABI/INFORM Global database.

Park C. W., Im, G., & Keil, M. (2008). Overcoming the mum effect in it project reporting: impacts of fault responsibility and time urgency*. Journal of the Association for Information Systems, 9(7), 409-431.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Pendharkar, P. C.,  & Rodger, J. A. (2007). An empirical study of the impact of team size on software development effort. Information Technology and Management, 8(4), 253-262.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Robey, D.,  Im, G., & Wareham, J. D. (2008). Theoretical foundations of empirical research on interorganizational systems: assessing past contributions and guiding future directions. Journal of the Association for Information Systems, 9(9), 497-518.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Shneiderman, B., Brethauer, D., Plaisant, C., & Potter, R. (1989). Evaluating three museum installations of a hypertext system. Journal of the American Society for Information Science (1986-1998), 40(3), 172.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Stegelin, F. E., & Novak, J. L. (1986). Attitudes of agribusiness toward microcomputers. Agribusiness (1986-1998), 2(2), 225.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Sugumaran, V., Tanniru, M., & Storey V. C. (2008). A knowledge-based framework for extracting components in agile systems development. Information Technology and Management, 9(1), 37-53.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Tam, K. Y., & Ho, S. Y.. (2005). Web personalization as a persuasion strategy: an elaboration likelihood model perspective. Information Systems Research, 16(3), 271-291.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Taylor, W. A.  (2004). Computer-mediated knowledge sharing and individual user differences: an exploratory study. European Journal of Information Systems, 13(1), 52-64.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Tsai, M. T., & Su, W. (2007). The impact of cognitive fit and consensus on acceptance of collaborative information systems. The Business Review, Cambridge, 8(2), 184-190.  Retrieved March 1, 2009, from ABI/INFORM Global database.

Van Pham K.  (2006). Strategic off shoring from a decomposed COO's perspective: a cross-regional study of four product categories. Journal of American Academy of Business, Cambridge, 8(2), 59-66.  Retrieved March 1, 2009, from ABI/INFORM Global database.