Input and output attacks

I’ve recently started leafing through one of the testing books available in the 28th floor bookshelf. The book is entitled How to Break Software by James Whittaker. And it conveniently has a summary of input and output attacks.

Summary of the Input/Output Attacks — A Checklist for Battle

  1. Make sure you see all error messages at least once by applying invalid input. Think of invalid inputs that the developers might have missed.
  2. Force the software to assign its default values for any internal variable that can be set through the user interface. First display and accept existing values. Then assign bogus values to force the software to calculate good ones.
  3. For every input field, enter values of the wrong type and values that represent strings that may be treated in a special way. Study the OS and programming language and make a list of possible problematic strings. Apply them all in every test entry field.
  4. In every input field enter the maximum number of characters allowed.
  5. Find input panels where a number of inputs are entered before pressing “OK”. Determine legal values for each individual field and try combinations that represent an illegal set of inputs.
  6. Find places where inputs are accepted and apply the same input or series of inputs over and over. Choose inputs that cause some underlying computation or data manipulation over inputs that are simply displayed on the screen.
  7. Pick an input, apply it to the software under test, and note the output. Think about other outputs that could occur when this input is applied in other situations. Apply the inputs in these other situations to ensure that each such output is observed during testing.
  8. Think of outputs that the software cannot or should not generate. Find a combination or sequence of inputs that will cause one of these illegal outputs to be generated.
  9. Apply an input that generates an output with some observable and changeable property, such as size. Force the property to change.
  10. Determine when the software under test is refreshing the screen. Create situations where the software refreshes the screen too often or in which it fails to refresh what it should.

Below’s my elaboration on the attacks. But, of course, the book provides better and more comprehensive details.

Input Attacks

1. Make sure you see all error messages at least once by applying invalid input. Think of invalid inputs that the developers might have missed.

Common examples of bugs found through this attack include unhandled exceptions. Instead of seeing the project standard and more user-friendly error messages, we sometimes come across ORA- errors, System.IO.IOException, NullReferenceException, etc. Another example of bugs found through this attack is when we come across incorrectly written error messages that have typos or need to be reworded.

In one search screen that had a date field as one of its search parameters, I entered a non-existent date value (e.g., Nov 31, Feb 29 on a non-leap-year). When I ran the search, instead of getting an error message informing me that the date I entered was invalid, an error was raised saying my search retrieved too many rows and it asked me to narrow it down by specifying additional criteria.

2. Force the software to assign its default values for any internal variable that can be set through the user interface. First display and accept existing values. Then assign bogus values to force the software to calculate good ones.

In one of my projects, we store info into XML files. What happens is that on load of the screen, the values are retrieved from the XML file. What we tried to do is simulate cases wherein our application had to detect that there was a problem in the XML file and it had to recreate the XML file.

3. For every input field, enter values of the wrong type and values that represent strings that may be treated in a special way. Study the OS and programming language and make a list of possible problematic strings. Apply them all in every test entry field.

In one of the projects, there was this ID field that contained numeric data. However, once the ID reached a value greater than 999, the field got reformatted such that it contained a comma e.g., 1000 got translated to “1,000”. This then triggered an error since the program was trying to insert a string value into a numeric field.

Some examples of special characters to consider are single- or double-quote marks which could be misinterpreted as the closing quotation mark, or two dashes which might be mistaken for the start of a comment. When working with screens which accept values and append them into the URL, playing around strings containing an ampersand might be a good idea since ampersands are sometimes used to separate sets of parameters in the URL.

Unicode characters can also be potentially problematic. In one of my projects, certain unicode characters have yielded quite unexpected results — from triggering an unhandled exception to an actual lost of data.

4. In every input field enter the maximum number of characters allowed.

This can help capture inconsistencies between the screens and the db.  Say, the screen allows 10 chars from one field, while db field can only hold 5. Likewise, this can also help detect if there’s an inconsistency the size of fields between the main table and the corresponding journal table.

5. Find input panels where a number of inputs are entered before pressing “OK”. Determine legal values for each individual field and try combinations that represent an illegal set of inputs.

In one project, there were several views that can be enabled or disabled, and the display of screen items depend on the enabled views. Certain combination of views had triggered some erroneous behavior.

6. Find places where inputs are accepted and apply the same input or series of inputs over and over. Choose inputs that cause some underlying computation or data manipulation over inputs that are simply displayed on the screen.

Using PowerPoint, it is possible to have nested bullet points (i.e., bullet point items under a bullet point item which is under another bullet point item and so on). When I tried a PowerPoint-to-Flash converter, it turns out that the converter couldn’t manage the nested bullet points in my slide notes.

Output Attacks

7. Pick an input, apply it to the software under test, and note the output. Think about other outpus that could occur when this input is applied in other situations. Apply the inputs in these other situations to ensure that each such output is observed during testing.

In one web app, I noticed that the value I entered into one of the fields got set to upper case on blur. There was a second way of entering values to that field but it didn’t set the value to upper case. For the same field, the input value ought to have been treated the same way.

In the same web app, I once tried searching for pending records that were due on a particular date. The search went without any hitch. I then tried searching for resolved records that were due on the same date. I got disoriented a bit when the results indicated ‘No due date’ instead of displaying the actual due date.

8. Think of outputs that the software cannot or should not generate. Find a combination or sequence of inputs that will cause one of these illegal outputs to be generated.

When testing web apps, we sometimes try to inject some html into the fields because sometimes upon retrieval, the screen renders the html rather than display the actual string that we entered. For instance, you enter “<b>happy birthday</b>” and once you retrieve it, you only get “happy birthday” in bold instead.

This item also reminds me of the NaN (not a number) bug in the date picker. For instance, I enter a valid date into the a date field edit box and then I click on the date picker icon, what happens is the date I entered is the date selected in the date picker.  But as Dhon had shared, if you enter “01-0a-2008” and then click on the date picker icon, “NaN” is displayed for all month and day values in the date picker.

9. Apply an input that generates an output with some observable and changeable property, such as size. Force the property to change.

In one web app, for a more personalized touch, it displays the user’s nickname on the screen. In one instance, it had displayed “Guest” instead of my nickname even though I was logged in. In some other instance, right after I modified the nickname value, the old value still got displayed instead of the new one.

10. Determine when the software under test is refreshing the screen. Create situations where the software refreshes the screen too often or in which it fails to refresh what it should.

There was this case wherein a record got retrieved, and then updates were made onto the record. Upon saving, an error was raised and the page refreshed with the values retrieved from the db. Instead of being able to make corrections based on the error message, the user would have to reapply all of the unsaved changes from before.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s