Assignment 02

This assignment is due at 1700 on Friday, 19 September.

The goal of this problem is to write a program that solves an optimization problem involving hanging paintings on a wall. Your program will try to cover as much of the wall as possible without overlapping any paintings; in a variation of the problem, you’ll try to hang a set of the largest paintings that you can, again with no overlap.

I will be updating the assignment with questions (and their answers) as they are asked.

Updated 09-15 with a validator and sample problems. See below for details.

Update 09-16: There is now a blog post describing a simple hill climber.

Update 09-21: Grading criteria have been relaxed.

Pictures, pictures, on the wall

~~My dad~~ An eccentric art collector has a large collection of N paintings. The paintings are rectangular, and of varying sizes.

If there is an empty wall of dimension W units wide by H units tall, the art collector could hang some subset of the N paintings on the wall.

For example, given a 5 x 5 wall, and a set of paintings names and sizes (width x height) as follows:

A: 4 x 3
B: 1 x 4
C: 2 x 2
D: 1 x 2
E: 4 x 1
F: 3 x 1

one possible arrangement the art collector might choose can be depicted as:

A A A A B
A A A A B
A A A A B
C C D . B
C C D . .

where each cell in the 5x5 grid is a character (e.g, A), if it’s occupied by the corresponding painting, or . if it’s empty.

For this assignment, you will write a program to aid the collector in creating arrangements of paintings on walls.

Input data format

We will test your programs using data in the following text-based format.

The input to your program will consist of a description of a wall and zero or more paintings. All numerical values will be integers.

The wall will be described on a single line. This line will consist of width then height, separated by whitespace. For example, a wall 120 units wide and 200 units high would be represented as:

120 200

The (possibly empty) list of paintings will be represented as a sequence of zero or more lines.

Each painting is represented on a single line.
A painting has three parts, separated by whitespace.
The first part is the name of the painting. It is a string, consisting of one or more non-whitespace characters. Each painting’s name is unique.
The second and third parts are the painting’s dimensions, width then height. Some paintings may be of identical dimensions.

For example, one painting might be represented as:

mona_lisa 53 77

The full input is a concatenation of the wall description line and the painting description lines. For example, an input that represents the example problem given above looks like this:

Output data format

Your program should output a wall size, as above, and a (possibly empty) list of paintings and hanging locations, as a series of zero or more lines.

Each painting/hanging location is represented on a single line.
A painting/hanging location has five parts, separated by whitespace.
The first part is the name of the painting, as above.
The second and third parts are the painting’s dimensions, as above.
The fourth and fifth parts are the painting’s integer offset from a fixed corner of the wall. (Because of symmetry, it doesn’t matter which, as long as it is consistent.) The horizontal offset is first, followed by the vertical offset.

For example, the solution given above could be output as:

Evaluating arrangements

The collector is fickle. Of course, he cares not a whit for the aesthetic value of each painting — that’s why the input doesn’t contain a value field. (Though he won’t stoop to rotating paintings. Your program must not swap a painting’s width and height in an attempt to pack more onto the wall.)

But, he cares about the arrangement itself. Here are two possible ways the collector might evaluate a given arrangement. In each case, the collector is attempting to maximize the evaluation function, f.

wall coverage: For a given arrangement, f is equal to the number of square units of the wall that are covered by paintings. For the above example, f = 22.
bigger is better: Assign each painting a value equal to the square of the smaller of its two dimensions. For a given arrangement, f is equal to the sum of the values of each painting in the arrangement. For the above example, f = 3² + 1² + 2² + 1² = 9 + 1 + 4 + 1 = 15.

The worst vice is advice

Some advice and comments, in no particular order:

This is a combinatorial optimization problem (COP). You can solve it with search. As discussed in class, there may be attributes of the problem that will direct you away from general (BFS, IDDFS, A*) search, and toward some kind of local search. Exactly what you do is up to you. In an amusingly meta-moral-of-the-story, there’s not necessarily a best answer, but a number of good-enough ones.
This problem has similarities to the knapsack problem, tiling puzzles, and other COPs. It might help to think about those problems when formulating heuristics and considering your approach.
Many search algorithms, particularly local ones, can find one answer quickly, then keep refining it. This behavior is the defining characteristic of an anytime algorithm. If your solution involves an anytime algorithm, consider writing in such a way that it terminates after around twenty seconds and returns its best answer at that point. Here’s a sketch in Java of a simple way to stop after a timer expires:

public class Arranger {
    private final int ONE_SECOND = 1000;
    private final int TIME_LIMIT = 5;
    private volatile boolean shouldStop = false;

    private int arrange() {
        Thread timer = new Thread() {
            public void run() {
                try {
                    sleep(TIME_LIMIT * ONE_SECOND);
                } catch (InterruptedException e) {
                }
                shouldStop = true;
            }
        };
        timer.start();

        int bestSolution = 0;

        while (!shouldStop) {
            // do some work! or just sleep for a tenth of a second,
            // since this is a demo
            try {
                Thread.sleep(ONE_SECOND / 10);
            } catch (InterruptedException e) {
            }
            bestSolution++;
        }
        return bestSolution;
    }

    public static void main(String[] args) {
        Arranger a = new Arranger();
        System.out.println(a.arrange());
    }
}

You could also check out the Timer and TimerTask classes.

There are two main challenges in attacking this problem, and they’re interrelated.
- Search. You should select a search algorithm that will run in a reasonable amount of time and space given the problem description. As noted at above, you’ll probably use a local search method that explores the state space heuristically.
- Representation. Your decision of how to represent state may seriously impact your program’s memory use and run time. You need to consider both how much space your representation takes, and how easy it is to find successors based upon the current state.
  
  If you’ve made the choice to find good-enough rather than optimal solutions, it’s OK to make other simplifying choices, too. To improve runtime (or just simplify the programming), your representation might not generate all possible successors. For example, each time you add a painting, you might divide the wall into rectangular sub-walls that are either empty or mostly-filled. Then you can recursively examine each empty sub-wall. This approach can’t represent many possible states, but can still find good arrangements in many cases. Of course, if you’re most comfortable with a W x H array representing the wall, that’s fine too.
You may know approaches other than search (e.g., dynamic programming) that you could apply to this problem, especially if you’ve taken CMPSCI 311. I’m not forbidding those approaches, but instead I ask you to consider two things, one practical and one philosophical. First, will your algorithm terminate (or return a valid answer) in less than twenty seconds? And second, since you’re taking an AI class and have to do this assignment anyway, your education is likely best served by learning about and implementing a general method for COPs used in AI. Why not give it a shot?
You are not required to write a validator for this assignment, but doing so is probably a good idea. In particular, paintings must not overlap, nor may they fall outside the bounds of the wall. If you’re feeling generous, you could post your validator (or a link to it) on Moodle. Update: I’ve posted a bare-bones validator/visualizer.
Writing a visualizer might also be a good idea. For example, given a restricted output format where paintings only have single-character names, a visualizer might output a depiction similar to the one at the start of this assignment. This visualization will help you see what your arranger is doing and save you time that you might otherwise be spending making tedious drawings on graph paper. As with the validator, you’re welcome to post or share your visualizer.

A validator / visualizer

You can download a simple validator here: a02_validator.py.

It requires Python 3+, which is installed as ~~python3.4~~ python3.2 on the edlab Linux machines. It takes three arguments: an evaluation function (coverage or bigger), a valid problem description, and a solution. It attempts to validate the solution, prints a visualization of the solution (only if each picture name is a single character in length), and prints the value of the evaluation function on the solution.

The error-checking is fairly bare-bones, and it will just exit with an AssertionError if the solution is invalid.

Please email me if you notice any problems.

Sample problems

I’ve generated some sample problems, and produced solutions for them, as generated by a simple hill climber. You can download them here: a02_samples.tar.gz.

Having your program work (or outperform the SHC) on these samples does not guarantee that it will do likewise on all grading cases. But they should give you a sense of your target. Some grading cases will differ (larger walls, more or fewer paintings, etc.).

What to submit

You should submit two things: an arrangement generator and a readme.txt.

Your arranger should use its first command line argument to determine the evaluation function (either coverage or bigger). It should use the second command line argument as the path to an input file. If, for example, your arranger’s main method is in a Java class named Arranger, I should be able to use java Arranger coverage /Users/liberato/testcase to direct your program to read the input in the file located at /Users/liberato/testcase.
Your arranger should print its generated arrangement to standard output in the format described above.

Submit the source code of your arranger, written in the language of your choice. Name the file containing the main() method Arranger.java or your language’s equivalent. If the file you submit depends upon other files, be sure to submit these other files as well.

As in the previous assignment, while you may use library calls for data structures and the like, you must implement the optimization method you use yourself. Do not use a library for search or optimization. I will consider it plagiarism if you do. Check with me if you think there’s any ambiguity.

Your readme.txt should contain the following items:

your name
if the language of your choice is not Java, Python, Ruby, node.js-compatible JavaScript, ANSI C or C++ (or if you’re concerned it’s not completely obvious to me how to compile and execute it), a description of how to compile and execute the submitted files
a description of your optimization strategy, your representation, and how they interact
a description of what you got working, what is partially working and what is completely broken

If you’re using language features that require a specific version of your language or runtime, check for that version at program start and fail if it’s not present, emitting an understandable error message indicating this fact. Your program must compile and execute on the Edlab Linux machines.

If your program does not compile or execute, you will receive no credit. Check with me in advance if you’re concerned.

Grading

We will run your program on a variety of test cases. The test cases will not be available to you before grading. You are welcome to write and distribute your own test cases (or test case generators).

If your program generates valid arrangements that are about as good ~~(+/- 5%)~~ (at least 95% as good) as those generated by a program that implements a simple, single run of hill climbing, and does so on at least 75% of the test cases, you will earn 75% of the available points. You will earn proportionately fewer points for a program that fails to do as well as as simple hill climber (SHC) on some proportion of test cases.

~~If your program outperforms a SHC on 85% or more of the test cases, you will earn 85% of the possible points.~~ For each percentage point of the test cases above 75% that your program’s performance equals or exceeds that of a SHC, you will earn an additional 1% of the available points, up to 95%.

If your program significantly outperforms (by ~~10%~~ 5% or more) the simple hill climber on ~~all~~ 75% or more of the test cases, you will earn all available points.

If your readme.txt is missing or judged insufficient, you may be penalized up to ten percentage points.

We’re not going to feed your program incorrectly formatted input, so you need only concern yourself with handling input in the format described in the assignment. We expect valid output. Generating output that is not in the format described in the assignment will result in a failed test case.

You should look for edge cases and make sure your program doesn’t fail or behave pathologically on them.

Some of the test cases may push at the boundaries of what your program will be capable of, depending upon your choice of search strategy. If your program exceeds available heap memory (which we’ll set to 1 GB in Java, using the -Xmx1024M argument if necessary), or if it does not terminate in twenty seconds, we will consider the test case failed.

Questions and answers

We need to keep it under 20 seconds, but there’s a little bit of set-up and cleanup that needs to be done. What do you think a safe timeout is assuming we start it before the set-up and have it interrupt before the clean up while still giving as many search iterations as possible? I was thinking something like 18 or so seconds leaving the remaining two to set-up and clean-up.

Yes, that’s fine. Typically a hillclimber improves asymptotically toward the optimal solution in, for example, 80/20 way. That is, on average, they’ll get 80% of the way from an average solution to an optimal solution in the first 20% of their tries. (Of course, it might be a lower ratio, but you get the idea).

On expectation, the last second or two of search are not as important for improvement as the first few seconds. You might find a better solution at any time (assuming you haven’t found an optimal one yet), but your best-known solution is improving over time, so the amount of improvement is likely to be smaller the longer you search.

A good idea on how to write test-cases for the coverage problem is to hand-divide a wall and see if it comes up with the optimal solution.

That’s true, though there’s no guarantee any particular instance of the coverage problem will have an optimal solution that fully covers the wall.

Are we able to assume that there are no duplicate paintings in the sets that we’re given? I don’t think most hill climbing algorithms would have much trouble dealing with duplicate paintings popping up, but was more asking out of curiosity.

It depends upon what you mean by “duplicate.” Each painting’s name is unique (as noted under “Input data format”), though not necessarily unique in its dimensions. So one valid input might be:

10 10
a 1 2
b 1 2

But the following is invalid:

10 10
a 1 2
a 1 2

as is this:

10 10
a 1 2
a 3 4

In both, the unique name constraint is violated. I won’t test your program on such invalid inputs.

Which evaluation function was used by the simple hill climber when solving the sample problems?

It assigns values by picture area, that is, it sorts the candidates list by (width * height), the same as the hill climber we wrote in class.

It is, however, slightly smarter about sub-wall bisection than the version we wrote in class.

It can certainly be outperformed by the margins suggested in the grading guidelines, above, using the techniques we discussed in class, and that are outlined in Russell and Norvig.

But if people find it too difficult to match, I may degrade its performance to that of the hill climber we wrote in class.

There are two ways of evaluating as stated in the documentation: “wall coverage” and “bigger is better”. Which one is more important as far as our heuristic is concerned?

We will evaluate both during the grading. If you are running short on time, then focus on coverage, as it’s more straightforward. In many instances, it will do fairly well on bigger-is-better as well. Regardless, though, it should be a minimal change for any local search algorithm to swap out the evaluation function. So I’d start by picking the heuristic to be the evaluation function itself, and only change if you think you can develop something better.

And the grading is based on comparing our results against the results from a simple hill climbing implementation unknown to us (different from the one we did in class), right?

Yes. It’s very similar to the one we did in class. In particular, it chooses the orientation of sub-walls such that the bigger region is oriented along the same axis as whichever has more space available.

I tried running the validator using python3.2 a02_validator.py coverage 10x10-01.problem 10x10-01.solution, but it just outputs random quotation marks and #s.

It’s not random: that’s the output.

It’s printing the paintings by name, and some of them have names that are punctuation marks (as they’re all single characters). Look at the .solution file to see:

The below is the output you’d expect on the input you’ve described. f is the value of the coverage heuristic for this layout.

"""""####&
"""""####
...%(*0/
...%(*0/
...%(*0/
!!!%(*0/
!!!%(*0
   %(*0
   %(*0
   %
f = 67

CMPSCI 383: Artificial Intelligence

Fall 2014 (archived)