While this (frankly, monster of a) blog post represents my own perspective, it benefitted hugely from feedback from members of the broad AIIDE community. Special thanks to Max Kreminski, Markus Eger, Nathan Sturtevant, Jurie Horneman, Gillian Smith, Arnav Jhala, Mike Cook, Chris Martens, Levi Santana de Lelis, and Gema Parreño. I greatly appreciate your insightful comments and questions!
If you’d like to see the original draft blog post Google Doc with the comments or follow along with it as you read this blog post you can find that here.
So You Want to Write an AIIDE Paper
The AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE) is my favorite conference. It’s a smaller conference, with around 120 attendees, making it a tight-knit (but welcoming!) community of academics, practitioners, and others interested in AI and digital media. However, it’s still a fairly selective venue, with an acceptance rate at ~25%. It’s this acceptance rate that prompted me to write this blogpost, in the hopes that it might be help to bring more people into this excellent community.
In case you see that ~25% figure and balk, I should note that while only 25% of AIIDE submissions are accepted for a full oral presentation, a further 25% are accepted as poster presentations, which have the same page limit and are included in the proceedings. This means you still have a good shot at a publication, despite the selective oral presentation acceptance rate.
First, in this blogpost I’m going to address how to know if a research project might make a good AIIDE paper. From there, I’ll cover my personal general structure for AIIDE papers and some tips on what to put in each section.
Before I get started, you might be wondering why you should listen to me. While I’ll note that this is only my personal take on the subject, I do believe that I’m pretty good at writing AIIDE papers. In the past five years I’ve had a personal AIIDE acceptance rate of ~70%. Further, two of these papers have been nominated for AIIDE’s best paper award, and I was on AIIDE’s organizing committee for 2019 and 2020.
How to Know if Your Research Might Make a Good AIIDE Paper
The general guide I use to decide whether some research would be good for AIIDE is whether it led to learning novel and valuable knowledge at the intersection of Artificial Intelligence and Interactive Digital Entertainment. This might seem relatively straightforward (given the name of the conference), but I find that it’s helpful inclusion/exclusion criteria. Every section of an AIIDE submission should be about proving your work’s relevance to AIIDE, the novelty of the learned knowledge, and the value of said knowledge. More on that later.
In terms of getting a bit more clarity on what topics are relevant to AIIDE, I’d recommend checking out the Topics of Interest section of the annual Call for Papers. These topics come from the organizers themselves, based on prior topics at AIIDE. You can also check the topics of the sessions at prior AIIDEs. For example, the program for AIIDE 2020 had the following session topics: Generation from Few Examples, Increased Understanding, Better Characters, Improved Accessibility, Narrative Generation, Platformer Level Generation, Understanding Algorithms, and Bridge to Industry.
In comparison, the program for AIIDE 2019 had: Game Design, Search & Planning, Reinforcement Learning, Learning in Games, and Computational Narrative. The organizers create these topics by grouping related, accepted papers, and so they tend to be more community-driven and can vary more from year-to-year.
But only looking at the past isn’t going to help push a field forward or lead to innovative research. From my years at AIIDE I’d suggest that if your research involves AI for generating, adapting, playing, or analyzing any part of a game, interactive narrative, or other experimental interactive digital environment, it’d be a good fit. This can get complicated or fuzzy at the edges. For example, Smith’s Generative Design for Textiles: Opportunities and Challenges for Entertainment AI (on generating textile patterns digitally), or Lamb’s Crowdsourced Social Media Poetry (on generating poetry based on Twitter text), or Ferreira et al.’s Computer-generated Music for Tabletop Role-playing Games (on generating music to match the tone of a typically non-digital game). All of these had to do with AI and Entertainment, but none involved a traditional video game. In the end, if you’re not sure if your research matches AIIDE, feel free to reach out to regular AIIDE Attendees or that year’s Program Chair.
AIIDE Paper Outline
As I mentioned above, the top two things when it comes to your AIIDE paper (after ensuring relevance to the community) are the novelty and value of the knowledge you gained through your research. As an example of how to structure an argument for your work’s novelty and value I’ll use my personal, general AIIDE paper outline, giving community-specific tips as I go. Since this is my personal outline. Any paper I helped author will likely fit this fairly closely. For example, feel free to follow along with Guzdial and Riedl’s Automated Game Design via Conceptual Expansion or Sturtevant et al.’s The Unexpected Consequence of Incremental Design Changes (nominated for a 2020 Best Paper Award), though the latter follows this outline less precisely since I wasn’t the first author.
Pretty straightforward and nothing out of the ordinary here: a summary of your work assuming the reader hasn’t read your paper yet.
Unsurprisingly, you want to start with an Introduction. I’d suggest your goals for the introduction should be: (1) to situate your work in a broader context, (2) identify problem(s) or lack(s) (a single problem/lack/gap/opportunity is ideal but some papers might identify 2-3) in said context, and (3) pitch your research as working towards solving this problem or filling in this lack. You want a reader at the end of the introduction to believe (or to be willing to believe that) your work has a topic that fits AIIDE, and that it is novel and valuable (because it works toward solving an important problem or addressing an important lack).
I even have a pretty consistent format that I start with when outlining introductions:
Paragraphs 1-2: Situate your work in a broader field (PCG, automated game playing, experience management, etc.) and identify a problem or lack in that field. At AIIDE you don’t need to start too broadly, which is a common mistake that newcomers make. These authors might structure their first introduction paragraph like (a) Video Games are important (b) Video Games make a lot of money/reach a lot of people (c) Here’s a specific problem/field related to Video Games. At AIIDE you can skip right to (c).
In terms of finding your (c) or “broader field”, there’s a few ways to go about it. But before that, it’s important to note that there’s no “true” list of fields somewhere, these are all just areas that researchers (and not necessarily all researchers) agree exist. Think of it like the genre of your paper. In terms of finding your (c)/broader field/genre, you can: (1) take a look at similar prior papers and see how they position themselves, (2) look at topic lists like those above and find the closest one, or (3) ask someone who has published at AIIDE for advice (if they’re willing).
I recommend ending this paragraph/paragraphs with a pitch as to what solving the problem or filling the lack you’ve identified would accomplish. This is the highest level argument about the value of your research (or future value of your research). If you can get it down to a single sentence, that’s ideal.
Paragraphs 2-3: Explain how existing work in this broad field has not been able to solve the problem or fill the lack from paragraph 1. This shouldn’t be a full Related Work or Background section, but should identify how the 2-3 most common strategies fall short so readers don’t think “Hey, isn’t this solved already?” This doesn’t require having to be critical of these strategies or existing approaches, it’s very possible they just have a different focus.
Paragraphs 3-4: Pitch your work. Make clear what it does and what it does not do in terms of the problem/lack. Don’t make claims not supported by your results. This is another pretty common issue I see in AIIDE submissions. You might be confident that your approach will fundamentally change interactive narrative, but don’t make that claim unless you have the evidence to back it up.
It can be helpful to have a literal, bulleted list with your stated contributions. These are the distinct ideas you’re arguing are novel and valuable about your research work. It can be helpful (for both you and readers/reviewers) to state them explicitly as it can serve as a checklist to ensure you’re presenting evidence for each of these contributions later in the paper.
Newcomers can struggle with this as well, either thinking an entire research project is one contribution or restating the same contribution in different ways for multiple bullets. I’d recommend thinking about this in terms of what distinct ideas or results another researcher could take from your paper and adapt for their own work. A novel AI system might be one contribution, but if it has two innovative components distinct from prior work it might be two contributions, while the evaluation of that system is another, distinct contribution.
Paragraph 5: (Optional) Outline the remainder of the paper for your reader. Some reviewers/readers get annoyed by this, since many papers have the same structure (see this outline). But it can be helpful if your paper has an unusual structure in some way. I also recommend it for new students. You can always cut this part in a future draft if you need the space, but for new students especially it can be helpful to give the guidelines of an outline.
(2) Related Work
The related work should come next. To my mind this section is primarily defensive and is about making a further argument for novelty. It’s meant to protect you from a reviewer saying “Didn’t [so and so] do this?” or “Ah, but you didn’t cite seminal work [Blah] (by me)”. Less cynically, you want to give readers a sense of how your work is in dialogue with prior work. Your goal here should be that after reading this section, a reader should have a sense of the most related areas of prior work and why your work is distinct from that work.
Processes for identifying related work or conducting literature reviews are out of the scope for this blog post. However, I can direct you to the excellent work of Dr. Mark Nelson, who has mapped out related conferences to AIIDE.
I don’t have as clear an outline to give you for this section as the Introduction. But as a general rule I like to separate related work sections into subsections focused on one area of prior work. Each of these subsections tends to have 1-2 paragraphs devoted to it. I then use the following structure per subsection:
- Introduce this specific area or group of prior work. Citing a survey or several papers can be helpful here.
- Discuss the overall most common approaches in this area or essentially subgroups that exist in this area of prior work.
- Identify the 1-3 most related pieces of prior work (papers generally, but could be blog posts or anything else) in this area, and describe them in no more than 1-2 sentences each. Your goal here is not to summarize the entire paper (another common newcomer issue), but only to identify the thing in that paper that most relates to this paper you’re writing.
- Clarify, either as part of the above 1-2 sentences or as a final sentence (or two) end the coverage of this area, how your work is distinct/different from the prior work you described. This is the most important part of each subsection. With the difference/distinction try to be as specific as you can. Saying “We apply [X] to a new game” is vague, being able to point to a particular technical or methodological difference is ideal. Importantly, this doesn’t mean being critical of prior work or downplaying it’s accomplishments, just differentiating your work from it (after all, an author of that prior work might be your reviewer!)
I recommend aiming for at least 20 citations in your final paper, but this is a bit controversial and should not be taken as a hard rule or threshold. The end goal is to have a good coverage of prior work and situate your work in relation to it. Some AIIDE authors cite far fewer papers than this, or primarily cite their own work. While this can make sense for exceptionally novel research, or projects that strongly build on an author’s prior work, it’s generally a good idea to diversify your citations (and the authors that you’re citing). This will give readers confidence that you haven’t missed some prior work that already had the same contributions.
(3) Section Overview or Methodology
At this point papers that come to AIIDE start to differ quite a bit (see the title of this section), but the basic idea is the same: describe your approach/research work in sufficient detail that someone could replicate or adapt your work by only reading this paper (and papers you cite). For some papers this will look like the outline of an AI system, for some it’ll look like the training process of a machine learning model, for some it’ll look like a description of a surveying methodology, for some it’ll look like a framework or encoding of design knowledge, and for others it’ll look like some other information gathering approach.
Regardless, this is the core of your paper, right at the center and the crux of your contributions. Readers should finish this section and understand exactly what you have done and what new knowledge you have brought to the area you identified in the Introduction.
It may be helpful, if possible, to consider a formal problem definition at the top of this section. The idea is to precisely define the expected input and output of your process or problem, along with what measures or effects can be gathered to evaluate performance. For example, if you’re working on a system to adapt game content to players, you might have input that is a specific representation of your player’s current state in the game (mechanically, emotionally, etc.), and an output that indicates what change should be made to adapt the game to this player. Then your measures or effects would be how this impacts the player experience.
I can’t offer an outline of any sort for this section but I have some general tips:
- Being able to pitch your approach visually is ideal. It’s more space efficient and often more intuitive. For example, Awiszus et al.’s TOAD-GAN: Coherent Style Level Generation from a Single Example (AIIDE 2020 Best Student Paper) or Ware and Young’s Glaive: A State-Space Narrative Planner Supporting Intentionality and Conflict.
- If you have a technical process I would recommend conveying it to readers in a way that they can replicate. Many conferences state this as “could a grad student in a different lab reimplement the work based on the description”. Essentially, what are the steps you would go through if you had to redo your project, rather than your entire research process. Newcomers often default to giving a history of the research project, but this is generally not necessary. Instead, give how you would replicate your final approach if you had to redo it. It can be nice to include your research process as well in terms of how you came to make certain decisions. But it saves space and can be more clear to readers if you focus only on the important parts of the process.
- Related to the above, anytime you describe a decision you made you should justify it. This might be as simple as saying “this was determined empirically” or “this is the standard approach” with a citation. As a general rule, don’t make claims not supported by citations or evidence in the paper. Newcomers can sometimes struggle to come up with justifications that feel “good enough”. For example, what if you did what everyone else does (from prior work) or the first thing you tried just worked? These decisions can still be justified: citing prior work to back up why that approach is commonly accepted, or stating that a simple, initial approach worked sufficiently, and so there was no need to try more complex ones.
(4) Experiments, Evaluation, or Study Methodology
As I stated above, these remaining sections will be less specific in terms of the feedback I can give. In the next section, you will have to present evidence that your claims/contributions are true. In this section, you need to present an argument as to why the way you got those results (your experiments/evaluation) is appropriate, and why readers should believe your results will clarify whether you addressed the problem(s)/lack(s) from the Introduction. If you made a list of contributions, explicitly going through each point on that list and explaining how your evaluation/experiments/etc. support or reflect them is ideal. This linking can also suggest what your experimental design should look like. For example, if you claim that your contributions include a new approach to improve player engagement, you should probably have a human subject study, or at least an argument as to why you don’t need one.
You can and likely should also back up the “appropriateness” argument in this section with references to prior papers. As in, you should be able to point to a prior paper that ran a similar experiment, or employed a similar approach to derive its results. But it’s possible (though unlikely) that no prior paper has ever had an evaluation like yours. At that point, it’s even more important to walk the reader through why your specific, novel evaluation was ideal, and why you couldn’t use a more traditional one (this may even count as a contribution! A new way to structure evaluations).
It’s worth noting for newcomers that this section does not need to look like a quantitative evaluation with a baseline and metrics “with numbers that go up and to the right”. In fact, it may not be possible to determine if you’ve addressed your problem/lack with a quantitative evaluation. For example, Summerville et al.’s Gemini: Bidirectional Generation and Analysis of Games via ASP made use of two case studies applying their tool, whereas Jacob et al.’s (AIIDE 2020’s Best Paper) “It’s Unwieldy and It Takes a Lot of Time”—Challenges and Opportunities for Creating Agents in Commercial Games used a set of interviews with game developers about their perspectives on creating AI agents. However, quantitative evaluations are still the most common type presented at AIIDE (see the majority of example papers above), so don’t feel bad if you do employ one.
(5) Results, Case Studies, or Takeaways
Fairly straightforwardly (despite the above title), in this section you give the results. The most common way to do this is with a number of tables summarizing your results, but graphs, plots, and other figures are also common. I’d recommend giving your results in some summarizing way, and then walking readers through the results in the text. In this way you can point out any surprising or subtle findings that a reader may not necessarily recognize on their own.
(6) Discussion, Limitations, and/or Future Work
This section is so varied across AIIDE papers that some don’t even have it, and some have multiple sections in its place. In the “Discussion” section, authors may include analysis (or additional analysis) of the results. In the “Limitations” section, authors may include a discussion of where their current approach falls short in terms of fully solving or addressing the problem(s) or lack(s) from the Introduction. Often this section will have a defensive purpose for peer review, getting ahead of/anticipating any criticisms from reviewers and explaining why they weren’t addressed by this work. In the “Future Work” section authors may include potential next steps for this research. These do not necessarily have to be what you/the authors plan to do next. In fact, they can themselves be contributions in terms of identifying projects for other researchers.
Sometimes these sections can be combined. For example “Limitations and Future Work” can identify limitations of the current research and propose new research projects that could address them.
Pretty straightforward and nothing out of the ordinary here: a summary of your work under the assumption that the reader has read your whole paper. I often recommend students include the 2-4 big points you most want readers to remember from the paper (even if they’ve forgotten everything else).
The purpose of this blog post was to give my personal perspective on how to know if your research might make a good AIIDE paper, and then how to write that paper. However, this should not be taken as a surefire way to get a paper accepted to AIIDE. I hope that this can help improve your AIIDE submission, but ultimately the decision comes down to the reviewers assigned to your paper and whoever is the Program Chair for the year of your submission.
Sometimes reviewers will point out a fundamental issue in the paper you didn’t notice, but sometimes you just get unlucky, and reviewers don’t understand your contributions or results despite your best efforts. This is less likely at AIIDE than a larger conference, since there’s a smaller and more consistent PC, but it still happens. But even in that case, there’s usually some way the confusion in the reviews can improve your paper. If nothing else than by explicitly including the misunderstanding from your reviewers in your next paper draft, along with comments to clarify it, so that others are less likely to have the same misunderstanding.
Even if your paper is rejected from AIIDE there’s a number of excellent AIIDE workshops (e.g. EXAG and INT) with higher acceptance rates and deadlines soon after AIIDE notifications. I’ve personally turned a rejected AIIDE paper into a well-received EXAG paper.
If you have further questions, I’m happy to try my best to answer them if I have the time (feel free to bother me on Twitter @MatthewGuz), though it’s probably better to check AIIDE’s organizers for your year (aiide.org), and to send a message to the Program Chair.
With that, all that’s left to say is: best of luck on your AIIDE submissions!