An Annotated Statement of Purpose
I’ve recently found myself giving a lot of similar advice to PhD applicants regarding the statement of purpose. While doing this, I often found myself wanting to provide examples of how something could be done, rather than simply describing what I think would be nice to have. So, I decided to annotate my own statement of purpose with some more information of why I decided to write it this way.
Based on advice I received from my mentors, the qualities I wanted to highlight in my SOP are:
- Initiative and independence I wanted to show that I was capable of coming up with interesting ideas on my own, and finding ways to bring these ideas to fruition. This aspect of highlighting independence and initiative was something that was strongly emphasized in the feedback I got.
- Technical proficiency I wanted to show that I was capable of dealing with complex research ideas and had the skills to apply my understanding of technical concepts to solving the problems I was faced with.
- Collaboration and communication I wanted to highlight that I could work well with others on research problems, as well as effectively work as part of the broader research community.
My initial drafts hardly touched upon these topics, and it was through a lot of feedback that I was able to understand why and how I should write about these, and I hope to shed a little light on this through this post.
At the end of the post, I have some more thoughts, resources, and context about the application process and statement of purpose more broadly.
Language allows humans to make infinite use of finite means. We can understand situations and ideas we have never encountered from structured messages composed of familiar units. We combine this with reasoning about other agents’ communication strategies to understand their beliefs and intentions. This makes language a powerful means of acquiring knowledge about the world and agents in it that we can use to perform other tasks. I am interested in studying how this linguistic structure and communication can be used to learn about the world.
In the opening paragraph, I laid out a broad vision for what I wanted to work on – my big question. When I wrote it, I tried to find a frame which captured a varying set of interests that I had, but was still specific and concrete enough to make sense. I didn’t write about everything I was interested in. I was (and still am) excited about a very broad range of things related to language, learning, and cognition. However, I tried to articulate a more coherent research direction and argue why I was equipped to work on these specific problems based on my experience. I think trading off the coherence of the research direction (your “purpose”) against how comprehensively it represents your interests is one of the most challenging things to do when writing the SOP.
I am drawn to these problems based on my research experience so far. I studied the influence of domain-specific concepts on learning using program synthesis, and wanted to find more flexible ways of acquiring these concepts. I studied how pragmatic reasoning allowed making sense of ambiguous instruction in program synthesis, and saw how resolving ambiguity was crucial to learning from other agents. In an ongoing project, I am examining the processing of tasks specified in text by neural language models, and understanding the different ways linguistic meaning is used in an interaction. Pursuing a PhD at the intersection of natural language processing (NLP) and machine learning (ML) would provide opportunities to study how linguistic structure and communication can be modeled computationally, and how this can be used to acquire knowledge to solve other tasks.
Here, I grounded the questions I wanted to work on in takeaways from my prior work. I gave a very brief description of different projects that I discuss later, followed by a very brief statement of how what I learned from that informs my broader interest. The idea here was to lay out all my cards at the beginning, and substantiate and reinforce these points in the rest of the statement. It is a well-known fact that people reading your application have hundreds of others to get through, and I thought being as direct as possible would help people get a complete picture of my ideas in a short time.
Program synthesis for linguistic rules. In my first research project, I worked towards solving phonology puzzles from the Linguistics Olympiads. These puzzles involve inferring abstract rules given only a few examples from a language, and pose a challenging problem in learning from very little data. I approached this problem by using program synthesis to learn rules in the form of programs in a domain-specific language (DSL). Not having had any background in the field of programming languages, I taught myself the basics of program synthesis, and adapted algorithms to implement a synthesizer. I designed experiments to understand how access to linguistic concepts helped synthesis, and developed methods to analyze and visualize the results. Through this experimentation and analysis, I found that linguistic concepts led to solutions with fewer, more generalizable rules. This work led to a paper at the SIGMORPHON Workshop at ACL 2021 [W1]. I followed up with a more specific study of phonological stress published at ICON 2021 [C1].
In this section, I described my first substantial research project. By the time I was applying, I was fairly certain that the specific themes of this project (search-based program synthesis, phonology and morphology) weren’t central to my interests. So I took the opportunity here to talk about what I learned about the research process.
This can be valuable in highlighting that you know what research looks like behind the scenes (of a research paper), and what challenges it entails. It’s also a peek at how I think about research that you cannot see on my CV, or in the papers I’ve written (I’ve heard it referred to as a “director’s cut” of research). It took me a while to wrap my head around this idea, but I do think it’s important and one of the reasons I’m writing this post is to highlight how I tried to do this in my own statement.
Here, I shared how I got up to speed with a new topic (highlighting my technical proficiency). I am primarily excited about (natural) language, so learning about program synthesis was both an example of trying to show I could quickly adapt to new topics and ideas, and that I had broader base of knowledge. I also wrote about the various things I learned about building an argument in a research paper from working on this project (highlighting both proficiency in research and aspects of research communication). I tried to weave this into the description of the technical side of the story to make it seem like a coherent whole.
I think different projects/experiences can serve different purposes in building your SOP. Here, I discussed this project even over other work that more closely resembles modern NLP research since I thought it allowed me to highlight these aspects better.
Through this project, I became familiar with the research process. I learned how to approach a broad goal by identifying small, concrete sub-problems. I came to appreciate the importance of communicating my findings, and refined my ability to write a research paper. I also saw how having access to the right abstractions (here, from the DSL) could help with learning from very little data. However, specifying these abstractions manually is laborious, and as I came to see, not sufficiently flexible. This is where I see the potential to discover these abstractions from language, which humans use powerfully and flexibly to refer to complex concepts.
Then, I explicitly highlight my takeaways both related to the problem itself, and about the research process. This was again meant to highlight and reinforce the salient ideas I wanted to get across. This is something I did for every section of this statement.
Pragmatic program synthesis. A central problem in program synthesis is choosing among multiple programs consistent with the given examples. To learn linguistic rules, I approached this by engineering a preference for linguistic abstractions into the DSL. However, a paper at NeurIPS 2020 cast synthesis as a reference game, and showed how principles of pragmatic communication could be used to choose among programs. While their framing of the problem was insightful, I also saw that they used pragmatic reasoning only after synthesis, leading to an approach that scaled poorly.
In opening this section, I tried to make a smooth transition from the previous topic by discussing how these ideas were related by the notion of disambiguating programs. This specific connection is something that I saw when reflecting on the different things I had worked on, and by no means something that directly motivated the project when I started out. Writing the SOP provides an opportunity to engage in such reflection, and identifying connections like this helps tell a nicer story.
I reached out to the authors of the paper, and led a collaborative effort to see if pragmatics could be used during synthesis, towards a more efficient and pragmatic synthesizer. We found that a simple decomposition of the program synthesis problem lent itself to pragmatic specification of intent, and made pragmatic inference over these specifications less expensive. We presented these findings as a talk at the Meaning in Context Workshop at NeurIPS 2021 [P1].
Here, I tried to highlight how I took initiative and independently pursued this idea of my own volition. I wrote about how I acted upon my excitement about the aforementioned NeurIPS paper and turned it into a substantial research project. I thought initiating a collaboration based on a conference paper was somewhat unique, and I think it’s really helpful to find these unique aspects of your experience and discuss them in your SOP.
This project gave me first-hand experience initiating and building a collaboration to improve upon prior work. Starting with an intuitive idea, I was able to discuss, refine, and iterate on it until we got to a formal solution. I also understood how much humans communicate by choosing not to specify something, and how modelling pragmatics can enable understanding such communication as well.
Again, concluding the section by highlighting and reinforcing my takeaways.
Representations of semantics in NLP models. In addition to providing directions for improving models, I also see the value of frameworks from linguistics and cognitive science in analyzing representations in NLP models. I had a chance to do this during my internship at MILA when I joined an effort to investigate language models trained to complete tasks or instructions specified in text. My attention was quickly drawn to the question of how these models represented a task that was specified in text. Inspired by recent work that used linear probing to examine language models for representations of worlds described in text, I turned to a framework of meaning called dynamic semantics to formalize this question. In this ongoing project, I am working with people with expertise in NLP and text-based games to design experiments in simple text domains, and studying the linguistics literature on how the meaning of tasks or instructions may be represented. Through this, I am gaining experience with research that synthesizes different forms of expertise in answering a question.
Here, I wrote about ongoing (at the time of writing) work. The goal here was to highlight how I was someone who was interested in bridging disciplines both in terms of the content of the research (discussing how I wanted to use ideas from linguistics), and in terms of bringing people together (discussing working with people with various expertise). As with the discussion before, I tried to focus on things that may not be apparent from the content of a research paper (in this case one that wasn’t even written), and tried to highlight some aspects of “research taste” – what kind of research I wanted to work on.
Beyond the specific question of how language models represent tasks, I am also interested in broader questions of how we look for linguistic ability in large, learned models. To ponder this question with a broader perspective, I started a reading group with colleagues at MILA to discuss perspectives from machine learning, NLP, and linguistics on what linguistic abilities we look for in models of language, and why these are useful to solving the tasks for which these models were designed.
Extending the idea of writing about what sort of questions I am interested in from the previous paragraph, I wrote about how I was also interested in taking a step back and looking at the field from a broader perspective to examine how and why current models are successful. In addition to highlighting this interest, I also wanted to describe another instance of showing initiative and leadership. Pointing to concrete examples like this can be an effective way to demonstrate that you have these qualities. This also doesn’t need to be as ostensibly research-focused as a reading group, it can be any instance of you taking initiative in a matter related to your research or to your academic or professional career.
At LTI, I am interested in working with Prof. Daniel Fried, Prof. Graham Neubig, and Prof. Yonatan Bisk. A central challenge in leveraging language to acquire knowledge, especially when interacting with other agents, is correctly interpreting the meaning in context. I saw in my own work on pragmatic program synthesis how using models of human communication can help resolve communicative ambiguity. I hope to work with Prof. Fried on ways to resolve linguistic ambiguity, and enable learning from what others convey in language. I am also interested in working with Prof. Neubig on modelling linguistic interactions, and creating better interfaces that allow machines to learn from language to perform tasks like program synthesis and question answering. I also hope to work with Prof. Bisk on modelling the semantics of the world, and how this can enable acquisition of world knowledge from language.
This is the “institution-specific” portion of the statement. I first listed faculty in bold (to make things obvious), and then for each faculty wrote a line or two about why I was interested in working with them. In each case, I explicitly drew connections between their research and what I had written before in the statement, which is a way of demonstrating a strong fit. I typically listed 2-3 faculty, since I think that’s a number I think is large enough to not “put all your eggs in one basket” and small enough to not seem like you’re confused.
More broadly, LTI is home to faculty and students exploring the connection between language and computation in a variety of ways. I hope to engage actively with this range of perspectives on NLP research, and believe the vibrant environment makes CMU an excellent place for me to do my PhD.
I also chose to write about why I was interested in the department more broadly. I tried to highlight things that appealed to me and that were unique when possible. At LTI, the sheer breadth of NLP work is unique, and I chose to highlight that.
Note that this isn’t meant to be a how-to guide, and it is simply one example of a statement of purpose for computer science PhD applications. I’m sharing this perspective in the hope that applicants (and prospective applicants) may find it helpful to have a bit more than just the surface form of my application materials. There is a lot of advice about the SOP, and the applications process as a whole out there. I think this Twitter thread does a great job of summarizing what makes a good SOP, and I think it applies across fields (it’s written by a professor of psychology, and I thought it would be really useful for CS too). Shaily Bhatt and Kalpesh Krishna have also curated useful sets of resources. A number of PhD students also shared their perspectives on the application process as a whole. There are also other annotated examples of SOPs available online, but I hope to share a more detailed and personal experience here.
I also strongly believe that there are many ways to build a successful PhD application, and mine is only one. Based on the rest of your profile, other parts of your application such as your publications or your letters of recommendation may play a more important role. My perspective may not be the most useful for you to take based on your background and experience, and I strongly encourage applicants to do what they feel is best in their situation. I had various prior research experiences, and I primarily used these experiences to motivate my future work. However, your path to applying for grad school may have been very different, and you should highlight what you believe to be the strongest and most unique aspects of your experience. All the best!
Acknowledgements
Huge thanks to Evan Pu, Vijay Viswanathan, Simran Khanuja, Shaily Bhatt, Akshay Goindani, and Mahathi Vempati for valuable feedback on earlier drafts of this post!