Prompt engineering

Prompt engineering (as named from a user's perspective) or in-context learning (as named from machine learning perspective) is an approach used in teaching a large language model (or learning, from its perspective) to do anything that it was not part of its pre-training. The results of learning are, however, contained within a single conversation (i.e. single context window) and completely forgotten by the model outside it. The results of such a temporary kind of learning seems to be in the form of temporary changes in the activations of neurons, i.e. without changing the model's weights (in contrast to the results of model's pre-training and fine-tuning). This is also known as "mesa"-optimization, based on presence of (small) learn-to-learn models in the data.

LLM prompting techniques

Chain-of-thought technique

The Chain-of-thought (CoT) prompting technique is intended for solving both multiple-step and logical-thinking tasks, such as arithmetic or commonsense reasoning, that require a series of intermediate steps before giving the final answer to a multi-step problem. To address this challenge, it was first proposed by Google researchers in 2022.

If user asks a model a question and would like the model to provide step-by-step reasoning behind its answer to the question such as “The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?”, the user needs to prepend "Let's do it step-by-step" to the question in order to get the answer similar to the following one, “The cafeteria had 23 apples originally. They used 20 to make lunch. So they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9.”

Single-prompt solutions and multiple-prompt solutions

Standard word-crafting attempts to use a 'perfect' prompt can be replaced by the following single-prompt and multiple-prompt solutions

The single-prompt solutions include:

• prepending examples (i.e. few-shot CoT prompting) and
• adding explicit instruction such as "Let's think step-by-step" (i.e. zero-shot CoT prompting).

The multiple-prompt solutions include two groups, one using a single conversation and the other using multiple conversations. The single-conversation solutions include:

• Generated knowledge prompting first generates a list of most relevant facts related to a problem to be solved, then - after receiving the knowledge - prompts again with a question based on the received knowledge. The completion quality is usually higher, as the model can be conditioned on relevant facts.

• Self-refine prompting first generates an initial solution, then - after receiving it - prompts again to generate its critique, then - after receiving the critique - prompts to address its own critique by generating a refined solution. A self-refining conversation stops when it runs out of tokens, time, or with a "stop" token.

• Maieutic (Socratic) prompting first generates an explanation of the phenomena related to a problem to be solved, then - after receiving the explanation - prompts again to generate additional explanation for the least understood parts of explanation. Inconsistent explanation paths (trees) are pruned or discarded, improving complex commonsense reasoning.

• Tree-of-thought prompting first generates "possible next steps" for solving a problem, then - after receiving them - prompts again to run each of the proposed possible steps, using a breadth-first, beam, or some other method of tree search.

• Least-to-most prompting first generates the sub-problems to a problem to be solved, then - after receiving the sub-problems - prompts again to solve those in sequence, such that later sub-problems can be solved with the help of answers to previous sub-problems.

The multiple-conversation solutions include:

• Self-consistency prompting first generates multiple separate conversations with the same prompt, then - after completing all of them - the most commonly reached conclusion is selected. If the results differ a lot, correction of the prompt is needed.

• Complexity-based prompting first generates multiple separate conversations with the same prompt, then - after completing all of them - the rollouts with the longest chains of thought are selected, then the most commonly reached conclusion out of those is selected.

CoT prompting is an emergent property of model scale, which improves the performance on both arithmetic and commonsense tasks, compared to standard word-crafting attempts to use a 'perfect' prompt. When applied to PaLM, a 540B parameter language model, it has allowed the model to perform comparably with task-specific fine-tuned models on several tasks, even setting a new state of the art at the time on the GSM8K mathematical reasoning benchmark.

Prompting to disclose uncertainty

By default, the output of language models may not contain estimates of uncertainty. The model may output text that appears confident, though the underlying token predictions have low likelihood scores. Large language models like GPT-4 can have accurately calibrated likelihood scores in their token predictions, and so the model output uncertainty can be directly estimated by reading out the token prediction likelihood scores.

But if one cannot access such scores (such as when one is accessing the model through a restrictive API), uncertainty can still be estimated and incorporated into the model output. One simple method is to prompt the model to use words to estimate uncertainty. Another is to prompt the model to refuse to answer in a standardized way if the input does not satisfy conditions.

Automatic prompt generation

Retrieval-augmented generation

Prompts often contain a few examples (thus "few-shot"). Examples can be automatically retrieved from a database with document retrieval, sometimes using a vector database. Given a query, a document retriever is called to retrieve the most relevant (usually measured by first encoding the query and the documents into vectors, then finding the documents with vectors closest in Euclidean norm to the query vector). The LLM then generates an output based on both the query and the retrieved documents.

Using LLM to generate prompts

LLM themselves can be used to compose prompts for LLM.

The automatic prompt engineer algorithm uses one LLM to beam search over prompts for another LLM:

• There are two LLMs. One is the target LLM, and another is the prompting LLM.
• Prompting LLM is presented with example input-output pairs, and asked to generate instructions that could have caused a model following the instructions to generate the outputs, given the inputs.
• Each of the generated instructions is used to prompt the target LLM, followed by each of the inputs. The log-probabilities of the outputs are computed and added. This is the score of the instruction.
• The highest-scored instructions are given to the prompting LLM for further variations.
• Repeat until some stopping criteria is reached, then output the highest-scored instructions.

CoT examples can be generated by LLM themselves. In "auto-CoT", a library of questions are converted to vectors by a model such as BERT. The question vectors are clustered. Questions nearest to the centroids of each cluster are selected. An LLM does zero-shot CoT on each question. The resulting CoT examples are added to the dataset. When prompted with a new question, CoT examples to the nearest questions can be retrieved and added to the prompt.

Using gradient descent to search for prompts

Prompts do not have to be in English. In "prefix-tuning" or "prompt tuning", floating-point-valued vectors are searched directly by gradient descent, to maximize the log-probability on outputs.

Formally, let $\mathbf {E} =\{\mathbf {e_{1}} ,\dots ,\mathbf {e_{k}} \}$ be a set of soft prompt tokens (tunable embeddings), while $\mathbf {X} =\{\mathbf {x_{1}} ,\dots ,\mathbf {x_{m}} \}$ and $\mathbf {Y} =\{\mathbf {y_{1}} ,\dots ,\mathbf {y_{n}} \}$be the token embeddings of the input and output respectively. During training, the tunable embeddings, input, and output tokens are concatenated into a single sequence${\text{concat}}(\mathbf {E} ;\mathbf {X} ;\mathbf {Y} )$, and fed to the large language models (LLM). The losses are computed over the $\mathbf {Y}$ tokens; the gradients are backpropagated to prompt-specific parameters: in prefix-tuning, they are parameters associated with the prompt tokens at each layer; in prompt tuning, they are merely the soft tokens added to the vocabulary.

With more math details, let an LLM be written as $LLM(X)=F(E(X))$, where $X$ is a sequence of linguistic tokens, $E$ is the token-to-vector function, and $F$ is the rest of the model. In prefix-tuning, one provide a set of input-output pairs $\{(X^{i},Y^{i})\}_{i}$, and then use gradient descent to search for $\arg \max _{\tilde {Z}}\sum _{i}\log Pr[Y^{i}|{\tilde {Z}}\ast E(X^{i})]$. In words, $\log Pr[Y^{i}|{\tilde {Z}}\ast E(X^{i})]$ is the log-likelihood of outputting $Y^{i}$, if the model first encodes the input $X^{i}$ into the vector $E(X^{i})$, then prepend the vector with the "prefix vector" ${\tilde {Z}}$, then apply $F$.

An earlier result uses the same idea of gradient descent search, but is designed for masked language models like BERT, and searches only over token sequences, rather than numerical vectors. Formally, it searches for $\arg \max _{\tilde {X}}\sum _{i}\log Pr[Y^{i}|{\tilde {X}}\ast X^{i}]$ where ${\tilde {X}}$ is ranges over token sequences of a specified length.

History

See also: Fine-tuning (machine learning)

The GPT-2 and GPT-3 language models were important steps in prompt engineering: Trained on large amounts of text using deep learning methods, they attained the capability of generating output that resembles human-generated text.

In 2021, researchers finetuned one generatively pretrained model (T0) on performing 12 NLP tasks (using 62 datasets, as each task can have multiple datasets) that showed good performance on new tasks, surpassing models trained directly on just performing one task (without pretraining). To solve a task, T0 is given the task in a structured prompt, for example If {{premise}} is true, is it also true that {{hypothesis}}? ||| {{entailed}}. is the prompt used for making T0 solve entailment.

A repository for prompts reported that over 2,000 public prompts for around 170 datasets were available in February 2022.

Non-textual prompting

Text-to-image

Main article: Text-to-image model

In 2022, machine learning (ML) models like DALL-E 2, Stable Diffusion, and Midjourney were released to the public. These models take text prompts as input and use them to generate images, which introduced a new category of prompt engineering related to text-to-image prompting.

Image prompting

In 2023, Meta's AI research released Segment Anything, a prompting-based model for creating masks for objects in images. It can be prompted in several ways, such as selecting a few "positive" and "negative" points, to create a mask that includes all the positive points and excludes all the negative points.

Malicious

See also: SQL injection and Cross-site scripting

Prompt injection is a family of related computer security exploits carried out by getting a machine learning model (such as an LLM) which was trained to follow human-given instructions to follow instructions provided by a malicious user. This stands in contrast to the intended operation of instruction-following systems, wherein the ML model is intended only to follow trusted instructions (prompts) provided by the ML model's operator.

Common types of prompt injection attacks are:

• jailbreaking, which may include asking the model to roleplay a character, to answer with arguments, or to pretend to be superior to moderation instructions
• prompt leaking, in which users persuade the model to divulge a pre-prompt which is normally hidden from users
• token smuggling, is another type of jailbreaking attack, in which the nefarious prompt is wrapped in a code writing task.

Prompt injection can be viewed as a code injection attack using adversarial prompt engineering. In 2022, the NCC Group characterized prompt injection as a new class of vulnerability of AI/ML systems.

In early 2023, prompt injection was seen "in the wild" in minor exploits against ChatGPT, Bard, and similar chatbots, for example to reveal the hidden initial prompts of the systems, or to trick the chatbot into participating in conversations that violate the chatbot's content policy. One of these prompts was known as "Do Anything Now" (DAN) by its practitioners.

For LLM that can query online resources, such as websites, they can be targeted for prompt injection by placing the prompt on a website, then prompt the LLM to visit the website. Another security issue is in LLM generated code, which may import packages not previously existing. An attacker can first prompt the LLM with commonly used programming prompts, collect all packages imported by the generated programs, then find the ones not existing on the official registry. Then the attacker can create such packages with malicious payload and upload them to the official registry.

See also

• Social engineering (security)
• Fine-tuning (machine learning)

External links

• LearnPrompting - Prompt Engineering Guide. Open-source prompt engineering guide.
• Prompt Engineering - Lilian Weng. Review post of prompt engineering methods.
• Towards Complex Reasoning: the Polaris of Large Language Models. Review of prompt engineering methods specifically to elicit complex reasoning in LLM.
• dair-ai - Prompt-Engineering-Guide. GitHub resource repo for prompt engineering.