A recent arXiv preprint titled “Deep Research Agents can be poisoned by User-Generated Content” describes a capacity for using user-generated content to shape the output of research agents. As Tingwei Zhang, Harold Triedman, and Vitaly Shmatikov write in their abstract in relation to these deep research agents.

We show that for many common search topics, they repeatedly retrieve the same user-generated content (UGC) pages from platforms such as Reddit and Wikipedia. Next, we argue that this retrieval overlap creates a concentrated attack surface: an adversary who appends a short, crafted text to a single, frequently retrieved UGC page can cause the agent to cite attacker-chosen content and promote attacker-chosen entities across many related queries.”

Where a human user encountering a one-off Reddit comment would likely discount it, AIs launder the questionable rhetorical situation in which the statement appears and present it within its authoritative rhetoric. This makes it more difficult for the user to recognize the deception.

UGC poisoning is related to RAG poisoning (Retrieval Augmented Generation). RAG poisoning occurs when malicious or misleading information is secreted into a documents, databases, and such and that information is called upon as part of an AI research process. For example, if you were a university administrator and you were drawing on a database of institutional policies as part of an AI research process, then you might encounter the sabotage of that database as RAG poisoning.

A simplistic version of RAG poisoning (really prompt poisoning) occurs when teachers insert hidden text into assignments for students to unknowingly upload to their chatbot. (That old chestnut.)

UGC poisoning also attacks retrieval processes. However where RAG poisoning exploits our trust in the data in our own databases, UGC poisoning attacks AI research agent processes as they extract data from sites such as Reddit.

This leads to the first of two implications I see for rhetoric.

UGC poisoning reveals some of the contours of AI’s susceptibility to UGC rhetorical tactics. The study suggests this works best for research involving consumer advice and recommendations, procedures, and local opinion/knowledge (e.g., where’s a good taco joint in Buffalo?). Part of the strategy is matching the syntax of user prompts. So a Reddit post that says something like “You can find good tacos in Buffalo at XXX” would be primed for that question.

These strategies make particular sense for attacking AI search summaries and overviews from search engines. Success means more than visibility, as was the aim with SEO. Success means the integration of one’s poison message into the AI laundered summary. I imagine that this will result in some kind of cat-and-mouse technical competition.

Meanwhile, there is a second implication for rhetoric, which I think is more interesting. This research suggests that AI remains dependent on human evaluation as a filter. While our digital experiences are algorithmic, aside from AI slop, the content of the internet is human filtered in the sense that we made it. It is also filtered post-hoc in many human ways. There are moderators on Reddit, Wikipedia, and so on. And there are discourse communities on the web that regularize online genres in the spaces they occupy. The online encyclopedia entry, for example. The news story. The substack post and comment. All of these human labors prove to be integral to AI’s capacity to process and extract data.

On the one hand there is the AI origin story in which the models were built upon our work. This is different, but related. This points to AI’s ongoing reliance on human judgments about texts/media in order to function.

If we imagine a world of AI agents, this becomes a bottleneck. AI’s value proposition is speed: faster searches, analysis, and actions. But this research suggests that AI relies on slower human practices of verification. The agent needs humans before retrieval, because it relies upon a web already organized by human judgment (so to speak). It also needs humans after generation, because it cannot make an epistemic claim about its output. If, as this suggests, claims for the acceleration of research rest upon occluded labor for their credibility, we could almost be surprised…. This occluded labor becomes a reverse centaur story as well when the human validator for research agents is viewed as a bottleneck in AI throughput.

As we debate AI slop, hallucinations, alignments, guardrails, and safety. we should also consider this other problem: what happens when model’s interaction with agent-tools to retrieve from shifting human generated mediascapes. While poisoning is malicious, manipulating these retrieval agents can be pedagogical, critical, activist or otherwise motived to shape AI answers to human questions. In short the issue becomes that AI answers may be prefigured by human attempts to make some claims more retrievable by AI than others.

If we can shape AI responses in advance by shaping the media environment from which they retrieve, then what is the AI doing again?