Refine: Bringing Transparency to AI-Assisted Writing

14 April 2025 · Tarek Awwad

In this article, I will discuss AI-assisted writing and the importance of tools to assess its contributions, which enhances transparency in published works. I’ll also introduce Refine, a Notion integration I built to facilitate measuring AI involvement and promote responsible use of AI in the writing process.

AI-Assisted Writing - Where’s the Limit?

In recent years, the democratization of Large Language Models (LLMs) has reshaped how people research, question, understand, and communicate the ideas and topics they care about. The accessibility and ease of use of LLMs, dictated by its inherent ability to comprehend and generate human language, transformed them into natural companions to virtually any writing task - whether it’s writing articles, developing code, compiling reports, or more. Unfortunately, this convenience has also led to the proliferation of AI-generated written content which, without exaggeration, poses a real threat to the intellectual and artistic integrity of written expression

Relying on AI, in the form of LLMs, to fully express our thoughts in its own way puts this authenticity and diversity - as well as their natural evolution over time - at a significant risk of gradually eroding. It is therefore crucial to promote a responsible approach of AI in the writing process. One in which not only the ideas are dictated by the author, but also the style, the reasoning and the structure.

To support this, authors need to devise workflows that help them harness AI’s strengths without compromising the originality and integrity of their style. I believe that a key part of developing these workflows is the ability to quantify the AI generated contribution in a given text. Doing so allows authors to refine their workflows until AI’s involvement reaches a level they’re comfortable with and, just as importantly, to be transparent towards their readers about the extent of that involvement.

In the next section, I share my transparency workflow. It describes how a) I leverage LLMs to write my articles, b) how I keep an eye on the amount of AI contributed text, by continuously measuring and aggregating it and c) how I try to be transparent about it.

The Transparency Workflow

How do I use LLMs?

For all the reasons mentioned earlier, when it comes to writing, my use of LLMs is limited to enhancing the linguistic quality and clarity of my articles. My goal is simply to ensure that the reader isn’t distracted or overwhelmed by unclear language or awkward phrasing. To this end, the workflow I used - up until recently - can be summarized as follows:

1. The prompt: In ChatGPT (GPT-4), which I found to best “mimic” my writing style among available AI tools and models I’ve tested (the tests were neither exhaustive nor rigorous!), I start a new conversation with these instructions as the first prompt:

2. The writing process: when I write a paragraph, I iterate on it until the idea is clear enough and the text structure reflects the reasoning I want the reader to follow. I copy the paragraph into the conversation and ask ChatGPT to refine it. I then review the output to ensure it aligns with both the structure and content of the original text. While the output often meets my expectations, there are instances where it does not. This can happen for the following reasons: Either a) the initial instructions are not strict or clear enough - though this is now less likely since I have refined them as part of this workflow. Or b) my input text lacks clarity - small linguistic nuances can create significant ambiguities in conveying an idea. My rule of thumb is that if ChatGPT misunderstands it, most people would too. In such cases, I revise my input text to remove any ambiguity and refeed it into ChatGPT for refinement if needed. When the output is satisfactory, I copy both my input text and the generated output into a separate file for later use.
3. The transparency step: Once I complete the article, I use a Python script to compute similarity measures between the initial text (all input texts) and the final text (all output texts). The detail of this computation are described further in this article. I include this similarity score at the bottom of my article to give readers an idea - albeit self-computed and unverifiable - of the extent of assistance I received from the “AI”.

Couldn’t This Be Easier…?

This workflow is undeniably tedious, as it involves a significant amount of manual copying and pasting back and forth, which quickly adds friction to the writing process. One could argue that instead of refining paragraphs one by one, I could refine the entire finished article at once. However, this approach is not ideal. Refining the full text comes with several drawbacks:

• My writing process relies on sequentially building upon solid blocks: After laying out a rough plan for my article, I prefer to tackle each section paragraph by paragraph. It is important to me that each paragraph is clear and nearly finalized before moving on to the next. This ensures that the final product is free of redundant ideas and avoids excessive repetition of similar linguistic structures.
• Refining on a paragraph basis ensures alignment between the input and output texts: As mentioned earlier, refined text can sometimes semantically deviate from the input. It is much easier to detect and correct such deviations in a single paragraph than in a full article.
• Non-textual components in Notion: I use notion as my writing and publishing tool. One of Notion’s strengths is its ability to integrate a wide range of formatted components, both textual and non-textual. Copying an entire article results in raw text cluttered with “noise,” such as tags and formatting elements. These could either overwhelm ChatGPT unnecessarily or be removed if it detects that they are non-essential to the refined text - requiring manual reinsertion afterward.
• Not all paragraphs need refinement: In many cases, I know that a paragraph is already clear and contains writing style elements that I do not want to be altered. In fact, such paragraphs make up more than half of each article I write.

Although I write only in my free time as a hobby - with no publication deadlines or productivity pressure - streamlining this process can still be a great help. In the next section, I’ll share how I built Refine, a simple yet effective Notion integration designed to achieve this and more.

Refine - Streamlining The Transparency Workflow

Before jumping into the details of Refine, I believe I should clarify a key point: Refine is not an AI detector. Tools to detect whether a text is generated by an LLM have been around for some time. While their effectiveness is steadily dropping over time as LLM become better and prompt engineering practices improve among users, they can occasionally produce impressive results. Nonetheless, these tools do not achieve the goal that Refine was designed to accomplish. Refine is a framework designed to support and streamline AI-assisted writing. It serves two core goals: first, to create a feedback loop that helps writers calibrate how much support they receive from AI; and second, to offer a transparent way to communicate this level of assistance to readers.

A System Overview

The Requirements

Before I started building a PoC, I set the following requirement list:

• R1: It should be integrated to my workspace. That is Notion.
• R2: It needs to be feature extendable, especially w.r.t the similarity computation component.
• R3: It should not use a database that requires a standalone setup.
• R4: The similarity computation must be clear and explainable.

The functional and technical specifications

Notion is a powerful platform for creating written content. In addition to its wide range of native tools - such as formatting options, databases, and media embedding - it also allows users to connect their workspace to external tools like GitHub, Gmail, Slack, Jira, and more, extending its functionality. This is achieved through connections, or integrations. These can be either officially supported, like the ones mentioned above, or custom-built for a specific workspace. Integrations interact with the user’s workspace via the Notion API, and can be triggered by workspace events (e.g., changes or comments) via Webhooks. Consequently, building an integration seemed like the obvious choice to create a tool that seamlessly fits into my workflow and workspace, fulfilling the first requirement (R1). I decided to leverage comments as a way of interaction between me and the integration since Notion does not allow interacting with unofficial integration with slash commands for instance.

The integration needs to receive commands from within the workspace, query LLM APIs, return the refined content as a suggestion, and replace the original paragraph with the refined version once approved by the author (me!). Additionally, it should compute and track similarity scores between the original and refined texts, and provide an aggregated measure for the overall article once it’s complete. NodeJS was a natural fit for this task - primarily to leverage the official JavaScript Notion client - except for the similarity computation. For that part, I chose Python due to its richer NLP ecosystem, which allows for future extensions of this module (R2).

To track individual paragraph versions and their similarity scores, Refine uses a Notion database, natively available in the Notion client and requiring no additional setup beyond creating the table (R3). For each accepted suggestion generated by the LLM, a row is added to the database with the following attributes:

[block_id, version, similarity, timestamp,  word_count_before, word_count_after]

block_id is the paragraph ID (you are asking why is it called block?). version is incremented every time a given block is refined by AI. similarity is the similarity between the original text and the accepted suggestion and timestamp is self explanatory. word_count_before and word_count_after stores the word count in the paragraph before and after refinement.

All of this adds up to create the PoC architecture shown in the image below.

The elephant in the room!

The most complex aspect of this project is undeniably the similarity score computation. A human writing process, as opposed to an LLM process, is not a “sequential” process based on maximizing the probability of the next token. People do not write linearly! They have virtually unlimited ways of writing both from a content perspective (e.g., adding ideas, altering structure, removing words, changing style and so on) and from a temporal perspective (e.g., writing then revising, revising while writing, overwriting to shorten, summarizing to elaborate, a combination of all the above). This is even more true when we add yet another step, that is AI assistance, which can occur at any point of this complex writing process.

When it comes to measuring the level of involvement of AI in a given text using similarity metrics, the question that should be answered is the following :

Naturally , other questions emerges from this one, like what similarity metrics are adapted to this use case? Should the choice of metric depend on the writing process itself? How can a similarity metric be interpreted by a reader? and last but not least, how to evaluate and validate a proposed measurement methods?

Following is a description of the similarity computation method I use and report in my writings.

To recall, there are two types of similarity metrics to compute: first, the Per-Paragraph Similarity PPS and second, the Overall Article Similarity AOS. The former is what really reflect how much AI contributed to changing the input texts and the latter aggregates the individual PPS scores to a final sharable score.

For a given PPS set, we can compute AOS as a weighted average with weights reflecting the paragraph length in word count. This gives higher importance to longer paragraphs. Let’s call the set of textual blocks (i.e. a paragraph) in a notion page (i.e. an article) P, AOS can be computed as follows:

AOS = ( Σ_i∈P w_i · PPS_i ) / ( Σ_i∈P w_i )

Computing PPS is more tricky. It is in fact what answers our open question. Let’s recall that for each paragraph which has been refined at least once by an LLM, we store every refinement instance with the similarity score between the texts before and after the refinement. Let’s call this similarity a Per-Version-Similarity PVS, and for a given paragraph p, let’s call the set of versions V_p. PPS is the aggregation of all the similarity score in V_p

PPS = agg(PVS_v) , v ∈ V_p

PVS is a [0,1] normalized cosine similarity between the TF-IDF vectors of the paragraph before and after refinement. TF-IDF (Term Frequency - Inverse Document Frequency) is a basic text vectorizing technique. Its limitation, as opposed to a more complex embedding technique like Doc2Vec, is that it is not able to capture or represent the semantic context of a text. This limitation can however be seen as an advantage; two texts, completely different in their structure and formulation, will be very distant from a TFIDF perspective but can be seen as very similar when compared using Doc2Vec as long as they hold the same semantics and meaning. Such a similarity does not reflect to what extent the LLM has altered the text from a style and structure perspective. In the context of my usage this seems a reasonable approach to be adopted.

What remains to do is to define the function agg() . As mentioned earlier, the unpredictability of the writing process makes it impossible to identify one single, generalizable, methods to aggregate the version similarities for each paragraph. One option is to store the actual text content for every version - not only the similarity scores - and to track additions and deletions from the initial to the final version. In other words, to build a version control system. Aside from being out of scope, this would require a more robust, standalone database, which would conflict with both requirement R4 for this PoC and, more importantly, the principle of simplicity that guides this work.

Another option is to make assumptions regarding the writing process and to use an ad-hoc method that follows these assumptions. While not ideal, I would argue that this solution is fairly reasonable: The goal of this work is not to create a benchmark for objectively comparing written content across authors. Rather, it is to promote transparency in AI assisted writing. As long as the author is clear about the rules, assumptions and computation methods they use, the goal is largely reached.

To illustrate the type of assumptions one could make, let’s consider my writing workflow I described earlier. Since I tend to finish writing a paragraph before AI-refining it, we can assume that the content and length of paragraph won’t drastically change over the various versions. Hence, it is safe to consider the minimum similarity over the various versions of a paragraph as fair way of computing PPS. That is:

PPS_p = min(VPS_v) , v ∈ V_p

A Demo

Refine automates the manual AI-assisted writing process I described earlier. The following screenshots depicts the actual use of this tool:

To refine a paragraph, I create a new comment /refine on the paragraph (or any word in it) or add a comment in an existing discussion. The integration will fetch the block content and query the LLM for a refinement. The instruction prompt is pre-loaded by the code at launch but can be overwritten by the /setup comment.

Once accepted through a /accept comment, the original text is automatically replaced with the suggested text. For this PoC, the accepted version is always the last suggestion provided (we cannot pick an older suggestion). After the replacement, a tag is appended indicating the similarity score and the version of the block. The /*/^ 🤖 is a control sequence used to filter out this tag when needed (e.g. to re-refine a block!).

Finally, when the article is completed, refine aggregates the similarity score using the computation discussed earlier and returns the final score. It also removes the similarity and version tags from all the refined paragraphs.

Limitations

This PoC meets the objectives it was designed for but still lacks key features to qualify as an MVP. First, the LLM integration is fairly rudimentary. While it works for my use case, the absence of context handling and more advanced instruction refinement tools is a significant limitation. Despite being tightly constrained by the instructions, there are ways to direct LLMs to retain user feedback within a given context and adapt their style to produce coherent content across multiple refined paragraphs. Which is key for an enjoyable reading session!

Second, the available commands are not configurable. For example, although the similarity engine supports multiple similarity computation methods, there is currently no way to specify - through commands - which method should be used for a given article. This is an important feature to implement, as it plays a key role in helping authors calibrate and refine their writing workflow.

Some Closing Thoughts…

This work started as a small Saturday project, but I believe that by addressing a few existing limitations and by adding a proper evaluation process, it can evolve into real testing framework that enables comparison between different workflows, and more importantly, between different similarity computation methods. Which is, after all, the real open research question.

First published on the Sydyk blog, April 2025.