13 AI Content Detection Tools tested and AI Watermarks

13 AI Content Detection Tools tested and AI Watermarks

Percent Real - AI Content Detection and AI Watermarks

With the launch of ChatGPT at the end of November 2022, it became clear that large language models (LLM) like GPT-3.5 can replace a lot of text generation, more than even GPT-3 or older models. Organizations, government agencies, and universities have a strong interest in determining if a piece of content was generated using a language model or AI text tool.

Marketers and companies who purchase content may want to understand the extent to which the content they have acquired was generated using tools such as GPT-3 or Jasper, Writesonic, or copy.ai. A GPT-3 detector would be helpful, as GPT-3 is the most common model used in these tools.

Website owners and SEO specialists want to ensure that Google correctly indexes their AI-generated content. The question remains if and when such AI content generator tools would indeed apply AI watermarks, or if they are able to decide.

For anyone wanting to audit content for ChatGPT as a major source, a “ChatGPT detector” would be required, particularly an “OpenAI Detector.” However, so far, most detectors seem to be pure GPT-2 detectors, including the “OpenAI Content Detector.”

Now we’ll look at the state of the art in AI Content Detection and AI Content Watermarks.

How would a GPT-3 detector or ChatGPT detector work?

There are several ways to detect whether a piece of content was generated using a language model like GPT-3, ChatGPT, or tools like Jasper.ai or Writesonic. Some methods include:

  1. Checking for certain linguistic features or patterns that are common in machine-generated text. For example, AI-generated text might have higher levels of repetitiveness or lack the complexity and variability of human-written text.
  2. Checking for specific formatting or structural features that are common in machine-generated text, such as a uniform structure or lack of varied formatting.
  3. Checking for certain keywords or phrases commonly used in AI-generated text. In general, there are statistically significant patterns known and detectable based on word combinations a model like GPT-3 picks.
  4. Comparing the content in question to known examples of machine-generated text, which could be done manually or using machine learning techniques to analyze the content.

It is important to know that both GPT-3 and ChatGPT (GPT-3.5) have a much richer vocabulary, and produce more complex output, making them harder to distinguish from human-written content. Detecting whether a piece of content was generated using a language model or AI tool is not straightforward and may be challenging to ascertain definitively.

AI Content Detection Tools

Which automated tools exist to detect AI content? Most of these are based on GPT-2, not GPT-3 or GPT-3.5. Here’s a brief list, along with a quick “smoke test,” commentary, and testing of each tool:

The Input from ChatGPT

write a polite mail that a customer cannot come to the office for a coffee, because we're a remote-only company, but that we would be happy to do a Zoom call for a chat.

which generated a typical response:

Dear valued customer,

Thank you for your interest in meeting with us for coffee. Unfortunately, as a remote-only company, we are unable to offer in-person meetings at this time.

However, we would be happy to schedule a Zoom call for a chat instead. Please let us know a convenient time for you, and we will make sure to set it up.

We apologize for any inconvenience this may cause.

Sincerely, [Your Name]

ChatGPT Content Detection not working with GPTZero

GPTZero gave us a statement regarding the likelihood of the text being AI-generated:

Your GPTZero score corresponds to the likelihood of the text being AI generated 31.5

ChatGPT Content Detection Successful with PoemOfQuotes

One of the two tools that provided a clear statement “Our algorithms believe that this content was written by AI” is the GPT-3 AI Content Detector by PoemOfQuotes.

AI Content Detector by Writer.com

The AI Content Detector by Writer has thus far failed to provide accurate results.

AI Content Detection Tools Failures

The testing showed several failures by various detection tools, such as:

AI Watermarks

What are AI Watermarks?

AI watermarking ensures that content generated by AI can be detected. There have been multiple demands for AI watermarking by institutions and journalists. OpenAI has already experimented with watermarking and built a prototype to statistically watermark the outputs of a text model like GPT.

Why AI Watermarks?

AI watermarking is potentially a legal requirement for large language model (LLM) providers. Many LLMs can and should result in required watermarking for AI-generated works.

Conclusions

What AI Content Detection tools can we use today?

Currently, tools like GPTZero, PoemOfQuotes, and Originality.ai warrant further exploring. More systematic tests are required for determining their efficacy against varied samples of content.