The secrets of chatGPT analyzed with Anatella
Research firm IntoTheMinds, has recently published a new analysis on ChatGPT conducted using Timi and, more specifically, Anatella.
This analysis focused on a corpus of 1000 texts produced by ChatGPT, and their similarity was analyzed using Anatella’s unique features (Dice method). The results shed new light on the effect of prompts on ChatGPT’s functioning.
In this experiment, minimal instructions were provided to ChatGPT to generate a 500-word text on a given topic. The results were compared to those from a previous analysis where the instructions were more detailed.
The results are as follows:
- The similarity of texts produced by ChatGPT increases by about 5% when the level of detail in the prompt is increased. In other words, the less instructions you give to ChatGPT, the less creative it will be in its production. This conclusion applies to both regenerating answers to the same question and answering different questions with similarly minimal prompts.
- The texts regenerated by ChatGPT in response to the same question are extremely similar. The similarity rate, measured using the Dice method, ranges from 76% to 80%. In other words, even when regenerating the answer, the produced text will be largely similar to the previous one.
- Surprisingly, even when the questions are different, ChatGPT’s answers are similar. Twenty different questions were asked, and the average similarity rate between the answers was 65%.
- The only instruction given (on the length of the text) is followed throughout the corpus (1000 texts) with a precision of +/-1%.
This analysis is particularly interesting as it demonstrates the full range of possibilities offered by Anatella. Not only did Anatella solve complex data processing problems, but the corpus processing was also done in record time. This confirms the previously published results of the benchmark conducted by IntoTheMinds with other ETL solutions.
Source : IntoTheMind
Photo from Daniel Perunov on Unsplash
