After exporting the chatgpt data, I would like to recreate the conversations similar to what is available in chat.html. It is not immediately clear to me how the mapping is structured in conversations.json. Is this documented somewhere?
I also tried to copy/paste chat.html into ChatGPT to see if I could get ChatGPT to figure out what the html code was doing, but I got a violation of terms acknowledgement window. Does anyone know why that might be?
Nevermind on part 2. It was related to having a really long line of json data apparently.
Finally got it working. Here is the conversation:
https://chat.openai.com/share/761c4e0c-b4a8-4c1b-b4bf-3ce32e9e3602
I was able to eventually get ChatGPT to get it. I didn’t have to understand much of the code at all, but it did take a few hours to navigate the straying, etc.
The code at the bottom seems to work ok.
Hi, I’m working on the same thing; I had chatgpt write a python script to convert the json file into something I would work with using tiddlywiki as an importer. Any progress on this? Does anyone know if my conversations.json file is available via API?
If you log into the site, you can export your conversations (via settings). The files are produced and a link/ url is emailed to you. Click on the link and it will download. But be aware that the conversations.json file format has recently changed . The example code here is out of date.
I’m working on a tool to do the sort of thing described here (in my case, it’s to create local markdown files from an export). It’s working great so far but there’s still a little more to do. If you’d like to try it out, shoot me a message at marty at martiansoftware dot com and I’ll send you a copy as soon as it’s ready.
Mate, thanks a lot! Works like a charm. I did a little polishing so one can use it easily as CLI. Also changed output name format so chat files are sorted by date when already sorted by name ( Github Gist fd351f01a5d561d433ae852fba8eca0a ):
"""
This script processes conversation data from a JSON file, extracts messages,
and writes them to text files. It also creates a summary JSON file with a summary
of the conversations. The script is designed to be run as a command-line interface (CLI),
allowing the user to specify the input JSON file and output directory.
Usage:
python script_name.py /path/to/conversations.json /path/to/output_directory
"""
import unicodedata
import json
import re
import argparse
from datetime import datetime
from pathlib import Path
def extract_message_parts(message):
"""
Extract the text parts from a message content.
Args:
message (dict): A message object.
Returns:
list: List of text parts.
"""
content = message.get("content")
if content and content.get("content_type") == "text":
return content.get("parts", [])
return []
def get_author_name(message):
"""
Get the author name from a message.
Args:
message (dict): A message object.
Returns:
str: The author's role or a custom label.
"""
author = message.get("author", {}).get("role", "")
if author == "assistant":
return "ChatGPT"
elif author == "system":
return "Custom user info"
return author
def get_conversation_messages(conversation):
"""
Extract messages from a conversation.
Args:
conversation (dict): A conversation object.
Returns:
list: List of messages with author and text.
"""
messages = []
current_node = conversation.get("current_node")
mapping = conversation.get("mapping", {})
while current_node:
node = mapping.get(current_node, {})
message = node.get("message") if node else None
if message:
parts = extract_message_parts(message)
author = get_author_name(message)
if parts and len(parts) > 0 and len(parts[0]) > 0:
if author != "system" or message.get("metadata", {}).get(
"is_user_system_message"
):
messages.append({"author": author, "text": parts[0]})
current_node = node.get("parent") if node else None
return messages[::-1]
def create_directory(base_dir, date):
"""
Create a directory based on the date.
Args:
base_dir (Path): Base output directory.
date (datetime): The date to base the directory name on.
Returns:
Path: The path of the created directory.
"""
directory_name = date.strftime("%Y_%m")
directory_path = base_dir / directory_name
directory_path.mkdir(parents=True, exist_ok=True)
return directory_path
def sanitize_title(title):
"""
Sanitize the title to create a valid file name, preserving non-ASCII characters.
Args:
title (str): The title of the conversation.
Returns:
str: Sanitized title.
"""
title = unicodedata.normalize("NFKC", title)
title = re.sub(r'[<>:"/\\|?*\x00-\x1F\s]', '_', title)
return title[:140]
def create_file_name(directory_path, title, date):
"""
Create a sanitized file name.
Args:
directory_path (Path): The directory where the file will be saved.
title (str): The title of the conversation.
date (datetime): The date to base the file name on.
Returns:
Path: The path of the created file.
"""
sanitized_title = sanitize_title(title)
return (
directory_path / f"{date.strftime('%Y_%m_%d')}_{sanitized_title}.txt"
)
def write_messages_to_file(file_path, messages):
"""
Write messages to a text file.
Args:
file_path (Path): The path of the file to write to.
messages (list): List of messages to write.
"""
with file_path.open("w", encoding="utf-8") as file:
for message in messages:
file.write(f"{message['author']}\n")
file.write(f"{message['text']}\n")
def update_conversation_summary(summary, directory_name, conversation, date, messages):
"""
Update the conversation summary dictionary.
Args:
summary (dict): The conversation summary dictionary.
directory_name (str): The name of the directory.
conversation (dict): The conversation object.
date (datetime): The updated date of the conversation.
messages (list): List of messages in the conversation.
"""
if directory_name not in summary:
summary[directory_name] = []
summary[directory_name].append(
{
"title": conversation.get("title", "Untitled"),
"create_time": datetime.fromtimestamp(
conversation.get("create_time")
).strftime("%Y-%m-%d %H:%M:%S"),
"update_time": date.strftime("%Y-%m-%d %H:%M:%S"),
"messages": messages,
}
)
def write_summary_json(output_dir, summary):
"""
Write the conversation summary to a JSON file.
Args:
output_dir (Path): The output directory.
summary (dict): The conversation summary to write.
"""
summary_json_path = output_dir / "conversation_summary.json"
with summary_json_path.open("w", encoding="utf-8") as json_file:
json.dump(summary, json_file, ensure_ascii=False, indent=4)
def write_conversations_and_summary(conversations_data, output_dir):
"""
Write conversation messages to text files and create a conversation summary JSON file.
Args:
conversations_data (list): List of conversation objects.
output_dir (Path): Directory to save the output files.
Returns:
list: Information about created directories and files.
"""
created_directories_info = []
conversation_summary = {}
for conversation in conversations_data:
updated = conversation.get("update_time")
if not updated:
continue
updated_date = datetime.fromtimestamp(updated)
directory_path = create_directory(output_dir, updated_date)
title = conversation.get("title", "Untitled")
file_name = create_file_name(directory_path, title, updated_date)
messages = get_conversation_messages(conversation)
write_messages_to_file(file_name, messages)
update_conversation_summary(
conversation_summary,
directory_path.name,
conversation,
updated_date,
messages,
)
created_directories_info.append(
{"directory": str(directory_path), "file": str(file_name)}
)
write_summary_json(output_dir, conversation_summary)
return created_directories_info
def main():
"""
Main function to parse arguments and process the conversations.
"""
parser = argparse.ArgumentParser(
description="Process conversation data from a JSON file."
)
parser.add_argument(
"input_file", type=Path, help="Path to the input conversations JSON file."
)
parser.add_argument(
"output_dir", type=Path, help="Directory to save the output files."
)
args = parser.parse_args()
if not args.input_file.exists():
print(f"Error: The input file '{args.input_file}' does not exist.")
return
with args.input_file.open("r", encoding="utf-8") as file:
conversations_data = json.load(file)
created_directories_info = write_conversations_and_summary(
conversations_data, args.output_dir
)
for info in created_directories_info:
print(f"Created {info['file']} in directory {info['directory']}")
if __name__ == "__main__":
main()
I just released the tool I mentioned above to convert ChatGPT exports into local Markdown files. It’s called ChatKeeper, and I’d love any feedback or suggestions for improvement you might have.
Links are not allowed here but it can be found on my website at martiansoftware dot com.
why wouldn’t you build a chrome extension type of tool?
Because I wanted to act on the full conversation history, not just the current conversation, and I wanted full access to the local filesystem (which might be possible in a chrome extension but I’m not sure).
Also I don’t use chrome. 
And didn’t want to make something specific to one browser.
For those who are interested in using PowerShell here is a script that does the same thing:
# Prompt the user for the directory containing the conversations.json file
$inputDirectory = Read-Host "Please enter the directory where conversations.json is located"
# Define the function to get conversation messages
function Get-ConversationMessages {
param ($conversation)
$messages = @()
$currentNode = $conversation.current_node
$mapping = $conversation.mapping
while ($currentNode) {
$node = $mapping.$currentNode
$message = $node.message
$content = $message.content
$author = if ($message.author) { $message.author.role } else { "" }
if ($content -and $content.content_type -eq "text") {
$parts = $content.parts
if ($parts.Count -gt 0 -and $parts[0].Length -gt 0) {
if ($author -ne "system" -or ($message.metadata).is_user_system_message) {
if ($author -eq "assistant") { $author = "ChatGPT" }
elseif ($author -eq "system") { $author = "Custom user info" }
$messages += [pscustomobject]@{
Author = $author
Text = $parts[0]
}
}
}
}
$currentNode = $node.parent
}
return $messages | Sort-Object -Descending
}
# Define the function to write conversations and create pruned.json
function Write-ConversationsAndJson {
param ($conversationsData)
# Get the directory where the script is saved
$outputDirectory = $PSScriptRoot
$createdDirectoriesInfo = @()
$prunedData = @{}
foreach ($conversation in $conversationsData) {
$updated = $conversation.update_time
if (-not $updated) { continue }
# Convert Unix timestamp to DateTime
$updatedDate = Get-Date ([DateTimeOffset]::FromUnixTimeSeconds($updated).DateTime)
$directoryName = $updatedDate.ToString("MMMM_yyyy")
$directoryPath = Join-Path $outputDirectory $directoryName
if (-not (Test-Path $directoryPath)) {
New-Item -Path $directoryPath -ItemType Directory | Out-Null
}
$title = if ($conversation.title) { $conversation.title } else { "Untitled" }
$sanitizedTitle = $title -replace "[^a-zA-Z0-9_]", "_" -replace "^.{120}", "$&"
$fileName = "$directoryPath/$sanitizedTitle_$($updatedDate.ToString('dd_MM_yyyy_HH_mm_ss')).txt"
$messages = Get-ConversationMessages $conversation
$messageContent = $messages | ForEach-Object { "$($_.Author)`n$($_.Text)`n" }
Set-Content -Path $fileName -Value $messageContent -Encoding UTF8
if (-not $prunedData[$directoryName]) {
$prunedData[$directoryName] = @()
}
$prunedData[$directoryName] += @{
Title = $title
Create_Time = (Get-Date ([DateTimeOffset]::FromUnixTimeSeconds($conversation.create_time).DateTime)).ToString("yyyy-MM-dd HH:mm:ss")
Update_Time = $updatedDate.ToString("yyyy-MM-dd HH:mm:ss")
Messages = $messages
}
$createdDirectoriesInfo += @{
Directory = $directoryPath
File = $fileName
}
}
$jsonPrunedData = $prunedData | ConvertTo-Json -Depth 4 -Compress
Set-Content -Path (Join-Path $outputDirectory "pruned.json") -Value $jsonPrunedData -Encoding UTF8
return $createdDirectoriesInfo
}
# Load conversations.json data after prompting the user for the directory
$conversationFilePath = Join-Path $inputDirectory "conversations.json"
if (Test-Path $conversationFilePath) {
$conversationData = Get-Content $conversationFilePath -Raw | ConvertFrom-Json
$createdDirectoriesInfo = Write-ConversationsAndJson -conversationsData $conversationData
Write-Host "Processing complete. Files saved in: $PSScriptRoot"
} else {
Write-Host "Error: conversations.json not found in the provided directory."
}
That’s a cool PowerShell script and although I haven’t tried it, it looks useful for some basic use. Reading through it, however, I can confidently say it doesn’t do the same thing as my software. But it might be a great alternative for some folks using Windows. The more options for backing up and having useful local copies of our conversations, the better.
I actually came here to say that there’s a beta version of ChatKeeper (at martiansoftware dot com) available now for Mac users in case anyone was disappointed not to see Mac support earlier. I hope it’s helpful to folks.
I’ve been working on some new features and improvements that I’d love for folks to try out in a new release candidate for ChatKeeper version 1.1.0.
Rendering to Markdown any conversations that use the new Canvas and Search features requires a lot of json processing that even OpenAI’s own export viewer (html and javascript included in the export) doesn’t do.
I’m not aware of any current issues with this version, but I’m still finalizing testing for the official 1.1.0 release… so it might have issues I haven’t found yet. So while I’m testing I’m also very interested to hear how it works for you.
- Marty
#!/home/jack/miniconda3/envs/cloned_base/bin/python
import json
import logging
import os
import glob
import subprocess
import os
import string
‘’’
create working directory in it a directory called CHATGPT
in CHATGPT folder unzip the downloaded zip file of chatgpt dataset
run this script from the project folder it will create a three folders called:
directory1 = ‘CHATGPT/JSON’
make_path_exist(directory1)
directory2 = ‘CHATGPT/HTML’
make_path_exist(directory2)
directory3 = ‘CHATGPT/TEXT’
make_path_exist(directory3)
it will convert the data into three forms txt, html and json
it will create a database called CHATGPT_files.db
it will create a table called files
it will insert the data into the table
it will close the connection
‘’’
def clean_title(title):
valid_chars = set(string.ascii_letters + string.digits + string.whitespace)
cleaned_title = ‘’.join(char if char in valid_chars else ‘’ for char in title)
cleaned_title = cleaned_title.replace(’ ', '’) # Replace spaces with underscores
return cleaned_title.strip()
‘’’
This code defines a function make_path_exist that takes a directory path as input and creates the directory if it does not already exist. It then calls this function three times with different directory names.
‘’’
def make_path_exist(directory):
path = os.path.join(os.getcwd(), directory)
if not os.path.exists(path):
os.makedirs(path)
def split_and_save_and_convert(conversations_file):
directory1 = ‘CHATGPT/JSON’
make_path_exist(directory1)
directory2 = ‘CHATGPT/HTML’
make_path_exist(directory2)
directory3 = ‘CHATGPT/TEXT’
make_path_exist(directory3)
try:
with open(conversations_file, ‘r’, encoding=‘utf-8’) as file:
data = json.load(file)
for conversation in data:
title = conversation.get('title', 'Unknown_Title')
title_with_underscores = clean_title(title)
chapter_filename = f"{title_with_underscores}.json"
chapter_filepath = os.path.join(directory1, chapter_filename)
logging.info(f"Saving data for conversation '{title}' to {chapter_filepath}")
with open(chapter_filepath, 'w', encoding='utf-8') as chapter_file:
json.dump([conversation], chapter_file, indent=2)
# Convert JSON to HTML
html_output_file = os.path.join(directory2, f"{title_with_underscores}.html")
convert_to_html(chapter_filepath, html_output_file)
# Convert JSON to TXT
txt_output_file = os.path.join(directory3, f"{title_with_underscores}.txt")
convert_to_txt(chapter_filepath, txt_output_file)
except FileNotFoundError:
logging.error(f"File not found: {conversations_file}")
except json.JSONDecodeError:
logging.error(f"Error decoding JSON in file: {conversations_file}")
except Exception as e:
logging.error(f"An unexpected error occurred: {e}")
def convert_to_html(json_file, html_output_file):
with open(json_file, ‘r’, encoding=‘utf-8’) as file:
json_data = json.load(file)
result_str = get_conversation_result(json_data)
with open(html_output_file, "w", encoding='utf-8') as html_output:
result_html = result_str.replace("/n", "XXXXXXX\n")
result_html = result_html.replace("<", "<")
result_html = result_html.replace(">", ">")
for line in result_html.split("XXXXXXX"):
line = line.replace("\n", "<br />\n")
html_output.write(line)
def convert_to_txt(json_file, txt_output_file):
with open(json_file, ‘r’, encoding=‘utf-8’) as file:
json_data = json.load(file)
result_str = get_conversation_result(json_data)
with open(txt_output_file, "w", encoding='utf-8') as txt_output:
result_txt = result_str.replace("/n", "XXXXXXX\n")
for line in result_txt.split("XXXXXXX"):
txt_output.write(line)
def get_conversation_result(json_data):
result_str = “”
for conversation in json_data:
title = conversation.get(‘title’, ‘’)
messages = get_conversation_messages(conversation)
result_str += title + '\n'
for message in messages:
result_str += message['author'] + '\n' + message['text'] + '\n'
result_str += '\n'
return result_str
def get_conversation_messages(conversation):
messages =
current_node = conversation.get(‘current_node’)
while current_node:
node = conversation[‘mapping’][current_node]
message = node.get(‘message’)
if (message and message.get(‘content’) and message[‘content’].get(‘content_type’) == ‘text’ and
len(message[‘content’].get(‘parts’, )) > 0 and len(message[‘content’][‘parts’][0]) > 0 and
(message[‘author’][‘role’] != ‘system’ or message.get(‘metadata’, {}).get(‘is_user_system_message’))):
author = message[‘author’][‘role’]
if author == ‘assistant’:
author = ‘ChatGPT’
elif author == ‘system’ and message[‘metadata’].get(‘is_user_system_message’):
author = ‘Custom user info’
messages.append({‘author’: author, ‘text’: message[‘content’][‘parts’][0]})
current_node = node.get(‘parent’)
return messages[::-1]
conversations_file_path = ‘CHATGPT/conversations.json’
#output_folder = ‘CHATDPT/output_txt_html_json’
#os.makedirs(output_folder, exist_ok=True)
logging.basicConfig(level=logging.INFO)
split_and_save_and_convert(conversations_file_path)
import sqlite3
import os
import hashlib
db_path2 = ‘CHATGPT_files.db’
conn = sqlite3.connect(db_path2)
cursor = conn.cursor()
cursor.execute(‘’’
CREATE TABLE IF NOT EXISTS files (
id INTEGER PRIMARY KEY,
filename TEXT NOT NULL,
content BLOB NOT NULL,
text_content TEXT NOT NULL,
hash_value TEXT NOT NULL,
format TEXT NOT NULL
)
‘’')
conn.commit()
conn.close()
def calculate_hash(file_path):
sha256 = hashlib.sha256()
with open(file_path, ‘rb’) as file:
while chunk := file.read(8192): # Read in 8KB chunks
sha256.update(chunk)
return sha256.hexdigest()
def insert_file(filename, content, text_content, hash_value, file_format):
conn = sqlite3.connect(db_path2)
cursor = conn.cursor()
cursor.execute(‘INSERT INTO files (filename, content, text_content, hash_value, format) VALUES (?, ?, ?, ?, ?)’,
(filename, content, text_content, hash_value, file_format))
conn.commit()
conn.close()
def insert_text_files(directory):
for filename in os.listdir(directory): # Corrected variable name
if filename.endswith(‘.txt’):
file_path = os.path.join(directory, filename) # Construct full file path
with open(file_path, ‘rb’) as file:
print(file_path)
file_content = file.read()
text_content = file_content.decode('utf-8', errors='ignore') # Convert bytes to string
hash_value = calculate_hash(file_path)
insert_file(filename, file_content, text_content, hash_value, 'txt') # Corrected insertion
print(f"Inserted: {filename}")
input_folder = ‘CHATGPT/TEXT’
insert_text_files(input_folder)
print(‘Insertion process completed.’)
#---------------------------------------------------
def clean_title(title):
valid_chars = set(string.ascii_letters + string.digits + string.whitespace)
cleaned_title = ‘’.join(char if char in valid_chars else ‘’ for char in title)
cleaned_title = cleaned_title.replace(’ ', '’) # Replace spaces with underscores
return cleaned_title.strip()
‘’’
This code defines a function make_path_exist that takes a directory path as input and creates the directory if it does not already exist. It then calls this function three times with different directory names.
‘’’
def make_path_exist(directory):
path = os.path.join(os.getcwd(), directory)
if not os.path.exists(path):
os.makedirs(path)
def split_and_save_and_convert(conversations_file):
directory1 = ‘CHATGPT/JSON’
make_path_exist(directory1)
directory2 = ‘CHATGPT/HTML’
make_path_exist(directory2)
directory3 = ‘CHATGPT/TEXT’
make_path_exist(directory3)
try:
with open(conversations_file, ‘r’, encoding=‘utf-8’) as file:
data = json.load(file)
for conversation in data:
title = conversation.get('title', 'Unknown_Title')
title_with_underscores = clean_title(title)
chapter_filename = f"{title_with_underscores}.json"
chapter_filepath = os.path.join(directory1, chapter_filename)
logging.info(f"Saving data for conversation '{title}' to {chapter_filepath}")
with open(chapter_filepath, 'w', encoding='utf-8') as chapter_file:
json.dump([conversation], chapter_file, indent=2)
# Convert JSON to HTML
html_output_file = os.path.join(directory2, f"{title_with_underscores}.html")
convert_to_html(chapter_filepath, html_output_file)
# Convert JSON to TXT
txt_output_file = os.path.join(directory3, f"{title_with_underscores}.txt")
convert_to_txt(chapter_filepath, txt_output_file)
except FileNotFoundError:
logging.error(f"File not found: {conversations_file}")
except json.JSONDecodeError:
logging.error(f"Error decoding JSON in file: {conversations_file}")
except Exception as e:
logging.error(f"An unexpected error occurred: {e}")
def convert_to_html(json_file, html_output_file):
with open(json_file, ‘r’, encoding=‘utf-8’) as file:
json_data = json.load(file)
result_str = get_conversation_result(json_data)
with open(html_output_file, "w", encoding='utf-8') as html_output:
result_html = result_str.replace("/n", "XXXXXXX\n")
result_html = result_html.replace("<", "<")
result_html = result_html.replace(">", ">")
for line in result_html.split("XXXXXXX"):
line = line.replace("\n", "<br />\n")
html_output.write(line)
def convert_to_txt(json_file, txt_output_file):
with open(json_file, ‘r’, encoding=‘utf-8’) as file:
json_data = json.load(file)
result_str = get_conversation_result(json_data)
with open(txt_output_file, "w", encoding='utf-8') as txt_output:
result_txt = result_str.replace("/n", "XXXXXXX\n")
for line in result_txt.split("XXXXXXX"):
txt_output.write(line)
def get_conversation_result(json_data):
result_str = “”
for conversation in json_data:
title = conversation.get(‘title’, ‘’)
messages = get_conversation_messages(conversation)
result_str += title + '\n'
for message in messages:
result_str += message['author'] + '\n' + message['text'] + '\n'
result_str += '\n'
return result_str
def get_conversation_messages(conversation):
messages =
current_node = conversation.get(‘current_node’)
while current_node:
node = conversation[‘mapping’][current_node]
message = node.get(‘message’)
if (message and message.get(‘content’) and message[‘content’].get(‘content_type’) == ‘text’ and
len(message[‘content’].get(‘parts’, )) > 0 and len(message[‘content’][‘parts’][0]) > 0 and
(message[‘author’][‘role’] != ‘system’ or message.get(‘metadata’, {}).get(‘is_user_system_message’))):
author = message[‘author’][‘role’]
if author == ‘assistant’:
author = ‘ChatGPT’
elif author == ‘system’ and message[‘metadata’].get(‘is_user_system_message’):
author = ‘Custom user info’
messages.append({‘author’: author, ‘text’: message[‘content’][‘parts’][0]})
current_node = node.get(‘parent’)
return messages[::-1]
conversations_file_path = ‘CHATGPT/conversations.json’
#output_folder = ‘CHATDPT/output_txt_html_json’
#os.makedirs(output_folder, exist_ok=True)
logging.basicConfig(level=logging.INFO)
#split_and_save_and_convert(conversations_file_path)
json_file=‘CHATGPT/conversations.json’
txt_output_file =“conversations_2_text.txt”
convert_to_txt(json_file, txt_output_file)
Insert = open(“conversations.txt”,“a”)
with open(“conversations_2_text.txt”,“r”) as data:
lines = data.read()
line = lines.replace(“user\n”,“CHAT_DIALOGUEuser\n”)
Insert.write(line)
import sqlite3
import logging
logging.basicConfig(level=logging.DEBUG)
def connect_to_database(database_name):
“”"
Connect to the SQLite database.
Args:
database_name (str): The name of the SQLite database file.
Returns:
sqlite3.Connection or None: The database connection or None if connection fails.
"""
try:
conn = sqlite3.connect(database_name)
logging.info("Connected to the database successfully.")
return conn
except Exception as e:
logging.error(f"Failed to connect to the database: {e}")
return None
def create_table(conn):
“”"
Create the dialogue table in the database.
Args:
conn (sqlite3.Connection): The SQLite database connection.
“”"
try:
if conn:
c = conn.cursor()
c.execute(‘’‘CREATE TABLE IF NOT EXISTS dialogue (
id INTEGER PRIMARY KEY,
user_ChatGPT_PAIR TEXT,
user_ChatGPT_PAIRb BLOB
)’‘’)
conn.commit()
logging.info(“Table ‘dialogue’ created successfully.”)
except Exception as e:
logging.error(f"Failed to create table ‘dialogue’: {e}“)
def insert_dialogue(conn, dialogue_data):
“””
Insert dialogue data into the database.
Args:
conn (sqlite3.Connection): The SQLite database connection.
dialogue_data (str): The dialogue data to insert into the database.
“”"
try:
if conn:
c = conn.cursor()
c.execute(“INSERT INTO dialogue (user_ChatGPT_PAIR, user_ChatGPT_PAIRb) VALUES (?,?)”, (dialogue_data,dialogue_data.encode(‘utf-8’),))
conn.commit()
logging.info(“Dialogue inserted into the database successfully.”)
except Exception as e:
logging.error(f"Failed to insert dialogue into the database: {e}")
file_path = ‘conversations.txt’
try:
with open(file_path, “r”) as file:
file_contents = file.read()
dialogue_parts = file_contents.split(“CHAT_DIALOGUE”)
conn = connect_to_database(‘dialogueEXP2.db’)
if conn:
create_table(conn)
for dialogue_part in dialogue_parts:
insert_dialogue(conn, dialogue_part.strip())
print(“.”, end=“-”)
conn.close()
except Exception as e:
logging.error(f"An error occurred while reading or processing the file: {e}")
@jack.northrup.ph - nice. You’ll find it gets much more involved with conversations that take advantage of newer features, like canvas and search. And I have no idea what all of their new features announced this month will entail.
I’ve done a bunch of reverse engineering of their format and intend to keep up with it, so I have some work ahead of me with all of this month’s announcements. It should be fun to figure out.
In the meantime, ChatKeeper has graduated to its 1.1.0 release for anyone interested in checking it out after a quick download.
If you have a github repo or other location I can link to for what you posted, I am planning to add an “alternatives” list to the ChatKeeper site. If you don’t then I can just link to your message. Shoot me an email if you like (email is on the chatkeeper site). That goes for anyone else with an alternative too.
- Marty
Wow, this works amazingly well! Thank you!
Along the lines of converting export data into more useful local files, ChatKeeper 1.2.0 is now available.
ChatKeeper converts your entire conversation history to local markdown files. You can rename or reorganize them however you like, and ChatKeeper will find and update them in place if you run it again later on an updated export.
New in this version:
Full details are on the ChatKeeper Download Page.
- Marty
This was very helpful. VERY helpful. Thank you so much, and for me it worked pretty much on the first try! For someone who isn’t a deep coder like some of you may be. Thank you!
Hello all, I have a quick update for anyone interested in ChatKeeper.
ChatKeeper 1.2.2 is now available for download.
This version adds “Chain of Thought” to the markdown output. This is the text that you might see flashing by while it’s “Reasoning”, followed by something like “Thought for 15 seconds.”
The markdown ends up looking something like this in Obsidian (but it’s just markdown and will work with any renderer):
Handling this chain of thought also eliminates any warning messages about “unhandled message types” for the types “thoughts” and “reasoning_recap”. (Note to self: I really need to make those warning messages friendlier.)
There are some minor bug fixes in this version as well, plus a change for better handling of conversations that use ChatGPT’s Canvas.
I read and respond to all feedback, so if you give it a try, please don’t hesitate to reach out and share your thoughts…
- Marty
