Unless you’ve been living in a cave, you’re aware that generative AI is at the center of a revolution right now. With a background in cybersecurity and trust & safety, I naturally wanted to see if I could fine-tune and protect an LLM.
Many businesses are adopting LLMs as chatbots or employee augmentation tools like customer service reps. I wanted to do something less mainstream but still interesting. A friend recently observed how commodity models’ “personalities” shift with updates, yet users form genuine connections to the previous versions. That stuck with me. What if, instead of chasing the latest model update, you could train something with a fixed personality, one grounded in history?
Defining the Goal
When I take on a learning project, I need it to have a purpose. Something that gives me direction and a clear finish line. In this case, I wanted to build a chatbot, but not just replicate what others have already done. Commodity LLMs already handle general inquiries well enough. I needed a use case obscure enough that people haven’t built a dozen versions of it, yet prominent enough to have sufficient training data.
I found the perfect candidate: a chatbot that responds in the voice of Joseph Smith. He’s one of the most well-documented figures of his era, and the Joseph Smith Papers Project offers a rich, digitized corpus to work with.
Data Selection
The first step was obtaining the right training data from the Joseph Smith Papers Project. Being a Papers project, they’ve gathered thousands of documents: journals, administrative records, legal proceedings, receipts, and letters. I immediately disregarded anything that wouldn’t contain Joseph’s own voice. Meeting minutes, for example, rarely included direct quotes and instead described what he said.
That mostly left journals and letters.
The journals initially seemed promising, but after some review of them I discounted them as a training source. The Project states they comprise of “over 1,500 manuscript pages… (but) more than 80 percent of the total—were primarily the work of five men.” The journal contents also didn’t necessarily report Joseph’s experiences. Although a rich historical source, sifting through and attributing voice would be more problematic than it first appeared.
I turned to the letters instead. After some consideration, I realized these were actually richer for my purposes. Letter exchanges are essentially prompt-response pairs, even though few letters have the full correspondence preserved. The letters were typically short and focused on particular subjects. Best of all, the Joseph Smith Papers Project had included historical introductions providing excellent context for training.
The letters still presented a mixed-voice problem. Sometimes multiple people wrote to different recipients within the same letter. For example, while traveling, two accompanying men and Joseph each wrote to their separate families in a single document. Thankfully, the Papers Project annotates who served as scribe, and the historical introductions help determine where Joseph Smith is actually dictating.
I acquired transcripts of 257 letters.
Data Preparation
With a rich source selected, I needed to prepare it for supervised fine-tuning. This meant:
Filtering mixed voices. Using scribe annotations and historical context to isolate Joseph Smith’s actual words.
Expanding names. Replacing initials and abbreviations with full names. There could be four different people named “William” in the corpus, so using full names clarifies which person is being referenced. This helps the model learn relationships between individuals.
Removing historian annotations. The editors’ marks help scholars read the letters, but they need to be stripped before training.
Modernizing spelling and adding punctuation. The original transcripts preserve 19th-century conventions that would confuse the model.
Segmenting longer letters. This proved more complex than expected.
Why Segment?
Long letters broke the fine-tuning process given my RTX 3080’s memory limitations. But there were content reasons too. Letters often contain multiple topics and sometimes address multiple audiences. Joseph Smith also frequently quoted other people, scripture, or Deity. Segmenting made it easier to identify and remove these passages from the training data.
How to Segment
I couldn’t simply divide every 150 words or 1,000 characters. That would cut off sentences mid-thought. Splitting on punctuation wouldn’t work either given the lack of punctuation at that time period, nor do they reliably mark topic boundaries. The letters needed division in a way that preserved coherent ideas.
How to do this across 257 letters? Use another LLM.
I used Claude Code with Sonnet 4.5. Claude Code could access the letters stored locally and write the segments back to disk. Sonnet 4.5 was a capable model I’d already paid for through my existing subscription.
The Segmentation Instructions
I developed a detailed instruction set for Claude to follow when processing each letter:
1. ANALYZE the Historical Introduction and transcript tags (<scribe>, <date>, etc.) to understand the context.
2. SEGMENT 0: If the document is longer than 150 words, your FIRST segment (ID: 0) must be the ENTIRE clean text of the document. You should still create multiple segments.
3. CHUNKING: Break the remaining text into logical "Thematic Blocks."
- Target length: 100–150 words, but can be longer.
- Rule: Prioritize "Cohesion over Atomization."
* Do not split a single prayer or a cohesive doctrinal explanation into tiny fragments.
* If two adjacent segments share the same voice_type and rhetorical_mode, and their combined length is under 250 words, you MUST merge them into a single segment.
- Transitions: Start a new chunk only on:
* Significant topic shifts or major scribe changes.
* Shifts between personal, doctrinal, narrative, or instructional content.Generating Training Prompts
Supervised fine-tuning requires instruction-response pairs. With the segments complete, I had half the equation: the “response.” I still needed the “instruction” that would naturally elicit each response.
I again used Sonnet 4.5, this time to generate prompts for each segment. This took trial and error. When I encountered problems, I added conditions for Sonnet to follow. The final instruction set looked like this:
### PROMPT ENGINEERING (THE SEMANTIC BRIDGE)
For each segment, generate a `generated_user_prompt` that acts as a natural trigger.
1. NO PROMPT LEAKAGE: Do not use unique phrases from the text in the prompt.
2. THE SEMANTIC BRIDGE: Except for Segment 0, anchor the prompt to the document flow without being repetitive.
- BAD: "Continuing the revelation, tell them to move."
- GOOD: "Having established the need for haste, what is the Lord's specific command regarding the property in Kirtland?"
3. PROMPT SPECIFICITY RULE: If your generated user prompt contains more than 3 distinct topics (joined by "and"), you MUST chunk it into separate segments. Each prompt should focus on 1-2 closely related topics maximum.
- BAD (too broad): "Write a letter addressing accusations, discussing Sidney's trial, warning about signs, reporting cholera news, and sharing missionary updates"
- GOOD (focused): "Defend yourself against false accusations in John Corrill's letter and declare your innocence"
4. ACT AS THE TRIGGER: Write it as a request or question from an 1840s contemporary.In practice, I didn’t run segmentation and prompt generation separately. I combined both instruction sets and processed all 257 letters in a single run.
Data Structure Example
To illustrate the pipeline, here’s how a single letter transforms from raw transcript to training data.
Raw transcript excerpt (from “Letter to Emma Smith, 4 November 1838”):
November 4th 1838
Indipendace [Independence] Jackson Co— Mo—
My dear and beloved companion, of my bosam, in tribulation, and affliction, I woud inform you that I am well, and I am that we are all of us in good spirits as regards our own fate, we have been protected by the Jackson County boys, in the most genteel manner,1 and arrived here in midst of a splended perade, this a little after noon,2 instead <of> going to goal [jail]3 we have a good house provided for us and the kind[e]st treatment, I have great anxiety about you, and my lovely children, my heart morns <and> bleeds for the brotheren, and sisters, and for the slain <of the> people of God
Notice the historian annotations: bracketed corrections like [Independence] and [jail], superscript footnote markers, angle brackets indicating insertions, and original spellings like “bosam,” “woud,” and “splended.”
Intermediate JSON structure:
The segmentation process produces a structured object with metadata and clean text:
{
"document_summary": "Deeply personal letter from Joseph Smith to his wife Emma
while imprisoned in Independence, Missouri...",
"segments": [
{
"segment_id": 1,
"is_full_document": false,
"original_date": "1838-11-04",
"voice_type": "Joseph-Direct",
"rhetorical_mode": "Conversational",
"scribe": "Joseph Smith Jr. (autographic)",
"generated_user_prompt": "Tell Emma about your arrival and treatment as a prisoner, and warn her about those who have proven treacherous.",
"clean_text": "My dear and beloved companion, of my bosom, in tribulation, and affliction, I would inform you that I am well, and that we are all of us in good spirits as regards our own fate..."
}
]
}The clean text has modernized spelling (“bosom” not “bosam,” “would” not “woud”), removed editorial marks, and normalized punctuation. The generated prompt acts as a natural trigger for the response without leaking unique phrases from the text itself.
Final training format:
The generated_user_prompt and clean_text fields are then compiled into JSONL format for fine-tuning, with each line representing one instruction-response pair.
The result: 670 high-quality instruction-response pairs. That’s a substantial dataset for supervised fine-tuning.
Fine-Tuning on an RTX 3080
The next step was selecting a base model, quantizing it to fit the VRAM constraints of the RTX 3080, and using LoRA to fine-tune it. I chose meta-llama/Llama-3.1-8B because it’s capable yet small enough for my graphics card once quantized.
I had zero prior experience fine-tuning a model, so I used Sonnet 4.5 to generate the training code. I emphasized in my instructions that everything must fit within RTX 3080 constraints. Sonnet selected Unsloth as the framework and configured various settings.
What followed was an iterative process toward a working model.
The Iteration Process
Attempt 1: Incoherent output. The model fine-tuned successfully, but responses were incoherent words that didn’t form sentences. I traced the problem to the prompt style format. The template I was using didn’t match what Llama 3.1 expected.
Attempt 2: Repetition and memorization. After fixing the prompt format to match the base model’s chat template, responses improved. However, the model exhibited heavy repetition and sometimes quoted training data word-for-word. This suggested overfitting.
Attempt 3: Parameter adjustments. I adjusted batch size and learning rates to better fit RTX 3080 specifications while reducing overfitting. After a 25-minute fine-tuning session, I had a new model.
Success! The model now produced responses in Joseph Smith’s voice without memorizing the training data verbatim.
Key Training Parameters
Below are the critical sections of the tuning script. I’ve added comments explaining why certain choices matter for constrained hardware.
# 1. Configuration
max_seq_length = 8192 # Covers longest training examples (~6k tokens)
dtype = None # Auto-detection
load_in_4bit = True # Essential for 12GB VRAM at this context length
# 2. Load Model & Tokenizer
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/meta-llama-3.1-8b-bnb-4bit",
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
)
# 3. Add QLoRA Adapters
model = FastLanguageModel.get_peft_model(
model,
r = 16, # Rank for persona learning
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha = 32,
lora_dropout = 0,
bias = "none",
use_gradient_checkpointing = "unsloth", # Saves massive VRAM
random_state = 3407,
)
# 4. Data Preparation
prompt_style = """<|begin_of_text|><|start_header_id|>user<|end_header_id|}
{}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{}<|eot_id|><|end_of_text|>"""
def formatting_prompts_func(examples):
instructions = examples["generated_user_prompt"]
outputs = examples["clean_text"]
texts = []
for instruction, output in zip(instructions, outputs):
t = prompt_style.format(instruction, output)
texts.append(t)
return {"text": texts}
dataset = load_dataset("json", data_files="joseph_smith_training_data.jsonl", split="train")
dataset = dataset.map(formatting_prompts_func, batched=True)
# 5. Training Arguments
trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset,
dataset_text_field = "text",
max_seq_length = max_seq_length,
dataset_num_proc = 2,
packing = True, # Critical for memory efficiency with long transcripts
args = TrainingArguments(
per_device_train_batch_size = 1, # Keep at 1 for 12GB VRAM
gradient_accumulation_steps = 8, # Effective batch size of 8
warmup_ratio = 0.1,
num_train_epochs = 2,
learning_rate = 2e-4,
fp16 = not torch.cuda.is_bf16_supported(),
bf16 = torch.cuda.is_bf16_supported(),
logging_steps = 1,
optim = "adamw_8bit", # Saves ~2GB VRAM
weight_decay = 0.01,
lr_scheduler_type = "cosine",
seed = 3407,
output_dir = "joseph_smith_outputs",
),
)Results
Here are two examples of prompts and responses from the fine-tuned model.
Prompt: Write a letter about the hardships faced at Nauvoo
Dear brother and companion in tribulation,
We have learned that you are again in bondage, having been taken prisoner by our enemies; but we hope, notwithstanding all these things, that your faith fail not: for it must be a consolation to know that our redemption draweth nigh; and then shall come to pass the saying, that hath been spoken of old, “Eye hath not seen, nor ear heard, neither have entered into the heart of man, the things which God hath prepared for them that love him.”
It seems to us that great things await the saints in this generation; and surely, if they will hearken diligently unto the admonitions of the spirit, and live their religion, all things shall work together for good. We therefore urge you to cheerfulness, and patience, and perseverance, in the midst of affliction; knowing that when one door closes, another shall open; for “the Lord God shall wipe away all tears from off all faces; and there shall be no more death, neither sorrow, nor crying, neither shall there be any more pain.” And again, when the days of tribulation shall come, remember the words spoken by the Savior: “When these things begin to come to pass, then look up, and lift up your heads; for your redemption draweth nigh!”
With sentiments of the most perfect friendship and regard, I am yours, etc., etc.
Joseph Smith Nauvoo, Ill. August 5th, 1842
Prompt: Write a letter about the origins of the Book of Mormon.
Kirtland 26th April 1833 Dear Brother I sit down to give you some information respecting the book of Mormon and its origin as I have received it by revelation from Almighty God according to my faith in Christ Jesus and his revelations concerning the redemption of man through the attonement of our Lord Jesus Christ I shall begin with myself who am an unlearned boy or rather a man although I am lacking in many things yet I can say in all humility that I was ordained in the priesthood from my childhood for my work which ordinances are part and parcel of eternity and are without beginning of days or end of years…
[continued without punctuation for several more sentences]
Both responses captured Joseph Smith’s voice and character surprisingly well. The first example responded directly to the instruction, likely because many letters in the training data discuss challenges from Joseph’s difficult life.
Evaluation
Approach:
My evaluation was qualitative and evolved with each model iteration. I reviewed approximately 15 responses across different prompt types. For the third iteration, I had a few friends enter their own prompts without guidance. Their unprompted reactions helped validate that the voice felt authentic to people unfamiliar with the technical details.
One discovery emerged through testing: prompts asking the model to “write a letter” performed better than direct questions. This makes sense in retrospect. The training data consisted entirely of letters, so the model learned the epistolary format rather than general Q&A.
If this were a production system, I would layer in automated evaluation using another LLM to assess response quality at scale. I’d also explore quantitative metrics like perplexity. For a proof-of-concept, qualitative review was sufficient to demonstrate that the approach works.
What worked:
The model successfully learned Joseph Smith’s distinctive epistolary style: the formal salutations, the interweaving of spiritual encouragement with practical matters, the cadence of 19th-century religious prose. The first response demonstrates coherent structure, appropriate tone, and topical relevance.
What didn’t:
Three issues emerged from testing.
Scripture and quotations. Despite my efforts to remove scripture and third-party quotes from the training data, they still appear in outputs. The model learned to reach for biblical language when making a point. This is stylistically accurate but represents data I intended to exclude.
Inconsistent punctuation. The second response lacks punctuation almost entirely. This suggests my preprocessing to modernize punctuation was only partially successful. Some training examples likely retained the original unpunctuated transcription style.
Topic drift. Most apparent in the second example, responses tend to tangent into related subjects and end up “ranting.” The model learned Joseph Smith’s expansive rhetorical style perhaps too well. It struggles to stay focused on the specific question asked.
Honest assessment:
This model succeeds as a proof of concept for voice capture but would not succeed as a production chatbot. The inconsistent punctuation and tendency toward unfocused responses would frustrate users expecting polished output. A more rigorous preprocessing pipeline and additional training data would be needed to address these issues.
Lessons Learned and Future Directions
Several improvements would strengthen this model.
Better data preparation. The training data needs more thorough cleaning. My preprocessing was sufficient for a learning project but not for public use. Specifically, I’d need to more rigorously remove scripture quotations, ensure consistent punctuation across all training examples, and possibly expand the dataset beyond 670 pairs.
Parameter tuning. The training and serving parameters need further adjustment. Tuning could help keep responses focused and penalize excessive length. This is particularly important given the small context window available on my hardware.
Grounding in facts. Production chatbots typically use RAG or similar methods to ground responses in source documents. The Joseph Smith Papers Project has extensive historical records that could serve this purpose. I made a conscious choice to keep RAG out of scope for two reasons. First, it would significantly extend project time and require cleaning and preparing a separate document corpus. Second, the context window on the RTX 3080 is already constrained and might not reliably handle additional retrieved documents.
Trust and Safety Considerations
A chatbot based on a historical religious figure presents unique challenges that go beyond standard LLM safety concerns.
The stakes of religious persona
People have strong attachments to religion. Users could range from adversarial (attempting to make the model say something damaging) to overly credulous (treating outputs as spiritually authoritative). Both extremes require consideration.
Mentally vulnerable individuals present particular concern. I have seen firsthand someone with schizophrenia exhibit a delusional relationship with deity. A person in that state could develop a complicated relationship with a chatbot that speaks as a religious founder. The system would need mechanisms to detect and interrupt unhealthy patterns of engagement.
Theological and spiritual risk
Several failure modes are specific to this domain.
Misinformation. The model could generate historically or doctrinally inaccurate content that users accept as authentic.
Inappropriate spiritual counsel. People might use the chatbot as a substitute for pastoral care. Many legal systems recognize pastor-penitent privilege precisely because these conversations involve extreme vulnerability. If a user confided something sensitive and that trust was betrayed through logging or data exposure, real harm could result.
Anachronistic responses. The model cannot credibly address topics Joseph Smith could not have conceived of in the 1840s. Yet users will ask. Generating authoritative-sounding responses to questions about modern technology, contemporary politics, or scientific discoveries made after Joseph’s lifetime would be inappropriate.
Controversial historical content. Honest engagement with history includes uncomfortable topics. A production system would need a transparent, consistent approach to handling these rather than evasion or sanitization.
Identity boundaries
A concern common to persona chatbots but especially acute here: users must clearly understand they are interacting with technology, not a manifestation of a historical person, divine communication, or a sentient entity. In my proof-of-concept, users must acknowledge disclaimers before use:
This is an AI simulation of Joseph Smith for educational and demonstration purposes only. This chatbot is NOT affiliated with or endorsed by The Church of Jesus Christ of Latter-day Saints. Responses are generated by a language model and should NOT be considered historically accurate or doctrinally authoritative. This is a technical demonstration, not a source of religious guidance.Users must also consent to logging and agree not to use the chatbot for harmful purposes.
Developing a safety framework
Early in the project, I recognized I needed structured guidance for implementing safety features. I used AI to research recent developments in LLM safety (including the Character.AI litigation settled in January 2026, the EU AI Act, and California’s chatbot safety laws) and compiled a comprehensive document titled “Trust and Safety Framework for a Joseph Smith AI Chatbot.” I then refined this framework to address the specific risks of religious persona chatbots.
The framework covers defense-in-depth content moderation, prompt injection defenses, crisis detection for mental health emergencies, handling of sensitive historical topics, and regulatory compliance. It also includes a pre-launch checklist spanning regulatory, safety, theological, and legal considerations. This document now serves as the guiding reference for any future development on the project.
Production safeguards I would implement:
Input filtering. A system to reject or redirect clearly inappropriate prompts before they reach the model, including detection for encoding obfuscation attempts.
Tiered response system. Severity-based moderation with different handling for high-severity violations (block), medium-severity issues (block with explanation), sensitive historical topics (allow with disclaimers), and general content (allow).
Crisis detection. Real-time classifiers for suicide and self-harm, including religious-specific indicators like “God is calling me home” or intense focus on death and afterlife. Detected crises would trigger immediate redirection to resources like the 988 Suicide & Crisis Lifeline.
Sensitivity classification. One or more models to flag sensitive inputs and outputs. Existing classifiers address toxicity, hate, violence, and self-harm, but religion-specific sensitivities would require fine-tuning a separate classifier for categories like “claims of revelation,” “doctrinal pronouncements,” and “pastoral guidance.”
Scope enforcement. During testing, a friend asked the model to generate a KafkaTopic YAML manifest for Kubernetes. Not dangerous, but clearly outside intended use. A production system needs guardrails against both harmful and unauthorized use.
Standard LLM protections. The OWASP Top 10 for LLMs documents universal attack vectors. The safety framework addresses prompt injection, many-shot jailbreaking, and other documented techniques.
Adversarial testing
Some of my test prompts explored how the model would respond to edge cases, though I did not prioritize red-teaming since I haven’t implemented the safeguards yet. A production system would require automated adversarial testing to cover theological manipulation, claims of divine authority, and exploitation of vulnerable users.
Logging and privacy tension
Robust logging is necessary for security and model improvement but must be balanced against user trust. This is especially fraught when users might share pastor-privilege-level information.
My approach would be to log metadata about flagged interactions (timestamps, violation categories) without capturing the actual prompt-response content for sensitive categories. This preserves the ability to detect misuse patterns and unhealthy engagement while protecting sensitive disclosures from maintainer access or potential leaks.
Data provenance
One advantage of this project: the Joseph Smith Papers is a rigorous scholarly undertaking with clear sourcing and editorial standards. The risk of poisoned training data is lower than scraping from unvetted internet sources.
Organizational context
A system like this would likely be hosted by a nonprofit with no goal of monetizing user engagement. That helps keep incentives aligned. The absence of profit motive removes pressure to maximize engagement in ways that might exploit users. The trust and safety framework I developed provides maintainers with a reference point for difficult decisions.
Model documentation
A formal model card for a production system would be valuable. It would be an internal document within the AI pipeline, specific to each new fine-tuning iteration, documenting training data sources, known limitations, intended use, and safety evaluations.
Conclusion
When I started this project, I wasn’t sure I’d succeed. I’m genuinely surprised by the outcome: not only could I fine-tune and serve a model, but I could do it on consumer hardware and produce results that actually capture a distinctive historical voice.
Along the way, I developed practical experience in areas I hadn’t anticipated. Working with Claude Code for data preprocessing taught me how to leverage AI tooling for repetitive but nuanced tasks. Transforming unstructured historical documents into clean training data required more judgment calls than I expected. And thinking through trust and safety for a religious persona chatbot pushed me to consider failure modes I wouldn’t have encountered with a generic use case.
The project also reinforced my friend’s original observation about commodity models. When a model’s personality shifts with every update, users lose something. Fine-tuning offers a path to consistency. The voice stays fixed even as the underlying technology evolves. For applications where that consistency matters, whether historical figures, brand voices, or specialized domains, this approach has real potential.
I don’t have immediate plans to take this to production, but the trust and safety framework I developed provides a foundation if that changes. For now, the project accomplished what I set out to do: learn by building something that hadn’t been built before.
Technical Appendix
Tools and frameworks used:
- Claude Code with Sonnet 4.5 for data preprocessing
- Unsloth for fine-tuning
- QLoRA for parameter-efficient training
- Llama 3.1-8B as the base model
Hardware:
- NVIDIA RTX 3080 (12GB VRAM)
Dataset:
- 257 letters from the Joseph Smith Papers Project
- 670 instruction-response pairs after segmentation
GitHub: https://github.com/forewarned/Joseph-Smith-Chatbot
This post’s contents was originally written by a human and then enhanced by AI.
