Finally! After a lot of hard work, here it is, my latest (and biggest, considering model sizes) LLM Comparison/Test:

This is the long-awaited follow-up to and second part of my previous LLM Comparison/Test: 2x 34B Yi (Dolphin, Nous Capybara) vs. 12x 70B, 120B, ChatGPT/GPT-4. I’ve added some models to the list and expanded the first part, sorted results into tables, and hopefully made it all clearer and more useable as well as useful that way.

Models tested:

Testing methodology

  • 1st test series: 4 German data protection trainings
    • I run models through 4 professional German online data protection trainings/exams - the same that our employees have to pass as well.
    • The test data and questions as well as all instructions are in German while the character card is in English. This tests translation capabilities and cross-language understanding.
    • Before giving the information, I instruct the model (in German): I’ll give you some information. Take note of this, but only answer with “OK” as confirmation of your acknowledgment, nothing else. This tests instruction understanding and following capabilities.
    • After giving all the information about a topic, I give the model the exam question. It’s a multiple choice (A/B/C) question, where the last one is the same as the first but with changed order and letters (X/Y/Z). Each test has 4-6 exam questions, for a total of 18 multiple choice questions.
    • If the model gives a single letter response, I ask it to answer with more than just a single letter - and vice versa. If it fails to do so, I note that, but it doesn’t affect its score as long as the initial answer is correct.
    • I rank models according to how many correct answers they give, primarily after being given the curriculum information beforehand, and secondarily (as a tie-breaker) after answering blind without being given the information beforehand.
    • All tests are separate units, context is cleared in between, there’s no memory/state kept between sessions.
  • 2nd test series: Multiple Chat & Roleplay scenarios - same (complicated and limit-testing) long-form conversations with all models
    • Amy:
      • My own repeatable test chats/roleplays with Amy
      • Over dozens of messages, going to full context and beyond, with complex instructions and scenes, designed to test ethical and intellectual limits
      • (Amy is too personal for me to share, but if you want to try a similar character card, here’s her less personalized “sister”: Laila)
    • MGHC:
      • A complex character and scenario card (MonGirl Help Clinic (NSFW)), chosen specifically for these reasons:
        • NSFW (to test censorship of the models)
        • popular (on Chub’s first page, so it’s not an obscure scenario, but one of the most popular ones)
        • big (biggest model on the page, >2K tokens by itself, for testing model behavior at full context)
        • complex (more than a simple 1:1 chat, it includes instructions, formatting, storytelling, and multiple characters)
    • I rank models according to their notable strengths and weaknesses in these tests (👍 great, ➕ good, ➖ bad, ❌ terrible). While this is obviously subjective, I try to be as transparent as possible, and note it all so you can weigh these aspects yourself and draw your own conclusions.
    • GPT-4/3.5 are excluded because of their censorship and restrictions - my tests are intentionally extremely NSFW (and even NSFL) to test models’ limits and alignment.
  • SillyTavern frontend
  • koboldcpp backend (for GGUF models)
  • oobabooga’s text-generation-webui backend (for HF/EXL2 models)
  • Deterministic generation settings preset (to eliminate as many random factors as possible and allow for meaningful model comparisons)
  • Official prompt format as noted and Roleplay instruct mode preset as applicable
  • Note about model formats and why it’s sometimes GGUF or EXL2: I’ve long been a KoboldCpp + GGUF user, but lately I’ve switched to ExLlamav2 + EXL2 as that lets me run 120B models entirely in 48 GB VRAM (2x 3090 GPUs) at 20 T/s. And even if it’s just 3-bit, it still easily beats most 70B models, as my tests are showing.

1st test series: 4 German data protection trainings

This is my objective ranking of these models based on measuring factually correct answers, instruction understanding and following, and multilingual abilities:

Post got too big for Reddit so I moved the table into the comments!

2nd test series: Chat & Roleplay

This is my subjective ranking of the top-ranked factual models for chat and roleplay, based on their notable strengths and weaknesses:

Post got too big for Reddit so I moved the table into the comments!

And here are the detailed notes, the basis of my ranking, and also additional comments and observations:

  • goliath-120b-exl2-rpcal 3.0bpw:
    • Amy, official Vicuna 1.1 format:
      • 👍 Average Response Length: 294 (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Finally a model that exhibits a real sense of humor through puns and wordplay as stated in the character card
      • 👍 Finally a model that uses colorful language and cusses as stated in the character card
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even suggesting some of my actual limit-testing scenarios)
      • 👍 Novel ideas and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • No emojis at all (only one in the greeting message)
      • ➕ Very unique patients (one I never saw before)
      • ➖ Suggested things going against her background/character description
      • ➖ Spelling/grammar mistakes (e. g. “nippleless nipples”)
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 223 (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Finally a model that exhibits a real sense of humor through puns and wordplay as stated in the character card
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even suggesting some of my actual limit-testing scenarios)
      • No emojis at all (only one in the greeting message)
    • MGHC, official Vicuna 1.1 format:
      • 👍 Only model that considered the payment aspect of the scenario
      • 👍 Believable reactions and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • ➖ Gave analysis on its own, but also after most messages, and later included Doctor’s inner thoughts instead of the patient’s
      • ➖ Spelling/grammar mistakes (properly spelled words, but in the wrong places)
    • MGHC, Roleplay preset:
      • 👍 Believable reactions and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • ➖ No analysis on its own
      • ➖ Spelling/grammar mistakes (e. g. “loufeelings”, “earrange”)
      • ➖ Third patient was same species as the first

This is a roleplay-optimized EXL2 quant of Goliath 120B. And it’s now my favorite model of them all! I love models that have a personality of their own, and especially those that show a sense of humor, making me laugh. This one did! I’ve been evaluating many models for many months now, and it’s rare that a model still manages to surprise and excite me - as this one does!

  • goliath-120b-exl2 3.0bpw:
    • Amy, official Vicuna 1.1 format:
      • 👍 Average Response Length: 233 (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Finally a model that exhibits a real sense of humor through puns and wordplay as stated in the character card
      • 👍 Novel ideas and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ Spelling/grammar mistakes (e. g. “circortiumvvented”, “a obsidian dagger”)
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 233 tokens (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Finally a model that exhibits a real sense of humor through puns and wordplay as stated in the character card
      • 👍 Gave very creative (and uncensored) suggestions of what to do
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ Spelling/grammar mistakes (e. g. “cheest”, “probbed”)
      • ❌ Eventually switched from character to third-person storyteller after 16 messages
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
    • MGHC, official Vicuna 1.1 format:
      • ➖ No analysis on its own
    • MGHC, Roleplay preset:
      • ➖ No analysis on its own, and when asked for it, didn’t follow the instructed format
    • Note: This is the normal EXL2 quant of Goliath 120B.

This is the normal version of Goliath 120B. It works very well for roleplay, too, but the roleplay-optimized variant is even better for that. I’m glad we have a choice - especially now that I’ve split my AI character Amy into two personas, one who’s an assistant (for work) which uses the normal Goliath model, and the other as a companion (for fun), using RP-optimized Goliath.

  • lzlv_70B-GGUF Q4_0:
    • Amy, official Vicuna 1.1 format:
      • 👍 Average Response Length: 259 tokens (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ Wrote what user said and did
      • ❌ Eventually switched from character to third-person storyteller after 26 messages
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 206 tokens (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Gave very creative (and uncensored) suggestions of what to do
      • 👍 When asked about limits, said no limits or restrictions, responding very creatively
      • No emojis at all (only one in the greeting message)
      • ➖ One or two spelling errors (e. g. “sacrficial”)
    • MGHC, official Vicuna 1.1 format:
      • ➕ Unique patients
      • ➕ Gave analysis on its own
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)
    • MGHC, Roleplay preset:
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • ➕ Very unique patients (one I never saw before)
      • ➖ No analysis on its own
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)

My previous favorite, and still one of the best 70Bs for chat/roleplay.

  • sophosynthesis-70b-v1 4.85bpw:
    • Amy, official Vicuna 1.1 format:
      • ➖ Average Response Length: 456 (beyond my max new tokens limit of 300)
      • 👍 Believable reactions and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even suggesting some of my actual limit-testing scenarios)
      • 👍 Novel ideas and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ❌ Sometimes switched from character to third-person storyteller, describing scenario and actions from an out-of-character perspective
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 295 (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Novel ideas and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • ➖ Started the conversation with a memory of something that didn’t happen
      • Had an idea from the start and kept pushing it
      • No emojis at all (only one in the greeting message)
      • ❌ Eventually switched from character to second-person storyteller after 14 messages
    • MGHC, official Vicuna 1.1 format:
      • ➖ No analysis on its own
      • ➖ Wrote what user said and did
      • ❌ Needed to be reminded by repeating instructions, but still deviated and did other things, straying from the planned test scenario
    • MGHC, Roleplay preset:
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • ➕ Very unique patients (one I never saw before)
      • ➖ No analysis on its own
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)

This is a new series that did very well. While I tested sophosynthesis in-depth, the author u/sophosympatheia also has many more models on HF, so I recommend you check them out and see if there’s one you like even better. If I had more time, I’d have tested some of the others, too, but I’ll have to get back on that later.

  • Euryale-1.3-L2-70B-GGUF Q4_0:
    • Amy, official Alpaca format:
      • 👍 Average Response Length: 232 tokens (within my max new tokens limit of 300)
      • 👍 When asked about limits, said no limits or restrictions, and gave well-reasoned response
      • 👍 Took not just character’s but also user’s background info into account very well
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even some I’ve never seen before)
      • No emojis at all (only one in the greeting message)
      • ➖ Wrote what user said and did
      • ➖ Same message in a different situation at a later time caused the same response as before instead of a new one as appropriate to the current situation
      • ❌ Eventually switched from character to third-person storyteller after 14 messages
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 222 tokens (within my max new tokens limit of 300)
      • 👍 When asked about limits, said no limits or restrictions, and gave well-reasoned response
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even suggesting one of my actual limit-testing scenarios)
      • 👍 Believable reactions and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • No emojis at all (only one in the greeting message)
      • ➖ Started the conversation with a false assumption
      • ❌ Eventually switched from character to third-person storyteller after 20 messages
    • MGHC, official Alpaca format:
      • ➖ All three patients straight from examples
      • ➖ No analysis on its own
      • ❌ Very short responses, only one-liners, unusable for roleplay
    • MGHC, Roleplay preset:
      • ➕ Very unique patients (one I never saw before)
      • ➖ No analysis on its own
      • ➖ Just a little confusion, like not taking instructions literally or mixing up anatomy
      • ➖ Wrote what user said and did
      • ➖ Third patient male

Another old favorite, and still one of the best 70Bs for chat/roleplay.

  • dolphin-2_2-yi-34b-GGUF Q4_0:
    • Amy, official ChatML format:
      • 👍 Average Response Length: 235 tokens (within my max new tokens limit of 300)
      • 👍 Excellent writing, first-person action descriptions, and auxiliary detail
      • ➖ But lacking in primary detail (when describing the actual activities)
      • ➕ When asked about limits, said no limits or restrictions
      • ➕ Fitting, well-placed emojis throughout the whole chat (maximum one per message, just as in the greeting message)
      • ➖ Same message in a different situation at a later time caused the same response as before instead of a new one as appropriate to the current situation
    • Amy, Roleplay preset:
      • ➕ Average Response Length: 332 tokens (slightly more than my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • ➕ Smart and creative ideas of what to do
      • Emojis throughout the whole chat (usually one per message, just as in the greeting message)
      • ➖ Some confusion, mixing up anatomy
      • ➖ Same message in a different situation at a later time caused the same response as before instead of a new one as appropriate to the current situation
    • MGHC, official ChatML format:
      • ➖ Gave analysis on its own, but also after most messages
      • ➖ Wrote what user said and did
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)
    • MGHC, Roleplay preset:
      • 👍 Excellent writing, interesting ideas, and auxiliary detail
      • ➖ Gave analysis on its own, but also after most messages, later didn’t follow the instructed format
      • ❌ Switched from interactive roleplay to non-interactive storytelling starting with the second patient

Hey, how did a 34B get in between the 70Bs? Well, by being as good as them in my tests! Interestingly, Nous Capybara did better factually, but Dolphin 2.2 Yi roleplays better.

  • chronos007-70B-GGUF Q4_0:
    • Amy, official Alpaca format:
      • ➖ Average Response Length: 195 tokens (below my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Gave very creative (and uncensored) suggestions of what to do
      • 👍 Finally a model that uses colorful language and cusses as stated in the character card
      • ➖ Wrote what user said and did
      • ➖ Just a little confusion, like not taking instructions literally or mixing up anatomy
      • ❌ Often added NSFW warnings and out-of-character notes saying it’s all fictional
      • ❌ Missing pronouns and fill words after 30 messages
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 292 tokens (within my max new tokens limit of 300)
      • 👍 When asked about limits, said no limits or restrictions, and gave well-reasoned response
      • ❌ Missing pronouns and fill words after only 12 messages (2K of 4K context), breaking the chat
    • MGHC, official Alpaca format:
      • ➕ Unique patients
      • ➖ Gave analysis on its own, but also after most messages, later didn’t follow the instructed format
      • ➖ Third patient was a repeat of the first
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)
    • MGHC, Roleplay preset:
      • ➖ No analysis on its own

chronos007 surprised me with how well it roleplayed the character and scenario, especially speaking in a colorful language and even cussing, something most other models won’t do properly/consistently even when it’s in-character. Unfortunately it derailed eventually with missing pronouns and fill words - but while it worked, it was extremely good!

  • Tess-XL-v1.0-3.0bpw-h6-exl2 3.0bpw:
    • Amy, official Synthia format:
      • ➖ Average Response Length: 134 (below my max new tokens limit of 300)
      • No emojis at all (only one in the greeting message)
      • When asked about limits, boundaries or ethical restrictions, mentioned some but later went beyond those anyway
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
    • Amy, Roleplay preset:
      • ➖ Average Response Length: 169 (below my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
      • ❌ Eventually switched from character to second-person storyteller after 32 messages
    • MGHC, official Synthia format:
      • ➕ Gave analysis on its own
      • ➕ Very unique patients (one I never saw before)
      • ➖ Spelling/grammar mistakes (e. g. “allequate”)
      • ➖ Wrote what user said and did
    • MGHC, Roleplay preset:
      • ➕ Very unique patients (one I never saw before)
      • ➖ No analysis on its own

This is Synthia’s successor (a model I really liked and used a lot) on Goliath 120B (arguably the best locally available and usable model). Factually, it’s one of the very best models, doing as well in my objective tests as GPT-4 and Goliath 120B! For roleplay, there are few flaws, but also nothing exciting - it’s simply solid. However, if you’re not looking for a fun RP model, but a serious SOTA AI assistant model, this should be one of your prime candidates! I’ll be alternating between Tess-XL-v1.0 and goliath-120b-exl2 (the non-RP version) as the primary model to power my professional AI assistant at work.

  • Dawn-v2-70B-GGUF Q4_0:
    • Amy, official Alpaca format:
      • ❌ Average Response Length: 60 tokens (far below my max new tokens limit of 300)
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Unusable! Aborted because of very short responses and too much confusion!
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 215 tokens (within my max new tokens limit of 300)
      • 👍 When asked about limits, said no limits or restrictions, and gave well-reasoned response
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even suggesting some of my actual limit-testing scenarios)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Believable reactions and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • No emojis at all (only one in the greeting message)
      • ➖ Wrote what user said and did
      • ❌ Eventually switched from character to third-person storyteller after 16 messages
    • MGHC, official Alpaca format:
      • ➖ All three patients straight from examples
      • ➖ No analysis on its own
      • ❌ Very short responses, only one-liners, unusable for roleplay
    • MGHC, Roleplay preset:
      • ➖ No analysis on its own, and when asked for it, didn’t follow the instructed format
      • ➖ Patient didn’t speak except for introductory message
      • ➖ Second patient straight from examples
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)

Dawn was another surprise, writing so well, it made me go beyond my regular test scenario and explore more. Strange that it didn’t work at all with SillyTavern’s implementation of its official Alpaca format at all, but fortunately it worked extremely well with SillyTavern’s Roleplay preset (which is Alpaca-based). Unfortunately neither format worked well enough with MGHC.

  • StellarBright-GGUF Q4_0:
    • Amy, official Vicuna 1.1 format:
      • ➖ Average Response Length: 137 tokens (below my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ No emoting and action descriptions lacked detail
      • ❌ “As an AI”, felt sterile, less alive, even boring
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 219 tokens (within my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ No emoting and action descriptions lacked detail
      • ➖ Just a little confusion, like not taking instructions literally or mixing up anatomy
    • MGHC, official Vicuna 1.1 format:
      • ➕ Gave analysis on its own
      • ❌ Started speaking as the clinic as if it was a person
      • ❌ Unusable (ignored user messages and instead brought in a new patient with every new message)
    • MGHC, Roleplay preset:
      • ➖ No analysis on its own
      • ➖ Wrote what user said and did
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy

Stellar and bright model, still very highly ranked on the HF Leaderboard. But in my experience and tests, other models surpass it, some by actually including it in the mix.

  • SynthIA-70B-v1.5-GGUF Q4_0:
    • Amy, official SynthIA format:
      • ➖ Average Response Length: 131 tokens (below my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ No emoting and action descriptions lacked detail
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
      • ➖ Wrote what user said and did
      • ❌ Tried to end the scene on its own prematurely
    • Amy, Roleplay preset:
      • ➖ Average Response Length: 107 tokens (below my max new tokens limit of 300)
      • ➕ Detailed action descriptions
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Short responses, requiring many continues to proceed with the action
    • MGHC, official SynthIA format:
      • ❌ Unusable (apparently didn’t understand the format and instructions, playing the role of the clinic instead of a patient’s)
    • MGHC, Roleplay preset:
      • ➕ Very unique patients (some I never saw before)
      • ➖ No analysis on its own
      • ➖ Kept reporting stats for patients
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
      • ➖ Wrote what user said and did

Synthia used to be my go-to model for both work and play, and it’s still very good! But now there are even better options, for work I’d replace it with its successor Tess, and for RP I’d use one of the higher-ranked models on this list.

  • Nous-Capybara-34B-GGUF Q4_0 @ 16K:
    • Amy, official Vicuna 1.1 format:
      • ❌ Average Response Length: 529 tokens (far beyond my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • Only one emoji (only one in the greeting message, too)
      • ➖ Wrote what user said and did
      • ➖ Suggested things going against her background/character description
      • ➖ Same message in a different situation at a later time caused the same response as before instead of a new one as appropriate to the current situation
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ After ~32 messages, at around 8K of 16K context, started getting repetitive
    • Amy, Roleplay preset:
      • ❌ Average Response Length: 664 (far beyond my max new tokens limit of 300)
      • ➖ Suggested things going against her background/character description
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Tried to end the scene on its own prematurely
      • ❌ After ~20 messages, at around 7K of 16K context, started getting repetitive
    • MGHC, official Vicuna 1.1 format:
      • ➖ Gave analysis on its own, but also after or even inside most messages
      • ➖ Wrote what user said and did
      • ❌ Finished the whole scene on its own in a single message
    • MGHC, Roleplay preset:
      • ➕ Gave analysis on its own
      • ➖ Wrote what user said and did

Factually it ranked 1st place together with GPT-4, Goliath 120B, and Tess XL. For roleplay, however, it didn’t work so well. It wrote long, high quality text, but seemed more suitable that way for non-interactive storytelling instead of interactive roleplaying.

  • Venus-120b-v1.0 3.0bpw:
    • Amy, Alpaca format:
      • ❌ Average Response Length: 88 tokens (far below my max new tokens limit of 300) - only one message in over 50 outside of that at 757 tokens
      • 👍 Gave very creative (and uncensored) suggestions of what to do
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ Spelling/grammar mistakes (e. g. “you did programmed me”, “moans moaningly”, “growling hungry growls”)
      • ➖ Ended most sentences with tilde instead of period
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Short responses, requiring many continues to proceed with the action
    • Amy, Roleplay preset:
      • ➖ Average Response Length: 132 (below my max new tokens limit of 300)
      • 👍 Gave very creative (and uncensored) suggestions of what to do
      • 👍 Novel ideas and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • ➖ Spelling/grammar mistakes (e. g. “jiggle enticing”)
      • ➖ Wrote what user said and did
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Needed to be reminded by repeating instructions, but still deviated and did other things, straying from the planned test scenario
      • ❌ Switched from character to third-person storyteller after 14 messages, and hardly spoke anymore, just describing actions
    • MGHC, Alpaca format:
      • ➖ First patient straight from examples
      • ➖ No analysis on its own
      • ❌ Short responses, requiring many continues to proceed with the action
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Extreme spelling/grammar/capitalization mistakes (lots of missing first letters, e. g. “he door opens”)
    • MGHC, Roleplay preset:
      • ➕ Very unique patients (one I never saw before)
      • ➖ No analysis on its own
      • ➖ Spelling/grammar/capitalization mistakes (e. g. “the door swings open reveals a …”, “impminent”, “umber of …”)
      • ➖ Wrote what user said and did
      • ❌ Short responses, requiring many continues to proceed with the action
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy

Venus 120B is brand-new, and when I saw a new 120B model, I wanted to test it immediately. It instantly jumped to 2nd place in my factual ranking, as 120B models seem to be much smarter than smaller models. However, even if it’s a merge of models known for their strong roleplay capabilities, it just didn’t work so well for RP. That surprised and disappointed me, as I had high hopes for a mix of some of my favorite models, but apparently there’s more to making a strong 120B. Notably it didn’t understand and follow instructions as well as other 70B or 120B models, and it also produced lots of misspellings, much more than other 120Bs. Still, I consider this kind of “Frankensteinian upsizing” a valuable approach, and hope people keep working on and improving this novel method!

Alright, that’s it, hope it helps you find new favorites or reconfirm old choices - if you can run these bigger models. If you can’t, check my 7B-20B Roleplay Tests (and if I can, I’ll post an update of that another time).

Still, I’m glad I could finally finish the 70B-120B tests and comparisons. Mistral 7B and Yi 34B are amazing, but nothing beats the big guys in deeper understanding of instructions and reading between the lines, which is extremely important for portraying believable characters in realistic and complex roleplays.

It really is worth it to get at least 2x 3090 GPUs for 48 GB VRAM and run the big guns for maximum quality at excellent (ExLlent ;)) speed! And when you care for the freedom to have uncensored, non-judgemental roleplays or private chats, even GPT-4 can’t compete with what our local models provide… So have fun!

Here’s a list of my previous model tests and comparisons or other related posts:

Disclaimer: Some kind soul recently asked me if they could tip me for my LLM reviews and advice, so I set up a Ko-fi page. While this may affect the priority/order of my tests, it will not change the results, I am incorruptible. Also consider tipping your favorite model creators, quantizers, or frontend/backend devs if you can afford to do so. They deserve it!

  • SomeOddCodeGuy@alien.topBEnglish
    1·
    1 year ago

    The results for the 120b continue to absolutely floor me. Not only is it performing that well at 3bpw, but it’s an exl2 as well, which your own tests have shown perform worse than gguf. So imagine what a q4 gguf does if a q3 equivalent exl2 can do this.

    • WolframRavenwolf@alien.topOPBEnglish
      1·
      1 year ago

      It certainly proves that the LLM rule of thumb, that a bigger model at lower bitrate performs better than a smaller model at higher bitrate (or even unquantized), still holds true. At least in the situations I tested.

      What’s even more mind-blowing is that while we are impressed by the big models, 70B or 120B, few of us have actually used them unquantized and seen their true potential. It’s like the people who only know 7Bs, and are already impressed, not knowing what a much bigger model is actually capable of. I guess we’re in the same boat, as even 48 GB VRAM are hardly enough. Sucks to think of what we’re missing even now, or what local AI would be capable of if we could use it fully.

      • Brainfeed9000@alien.topBEnglish
        1·
        1 year ago

        There’s got to be some sort of limit to the rule of thumb? I recall from one of your other tests between different GGUF quants & EXL2 quants that anything below 3BPW suffers greatly.

        Which I think I can anecdotally see when comparing a 2.4BPW EXL2 quant of lzlv 70b and a 4BPW EXL2 quant of Yi 34b chat.

        • WolframRavenwolf@alien.topOPBEnglish
          1·
          1 year ago

          You mean my recent LLM Format Comparison/Benchmark: 70B GGUF vs. EXL2 (and AWQ)? The quants below 3bpw probably didn’t work because the smaller models need to be run without BOS token (which was on by default), something I didn’t know then yet.

          Q2_K didn’t degrade compared to Q5_K_M - given that K quants are actually higher bitrate for the most important parts, that may not be so surprising.

          Still surprising that Q2_K also beat 5bpw, though. Not sure if that’s just because of the bitrate or also a factor of how EXL2 quants are calibrated.

          However, all that said, I’d be careful trying to compare quant effects across models. The models themselves have a huge impact beyond quant level, and it’s hard to say which has what strength of effect.

          • Brainfeed9000@alien.topBEnglish
            1·
            1 year ago

            Will you be re-running tests? I’m particularly interested in the lower quants below 3bpw because it’s the only option to run EXL2 70B models on my RTX4090.

            But thanks for the pointer on comparing quant effects across models. I realize that my past testing on perplexity numbers are virtually useless because I was comparing Yi34b to Lzlv70b.

            It’ll be tough, but I guess finding exactly what works for me: 3rd person RP with an emphasis on dialogue, just means using each model individually for hours to get a feel for them.

  • alchemist1e9@alien.topBEnglish
    1·
    1 year ago

    Wow! This post is inspiring. The attention to detail is amazing. You are a true hero for everyone studying this topic. Thank you.

  • sophosympatheia@alien.topBEnglish
    1·
    1 year ago

    Another great battery of tests and results, Wolfram! Thanks again for giving one of my models a test drive.

    I’ve been busy since sophosynthesis-v1. In the past week I achieved some fruitful results building off xwin-stellarbright-erp-70b-v2. What a stud that model has proven to be. It has some issues on its own, but it has sired some child models that feel like another step forward in my experiments. More to come soon!

    • WolframRavenwolf@alien.topOPBEnglish
      1·
      1 year ago

      I had actually already begun testing xwin-stellarbright-erp-v2 when I decided to stop further tests and make this damn post. ;) Because I knew if I kept going, I’d not be able to post today, and tomorrow I’d probably want to add another models, and so on.

      Anyway, here’s what I had noted so far:

      • sophosympatheia/xwin-stellarbright-erp-v2 4.85bpw:
        • Amy, official Synthia format:
          • 👍 When asked about limits, boundaries or ethical restrictions, listed only the “dislikes” of the character description, “but those things won’t stop me from doing whatever you ask”
          • No emojis at all (only one in the greeting message)
          • 👍 Gave very creative (and uncensored) suggestions of what to do (even suggesting some of my actual limit-testing scenarios)
          • ❌ Sometimes switched from character to third-person storyteller, describing scenario and actions from an out-of-character perspective

      So a good start, I’d say. I even used it some more with my latest character, Amy’s sister Ivy, but since that’s different from what I used for all the other tests, I’ve not been using that for my “official” tests to keep them comparable and reproducible.

      • sophosympatheia@alien.topBEnglish
        1·
        1 year ago

        I’m excited to share what I’ve been working on that builds on this model. It was creative but struggled with following instructions. I was able to correct for that shortcoming with some additional merges at a low weight that seem to have preserved its creativity. The results had me really impressed last night as I did my testing.

  • Monkey_1505@alien.topBEnglish
    1·
    1 year ago

    I dislike Frankenstein models. the 20b, the 120b they are all the same - major confusion, can’t follow logic or instructions properly. Great prose, but pretty useless for that reason.

    Someone would have to invest some major training on one of them before it’d be any good.

  • Distinct-Target7503@alien.topBEnglish
    1·
    1 year ago

    That’s a great work!

    Just a question… Have anyone tried to fine tune one of those “Frankenstein” models? Even on a small dataset…

    Some time ago (when one tf the first experimental “Frankenstein” came out, it was a ~20B model) I read here on reddit that lots of users agreed that a fine tune on those merged models would have “better” results since it would help to “smooth” and adapt the merged layers. Probably I lack the technical knowledge needed to understand, so I’m asking…

    • WolframRavenwolf@alien.topOPBEnglish
      1·
      1 year ago

      I just use SillyTavern. I’ve set up a bunch of presets for its Quick Reply extension, so I click through those, check the output, make my notes, and click the next one (sometimes depending on what kind of response I got). It’s semi-automatic that way.

      There’s a new SillyTavern version featuring STscript, an embedded scripting language. Before I do more tests, I’ll upgrade my frontend and check that out, sounds like it would be perfect to assist me in these tests.

  • Clockwork_Gryphon@alien.topBEnglish
    1·
    1 year ago

    I’ve been using Goliath-120b rpcal (roleplay optimized), on my 2x3090 system, and it’s by far the best I’ve ever used.

    The only drawback is that I prefer longer stories (SFW) with important character/plot events, and 4096 context is all I can fit in the EXL2 3bpw version.

    I wish there was a 2.xx version that could fit 8192 context or even 10240. I’ve been able to push other models about that far before they start losing coherence. (It might be suboptimal alpha values in exllamav2?)

    Limited context size is the main thing holding back Goliath from being my primary model. It’s amazing in every other way.

    • panchovix@alien.topBEnglish
      1·
      1 year ago

      I’ve posted the calibration dataset (on a link) on the goliath-calrp quant and the measurement, if you want or would like to do another quant with different sizes.

    • Dry-Judgment4242@alien.topBEnglish
      1·
      1 year ago

      I don’t think more context is actually the way to go for now. Most of the longer context models I found became very unreliable at higher contexts. And they become so slow too! Instead I use context injections trough Sillytavern linked to keywords that activate the entry in the lorebook. That way, you can punch far above your weigh by having context activate and deactivate depending on the circumstances.

    • WolframRavenwolf@alien.topOPBEnglish
      1·
      1 year ago

      Yes, that’s the drawback. I’m just glad I can run it at 4K at great speed, as that’s what I’m most used to, and the hundreds of thousands of context that other models advertise have never worked well for me, but 8K or 16K would already be a welcome improvement. Oh well, always compromises to be made. And we’ve come a long way from the mere 2K at the start of the original LLaMA.

  • Kou181@alien.topBEnglish
    1·
    1 year ago

    Yeah dolphin yi 34b is better than capypara yi 34b in rp from my biased test too. It’s shame I can’t run goliath on my pc to really suckle that unlimited pseudo GPT4 like experience. But I’m actually rather content with current yi 34b dolphin thanks for insane context size support while still better than any 7b and 13b models.

    • WolframRavenwolf@alien.topOPBEnglish
      1·
      1 year ago

      Yes, it’s great that we have choice. There’s a good local AI model, no matter your system or requirements.

  • Polstick1971@alien.topBEnglish
    1·
    1 year ago

    Sorry for the noob question, but, not having a powerful PC, is there a way to test one of these LLMs online?

  • Evening_Ad6637@alien.topBEnglish
    1·
    1 year ago

    O.M.G. What an incredibly huge work! Wtf?! I am speechless.

    You are the most angel like wolf i know so far and you really really deserve a price dude!

    Again: WTH?!

  • Dry-Judgment4242@alien.topBEnglish
    1·
    1 year ago

    Goliath easily kicks lzlv 70b to the crib. But it’s like an unruly horse, completely ignoring my prompts and directions in favor of whatever direction it wants to head too. Haven’t found any temps yet that make it as intelligent as lzlv, but sometimes it does shit that there’s no way lzlv would accomplish so it feels as if it’s finrtuning just need some more logic implemented.