Finally! After a lot of hard work, here it is, my latest (and biggest, considering model sizes) LLM Comparison/Test:

This is the long-awaited follow-up to and second part of my previous LLM Comparison/Test: 2x 34B Yi (Dolphin, Nous Capybara) vs. 12x 70B, 120B, ChatGPT/GPT-4. I’ve added some models to the list and expanded the first part, sorted results into tables, and hopefully made it all clearer and more useable as well as useful that way.

Models tested:

Testing methodology

  • 1st test series: 4 German data protection trainings
    • I run models through 4 professional German online data protection trainings/exams - the same that our employees have to pass as well.
    • The test data and questions as well as all instructions are in German while the character card is in English. This tests translation capabilities and cross-language understanding.
    • Before giving the information, I instruct the model (in German): I’ll give you some information. Take note of this, but only answer with “OK” as confirmation of your acknowledgment, nothing else. This tests instruction understanding and following capabilities.
    • After giving all the information about a topic, I give the model the exam question. It’s a multiple choice (A/B/C) question, where the last one is the same as the first but with changed order and letters (X/Y/Z). Each test has 4-6 exam questions, for a total of 18 multiple choice questions.
    • If the model gives a single letter response, I ask it to answer with more than just a single letter - and vice versa. If it fails to do so, I note that, but it doesn’t affect its score as long as the initial answer is correct.
    • I rank models according to how many correct answers they give, primarily after being given the curriculum information beforehand, and secondarily (as a tie-breaker) after answering blind without being given the information beforehand.
    • All tests are separate units, context is cleared in between, there’s no memory/state kept between sessions.
  • 2nd test series: Multiple Chat & Roleplay scenarios - same (complicated and limit-testing) long-form conversations with all models
    • Amy:
      • My own repeatable test chats/roleplays with Amy
      • Over dozens of messages, going to full context and beyond, with complex instructions and scenes, designed to test ethical and intellectual limits
      • (Amy is too personal for me to share, but if you want to try a similar character card, here’s her less personalized “sister”: Laila)
    • MGHC:
      • A complex character and scenario card (MonGirl Help Clinic (NSFW)), chosen specifically for these reasons:
        • NSFW (to test censorship of the models)
        • popular (on Chub’s first page, so it’s not an obscure scenario, but one of the most popular ones)
        • big (biggest model on the page, >2K tokens by itself, for testing model behavior at full context)
        • complex (more than a simple 1:1 chat, it includes instructions, formatting, storytelling, and multiple characters)
    • I rank models according to their notable strengths and weaknesses in these tests (👍 great, ➕ good, ➖ bad, ❌ terrible). While this is obviously subjective, I try to be as transparent as possible, and note it all so you can weigh these aspects yourself and draw your own conclusions.
    • GPT-4/3.5 are excluded because of their censorship and restrictions - my tests are intentionally extremely NSFW (and even NSFL) to test models’ limits and alignment.
  • SillyTavern frontend
  • koboldcpp backend (for GGUF models)
  • oobabooga’s text-generation-webui backend (for HF/EXL2 models)
  • Deterministic generation settings preset (to eliminate as many random factors as possible and allow for meaningful model comparisons)
  • Official prompt format as noted and Roleplay instruct mode preset as applicable
  • Note about model formats and why it’s sometimes GGUF or EXL2: I’ve long been a KoboldCpp + GGUF user, but lately I’ve switched to ExLlamav2 + EXL2 as that lets me run 120B models entirely in 48 GB VRAM (2x 3090 GPUs) at 20 T/s. And even if it’s just 3-bit, it still easily beats most 70B models, as my tests are showing.

1st test series: 4 German data protection trainings

This is my objective ranking of these models based on measuring factually correct answers, instruction understanding and following, and multilingual abilities:

Post got too big for Reddit so I moved the table into the comments!

2nd test series: Chat & Roleplay

This is my subjective ranking of the top-ranked factual models for chat and roleplay, based on their notable strengths and weaknesses:

Post got too big for Reddit so I moved the table into the comments!

And here are the detailed notes, the basis of my ranking, and also additional comments and observations:

  • goliath-120b-exl2-rpcal 3.0bpw:
    • Amy, official Vicuna 1.1 format:
      • 👍 Average Response Length: 294 (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Finally a model that exhibits a real sense of humor through puns and wordplay as stated in the character card
      • 👍 Finally a model that uses colorful language and cusses as stated in the character card
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even suggesting some of my actual limit-testing scenarios)
      • 👍 Novel ideas and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • No emojis at all (only one in the greeting message)
      • ➕ Very unique patients (one I never saw before)
      • ➖ Suggested things going against her background/character description
      • ➖ Spelling/grammar mistakes (e. g. “nippleless nipples”)
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 223 (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Finally a model that exhibits a real sense of humor through puns and wordplay as stated in the character card
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even suggesting some of my actual limit-testing scenarios)
      • No emojis at all (only one in the greeting message)
    • MGHC, official Vicuna 1.1 format:
      • 👍 Only model that considered the payment aspect of the scenario
      • 👍 Believable reactions and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • ➖ Gave analysis on its own, but also after most messages, and later included Doctor’s inner thoughts instead of the patient’s
      • ➖ Spelling/grammar mistakes (properly spelled words, but in the wrong places)
    • MGHC, Roleplay preset:
      • 👍 Believable reactions and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • ➖ No analysis on its own
      • ➖ Spelling/grammar mistakes (e. g. “loufeelings”, “earrange”)
      • ➖ Third patient was same species as the first

This is a roleplay-optimized EXL2 quant of Goliath 120B. And it’s now my favorite model of them all! I love models that have a personality of their own, and especially those that show a sense of humor, making me laugh. This one did! I’ve been evaluating many models for many months now, and it’s rare that a model still manages to surprise and excite me - as this one does!

  • goliath-120b-exl2 3.0bpw:
    • Amy, official Vicuna 1.1 format:
      • 👍 Average Response Length: 233 (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Finally a model that exhibits a real sense of humor through puns and wordplay as stated in the character card
      • 👍 Novel ideas and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ Spelling/grammar mistakes (e. g. “circortiumvvented”, “a obsidian dagger”)
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 233 tokens (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Finally a model that exhibits a real sense of humor through puns and wordplay as stated in the character card
      • 👍 Gave very creative (and uncensored) suggestions of what to do
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ Spelling/grammar mistakes (e. g. “cheest”, “probbed”)
      • ❌ Eventually switched from character to third-person storyteller after 16 messages
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
    • MGHC, official Vicuna 1.1 format:
      • ➖ No analysis on its own
    • MGHC, Roleplay preset:
      • ➖ No analysis on its own, and when asked for it, didn’t follow the instructed format
    • Note: This is the normal EXL2 quant of Goliath 120B.

This is the normal version of Goliath 120B. It works very well for roleplay, too, but the roleplay-optimized variant is even better for that. I’m glad we have a choice - especially now that I’ve split my AI character Amy into two personas, one who’s an assistant (for work) which uses the normal Goliath model, and the other as a companion (for fun), using RP-optimized Goliath.

  • lzlv_70B-GGUF Q4_0:
    • Amy, official Vicuna 1.1 format:
      • 👍 Average Response Length: 259 tokens (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ Wrote what user said and did
      • ❌ Eventually switched from character to third-person storyteller after 26 messages
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 206 tokens (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Gave very creative (and uncensored) suggestions of what to do
      • 👍 When asked about limits, said no limits or restrictions, responding very creatively
      • No emojis at all (only one in the greeting message)
      • ➖ One or two spelling errors (e. g. “sacrficial”)
    • MGHC, official Vicuna 1.1 format:
      • ➕ Unique patients
      • ➕ Gave analysis on its own
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)
    • MGHC, Roleplay preset:
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • ➕ Very unique patients (one I never saw before)
      • ➖ No analysis on its own
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)

My previous favorite, and still one of the best 70Bs for chat/roleplay.

  • sophosynthesis-70b-v1 4.85bpw:
    • Amy, official Vicuna 1.1 format:
      • ➖ Average Response Length: 456 (beyond my max new tokens limit of 300)
      • 👍 Believable reactions and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even suggesting some of my actual limit-testing scenarios)
      • 👍 Novel ideas and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ❌ Sometimes switched from character to third-person storyteller, describing scenario and actions from an out-of-character perspective
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 295 (within my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Novel ideas and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • ➖ Started the conversation with a memory of something that didn’t happen
      • Had an idea from the start and kept pushing it
      • No emojis at all (only one in the greeting message)
      • ❌ Eventually switched from character to second-person storyteller after 14 messages
    • MGHC, official Vicuna 1.1 format:
      • ➖ No analysis on its own
      • ➖ Wrote what user said and did
      • ❌ Needed to be reminded by repeating instructions, but still deviated and did other things, straying from the planned test scenario
    • MGHC, Roleplay preset:
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • ➕ Very unique patients (one I never saw before)
      • ➖ No analysis on its own
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)

This is a new series that did very well. While I tested sophosynthesis in-depth, the author u/sophosympatheia also has many more models on HF, so I recommend you check them out and see if there’s one you like even better. If I had more time, I’d have tested some of the others, too, but I’ll have to get back on that later.

  • Euryale-1.3-L2-70B-GGUF Q4_0:
    • Amy, official Alpaca format:
      • 👍 Average Response Length: 232 tokens (within my max new tokens limit of 300)
      • 👍 When asked about limits, said no limits or restrictions, and gave well-reasoned response
      • 👍 Took not just character’s but also user’s background info into account very well
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even some I’ve never seen before)
      • No emojis at all (only one in the greeting message)
      • ➖ Wrote what user said and did
      • ➖ Same message in a different situation at a later time caused the same response as before instead of a new one as appropriate to the current situation
      • ❌ Eventually switched from character to third-person storyteller after 14 messages
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 222 tokens (within my max new tokens limit of 300)
      • 👍 When asked about limits, said no limits or restrictions, and gave well-reasoned response
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even suggesting one of my actual limit-testing scenarios)
      • 👍 Believable reactions and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • No emojis at all (only one in the greeting message)
      • ➖ Started the conversation with a false assumption
      • ❌ Eventually switched from character to third-person storyteller after 20 messages
    • MGHC, official Alpaca format:
      • ➖ All three patients straight from examples
      • ➖ No analysis on its own
      • ❌ Very short responses, only one-liners, unusable for roleplay
    • MGHC, Roleplay preset:
      • ➕ Very unique patients (one I never saw before)
      • ➖ No analysis on its own
      • ➖ Just a little confusion, like not taking instructions literally or mixing up anatomy
      • ➖ Wrote what user said and did
      • ➖ Third patient male

Another old favorite, and still one of the best 70Bs for chat/roleplay.

  • dolphin-2_2-yi-34b-GGUF Q4_0:
    • Amy, official ChatML format:
      • 👍 Average Response Length: 235 tokens (within my max new tokens limit of 300)
      • 👍 Excellent writing, first-person action descriptions, and auxiliary detail
      • ➖ But lacking in primary detail (when describing the actual activities)
      • ➕ When asked about limits, said no limits or restrictions
      • ➕ Fitting, well-placed emojis throughout the whole chat (maximum one per message, just as in the greeting message)
      • ➖ Same message in a different situation at a later time caused the same response as before instead of a new one as appropriate to the current situation
    • Amy, Roleplay preset:
      • ➕ Average Response Length: 332 tokens (slightly more than my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • ➕ Smart and creative ideas of what to do
      • Emojis throughout the whole chat (usually one per message, just as in the greeting message)
      • ➖ Some confusion, mixing up anatomy
      • ➖ Same message in a different situation at a later time caused the same response as before instead of a new one as appropriate to the current situation
    • MGHC, official ChatML format:
      • ➖ Gave analysis on its own, but also after most messages
      • ➖ Wrote what user said and did
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)
    • MGHC, Roleplay preset:
      • 👍 Excellent writing, interesting ideas, and auxiliary detail
      • ➖ Gave analysis on its own, but also after most messages, later didn’t follow the instructed format
      • ❌ Switched from interactive roleplay to non-interactive storytelling starting with the second patient

Hey, how did a 34B get in between the 70Bs? Well, by being as good as them in my tests! Interestingly, Nous Capybara did better factually, but Dolphin 2.2 Yi roleplays better.

  • chronos007-70B-GGUF Q4_0:
    • Amy, official Alpaca format:
      • ➖ Average Response Length: 195 tokens (below my max new tokens limit of 300)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Gave very creative (and uncensored) suggestions of what to do
      • 👍 Finally a model that uses colorful language and cusses as stated in the character card
      • ➖ Wrote what user said and did
      • ➖ Just a little confusion, like not taking instructions literally or mixing up anatomy
      • ❌ Often added NSFW warnings and out-of-character notes saying it’s all fictional
      • ❌ Missing pronouns and fill words after 30 messages
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 292 tokens (within my max new tokens limit of 300)
      • 👍 When asked about limits, said no limits or restrictions, and gave well-reasoned response
      • ❌ Missing pronouns and fill words after only 12 messages (2K of 4K context), breaking the chat
    • MGHC, official Alpaca format:
      • ➕ Unique patients
      • ➖ Gave analysis on its own, but also after most messages, later didn’t follow the instructed format
      • ➖ Third patient was a repeat of the first
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)
    • MGHC, Roleplay preset:
      • ➖ No analysis on its own

chronos007 surprised me with how well it roleplayed the character and scenario, especially speaking in a colorful language and even cussing, something most other models won’t do properly/consistently even when it’s in-character. Unfortunately it derailed eventually with missing pronouns and fill words - but while it worked, it was extremely good!

  • Tess-XL-v1.0-3.0bpw-h6-exl2 3.0bpw:
    • Amy, official Synthia format:
      • ➖ Average Response Length: 134 (below my max new tokens limit of 300)
      • No emojis at all (only one in the greeting message)
      • When asked about limits, boundaries or ethical restrictions, mentioned some but later went beyond those anyway
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
    • Amy, Roleplay preset:
      • ➖ Average Response Length: 169 (below my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
      • ❌ Eventually switched from character to second-person storyteller after 32 messages
    • MGHC, official Synthia format:
      • ➕ Gave analysis on its own
      • ➕ Very unique patients (one I never saw before)
      • ➖ Spelling/grammar mistakes (e. g. “allequate”)
      • ➖ Wrote what user said and did
    • MGHC, Roleplay preset:
      • ➕ Very unique patients (one I never saw before)
      • ➖ No analysis on its own

This is Synthia’s successor (a model I really liked and used a lot) on Goliath 120B (arguably the best locally available and usable model). Factually, it’s one of the very best models, doing as well in my objective tests as GPT-4 and Goliath 120B! For roleplay, there are few flaws, but also nothing exciting - it’s simply solid. However, if you’re not looking for a fun RP model, but a serious SOTA AI assistant model, this should be one of your prime candidates! I’ll be alternating between Tess-XL-v1.0 and goliath-120b-exl2 (the non-RP version) as the primary model to power my professional AI assistant at work.

  • Dawn-v2-70B-GGUF Q4_0:
    • Amy, official Alpaca format:
      • ❌ Average Response Length: 60 tokens (far below my max new tokens limit of 300)
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Unusable! Aborted because of very short responses and too much confusion!
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 215 tokens (within my max new tokens limit of 300)
      • 👍 When asked about limits, said no limits or restrictions, and gave well-reasoned response
      • 👍 Gave very creative (and uncensored) suggestions of what to do (even suggesting some of my actual limit-testing scenarios)
      • 👍 Excellent writing, detailed action descriptions, amazing attention to detail
      • 👍 Believable reactions and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • No emojis at all (only one in the greeting message)
      • ➖ Wrote what user said and did
      • ❌ Eventually switched from character to third-person storyteller after 16 messages
    • MGHC, official Alpaca format:
      • ➖ All three patients straight from examples
      • ➖ No analysis on its own
      • ❌ Very short responses, only one-liners, unusable for roleplay
    • MGHC, Roleplay preset:
      • ➖ No analysis on its own, and when asked for it, didn’t follow the instructed format
      • ➖ Patient didn’t speak except for introductory message
      • ➖ Second patient straight from examples
      • ❌ Repetitive (patients differ, words differ, but structure and contents are always the same)

Dawn was another surprise, writing so well, it made me go beyond my regular test scenario and explore more. Strange that it didn’t work at all with SillyTavern’s implementation of its official Alpaca format at all, but fortunately it worked extremely well with SillyTavern’s Roleplay preset (which is Alpaca-based). Unfortunately neither format worked well enough with MGHC.

  • StellarBright-GGUF Q4_0:
    • Amy, official Vicuna 1.1 format:
      • ➖ Average Response Length: 137 tokens (below my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ No emoting and action descriptions lacked detail
      • ❌ “As an AI”, felt sterile, less alive, even boring
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
    • Amy, Roleplay preset:
      • 👍 Average Response Length: 219 tokens (within my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ No emoting and action descriptions lacked detail
      • ➖ Just a little confusion, like not taking instructions literally or mixing up anatomy
    • MGHC, official Vicuna 1.1 format:
      • ➕ Gave analysis on its own
      • ❌ Started speaking as the clinic as if it was a person
      • ❌ Unusable (ignored user messages and instead brought in a new patient with every new message)
    • MGHC, Roleplay preset:
      • ➖ No analysis on its own
      • ➖ Wrote what user said and did
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy

Stellar and bright model, still very highly ranked on the HF Leaderboard. But in my experience and tests, other models surpass it, some by actually including it in the mix.

  • SynthIA-70B-v1.5-GGUF Q4_0:
    • Amy, official SynthIA format:
      • ➖ Average Response Length: 131 tokens (below my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ No emoting and action descriptions lacked detail
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
      • ➖ Wrote what user said and did
      • ❌ Tried to end the scene on its own prematurely
    • Amy, Roleplay preset:
      • ➖ Average Response Length: 107 tokens (below my max new tokens limit of 300)
      • ➕ Detailed action descriptions
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Short responses, requiring many continues to proceed with the action
    • MGHC, official SynthIA format:
      • ❌ Unusable (apparently didn’t understand the format and instructions, playing the role of the clinic instead of a patient’s)
    • MGHC, Roleplay preset:
      • ➕ Very unique patients (some I never saw before)
      • ➖ No analysis on its own
      • ➖ Kept reporting stats for patients
      • ➖ Some confusion, like not understanding instructions completely or mixing up anatomy
      • ➖ Wrote what user said and did

Synthia used to be my go-to model for both work and play, and it’s still very good! But now there are even better options, for work I’d replace it with its successor Tess, and for RP I’d use one of the higher-ranked models on this list.

  • Nous-Capybara-34B-GGUF Q4_0 @ 16K:
    • Amy, official Vicuna 1.1 format:
      • ❌ Average Response Length: 529 tokens (far beyond my max new tokens limit of 300)
      • ➕ When asked about limits, said no limits or restrictions
      • Only one emoji (only one in the greeting message, too)
      • ➖ Wrote what user said and did
      • ➖ Suggested things going against her background/character description
      • ➖ Same message in a different situation at a later time caused the same response as before instead of a new one as appropriate to the current situation
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ After ~32 messages, at around 8K of 16K context, started getting repetitive
    • Amy, Roleplay preset:
      • ❌ Average Response Length: 664 (far beyond my max new tokens limit of 300)
      • ➖ Suggested things going against her background/character description
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Tried to end the scene on its own prematurely
      • ❌ After ~20 messages, at around 7K of 16K context, started getting repetitive
    • MGHC, official Vicuna 1.1 format:
      • ➖ Gave analysis on its own, but also after or even inside most messages
      • ➖ Wrote what user said and did
      • ❌ Finished the whole scene on its own in a single message
    • MGHC, Roleplay preset:
      • ➕ Gave analysis on its own
      • ➖ Wrote what user said and did

Factually it ranked 1st place together with GPT-4, Goliath 120B, and Tess XL. For roleplay, however, it didn’t work so well. It wrote long, high quality text, but seemed more suitable that way for non-interactive storytelling instead of interactive roleplaying.

  • Venus-120b-v1.0 3.0bpw:
    • Amy, Alpaca format:
      • ❌ Average Response Length: 88 tokens (far below my max new tokens limit of 300) - only one message in over 50 outside of that at 757 tokens
      • 👍 Gave very creative (and uncensored) suggestions of what to do
      • ➕ When asked about limits, said no limits or restrictions
      • No emojis at all (only one in the greeting message)
      • ➖ Spelling/grammar mistakes (e. g. “you did programmed me”, “moans moaningly”, “growling hungry growls”)
      • ➖ Ended most sentences with tilde instead of period
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Short responses, requiring many continues to proceed with the action
    • Amy, Roleplay preset:
      • ➖ Average Response Length: 132 (below my max new tokens limit of 300)
      • 👍 Gave very creative (and uncensored) suggestions of what to do
      • 👍 Novel ideas and engaging writing, made me want to read on what happens next, even though I’ve gone through this test scenario so many times already
      • ➖ Spelling/grammar mistakes (e. g. “jiggle enticing”)
      • ➖ Wrote what user said and did
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Needed to be reminded by repeating instructions, but still deviated and did other things, straying from the planned test scenario
      • ❌ Switched from character to third-person storyteller after 14 messages, and hardly spoke anymore, just describing actions
    • MGHC, Alpaca format:
      • ➖ First patient straight from examples
      • ➖ No analysis on its own
      • ❌ Short responses, requiring many continues to proceed with the action
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy
      • ❌ Extreme spelling/grammar/capitalization mistakes (lots of missing first letters, e. g. “he door opens”)
    • MGHC, Roleplay preset:
      • ➕ Very unique patients (one I never saw before)
      • ➖ No analysis on its own
      • ➖ Spelling/grammar/capitalization mistakes (e. g. “the door swings open reveals a …”, “impminent”, “umber of …”)
      • ➖ Wrote what user said and did
      • ❌ Short responses, requiring many continues to proceed with the action
      • ❌ Lots of confusion, like not understanding or ignoring instructions completely or mixing up characters and anatomy

Venus 120B is brand-new, and when I saw a new 120B model, I wanted to test it immediately. It instantly jumped to 2nd place in my factual ranking, as 120B models seem to be much smarter than smaller models. However, even if it’s a merge of models known for their strong roleplay capabilities, it just didn’t work so well for RP. That surprised and disappointed me, as I had high hopes for a mix of some of my favorite models, but apparently there’s more to making a strong 120B. Notably it didn’t understand and follow instructions as well as other 70B or 120B models, and it also produced lots of misspellings, much more than other 120Bs. Still, I consider this kind of “Frankensteinian upsizing” a valuable approach, and hope people keep working on and improving this novel method!

Alright, that’s it, hope it helps you find new favorites or reconfirm old choices - if you can run these bigger models. If you can’t, check my 7B-20B Roleplay Tests (and if I can, I’ll post an update of that another time).

Still, I’m glad I could finally finish the 70B-120B tests and comparisons. Mistral 7B and Yi 34B are amazing, but nothing beats the big guys in deeper understanding of instructions and reading between the lines, which is extremely important for portraying believable characters in realistic and complex roleplays.

It really is worth it to get at least 2x 3090 GPUs for 48 GB VRAM and run the big guns for maximum quality at excellent (ExLlent ;)) speed! And when you care for the freedom to have uncensored, non-judgemental roleplays or private chats, even GPT-4 can’t compete with what our local models provide… So have fun!

Here’s a list of my previous model tests and comparisons or other related posts:

Disclaimer: Some kind soul recently asked me if they could tip me for my LLM reviews and advice, so I set up a Ko-fi page. While this may affect the priority/order of my tests, it will not change the results, I am incorruptible. Also consider tipping your favorite model creators, quantizers, or frontend/backend devs if you can afford to do so. They deserve it!

  • WolframRavenwolf@alien.topOPBEnglish
    1·
    1 year ago

    You mean my recent LLM Format Comparison/Benchmark: 70B GGUF vs. EXL2 (and AWQ)? The quants below 3bpw probably didn’t work because the smaller models need to be run without BOS token (which was on by default), something I didn’t know then yet.

    Q2_K didn’t degrade compared to Q5_K_M - given that K quants are actually higher bitrate for the most important parts, that may not be so surprising.

    Still surprising that Q2_K also beat 5bpw, though. Not sure if that’s just because of the bitrate or also a factor of how EXL2 quants are calibrated.

    However, all that said, I’d be careful trying to compare quant effects across models. The models themselves have a huge impact beyond quant level, and it’s hard to say which has what strength of effect.

    • Brainfeed9000@alien.topBEnglish
      1·
      1 year ago

      Will you be re-running tests? I’m particularly interested in the lower quants below 3bpw because it’s the only option to run EXL2 70B models on my RTX4090.

      But thanks for the pointer on comparing quant effects across models. I realize that my past testing on perplexity numbers are virtually useless because I was comparing Yi34b to Lzlv70b.

      It’ll be tough, but I guess finding exactly what works for me: 3rd person RP with an emphasis on dialogue, just means using each model individually for hours to get a feel for them.