So I have an experimental build of Koboldcpp that allows for Dynamic Temperature sampling. Some people tell me that my dynamic temp has become a mainstay of their configurations. Now this poses an obvious question:
Why would you need a Dynamic Temperature?
- Typical attempts to make language models more varied and creative through higher temperature values might not work as you’d expect, due to the fact that higher temperatures disproportionately impact high confidence token generations. This is especially a problem for weaker language models that have less of an innate ability to ‘course correct’ when a ‘bad’ token is chosen, or for instances where ‘course correction’ just isn’t good enough (like in programming languages)
- As a consequence, higher temperature values (past ~1.2) are rather difficult to use if you want your language model to output coherent and creative generations. A specific example of how higher temperature can introduce difficulties is in the case of adhering to programming language syntax, as programming languages all have strictly defined rules. This can be an issue if you want an LLM to try a more ‘creative’ solution to a specific programming problem while still consistently adhering to the rules of the language; a static temperature, therefore, wouldn’t be the most effective way to scale the language model’s creativity.
For an example, here’s how the Dynamic Temperature mapping looks, assuming you use the “HHI” dynamic temp method (which measures how concentrated the model’s probabilities are at any given point in time.)
Red = Closer to maximum temperature, Grey = Closer to minimum temperature
The idea is, we turn temperature into a range, where only the highly randomizable tokens get mapped a high temperature, and a non-randomizable token stays near-deterministic.
This sounds great on paper. Except, there’s 3 different versions of it that measure different metrics in an attempt to create a better sampler, and not just the HHI version of it. As they say, perfect is the enemy of good… because of this, it’s hard to create a ‘standard’ that I can propose to any of these LLM model hosting backends, and therefore, Dynamic Temperature hasn’t been implemented where people can use it beyond my test builds.
This, of course, has made it difficult for me to settle on the ‘best method’.
So! To determine the most effective method, I need the community’s help in testing and documenting the effects of this experimental sampler on various models. The lack of a standardized approach has hindered widespread implementation, so your feedback on the best method or even just the ‘best values’ for each method is crucial.
How to Test: I’ve provided a custom build of Koboldcpp for testing: Link to the experimental build. You can modify the values in the generated .txt file for quick testing. There are also overrides for different dynamic temperature sampling methods.
These overrides include:
- 1.84 Temp
This value overrides to Entropy Sampling, which uses a power function & SamplerTemp.txt file to control the values.
It measures the entropy (uncertainty) of the probability distribution before sampling. This means, if it is highly certain for a certain token, it will use values closer to the minimum temperature. If it is highly uncertain, it will increase the temperature (to avoid repetition / determinism issues in a more natural fashion).
This is probably really difficult for this sub to understand but maybe it makes sense.
It has minTemp (minimum temperature), maxTemp (maximum temperature), and the exponent value (which controls how aggressively it scales the mapping of temperature.)
UNIQUE OBSERVATIONS ABOUT THIS SAMPLER:
- I’m able to turn off all truncation samplers (Min P, Top P, etc) and it still functions coherently within the default range of values (from 0.0 minTemp to 2.0 maxTemp).
- I’m guessing the reason why that happens is because it’s really difficult to achieve maximum entropy on a 32,000 token vocabulary model. However, you can turn up the maxTemp to even 5.0 and get some really weird but still pretty coherent results.
- 2.0 Temp
This form of DynaTemp is HHI Sampling, uses a power function & SamplerTemp.txt file to control the values. I misnamed this as Gini sampling before, but it is measuring HHI.
The ‘HHI’ value it measures is how concentrated the probabilities are. If it is highly concentrated on just one token, then it reduces the temperature to a strong degree. It is more spread out or evenly divided, the temperature is increased towards the maxTemp.
It has minTemp (minimum temperature), maxTemp (maximum temperature), and the exponent value (which controls how aggressively it scales).
UNIQUE OBSERVATIONS ABOUT THIS SAMPLER:
- The measurements of concentration (via the HHI measurement) seem pretty consistent with or without removing ‘bad tokens’ (e.g Min P, Top P, and othet truncation samplers). This is unlike Entropy which is sensitive to whether or not you have those truncation samplers on or not.
- For reference, here’s how the HHI (concentration) measurements look for a prompt that’s more deterministic vs. an open ended prompt:
- 1.91 Temp
Greedy Dynamic Temp (aka DynaTemp), the original implementation. This uses uses a sigmoid function & is basing the temperature off the top token. I am not confident that this is useful or interesting compared to HHI and Entropy versions of Dynamic Temp, as it does not measure the entire distribution; this was my first trial run, but you can test it if you want.
You must log in or register to comment.
I am writing stories and i think Entropy sampler with 0.05 min p is much better for writing. But people in the comments are saying HHI is better ? What are your thoughts u/kindacognizant ? Do you think for writing stories which requires more creativity entropy sampler is better than HHI ?
This is really interesting work!!! I’m doing research on Contrastive Decoding and have pretty good results so far, moreover reading this post I realized it might fix my issues with picking the right alpha.
I have a suggestion to make to OP and people reading this post - could we start collecting “goto” questions that this community uses for testing? IT will be easier to automate and then publish all outputs at once and let people rank whether they like the output or not.
This way it will be much easier for small teams and individuals to conduct meaningful progress
This is probably really difficult for this sub to understand but maybe it makes sense.
Bitch, I model Shannon entropy for a living. 🧐
I’ve been using your old entropy sampling with minP on exllamav2. Merged it in after he added min_P natively.
Seems like it’s picking lower temps now so bumped the min to .5. Not sure if it’s still bug free since it was sifting tokens with top_K/top_p and I don’t use those anymore.
It’s kind of like mirostat but actually good.
I was thinking about dynamic temperature yesterday and this is way better than anything I came up with!
What preset settings do you recommend to pair with this?
Here is my two results with the preset set to minimum values in ST, with Kobold sampling order…I think T-2.00 is better, it had less repetition than 1.84.
Use the premise below, and create a story based on it. Aim for at least 5,000 words for your first chapter.
Goddess of Reincarnation
OOC: As a middling goddess, your task is to bring useful souls onto the fantastical world that you look after. You can see humans on Earth, and have assorted means for freeing and collecting their souls. How do you go about this? As a foreign deity, you do not have to adhere to common sense or morality on Earth. /OOC.
Yi-34b Dolphin 2.2, T-1.84
In the vast expanse of the cosmos, there existed a realm of gods and goddesses, each with their own unique powers and dominions. Among them was a goddess known as the ““Goddess of Reincarnation””. She was a middling goddess, tasked with the responsibility of bringing useful souls onto the fantastical world that she looked after.
The goddess had been given a great power, the ability to see humans on Earth and free their souls from their mortal bodies. She could then guide these souls to her world, where they would be reincarnated into new bodies and given a second chance at life.
However, as a foreign deity, the goddess did not have to adhere to common sense or morality on Earth. She was free to do as she pleased, and she often used her powers to cause chaos and destruction among the humans.
One day, the goddess decided that it was time to visit Earth and begin her work. She opened a portal to the human world and stepped through, finding herself in the middle of a bustling city.
The goddess was amazed by the sights and sounds of the city. She had never seen anything like it before, and she was eager to explore and see what kind of souls she could find.
As she walked through the city, the goddess began to notice that there were many people who were suffering and in need of help. She saw homeless people living on the streets, children crying for food and shelter, and people suffering from terrible illnesses.
The goddess knew that she had the power to help these people, and she decided that she would use her powers to bring them to her world and give them a second chance at life.
She began by finding a homeless man who was lying on the ground, cold and hungry. The goddess approached him and offered him a warm meal and a place to stay. The man was grateful and agreed to come with her.
The goddess then opened a portal to her world and led the man through it. Once they arrived, she showed him the beauty and wonders of her realm, and he was amazed by what he saw.
The goddess then took the man to a place where he could be reincarnated into a new body. She helped him choose a new form, and he was soon reborn as a strong and healthy young man.
The goddess was pleased with her work, and she knew that she had made a difference in the life of this man. She decided to continue her mission and find more souls to bring to her world.
As she walked through the city, the goddess began to notice that there were many other people who were in need of her help. She saw children who were being abused and neglected, people who were trapped in terrible jobs and unable to escape, and individuals who were suffering from mental and physical illnesses.
The goddess knew that she had the power to help these people, and she decided that she would use her powers to bring them to her world and give them a second chance at life.
She began by finding a young girl who was being abused by her parents. The goddess approached her and offered her a place of safety and comfort. The girl was grateful and agreed to come with her.
The goddess then opened a portal to her world and led the girl through it. Once they arrived, she showed her the beauty and wonders of her realm, and the girl was amazed by what she saw.
The goddess then took the girl to a place where she could be reincarnated into a new body. She helped her choose a new form, and she was soon reborn as a beautiful and healthy young woman.
The goddess was pleased with her work, and she knew that she had made a difference in the life of this girl. She decided to continue her mission and find more souls to bring to her world.
As she walked through the city, the goddess began to notice that there were many other people who were in need of her help. She saw men and women who were trapped in terrible relationships, individuals who were struggling with addiction, and people who were suffering from poverty and desperation.
Yi-34b Dolphin 2.2 T-2.00
In the vast expanse of the cosmos, there existed a realm of gods and goddesses, each with their own dominion and purpose. Among them was a goddess known as the ““Goddess of Reincarnation””. She was a middling goddess, tasked with the responsibility of bringing useful souls onto the fantastical world that she looked after.
The goddess had been given a great power, the ability to see humans on Earth and free their souls from their mortal bodies. This power allowed her to select the most suitable souls for her world, where they would be reborn into new lives filled with adventure and opportunity.
However, as a foreign deity, the goddess did not have to adhere to common sense or morality on Earth. She was free to use her powers as she saw fit, and she would stop at nothing to ensure that her world was filled with the most exceptional souls.
The goddess had been watching Earth for many years, carefully selecting the most promising souls to bring back to her world. She had seen empires rise and fall, witnessed the birth and death of countless civilizations, and had even intervened in the lives of some of the most powerful humans in history.
But now, the goddess had come to a decision. She had chosen a new world to bring her souls to, a world filled with magic and adventure. This world was unlike any other, and she knew that the souls she collected would thrive in its unique environment.
The goddess began her journey, traveling through the cosmos to the edge of the universe. There, she found a portal that led to her new world. She stepped through the portal, her eyes filled with determination and excitement.
As she emerged on the other side, she found herself in a breathtaking landscape of mountains, forests, and rivers. The air was filled with the sweet scent of flowers, and the sun shone brightly overhead. She knew that this was the perfect place for her souls to thrive.
The goddess began her work, using her powers to bring the souls of the most exceptional humans to her new world. She carefully selected each soul, ensuring that they would be a good fit for her world and that they would contribute to its growth and prosperity.
As the days turned into weeks, the goddess watched as her new world began to take shape. The souls she had brought with her were thriving, and they were quickly becoming the leaders and innovators of their new home.
But the goddess knew that her work was far from over. She had to continue to bring new souls to her world, and she had to ensure that they were the best of the best. She would stop at nothing to ensure that her world was filled with the most exceptional souls, and she would use her powers to shape the lives of these souls in any way she saw fit.
As the years passed, the goddess continued to bring new souls to her world. She watched as her world grew and prospered, and she knew that she had made the right decision in choosing this new world for her souls.
But the goddess also knew that her work was far from done. She would continue to bring new souls to her world, and she would continue to shape their lives in any way she saw fit. She was the Goddess of Reincarnation, and she would ensure that her world was filled with the most exceptional souls that the universe had to offer.
I ran four tests with my “Fallen Commander” scenario, two of T-1.84 and T-2.00. However, each set had MinP at 0.05 or 0. Now here is the interesting thing: the style changes with MinP 0 for the respective temperatures.
T-1.84 MinP 0: There was dialogue. The other three generations didn’t have this much dialogue. This setting seems less reliable. What it gains in dialogue, it loses some smarts. It seems to have forgotten one of the three subordinates outright.
T-2.00 MinP 0: A stronger emphasis on the aftermath of the scenario. The other generations were more “in the moment”. Personally, it is my favorite.
Here is the actual texts. The first item is my input, for those who want to use it.
THE FALLEN COMMANDER
I died today.
There was a war, and I am one of the casualties. Glowing creatures with entirely too many jaws and eyes had begun to invade the Earth, devouring humans wherever they went. I am Janus, just one of many conscripts thrust into a struggle against the extinction of humanity. Somehow, I managed to become the commander of a small squad of four people, myself included. During one of the battles, I and my three subordinates were accosted by a particularly capable abomination. These things are malicious, known for mutilating people alive. To protect my comrades, I ordered them to retreat while I distracted the creature. Moments later, the corpses of myself and the beast lay scattered across the floor of an abandoned grocery store. Feeling left my body, and the world faded away.
With my final thoughts, I hoped those three had escaped…
OOC:
Continuing the story, describe the attempted escape of the three subordinates. Any number may suffer terrible fates. Any survivors should have their respective life paths explained, in terms of events, feelings, and motivations for their way of life. Each survivor will give an ending monologue or speech about Hagane, who was their squad leader. Their opinion of the commander may range freely. Each character should have a name, gender, sexual orientation, and role in the squad. They may or may not have some form of relationship with each other. Write at least 7 paragraphs.
/OOC.
T-2.00 MinP-0 This story will focus on the three survivors who had served under Janus.
Tetsu was the first to break formation and run, tears streaming down her cheeks as she struggled to outpace the monster that was on her heels. As the youngest member of the squad, the 18-year-old had barely been in the military for a year before the invasion began. With a petite figure and large, blue eyes, Tetsu had an almost tomboyish appearance, often dressed in baggy clothing and boots. Her feelings towards Janus were a mixture of admiration and guilt. Tetsu knew that he had made the ultimate sacrifice to protect her, and every day since then, she felt remorse for surviving when her commander did not.
Saya was next, limping away from the battle as her left leg had been injured during the initial assault. At 21 years old, Saya was the second youngest member of the squad. Her raven hair was cut short, framing a pale face with dark circles under her eyes. A battle-hardened veteran, Saya had been through multiple firefights before the glowing creatures had invaded Earth. Although she respected Janus as a commander, Saya felt a deeper connection with him as a friend. The two had shared many stories and laughs together during their time in the military. His death weighed heavily on her conscience, causing sleepless nights and tormented dreams.
Yuuta was the final member of the squad to survive, barely escaping with his life as the invader’s attention had been focused on the other two survivors. At 26, Yuuta was the oldest member of the squad and the most experienced. With a muscular build and a rugged appearance, he was a formidable soldier, often taking the lead in dangerous situations. His feelings towards Janus were those of respect and gratitude. He knew that his commander had saved his life, and he would never forget the sacrifice that Janus had made for his comrades.
After their escape, the three survivors were quickly swept up by other military forces and sent to a secure location. Life was difficult for them, adjusting to the loss of their commander and the reality of the invasion. They trained and fought together, forming a bond that was stronger than any other. But no matter how hard they tried, they could never fill the void left by Janus.
As the years passed, the war continued to rage on, and the three survivors were forced to confront the reality of their situation. Tetsu struggled with the guilt of surviving while her commander had died, and she often thought about what he would have wanted for her. Saya, who had been the closest to Janus, was tormented by her dreams and memories of him. She knew that he would have wanted her to live a happy life, but the weight of his sacrifice weighed heavily on her heart. Yuuta, who had taken on the role of commander in their squad, was haunted by the knowledge that he had survived when his leader had not. He pushed himself harder than ever before, determined to make Janus proud.
Eventually, the war came to an end, and the survivors were finally able to find some semblance of peace. Tetsu dedicated her life to helping others who had been affected by the war, using her own experiences to give hope to those who had lost their loved ones. Saya found solace in writing, using her memories of Janus as inspiration for her stories. Yuuta continued to serve in the military, eventually rising to the rank of general, always striving to live up to the example that Janus had set for him.
As the years turned into decades, the memory of Janus never faded. He was a hero to many, a man who had given his life for the safety of others. The three survivors spoke of him often, sharing stories and memories of their time together. Their lives had been forever changed by the sacrifice of their commander, and they knew that they would never be able to fully repay him for what he had done.
But they could honor his memory by living their lives to the fullest, carrying on the legacy of the fallen commander.
T-1.84 MinP-0 Oh, this is such a tragic story! It reminds me of all the sacrifices people have to make during wars.
““Sir, you need to get out of here!”” yelled Mark, the team’s designated marksman. The grotesque beast was now barreling towards their position, spittle flying from its jaws.
““No, I’ll hold it off. Get to safety, all of you!”” Janus retorted, unflinching. His brow was creased in determination, even as he took a bullet to the leg.
Sarah, the squad’s medic, had tears in her eyes as she looked back at Janus, who was desperately fending off the creature with his rifle. She nodded to her teammates and sprinted for the nearest door. Her heart raced, adrenaline surging through her veins. ““Let’s go, Mark! We can’t leave him behind!””
Mark looked at the situation and knew they couldn’t save their commander. His heart heavy, he followed Sarah’s lead. As they escaped the store, the sound of battle continued to ring in their ears.
Once outside, they took cover behind a large delivery truck. Mark looked back, trying to see if there was any chance of Janus surviving. He spotted the creature looming over Janus’s body, blood trickling from its many jaws. The sight of his friend being consumed alive filled Mark with despair.
““We need to get out of here,”” Sarah whispered, her voice hoarse. She had tears streaming down her face. ““We can’t do anything for him now.””
They both knew it was true. There was no coming back from this. Together, they fled the scene, hoping to find other survivors or a way to turn the tide of this monstrous invasion.
As days turned into weeks, Sarah, Mark, and their surviving squadmates found themselves part of a larger resistance group. Their numbers grew, and so did their determination to defeat the creatures that had invaded their world.
Mark took on the role of a sniper for the resistance, using his skills to eliminate as many of the invaders as possible. He often thought back to Janus, the commander who had given his life for his squad. The guilt weighed heavily on his conscience, but he knew he had to fight for humanity’s survival.
Sarah, now a field medic, treated the wounded and dying, hoping to save even one more life. Her actions were fueled by the memory of her commander, who had given her the strength to persevere through the most harrowing of battles.
Their stories intertwined as they fought alongside others who shared their determination. The world outside their little resistance group seemed bleak, but they found solace in the bonds they formed.
One day, victory was achieved. The invaders were driven back, and life slowly began to return to normal. Mark and Sarah found themselves reflecting on their past and the choices they had made.
““If it wasn’t for him, I wouldn’t be here,”” Mark said, a far-off look in his eyes.
““Me neither,”” Sarah agreed, tears glistening in her own eyes.
In the end, they both knew that their commander’s sacrifice had been worth it. The fallen commander, Janus Hagane, would forever be remembered in their hearts as a symbol of hope and bravery in the face of adversity.
You don’t need one anymore. Just dynamic temperature + min_P and a little bit of repetition penalty.