I tried to apply a lot of prompting techniques in 7b and 13b models. And no matter how hard I tried, there was barely any improvement.

  • phree_radical@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    2 years ago

    What you’re referring to as “prompt engineering” is more accurately described as how to get good interpolations between ChatGPT behaviors. Those are specific instructions and behaviors that OpenAI trains their models on, in careful proportion designed to reach good generalization on them

    And it’s not that the models are too small – Mistral 13b will be better than gpt-3.5-turbo. It’s all about the training

    Anyways that’s why I try to loudly proclaim the benefits of few-shot examples and completion instead of instruction, until we have models trained the way OpenAI’s are. If you’re willing to write examples and dodge the chatbot trained behaviors, you can pretty much perform any task without need for training