언어모델 미세조정 데이터셋 - 딥러닝 언어 모델

from datasets import load_dataset

dataset_name = 'HuggingFaceTB/smoltalk'
smoltalk = load_dataset(dataset_name, 'all')

import pandas as pd

pd.DataFrame(smoltalk['train'].shuffle(0).take(5))

chat = smoltalk.filter(lambda example: example['source'] == 'everyday-conversations')
for entry in chat['train']['messages'][:5]:
    for item in entry:
        print(f"{item['role']}: {item['content']}")
    print('-----' * 10)

user: Hi there
assistant: Hello! How can I help you today?
user: I'm looking for a healthy breakfast idea. What's a good option?
assistant: A fruit salad is a great choice. It's nutritious, delicious, and easy to make.
user: What fruits are good in a fruit salad?
assistant: You can use a mix of your favorite fruits, such as strawberries, bananas, grapes, and pineapple.
--------------------------------------------------
user: Hi there
assistant: Hello! How can I help you today?
user: I'm looking for a new pair of jeans. What are the current styles?
assistant: Currently, straight-leg and high-waisted jeans are popular. You can also consider boyfriend or slim-fit styles, depending on your preference.
user: What about colors? Are light washes still in style?
assistant: Yes, light washes are still popular, but you can also consider darker washes or trendy colors like black or gray with distressing.
user: Do you have any recommendations for brands?
assistant: Some popular brands for jeans include Levi's, American Eagle, and Gap. They offer a variety of styles and sizes to fit different tastes and budgets.
--------------------------------------------------
user: Hi there
assistant: Hello! How can I help you today?
user: I'm thinking of getting a pet, what are some popular pets?
assistant: Popular pets include dogs, cats, and fish. They are relatively low-maintenance and easy to care for.
user: What about pet insurance, is it necessary?
assistant: Yes, pet insurance is highly recommended. It helps cover unexpected veterinary expenses and can provide peace of mind for pet owners.
user: How much does pet insurance typically cost?
assistant: The cost of pet insurance varies depending on the type of pet, age, and coverage level. On average, it can range from $20 to $50 per month.
--------------------------------------------------
user: Hey!
assistant: Hello! How can I help you today?
user: I'm worried about the storm that's supposed to hit our area tonight. Will it be severe?
assistant: Yes, the forecast indicates a strong storm with heavy rain and wind gusts. It's expected to start around 8 PM.
user: Will the storm cause any power outages?
assistant: There's a possibility of power outages due to the strong winds, but the utility company is preparing for the storm and has crews on standby to quickly restore power if needed.
user: What's the best way to stay updated on the storm's progress?
assistant: You can check the local news website or download a weather app on your phone to get real-time updates on the storm's location and severity.
--------------------------------------------------
user: Hi.
assistant: Hello! How can I help you today?
user: I have a brother and a sister, and we always fight. Is that normal?
assistant: Yes, it's normal for siblings to fight sometimes. Sibling rivalry is a common thing in many families.
user: Why do you think we fight so much?
assistant: Siblings often fight because they may feel jealous or competing for attention from their parents. But it's also a way for siblings to learn how to resolve conflicts and develop their social skills.
user: That makes sense.
--------------------------------------------------