Дообучаем Llama 3.1 у себя дома

%pip install -U transformers # для использования модели
%pip install -U datasets # для подгрузки тестового датасетов
%pip install -U accelerate # вспомогательная библиотека
%pip install -U peft # QLoRA
%pip install -U trl # super-vised тренер
%pip install -U bitsandbytes # оптимизация весов модели
%pip install -U wandb # храним логи правильно

# Импорт библиотек
import os, torch, wandb

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    HfArgumentParser,
    TrainingArguments,
    pipeline,
    logging,
)
from peft import (
    LoraConfig,
    PeftModel,
    prepare_model_for_kbit_training,
    get_peft_model,
)

from datasets import load_dataset
from trl import SFTTrainer, setup_chat_format
from dataclasses import dataclass

# Настройка Weights & Bias для логирования при обучении

from huggingface_hub import login
from kaggle_secrets import UserSecretsClient
user_secrets = UserSecretsClient()

hf_token = user_secrets.get_secret("huggingface_token")

login(token = hf_token)

wb_token = user_secrets.get_secret("wandb_api_key")

wandb.login(key=wb_token)
run = wandb.init(
    project='Fine-tune Llama 3.1 8B on Medical Dataset', 
    job_type="training", 
    anonymous="allow"
)

# Вспомогательный класс для сохранения атрибутов

@dataclass
class Config:
    model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct"
    dataset_name = "ruslanmv/ai-medical-chatbot"
    new_model = "llama-3.1-8b-chat-doctor"
    torch_dtype = torch.float16
    attn_implementation = "eager"


# напишем вспомогательный класс для хранений атрибутов
cfg = Config() 

# cfg можно будет потом использовать вот так удобно
tokenizer = AutoTokenizer.from_pretrained(cfg.model_name)

model, tokenizer = setup_chat_format(model, tokenizer)
tokenizer.padding_side = 'right'
tokenizer.padding_token = '<|pad|>'

# Настройка адаптера (QLoRa) для быстрого обучения
# QLoRA config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch_dtype,
    bnb_4bit_use_double_quant=True,
)

# Load model
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation=attn_implementation
)

# LoRA config / адаптер 
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=['up_proj', 'down_proj', 'gate_proj', 'k_proj', 'q_proj', 'v_proj', 'o_proj']
)
# Сам адаптер удобно использовать при создании тренера,
# так что читаем дальше

# Подготовка датасета
def format_chat_template(row):
    row_json = [{"role": "user", "content": row["Patient"]},
               {"role": "assistant", "content": row["Doctor"]}]
    row["text"] = tokenizer.apply_chat_template(row_json, tokenize=False)
    return row


# Библиотека Datasets 🤗 очень приятна в использовании, 
# когда используете LLM. Так что берите на вооружение, чемпионы!
dataset = load_dataset(cfg.dataset_name, split="all")

dataset = dataset.map(
	format_chat_template,
	num_proc=4,
)


# Также для учебных целей урежем исходный датасет,
# чтобы сравнить результаты - читаем в самом конце
dataset_sh = dataset.shuffle(seed=2024).select(range(10_000))

I'm sorry to hear that you're experiencing severe headaches. While I can provide some general advice, it's important to note that I’m not a substitute for medical advice from a healthcare professional. If your headache is sudden, severe, or accompanied by other symptoms like vision changes, confusion, difficulty speaking, or weakness, you should seek immediate medical attention, as these could be signs of a serious condition.

As a medical professional, I would conduct a thorough evaluation to determine the cause of your headaches. This would include:

1. Medical history: A thorough review of your medical history, including any previous diagnoses, medications, and allergies.

2. Physical examination: A physical examination to assess your overall health and look for any signs of underlying conditions that may be contributing to your headaches.

3. Imaging tests: Imaging tests such as CT or MRI scans may be ordered to rule out any underlying conditions such as tumors, aneurysms, or other structural abnormalities.

4. Blood tests: Blood tests may be ordered to check for any underlying conditions such as infections, inflammation, or hormonal imbalances.

Treatment Options:

Based on the results of the evaluation, the following treatment options may be recommended:

1. Pain relief medications: Over-the-counter or prescription pain medications such as triptans, ergots, or corticosteroids may be recommended to help manage headache pain.

2. Migraine medications: If you are diagnosed with migraines, medications such as triptans or ergots may be prescribed to help manage symptoms.

3. Muscle relaxants: Muscle relaxants such as cyclobenzaprine or baclofen may be prescribed to help relax tense muscles and reduce headache pain.

4. Antidepressants: Antidepressants such as amitriptyline or nortriptyline may be prescribed to help manage stress and anxiety that may be contributing to headaches.

5. Lifestyle modifications: Lifestyle modifications such as maintaining a consistent sleep schedule, staying hydrated, and avoiding triggers may be recommended to help manage headaches.

Headache Prevention:

To prevent headaches, the following strategies may be recommended:

1. Maintain a headache diary: Keeping a headache diary can help identify triggers and patterns of headache occurrence.

2. Avoid triggers: Avoiding triggers such as certain foods, bright lights, or loud noises can help prevent headaches.

3. Stay hydrated: Drinking plenty of water throughout the day can help prevent dehydration headaches.

4. Exercise regularly: Regular exercise can help reduce stress and anxiety and improve overall health.

5. Manage stress: Stress management techniques such as meditation, deep breathing, or yoga can help reduce stress and anxiety that may contribute to headaches.

I can't provide medical advice or diagnose skin conditions, including cancer. If you are concerned about a mole, I suggest you schedule an appointment with a dermatologist. They can examine the mole, take a medical history, and perform any necessary tests to determine if the mole is cancerous.

As a dermatologist, I must emphasize that while it's possible to evaluate your mole at home, a thorough examination by a dermatologist is essential for an accurate diagnosis. That being said, I'll provide you with a detailed guide on how to assess your mole at home, and what to look for.

I use the ABCDE rule to evaluate moles for potential cancer risk:

1. A - Asymmetry: If the mole is not symmetrical, it could be a sign of cancer. Imagine drawing a line down the middle of the mole. If the two sides are not mirror images of each other, it's asymmetrical.

2. B - Border: A mole with an irregular, notched, or scalloped border may be a sign of cancer. A healthy mole has a smooth, well-defined border.

3. C - Color: Moles with multiple colors or an uneven color distribution could be a sign of cancer. Healthy moles are usually one color, such as brown, tan, or black.

4. D - Diameter: Moles that are larger in diameter than a pencil eraser (about 6mm) could be a sign of cancer. However, size alone is not a reliable indicator of cancer.

5. E - Evolving: If the mole is changing in size, shape, or color, it could be a sign of cancer.