Manage your Datasets

​

Supported File Types

• Text files (.txt) - Plain text documents
• Markdown files (.md) - Formatted text documents with markup
• CSV files (.csv) - Comma-separated value spreadsheets
• JSON files (.json) - Structured data in JSON format
• Parquet files (.parquet) - Columnar storage format (not supported for Hugging Face imports)
• PDF files (.pdf) - Portable document format files
• Word documents (.docx) - Microsoft Word documents
• Excel spreadsheets (.xlsx) - Microsoft Excel workbooks
• JPEG images (.jpg, .jpeg) - Compressed image files
• PNG images (.png) - Portable network graphics
• GIF images (.gif) - Graphics interchange format
• BMP images (.bmp) - Bitmap image files
• TIFF images (.tiff) - Tagged image file format
• ZIP archives - Compressed folders containing multiple files

​

Automatic OCR Processing

• PDF files with scanned or non-selectable text
• Image files (JPG, PNG, TIFF, BMP, GIF) containing text
• Documents with embedded images

• Works with any base model you select for inline tuning
• No manual configuration needed
• Seamlessly integrated into the data processing pipeline

​

Upload Size Limit

• You can upload up to 100 MB of data at a time. This applies to single files, multiple files, or integration datasets. A progress bar will display the total upload size.
• To upload more data, simply reopen the dataset and upload the next 100 MB batch.
  There is no limit on the overall dataset size, only on each individual upload batch.

​

Adding Data to Existing Datasets

​

Why Continuous Data Updates Matter

​

How to Add New Data

​

From the Datasets tab

1. Navigate to the Datasets tab
2. Locate the dataset you want to update
3. Click on the dataset to open it
4. Click Upload Additional Data
5. Choose your data source: Upload Files, Integrations, or URL.
6. Upload your new data based on the selected source.

​

From the Model tab

1. Navigate to Models tab
2. Select the model you want to update with additional data.
3. Click the three-dot menu (⋮) in the Actions column for the model you want to update with additional data.
4. Click the Add Training Data button in the shortcuts section
5. Click Upload Additional Files
6. Choose your data source: Upload Files, Integrations, or URL.
7. Upload your new data based on the selected source.
8. The data is processed and added to your model’s knowledge graph automatically

​

From the Control Center

1. Open your model in the Control Center
2. Click the Add Training Data button in the shortcuts section
3. Click Upload Additional Files
4. Choose your data source: Upload Files, Integrations, or URL.
5. Upload your new data based on the selected source.
6. The data is processed and added to your model’s knowledge graph automatically

​

From the Playground tab

1. Navigate to the Playground tab
2. Select the model you want to interact with
3. While querying or testing your model, click the + icon next to the message box
4. Choose your data source: Attach File or Add URLs.
5. Upload your new data based on the selected source.
6. The data is automatically processed and integrated into your model’s knowledge graph

​

Best practices for Data Updates

​

Building Comprehensive Datasets

​

Example: Training on Customer Support Excellence

• Product interfaces - Screenshots of your software, dashboard views, error messages, feature locations
• Troubleshooting visuals - Common configuration issues, installation steps, system architecture diagrams
• Documentation - Annotated screenshots showing workflows, setup guides, integration diagrams
• Error states - What customers see when things go wrong, loading states, failure modes
• Customer-submitted images - Photos of hardware issues, setup problems, packaging damage
• Competitor products - Interface comparisons, feature differences, migration guides

• Product documentation - Technical specifications, API references, user guides, release notes
• Internal knowledge bases - Troubleshooting playbooks, known issues, workaround procedures
• Policy documents - SLA agreements, refund policies, terms of service, data privacy guidelines
• Training materials - Onboarding docs for new support agents, escalation procedures, quality standards
• Industry context - Regulatory compliance guides, security best practices, industry standards
• Best practices - Customer service frameworks, communication guidelines, de-escalation techniques
• Competitive intelligence - How competitors solve similar problems, market positioning, feature comparisons

• Support ticket history - Ticket IDs, timestamps, issue categories, resolution times, customer satisfaction scores
• Customer data - Account types, subscription tiers, usage patterns, feature adoption rates
• Product usage analytics - Most-used features, error rates, session durations, drop-off points
• Response metrics - First response time, resolution time, reopened tickets, escalation rates
• Customer sentiment - NPS scores, CSAT ratings, survey responses, sentiment analysis results
• Seasonal patterns - Ticket volume by time/day/season, spike events, capacity planning data
• Agent performance - Resolution rates, customer satisfaction per agent, specialization areas

• Resolved tickets - Successful interactions showing problem identification and resolution
• Escalated cases - Complex issues requiring multiple touchpoints or specialist involvement
• Edge cases - Unusual requests, policy exceptions, creative problem-solving examples
• Tone variations - Professional responses, empathetic communications, frustrated customer de-escalation
• Multi-channel interactions - Email threads, chat transcripts, phone call summaries, social media responses
• Follow-ups - Post-resolution check-ins, proactive outreach, account management communications

• Call recordings - Customer support calls showing tone, pacing, active listening, problem resolution
• Product demos - Video walkthroughs of features, setup processes, advanced use cases
• Training sessions - Internal workshops, role-playing scenarios, best practice reviews
• Customer feedback sessions - User interviews, usability testing, feature request discussions

• Product roadmap - Upcoming features, deprecation schedules, beta programs
• Billing systems - Invoice examples, pricing tiers, renewal processes, refund workflows
• Integration documentation - Third-party connections, API partnerships, data sync processes
• Company information - Team structure, hours of operation, regional support coverage, contact escalation paths
• Legal & compliance - GDPR requirements, data handling procedures, audit trails, security protocols

​

Why this Matters

• Contextual problem-solving that understands not just what the issue is, but why it matters and how it impacts the customer’s business
• Tone awareness from seeing thousands of interactions, knowing when to be technical vs empathetic, formal vs conversational
• Pattern recognition identifying common issues before customers fully describe them, predicting follow-up questions
• Operational intelligence understanding SLAs, escalation paths, when to involve specialists, and business constraints
• Proactive guidance suggesting solutions based on similar past cases, usage patterns, and product knowledge

​

Tutorial: Fine-Tune a Model with Hugging Face Datasets

​

What you’ll build

• Understand domain-specific terminology and concepts
• Extract relevant information from your training data
• Provide accurate, contextual responses to queries in your domain

​

Prerequisites

• A 4Minds account with access to the platform
• A Hugging Face account with a generated access token
• Basic understanding of the 4Minds model creation workflow

​

Fine-tuning overview

• Create custom models tailored to your domain (e.g., financial analysis, customer support)
• Train on proprietary datasets to improve accuracy for specific tasks
• Deploy models via API or test them in the interactive Playground
• Monitor performance metrics including response time, token speed, and success rate

​

Model status types

​

Selecting a base model

​

Training data best practices

1. Provide diverse examples – Include variations of similar questions to improve generalization
2. Maintain consistency – Use a consistent format and tone across all training samples
3. Include edge cases – Add examples of boundary conditions and unusual queries
4. Quality over quantity – 500 high-quality examples often outperform 5,000 poor ones

​

Step 1: Access the data upload screen

1. Under Choose Data Source, ensure the Upload New Data tab is selected
2. You have three options under Add Files from Sources:
   • Upload Files - Local files from your computer
   • Integrations - External data sources
   • URL - Import from a web address
3. To import from Hugging Face, click the Integrations button

​

Step 2: Select Hugging Face integration

1. Scroll down and select Hugging Face from the list

​

Step 3: Search for your dataset

1. Enter your search query in the search bar (e.g., “finQA”)
2. Click Search
3. Browse the results using the available tabs:
   • Popular Datasets – Trending datasets on Hugging Face
   • My Datasets – Your personal Hugging Face datasets
   • Search Results – Results matching your query

• Dataset name and author
• Description
• Download count
• Size and format
• Task type and modality

​

Step 4: Configure dataset import settings

• Name and author
• Description
• Download statistics
• Task IDs, size, and format

• Configuration – Select the dataset configuration (e.g., “Default”)
• Split – Choose which data split to import (e.g., “Test”, “Train”, “Validation”)

​

Step 5: Review attached files

• File name
• Source (Hugging Face icon)
• File size
• Row count

• relevance.jsonl – 66.68 KB, 341 rows
• queries.jsonl – 137.71 KB, 705 rows
• corpus.jsonl – 1.44 MB, 7549 rows

​

Step 6: Review and launch training

​

Step 7: Monitor training progress

• Status – “New” badge with “Building Graph” progress indicator
• Parameters – Model size (e.g., 14b)
• Base – Base model used (e.g., Phi)
• Created – Timestamp

​

Step 8: Test in the Playground

1. From the model dashboard, click the ⋮ menu on any model
2. Select Run Model
3. Or navigate to Control Center → Playground and select your model

• Real-time responses – See model outputs as they generate
• Conversation history – Maintain context across multiple turns
• View Graph – Visualize model reasoning and token flow
• Clear All Chats – Reset the conversation history
• Add Model – Compare multiple models side-by-side

• “What is the ratio of operating income to total revenue?”
• “What is the total of all lease obligations?”
• “What was the percentage change in revenue from 2018 to 2019?”

​

Model actions

​

API integration

1. Click ⋮ on your model
2. Select API Access
3. Copy your API endpoint and authentication token

curl -X POST https://api.4minds.ai/v1/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "customer-faq-expert",
    "prompt": "How do I reset my password?",
    "max_tokens": 500
  }'

​

Performance optimization

1. Add more training data – Expand coverage of your use case
2. Refine existing data – Remove low-quality or contradictory examples
3. Adjust the persona – Use “Technical Expert” for specialized domains
4. Choose appropriate model size – Larger models (70b) handle complex reasoning better

1. Use smaller base models – Phi (14b) offers faster inference
2. Optimize prompt length – Shorter prompts reduce processing time
3. Enable caching – Reuse responses for common queries

​

Troubleshooting

​

FAQs

​

Supported file formats

​

Tips

• Choose appropriate splits – For fine-tuning, you typically want the “Train” split. Use “Test” or “Validation” for evaluation datasets.
• Check dataset size – Larger datasets may take longer to import and process.
• Enable Rsync – If you’re working with frequently updated datasets, enable Rsync to stay current automatically.

​

Tutorial: Fine-Tune a Model with Hugging Face Datasets via API

​

What you’ll build

​

Prerequisites

• A 4Minds account with API access
• Your API key (found in Account Settings)
• A Hugging Face account with a generated access token
• Python 3.7+ with the requests and datasets libraries installed

​

Step 1: Download the FinQA dataset from Hugging Face

from datasets import load_dataset
import json

# Load the FinQA dataset
dataset = load_dataset("ibm/finqa", split="train")

# Convert to JSON format for upload
data = [{"question": item["question"], "answer": item["answer"]} for item in dataset]

# Save to a local file
with open("finqa_training_data.json", "w") as f:
    json.dump(data, f, indent=2)

print(f"Saved {len(data)} training examples to finqa_training_data.json")

​

Step 2: Upload the dataset to 4Minds

import requests

API_KEY = "your_api_key_here"
BASE_URL = "https://api.4minds.ai/api/v1"

headers = {
    "Authorization": f"Bearer {API_KEY}",
}

# Create a new dataset with the uploaded file
with open("finqa_training_data.json", "rb") as f:
    response = requests.post(
        f"{BASE_URL}/user/dataset",
        headers=headers,
        files={"file": ("finqa_training_data.json", f, "application/json")},
        data={"name": "FinQA Training Data"}
    )

dataset_response = response.json()
dataset_id = dataset_response["id"]
print(f"Created dataset with ID: {dataset_id}")

​

Step 3: Create a model with the dataset attached

# Create a new model with the dataset
model_payload = {
    "name": "Financial QA Assistant",
    "description": "Fine-tuned on FinQA dataset for financial question answering",
    "dataset_id": dataset_id
}

response = requests.post(
    f"{BASE_URL}/user/model",
    headers={**headers, "Content-Type": "application/json"},
    json=model_payload
)

model_response = response.json()
model_id = model_response["id"]
print(f"Created model with ID: {model_id}")
print(f"Training status: {model_response['status']}")

​

Step 4: Monitor training progress

import time

while True:
    response = requests.get(
        f"{BASE_URL}/user/model/{model_id}",
        headers=headers
    )
    status = response.json()["status"]
    print(f"Training status: {status}")
    
    if status == "ready":
        print("Training complete!")
        break
    elif status == "failed":
        print("Training failed. Check the dashboard for details.")
        break
    
    time.sleep(30)  # Check every 30 seconds

​

Step 5: Test your model via API

# Send a test query
inference_payload = {
    "model_id": model_id,
    "message": "What was the revenue growth percentage year-over-year?"
}

response = requests.post(
    f"{BASE_URL}/user/model/{model_id}/inference",
    headers={**headers, "Content-Type": "application/json"},
    json=inference_payload
)

print("Model response:")
print(response.json()["response"])

​

Complete script

from datasets import load_dataset
import requests
import json
import time

# Configuration
API_KEY = "your_api_key_here"
BASE_URL = "https://api.4minds.ai/api/v1"
headers = {"Authorization": f"Bearer {API_KEY}"}

# Step 1: Download FinQA from Hugging Face
print("Downloading FinQA dataset...")
dataset = load_dataset("ibm/finqa", split="train")
data = [{"question": item["question"], "answer": item["answer"]} for item in dataset]
with open("finqa_training_data.json", "w") as f:
    json.dump(data, f, indent=2)

# Step 2: Upload to 4Minds
print("Uploading dataset to 4Minds...")
with open("finqa_training_data.json", "rb") as f:
    response = requests.post(
        f"{BASE_URL}/user/dataset",
        headers=headers,
        files={"file": ("finqa_training_data.json", f, "application/json")},
        data={"name": "FinQA Training Data"}
    )
dataset_id = response.json()["id"]

# Step 3: Create model
print("Creating model...")
response = requests.post(
    f"{BASE_URL}/user/model",
    headers={**headers, "Content-Type": "application/json"},
    json={
        "name": "Financial QA Assistant",
        "description": "Fine-tuned on FinQA for financial Q&A",
        "dataset_id": dataset_id
    }
)
model_id = response.json()["id"]

# Step 4: Wait for training
print("Waiting for training to complete...")
while True:
    response = requests.get(f"{BASE_URL}/user/model/{model_id}", headers=headers)
    status = response.json()["status"]
    if status == "ready":
        break
    time.sleep(30)

# Step 5: Test the model
print("Testing model...")
response = requests.post(
    f"{BASE_URL}/user/model/{model_id}/inference",
    headers={**headers, "Content-Type": "application/json"},
    json={"message": "What was the revenue growth percentage?"}
)
print(f"Response: {response.json()['response']}")