Jobs Failing

Troubleshoot failed jobs and understand error messages

Common Failure Types

Timeout

Error: "Execution timed out after X seconds"

Cause: Job took longer than the allowed time.

Solutions:

  • Reduce input size
  • Optimize your code
  • Request longer timeout in job parameters
  • Break job into smaller chunks

Out of Memory

Error: "OutOfMemoryError" or "MemoryError"

Cause: Job needed more RAM than available.

Solutions:

  • Process data in smaller batches
  • Use memory-efficient algorithms
  • Request nodes with more RAM in hardware requirements
  • Reduce model size (for ML workloads)

Invalid Parameters

Error: "Invalid parameters: X is required"

Cause: Missing or malformed job parameters.

Solutions:

  • Check the job type required parameters
  • Ensure parameters are correctly formatted
  • Validate input data before submission

File Not Found

Error: "File not accessible" or "URL returned 404"

Cause: Input file URL is invalid or inaccessible.

Solutions:

  • Verify the URL is correct
  • Ensure file is publicly accessible
  • Check file has not been deleted
  • Re-upload file and use new URL

Model Load Error

Error: "Failed to load model" or "Model not found"

Cause: ML model could not be loaded.

Solutions:

  • Verify model name/path is correct
  • Try a smaller model variant

Execution Error

Error: Various Python exceptions

Cause: Error in workload code or data.

Solutions:

  • Check the error traceback in job details
  • Validate input data format
  • For custom workloads: test code locally first

Viewing Error Details

Via Dashboard

Go to fabric.carmel.so

Navigate to Jobs

Find the failed job

Click to view details

Error message and traceback are shown

Retrying Failed Jobs

Via Dashboard:

Open the failed job

Click "Retry"

Job is resubmitted with same parameters

Preventing Failures

Validate Input

Before submitting:

  • Check file URLs are accessible
  • Ensure data is in expected format
  • Validate required fields are present

Set Appropriate Timeout

For long-running jobs, set a longer timeout in your job parameters.

Test Locally

For custom workloads:

Test with same Python version

Use same dependencies

Verify with sample data

Job Failure Rates

Some failure is normal:

  • Node crashes
  • Network interruptions
  • Resource contention

Failed jobs may be automatically retried on different nodes depending on failure type.

When to Contact Support

Contact support if:

  • All jobs of a specific type fail
  • Error message is unclear
  • Issue persists after fixes
  • You suspect a platform problem

Include:

  • Job IDs that failed
  • Error messages
  • Steps you have tried