> ## Documentation Index
> Fetch the complete documentation index at: https://docs.phones.farm/llms.txt
> Use this file to discover all available pages before exploring further.

# Graphics & OCR

> Computer vision API endpoints: multi-point color finding, template matching, and Apple Intelligence neural OCR.

## 1.Find Color

Find specified multiple color points in device screenshot, supports relative and absolute coordinate modes, configurable similarity threshold and finding all matching locations.

💡 Quick Color Picker

You can use the quick color picker in the console developer tools to quickly get screen coordinates and color value arrays.

![Quick Color Picker Tool](https://iosclick.com/images/dashboard_color_picker.png)

### Endpoint Description

#### Endpoint Type

```
findColor
```

#### Parameters

| Parameter | Type | Required | Description |
| - | - | - | - |
| deviceId | string | Yes | Device ID |
| colors | array | Yes | Color array, format: `[[x, y, "color"], ...]` or `[[x, y, "color\|similarity"], ...]` |
| rect | array | No | Search area `[x1, y1, x2, y2]`, required in relative coordinate mode, not needed in absolute coordinate mode |
| multiple | boolean | No | Whether to find all matching locations, default false, not needed in absolute coordinate mode |
| spacing | number | No | Minimum distance between results (pixels), default 50, only effective when multiple=true |

#### Response Payload

**Single Match (multiple = false or absolute coordinate mode):**

```
{
    x: 100,  // X coordinate of match point
    y: 200   // Y coordinate of match point
}
```

**Multiple Matches (multiple = true):**

```
[
    { x: 100, y: 200 },
    { x: 150, y: 250 },
    { x: 200, y: 300 }
]
```

**No Match Found:**

```
null
```

### Feature Description

#### Coordinate Modes

`findColor` supports two coordinate modes:

**1. Absolute Coordinate Mode**

In absolute coordinate mode, all points use screen absolute coordinates:

* **No need for `rect` and `multiple` parameters**
* If all color points match, returns the first point's coordinates

**Example:**

```
// Absolute coordinates: all points are screen absolute coordinates
colors: [
    [245, 509, "00B4F9|1"],   // Absolute coordinate (245, 509)
    [284, 745, "A49696|0.95"], // Absolute coordinate (284, 745)
    [290, 911, "9B8C89"]      // Absolute coordinate (290, 911)
]
```

**2. Relative Coordinate Mode**

When the first point of the color array is `[0, 0]`, relative coordinate mode is used:

* The first point `[0, 0]` serves as the reference point, searched within the `rect` area
* Other points' coordinates are relative to the first point
* **Can provide `rect` parameter** to define search area
* Supports `multiple` parameter to find all matches

**Example:**

```
// Relative coordinates: first point is [0,0], other points relative to first point
colors: [
    [0, 0, "00B4F9|1"],      // Reference point
    [39, 236, "A49696|0.95"], // Relative offset from reference point (39, 236)
    [45, 402, "9B8C89"]      // Relative offset from reference point (45, 402)
]
```

### Basic Usage

#### Example 1: Absolute Coordinate Mode

```
// Verify if multiple absolute coordinate points on screen match
const result = await apiInvoke('findColor', {
    deviceId: 'P72578581E07',
    colors: [
        [245, 509, "00B4F9|1"],   // Absolute coordinate point 1
        [284, 745, "A49696|0.95"], // Absolute coordinate point 2
        [290, 911, "9B8C89"]      // Absolute coordinate point 3
    ]
    // No need for rect and multiple parameters
});

if (result) {
    console.log(`All color points match, reference point at (${result.x}, ${result.y})`);
} else {
    console.log('Color points do not match');
}
```

#### Example 2: Relative Coordinate Mode - Find All Matches

```
// Find all matching locations
const results = await apiInvoke('findColor', {
    deviceId: 'P72578581E07',
    colors: [
        [0, 0, "FFFFFF|0.9"],
        [10, 10, "000000|0.9"]
    ],
    rect: [0, 0, 800, 1200],
    multiple: true,
    spacing: 50  // Minimum distance between results
});

if (results && results.length > 0) {
    console.log(`Found ${results.length} matching points`);
    results.forEach((point, index) => {
        console.log(`Match point ${index + 1}: (${point.x}, ${point.y})`);
    });
}
```

## 2.Find Image

Use OpenCV template matching technology to find specified image template in device screenshot, supports similarity control and multiple results finding.

### Endpoint Description

#### Endpoint Type

```
findImage
```

#### Parameters

| Parameter | Type | Required | Description |
| - | - | - | - |
| deviceId | string | Yes | Device ID |
| sprite | string | Yes | Template image (base64 string) |
| threshold | number | No | Similarity threshold (0-1), default 0.9 |
| rect | array | No | Search area `[x1, y1, x2, y2]`, default full screen |
| direction | number | No | Search direction 0-3, default 0 |
| multiple | boolean | No | Whether to find all matches, default false |
| distance | number | No | Minimum distance between results (pixels), default 8 |

#### Response Payload

**Single Match (multiple = false):**

```
{
    x: 100,          // Match area center point X coordinate
    y: 200,          // Match area center point Y coordinate
    confidence: 0.95 // Similarity (0-1)
}
```

**Multiple Matches (multiple = true):**

```
[
    { x: 100, y: 200, confidence: 0.95 },
    { x: 150, y: 250, confidence: 0.92 },
    { x: 200, y: 300, confidence: 0.90 }
]
// Sorted by similarity in descending order, max 88 results
```

### Feature Description

#### Search Direction

| direction | Description | Applicable Scenario |
| - | - | - |
| 0 | Best match | Find highest similarity position (default) |
| 1 | Right to left | Prefer finding target on right side of screen |
| 2 | Bottom to top | Prefer finding target at bottom of screen |
| 3 | Bottom-right to top-left | Prefer finding target at bottom-right corner |

#### Similarity Threshold

`threshold` controls matching precision:

* `1.0` - Exactly the same (hard to match)
* `0.95` - Almost identical
* `0.9` - Highly similar (recommended, default)
* `0.85` - Similar
* `0.8` - Basically similar
* `< 0.8` - Possible false match

#### Multiple Results Finding

Use `multiple = true` to find all matching locations:

* Automatically sorted by similarity in descending order
* Use `distance` parameter to filter results too close together
* Max 88 results returned

💡 Tip

Image matching uses OpenCV template matching algorithm based on grayscale images, so it has certain tolerance for color changes. It's recommended that template images contain obvious features and avoid solid colors or repetitive patterns to improve matching accuracy and speed.

## 3.Find Images (Multi)

Search multiple template images in a single screenshot using OpenCV template matching. Compared with `findImage`, this interface accepts a `sprites` array and returns the match result with the corresponding template `index`.

### Endpoint Description

#### Endpoint Type

```
findImages
```

#### Parameters

| Parameter | Type | Required | Description |
| - | - | - | - |
| deviceId | string | Yes | Device ID |
| sprites | array | Yes | Template images array (base64 string array) |
| threshold | number | No | Similarity threshold (0-1), default 0.9 |
| rect | array | No | Search area `[x1, y1, x2, y2]`, default full screen |
| direction | number | No | Search direction 0-3, default 0 |
| findAll | boolean | No | Whether to return all matches, default false (returns as soon as any match is found) |
| distance | number | No | Minimum distance between results (pixels), default 8 |

#### Response Payload

**Best match (findAll = false):**

* Returns `null` if nothing matches.
* Returns an object when matched:

```
{
    x: 100,          // Match center X
    y: 200,          // Match center Y
    confidence: 0.95,// Similarity (0-1)
    index: 0         // Index of matched template in sprites
}
```

**All matches (findAll = true):**

* Returns an empty array `[]` if nothing matches.
* Returns an array of results when matched:

```
[
    { x: 100, y: 200, confidence: 0.95, index: 0 },
    { x: 150, y: 250, confidence: 0.92, index: 2 },
    { x: 200, y: 300, confidence: 0.90, index: 1 }
]
// Sorted by confidence in descending order, max 88 results
```

### Feature Description

#### Search Strategy (findAll = false)

* When `direction = 0` (default): picks the global best match across all `sprites`.
* When `direction != 0`: tries templates in `sprites` order, returns the first template that matches according to the direction preference.

#### Search Direction

| direction | Description | Applicable Scenario |
| - | - | - |
| 0 | Best match | Find highest similarity position (default) |
| 1 | Right to left | Prefer finding target on right side of screen |
| 2 | Bottom to top | Prefer finding target at bottom of screen |
| 3 | Bottom-right to top-left | Prefer finding target at bottom-right corner |

#### Similarity Threshold

`threshold` controls matching precision:

* `1.0` - Exactly the same (hard to match)
* `0.95` - Almost identical
* `0.9` - Highly similar (recommended, default)
* `0.85` - Similar
* `0.8` - Basically similar
* `< 0.8` - Possible false match

#### Result De-duplication (distance)

When `findAll = true`, results that are too close (both X and Y within `distance` pixels) will be filtered to reduce duplicates, and only the higher-confidence point is kept.

💡 Tip

To improve accuracy and speed, keep template images small and distinctive (avoid large solid-color areas or repetitive patterns). It’s recommended to set `rect` to limit the search area whenever possible.

## 4.Get Color

Get the hexadecimal color value of a specific coordinate from the latest device screenshot.

### Endpoint Description

#### Endpoint Type

```
getColor
```

#### Parameters

| Parameter | Type | Required | Description |
| - | - | - | - |
| deviceId | string | Yes | Target device ID |
| x | number | Yes | X coordinate (0 ≤ x \< screen width) |
| y | number | Yes | Y coordinate (0 ≤ y \< screen height) |

#### Return Value

Returns a string in `RRGGBB` format (uppercase), representing the RGB color of the requested pixel. Example: `"2A9DF4"`.

### Basic Usage

```
const color = await apiInvoke('getColor', {
    deviceId: 'P72578581E07',
    x: 128,
    y: 256
});

console.log('Pixel color:', color); // e.g. "2A9DF4"
```

## 5.Compare Color

Check whether the pixel at a specific coordinate in the latest screenshot matches the expected color, with optional channel deviation.

### Endpoint Description

#### Endpoint Type

```
cmpColor
```

#### Parameters

| Parameter | Type | Required | Description |
| - | - | - | - |
| deviceId | string | Yes | Target device ID |
| x | number | Yes | X coordinate (0 ≤ x \< screen width) |
| y | number | Yes | Y coordinate (0 ≤ y \< screen height) |
| color | string | Yes | **6-digit hex color only**, e.g. **PROTECTED\_BLOCK\_80** |
| threshold | number | No | Default 1. Similarity = **PROTECTED\_BLOCK\_81** with the same value applied to all three channels |

#### Return Value

Returns `true` if the pixel color falls within the tolerance of any segment; otherwise `false`

## 6.Match Color Point

Check if all specified color points exist in the specified region of device screenshot, supports similarity threshold settings. Returns `true` if all colors are found, otherwise throws an error indicating which colors were not found.

### Endpoint Description

#### Endpoint Type

```
matchColorPoint
```

#### Parameters

| Parameter | Type | Required | Description |
| - | - | - | - |
| deviceId | string | Yes | Device ID |
| colors | array | Yes | Color array, format: \`\[color |
| rect | array | No | Search area `[x1, y1, x2, y2]`, default full screen |

#### Response Payload

**On Success:**

```
true  // All colors found in specified region
```

**On Failure:**

```
null
```

### Feature Description

#### Color Format

Each element in the color array has the format: `"color|threshold"`

* **Color**: 6-digit hexadecimal color value, e.g., `"FFFFFF"`
* **Threshold**: Similarity threshold, range 0-1, similarity = `round((1 - threshold) * 255)`, same for all three channels

#### Search Area

The `rect` parameter defines the search area:

* Format: `[x1, y1, x2, y2]`

### Use Cases

#### Case 1: Verify UI State

Check if system modal exists:

```
const _hasSysModal = await apiInvoke('matchColorPoint', {
    deviceId: 'P72578581E07',
    colors: ["FFFFFF|0.8", "222222|0.9", "1074EC|0.9"],
    rect: [200, 300, 400, 350]
});
```

💡 Tip

* Unlike `findMultiColor`, `matchColorPoint` does not return specific coordinates, only verifies if colors exist
* Suitable for UI state verification, conditional checks, and similar scenarios

## 7.Find Color(DM Tools Version)

Find specified colors and their offset colors in device screenshot, supports multi-point color recognition and tolerance matching.

### Endpoint Description

#### Endpoint Type

```
findMultiColor
```

#### Parameters

| Parameter | Type | Required | Description |
| - | - | - | - |
| deviceId | string | Yes | Device ID |
| color | string | Yes | Main color (hex format, e.g., `"FFFFFF"`) |
| offsetColor | string | Yes | Offset color group |
| rect | array | No | Search area `[x1, y1, x2, y2]`, default full screen |
| threshold | number | No | Color tolerance (0-1), default 0.9 |
| multiple | boolean | No | Whether to find all matching locations, default false |
| direction | number | No | Search direction 0-3, default 0 |

#### Response Payload

**Single Match (multiple = false):**

```
{
    x: 100,  // X coordinate of match point
    y: 200   // Y coordinate of match point
}
```

**Multiple Matches (multiple = true):**

```
[
    { x: 100, y: 200 },
    { x: 150, y: 250 },
    { x: 200, y: 300 }
]
```

### Basic Usage

```
// Find single color point
const result = await apiInvoke('findMultiColor', {
    deviceId: 'P72578581E07',
    color: '000000',              // Main color
    offsetColor: '10|0|f6f6f6'    // Offset color
});

if (result) {
    console.log(`Found color point at (${result.x}, ${result.y})`);
}
```

### Feature Description

#### Color Format

Colors use hexadecimal format:

* `888888` - Gray

#### Offset Color Format

Offset colors for multi-point color matching, format: `"x|y|color,x|y|color,..."`

#### Search Direction

| direction | Description |
| - | - |
| 0 | Left to right, top to bottom (default) |
| 1 | Right to left, top to bottom |
| 2 | Left to right, bottom to top |
| 3 | Right to left, bottom to top |

#### Color Tolerance

`threshold` parameter controls color matching accuracy:

* `1.0` - Perfect match, no deviation allowed
* `0.9` - Allow about 10% color deviation (recommended)
* `0.8` - Allow about 20% color deviation
* `0.7` - Allow about 30% color deviation

💡 Tip

Multi-point color matching is more accurate than single point. It's recommended to use 3-5 feature points to identify UI elements, effectively avoiding false matches. Offset color coordinates are relative to the main color point, pay attention to relative position when setting.

## 8.OCR Recognition

Use device-side GPU acceleration technology to recognize text content in specified screen area, supports multiple language recognition.

> [!NOTE]
>
> 💡 Tip
>
> System automatically recognizes phone language, no need to manually specify language type.
>
> Why not integrate third-party OCR? See [OCR Guide](https://docs.phones.farm/iphone-farm/common-issues.md) for details.

> [!WARNING]
>
> ⚠️ Important Limitation
>
> Due to iOS system's strict memory limits for background apps, **built-in OCR is not suitable for large-area document scanning work**.
>
> Recognizing too large areas or frequently recognizing large areas may cause:
>
> * Memory usage exceeds limit
> * Screen mirroring service crashes and disconnects
>
> **Recommendations:**
>
> * ✅ Suitable: Recognize buttons, titles, short text and other small area text
> * ❌ Not suitable: Full-page document scanning, large paragraph recognition, PDF text extraction, etc.
>
> For document scanning or large-area text recognition, use [Screen Push](#screen-image-push) interface with computer-side OCR service.

### Endpoint Description

#### Endpoint Type

```
ocr
```

#### Parameters

| Parameter | Type | Required | Description |
| - | - | - | - |
| deviceId | string | Yes | Device ID |
| rect | array | Yes | Recognition area `[x1, y1, x2, y2]` |
| model | string | No | auto paddle vision device |

#### Response Payload

Returns an array of text blocks, each text block contains:

```
[
    {
        text: "Recognized text",         // Recognized text content
        boundingBox: {                   // Text bounding box
            x: 100,                      // Top-left X coordinate
            y: 100,                      // Top-left Y coordinate
            width: 200,                  // Width
            height: 50                   // Height
        },
        confidence: 0.95,                // Recognition confidence (0-1)
        x: 200,                          // Text center point X coordinate
        y: 125                           // Text center point Y coordinate
    },
    // ... more text blocks
]
```

### Basic Usage

```
// Recognize text in specified screen area
const results = await apiInvoke('ocr', {
    deviceId: 'P72578581E07',
    rect: [100, 100, 500, 200]    // Recognition area: top-left(100,100) to bottom-right(500,200)
});

// Iterate through all recognized text blocks
results.forEach(item => {
    console.log(`Text: ${item.text}`);
    console.log(`Position: (${item.x}, ${item.y})`);
    console.log(`Confidence: ${item.confidence}`);
});

// Get all text content
const allText = results.map(item => item.text).join(' ');
console.log('Full text:', allText);
```

### Feature Description

#### Recognition Area

`rect` parameter defines recognition area, format is `[x1, y1, x2, y2]`:

* `x1, y1` - Top-left corner coordinates of area
* `x2, y2` - Bottom-right corner coordinates of area

**Area Setting Recommendations:**

* Area should contain complete text content
* Avoid including too much background and interference elements
* **Smaller area = faster recognition, less memory usage**
* Ensure area is within screen bounds
* **Avoid recognizing too large area** (recommend single recognition area not exceeding 1/2 of screen)

#### Language Support

System automatically detects text language, including but not limited to **18+ major languages**

### Use Cases

> [!NOTE]
>
> 💡 Suitable Scenarios
>
> Built-in OCR is suitable for recognizing:
>
> * ✅ UI interface button text, titles, prompts
> * ✅ Short message content, notification text
> * ✅ In-game text prompts, number information
> * ✅ Form fields, input box content
>
> **Not suitable:**
>
> * ❌ Full-page document scanning
> * ❌ Long article recognition
> * ❌ PDF text extraction
> * ❌ Large-area continuous text recognition
>
> For document scanning scenarios, it's recommended to use [Screen Push ](#screen-image-push)interface to send images to computer-side OCR service for processing.

## 9.Get Screenshot

Get real-time screenshot of specified device, supports various image formats and quality settings.

### Endpoint Description

#### Endpoint Type

```
getScreenShot
```

#### Parameters

| Parameter | Type | Required | Description |
| - | - | - | - |
| deviceId | string | Yes | Device ID |
| quality | number | No | Image quality (0-1), default 0.8 |
| format | string | No | Return format: `'bmp'` or default format |

#### Response Payload

Returns binary image data:

### Basic Usage

```
// Get screenshot (default settings)
const screenshot = await apiInvoke('getScreenShot', {
    deviceId: 'P72578581E07'
});

// Save as file
const fs = require('fs');
fs.writeFileSync('screenshot.png', screenshot);
```

## 10.Crop Screen Shot

Crop a specified area from the device screenshot and return the cropped image.

Important Note

This method **only returns BMP image format**, other formats are not supported.

### Endpoint Description

#### Endpoint Type

```
cropScreenShot
```

#### Parameters

| Parameter | Type | Required | Description |
| - | - | - | - |
| deviceId | string | Yes | Device ID |
| rect | number\[] | Yes | Crop area, format: `[x1, y1, x2, y2]`, unit: pixels |

#### Response Payload

Returns cropped image binary data (**BMP format**):

### Basic Usage

```
// Crop screenshot
// rect: [x1, y1, x2, y2] - top-left and bottom-right coordinates
const croppedImage = await apiInvoke('cropScreenShot', {
    deviceId: 'P72578581E07',
    rect: [100, 100, 300, 400]  // Area from (100, 100) to (300, 400)
});

// Save as file (Note: returns BMP format)
const fs = require('fs');
fs.writeFileSync('cropped.bmp', croppedImage);
```

### Notes

1. **Format Limitation**: This method always returns BMP format, other formats cannot be specified
2. **Coordinate Range**: Coordinates and dimensions in `rect` parameter must not exceed screen bounds
3. **Parameter Format**: `rect` must be an array of 4 numbers: `[x1, y1, x2, y2]`, where (x1, y1) is the top-left coordinate and (x2, y2) is the bottom-right coordinate

## 11.Screen Image Push

Push specified screen area image directly to custom API service for processing, suitable for various image recognition scenarios.

### Application Scenarios

This interface is particularly suitable for the following scenarios:

* 🔤 **OCR Text Recognition** - Integrate third-party OCR services (such as PaddleOCR, Tesseract, etc.)
* 🎯 **Captcha Recognition** - Use specialized captcha recognition services
* 🖼️ **Image Classification** - Identify image content, object detection
* 🤖 **AI Recognition** - Integrate any custom image AI model

💡 How It Works

**Streaming End → Capture Image → Direct POST → Your API → Return Result**

Images are sent directly from the streaming end to your API, saving decoding and redundant transmission steps, improving efficiency.

### Difference from getScreenShot

#### Comparison

| Feature | screenPush | getScreenShot |
| - | - | - |
| **Data Flow** | Streaming end → Your API | Streaming end → iClick main process → Caller |
| **Transmissions** | 1 (direct push) | 2 (first to main process, then to API) |
| **Resource Usage** | Low (not through main process) | High (requires decoding and passing in main process) |

### Endpoint Description

#### Endpoint Type

```
screenPush
```

#### Parameters

| Parameter | Type | Required | Description |
| - | - | - | - |
| api | string | Yes | Your API service address |
| args | object | No | Custom parameter object, will be passed as-is to API server |
| quality | number | No | Image quality (0-1), default 0.8 |
| scale | number | No | Scale ratio (0-1), default 1.0 |

#### Request Format

iClick will send POST request to your API. To save unnecessary resources, after the streaming end successfully pushes, it will not wait for the result and return it to the caller. screenPush will always return true, and you need to implement your own business callback logic.

**Request Headers:**

```
Content-Type: multipart/form-data
```

**Request Body:**

```
image: <Image binary data (screenshot)>
```

If `args` parameter is provided, all fields will be passed as additional form fields:

```
image: <Image binary data>
customField: "value"            // Your custom field
...
```

### Basic Usage

```
// Captcha recognition example
await apiInvoke('screenPush', {
    api: 'http://localhost:5000/resolve_captcha',
    args: {
        type: 'digits',
        ...
    }
});
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.