跪拜 Guibai
← Back to the summary

Apple Foundation Models: A Unified AI Framework for On-Device, Cloud, and Custom Models

Apple Foundation Models:Apple AI核心框架

AA2hzqlh.png

Apple Foundation Models is Apple's generative AI development framework. Developers can call on-device models, Private Cloud Compute (PCC) (Apple's private cloud model), or custom models through a unified API to accomplish tasks such as text generation, structured output, tool calling, and image understanding.

foundation-models-framework.jpg The core of the Apple Foundation Models framework is five objects:

  1. LanguageModel: Determines where the model runs and what capabilities it has.
  2. LanguageModelSession: Manages instructions, history, tools, and requests.
  3. @Generable: Outputs structured objects suitable for App processing.
  4. Tool: Lets the model accomplish tasks by leveraging App capabilities.
  5. DynamicProfile: Manages workflows.

一、先理解三个核心概念

1. 模型种类

模型 特点 适用场景
SystemLanguageModel Runs locally on the device (iOS macOS), emphasizing privacy and response speed Summarization, classification, simple Q&A
PrivateCloudComputeLanguageModel Runs on Apple's self-built data centers and self-developed chips, providing more complex capabilities and adopting a verifiable privacy protection mechanism Complex reasoning or larger contexts
自定义 LanguageModel Connects to other inference backends via a unified protocol Self-developed or third-party models

2. 会话(Session)

LanguageModelSession is the main entry point for using the model. A session typically contains:

After receiving instructions, input, and necessary context, the model can generate text, structured results, or tool calls. Tools are executed by the App, and the execution results are then passed back to the model to generate the final answer.

Therefore, the model does not directly access the App's database or execute arbitrary code; permissions remain controlled by the App.

3. 为什么需要结构化输出

Plain text suits direct display, while @Generable lets the model return Swift data types that the App can use.

4. 整体结构一览

image.png

This diagram corresponds to three key points: LanguageModelSession is the only entry point; LanguageModel only determines "where the model runs"; Tool execution power always remains in the App, and the model only initiates requests.

二、Foundation Models 基础调用

2.1 生成文本

The most basic call only requires creating a session and sending a prompt. The default session uses the system-provided on-device model.

import FoundationModels

let session = LanguageModelSession(
    instructions: "你是一位简洁、准确的中文助手。"
)

let response = try await session.respond(
    to: "用一句话解释什么是端侧 AI。"
)

print(response.content)

Before actual use, you should check the model's isAvailable. For long texts or long conversations, you also need to pay attention to contextSize, and trim or summarize history if necessary to avoid exceeding the context window.

2.2 生成结构化结果

@Generable describes the output structure, and @Guide supplements field semantics or constraints. The caller can directly read type-safe objects without manually parsing JSON strings.

@Generable
struct ArticleSummary {
    @Guide(description: "一句话概括文章主题")
    var summary: String

    @Guide(description: "最重要的三个要点")
    var keyPoints: [String]
}

let response = try await session.respond(
    to: "总结这篇文章:\(article)",
    generating: ArticleSummary.self
)

let summary: ArticleSummary = response.content
print(summary.summary)
print(summary.keyPoints)

2.3 调用App工具

Tools are suitable for querying real-time information, reading App data, or executing explicit operations.

The model generates parameters, and the App executes the actual code.

struct WeatherTool: Tool {
    let description = "查询指定城市的当前天气"

    @Generable
    struct Arguments {
        @Guide(description: "需要查询天气的城市名称")
        var city: String
    }

    func call(arguments: Arguments) async throws -> String {
        try await WeatherService.currentWeather(city: arguments.city)
    }
}

let session = LanguageModelSession(
    tools: [WeatherTool()],
    instructions: "需要实时天气时使用工具,不要猜测。"
)

let response = try await session.respond(
    to: "上海现在的天气怎么样?"
)

Here the model might generate city = "上海" and request to call the tool. After WeatherTool returns the result, the model organizes the result into a natural language answer.

The complete flow is as follows:

sequenceDiagram
    autonumber
    participant U as 用户
    participant A as App
    participant S as Session
    participant M as 模型
    participant T as Tool

    rect rgb(234, 242, 255)
    Note over U,A: ① 发起请求
    U->>A: 提出需求
    A->>S: respond(to:)
    S->>S: 组装 Instructions + Transcript + Prompt
    S->>M: 发起生成请求
    end

    rect rgb(255, 244, 229)
    Note over M,T: ② 模型只产出参数,App 真正执行
    M-->>S: 生成工具参数 city = "上海"
    S->>T: call(arguments:)
    T-->>S: 返回工具执行结果
    end

    rect rgb(233, 251, 240)
    Note over S,M: ③ 结果回灌,生成最终答案
    S->>M: 带上工具结果继续生成
    M-->>S: 最终答案 / 结构化对象
    end

    S-->>A: response.content
    A-->>U: 渲染结果

Note step 5 in the diagram: the model only produces parameters; the actual execution is Tool.call(arguments:), and the result is then fed back to the model to organize into natural language.

三、使用 DynamicProfile 编排会话

Fixed instructions suit single tasks. Complex Apps may need to switch between modes like "brainstorming" and "planning," and configure different instructions, models, and generation parameters for each mode.

DynamicProfile reads App state and generates the corresponding session configuration:

struct CraftProfile: LanguageModelSession.DynamicProfile {
    var orchestrator: CraftOrchestrator

    var body: some DynamicProfile {
        switch orchestrator.mode {
        case .brainstorming:
            Profile {
                Instructions {
                    "为手工项目提出多样、有创意的方案。"
                }
                BrainstormRecordTool()
            }
            // 创意阶段使用 PCC,温度能够提高输出的多样性
            .model(orchestrator.pccLanguageModel)
            .temperature(1)

        case .planning:
            Profile {
                Instructions {
                    "把选定的方案整理成清晰、可执行的教程。"
                }
            }
            // 规划阶段启用更深入的推理
            .model(orchestrator.pccLanguageModel)
            .reasoningLevel(.deep)

        case .reviewing:
            Profile {
                Instructions { "回顾项目并提出改进建议。" }
            }
            // 简单回顾可以回到端侧模型
            .model(orchestrator.systemLanguageModel)
        }
    }
}

let session = LanguageModelSession(
    profile: CraftProfile(orchestrator: orchestrator)
)

This example embodies the core of session orchestration: The business code still calls the same session, but the instructions, tools, and models used by the session can change with state.

When history is long, you can use historyTransform to adjust the actual content fed to the model without directly modifying the original session history.

The following example retains tool calls and tool outputs from the session history, plus the most recent response, reducing the context occupied by ordinary conversation:

struct HistoryManagingProfile: LanguageModelSession.DynamicProfile {
    var orchestrator: CraftOrchestrator

    var body: some DynamicProfile {
        Profile {
            Instructions { "结合已有工具结果,回顾项目并提出建议。" }
        }
        .model(orchestrator.systemLanguageModel)
        .historyTransform { history in
            // 找不到最近一次响应时,保持原历史不变
            guard let latestResponseIndex = lastResponseEntryIndex(history) else {
                return history
            }

            // 从会话历史中只保留工具调用和工具输出
            let filteredHistory = history[0..<latestResponseIndex].filter { entry in
                isToolCallsOrToolOutput(entry)
            }

            // 拼接筛选后的早期记录和最近一次响应
            return filteredHistory + history[latestResponseIndex...]
        }
    }
}

historyTransform only affects the history provided to the model this time; it does not delete the original session history Transcript stored in the App.

四、选择模型与接入自定义模型

4.1 在端侧模型和 PCC 之间切换

The session interface does not depend on a fixed model. After changing the model parameter, the patterns for prompts, structured output, and tool calls can remain essentially unchanged.

// 端侧模型
let localSession = LanguageModelSession(
    model: SystemLanguageModel()
)

// Private Cloud Compute 模型
let cloudSession = LanguageModelSession(
    model: PrivateCloudComputeLanguageModel()
)

let response = try await cloudSession.respond(
    to: "分析这篇文章的论点,并列出关键证据。\(article)"
)

When selecting a model, you need to consider task complexity, privacy requirements, context size, response speed, and available quotas comprehensively, rather than always using the stronger model for every request.

4.2 实现自定义 LanguageModel

Custom models are mainly divided into two parts:

The following code shows the basic structure:

public struct MyLanguageModel: LanguageModel {
    public typealias Executor = MyLanguageModelExecutor

    public var capabilities: LanguageModelCapabilities {
        LanguageModelCapabilities(capabilities: [
            .toolCalling,
            .guidedGeneration,
            .reasoning
        ])
    }

    public var executorConfiguration: Executor.Configuration {
        Executor.Configuration()
    }
}

public struct MyLanguageModelExecutor: LanguageModelExecutor {
    public typealias Model = MyLanguageModel

    public struct Configuration: Hashable, Sendable {
        public init() {}
    }

    public init(configuration: Configuration) throws {
    }

    public func respond(
        to request: LanguageModelExecutorGenerationRequest,
        model: MyLanguageModel,
        streamingInto channel: LanguageModelExecutorGenerationChannel
    ) async throws {
        // 1. 将 request 中的 Transcript 和生成参数转换为后端API请求
        // 2. 执行真正的模型推理
        // 3. 将生成的文本增量发送给App上层会话
        for try await token in runInference(request) {
            await channel.send(.response(action: .appendText(token)))
        }
    }
}

When integrating a custom model, you are not merely calling a backend/SDK API; you need to correctly complete four mappings: history messages, generation parameters, streaming output, and errors.

五、通过命令行和 Python 使用Apple模型

5.1 使用 fm 命令行工具

fm is suitable for quickly debugging prompts, image inputs, and structured outputs. The following example first generates a JSON Schema, then asks the model to return results according to the Schema:

# 定义输出结构:一个字符串数组 app_names
fm schema object --name AppsIdentified \
    --string app_names --array > schema.json

# 从截图中提取 App 名称,并按 Schema 输出
fm respond "What apps are visible in this screenshot?" \
    --model pcc \
    --image Screenshot.png \
    --schema schema.json

# 输出示例:{"app_names": ["Messages", "Mail", "Calendar"]}

Without using a Schema, you can also directly execute fm respond "你的提示词" to quickly verify model output.

5.2 使用 Python SDK

The Python SDK's calling flow is basically the same as Swift: create a session, send an asynchronous request, read the result. Structured output can be declared using @fm.generable.

import apple_fm_sdk as fm

@fm.generable("Article summary")
class ArticleSummary:
    summary: str = fm.guide("A one-sentence summary")
    key_points: list[str] = fm.guide("The three most important points")


async def summarize(article: str) -> ArticleSummary:
    session = fm.LanguageModelSession(
        instructions="You are a concise article summarizer."
    )

    return await session.respond(
        f"Summarize this article: {article}",
        generating=ArticleSummary
    )

The installation command is pip install apple_fm_sdk. Python also supports defining tools, letting the model query business data before generating an answer.

六、使用 SpotlightSearchTool 搜索App内容

SpotlightSearchTool lets the model search content that the App has written to Spotlight.

import CoreSpotlight
import FoundationModels

let searchTool = SpotlightSearchTool()

let session = LanguageModelSession(
    model: SystemLanguageModel(),
    tools: [searchTool],
    instructions: """
        使用搜索工具查找用户的徒步记录。
        只根据搜索结果回答,不要编造不存在的记录。
        """
)

let response = try await session.respond(
    to: "我去过哪些靠近水边的徒步路线?"
)