feat(runware): add Runware AI provider integration for image and video generation #139
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add initial spec for integrating Runware as an AI provider, covering: - SDK initialization and configuration - Image inference API (text-to-image, image-to-image) - Video inference API (text-to-video, image-to-video) - Available video models (Kling, PixVerse, MiniMax, Veo, etc.) - Provider-specific settings for video generation - Quick action support mapping - Implementation checklist
- Add parameter classification into UI-relevant and technical groups - Document dimension handling with aspect ratio mapping - Convert provider examples to generic templates for reusability - Add link to PROVIDERS.md for available models
Add PROVIDERS.md with complete catalog of Runware-available models: - Image generation models from 10+ providers (BFL, Google, OpenAI, etc.) - Video generation models (KlingAI, MiniMax, PixVerse, Veo, etc.) - AIR identifiers, capabilities, and specifications for each model
Move parameter classification and dimension handling documentation to PARAMETERS.md for better organization and maintainability.
Add comprehensive per-model specifications with OpenAPI-like input schemas for provider implementation. Each model now includes: - AIR identifier and capabilities - Dimension/resolution constraints - Input schema with proper UI component annotations Image models cover: BFL FLUX, Google Imagen, OpenAI GPT Image 1, ByteDance Seedream/SeedEdit, Ideogram 3.0 variants, KlingAI Kolors, and Bria FIBO. Video models cover: KlingAI, MiniMax Hailuo, PixVerse, Vidu, Google Veo, Runway Gen-4 Turbo, ByteDance Seedance, and Lightricks LTX. Schemas follow existing provider patterns with x-imgly-builder components, enum labels, and property ordering for consistent UI.
The AIR identifier was incorrectly documented as bfl:1@1. Per official Runware BFL documentation, FLUX.1.1 Pro uses bfl:2@1.
Document how Runware handles image uploads for seedImage and frameImages: - API-level imageUpload taskType with request/response format - Supported image formats (Data URI, Base64, URL) - JavaScript SDK usage patterns - Relation to inference method parameters - Implementation notes for IMG.LY plugin (no explicit upload needed)
Add new provider integration for Runware API with support for multiple image generation models including FLUX, Ideogram, Imagen, and more. Includes both text-to-image and image-to-image capabilities.
Replace @runware/sdk-js WebSocket SDK with direct HTTP REST API calls
for better consistency with other providers in the codebase.
Changes:
- createRunwareClient.ts: Implement HTTP client using fetch() to
POST to the Runware REST API endpoint
- types.ts: Remove SDK dependency, re-export RunwareClient type
- createImageProvider.ts: Support AbortSignal for request cancellation,
fix isNaN -> Number.isNaN lint error
- Flux11Pro.ts, Flux11ProUltra.ts: Prettier formatting only
The HTTP client sends requests in the standard Runware format:
[{ taskType: "imageInference", taskUUID: "<uuid>", model: "<air>", ... }]
The proxy URL (config.proxyUrl) handles authentication.
Use preset sizes with valid dimensions (128-2048px, divisible by 64) instead of complex custom dimension logic. Removes unused custom size support to comply with API requirements.
Add providers for: - Google: Imagen 3.0, Imagen 3.0 Fast, Nano Banana Pro - ByteDance: Seedream 3.0 - Bria: Bria 3.2 - Kling AI: Kolors 2.0 - BFL: FLUX.1 Krea [dev] - Other: HiDream-I1 Full, Qwen-Image, Flex.1-alpha
Add detailed specifications for Imagen 3.0, Imagen 3.0 Fast, Seedream 3.0, Bria 3.2, Kolors 2.0, HiDream-I1 Full, FLUX.1 Krea [dev], Qwen-Image, Flex.1-alpha, and Nano Banana Pro. Add "Tested" column to overview table for manual test tracking.
Add comprehensive documentation for AI provider development: - Architecture overview and provider patterns - UI guidelines for schema-driven forms - Schema templates for T2I, I2I, T2V, I2V providers - Quick action integration patterns - Runware-specific implementation notes (HTTP REST over WebSocket) - Runware provider catalog with model specifications
Remove spec files that are superseded by the new canonical specs in specs/providers/. The Runware integration specs have been consolidated and the other specs were for completed or abandoned features.
Provider generation now relies on specs/providers/ documentation instead of duplicated agent instructions.
…lementation Adds a Claude skill to manage the lifecycle of Runware AI providers: - Discovery phase: fetch Runware docs, compare against tracked models - Implementation phase: create providers following specs/providers/ patterns
Remove all hardcoded Runware image provider implementations in preparation for the partner-providers-runware skill workflow. Providers will now be dynamically added via the skill system rather than being maintained manually.
Add initial scaffolding for Runware video providers in the video generation plugin. This includes: - HTTP REST API client for Runware video inference - createVideoProvider factory with quick action support - Types and utilities for video aspect ratio handling - Package export for runware subpath Providers will be added dynamically via the partner-providers-runware skill.
…terns Add comprehensive documentation for provider Feature API and i18n integration: - feature-api.md: Feature flag naming conventions, registration patterns - i18n.md: Translation key conventions, required translations Rename schemas/ to patterns/ to better reflect content (implementation patterns, not just JSON schemas). Update all references accordingly.
…tterns Most providers don't need feature flags - removed plugin-level flag documentation that's irrelevant for provider authors. Now clearly states this is a rare pattern only needed for toggleable sub-features like Recraft's style groups.
Reduced to essential provider tasks: - Required: history label registration - Optional: custom enum translations (with schema alternative) - Reference to defaultTranslations.ts for adding reusable values
Rename ARCHITECTURE.md, PROVIDERS.md, and UI-GUIDELINES.md to lowercase for consistency. README.md remains uppercase per convention. Update all references in specs and skill files.
Add UI dropdown to switch between Fal.ai, OpenAI, and Runware provider partners. Extracts provider configurations into dedicated files for cleaner organization and easier partner-specific customization.
- Add reference to specs/providers/patterns/ui-guidelines.md as required reading before implementation (controls which params to expose in UI) - Add step 6 to register providers in examples/ai/src/runwareProviders.ts - Update spec file paths from schemas/ to patterns/ directory - Add note about not exposing advanced parameters (enhance_prompt, etc.)
Add a summary checklist at the end of implementation-notes.md to ensure all steps are completed, including the often-missed step of adding providers to the example app.
Implement first Runware provider for Black Forest Labs FLUX.2 [dev] model with text-to-image generation capability using standard aspect ratios.
- Rename modelAIR to modelId for clarity - Add comprehensive JSDoc documentation to provider options - Support custom headers parameter in createRunwareClient - Re-export RunwareProviderConfiguration from provider modules
Add Runware provider documentation including: - Provider configuration examples - API reference for all provider functions - Panel IDs for UI integration - Asset history source IDs
Update implementation checklist and skill documentation to require README updates when adding new Runware providers, including sections for provider configuration, API reference, panel IDs, and history sources.
Add support for multiple reference images in Runware providers, enabling the combineImages quick action. Also align naming and configuration patterns with fal.ai for consistency. Changes: - Add convertImageUrlArrayForRunware utility for processing image arrays - Update createImageProvider to accept image_urls in addition to image_url - Add deprecated middleware field for backwards compatibility - Standardize output field order in video provider - Enable combineImages quick action for all I2I providers that support multiple reference images (FLUX.2 dev/pro/flex, Seedream 4, Nano Banana 2 Pro, GPT Image 1)
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
maerch
changed the title
The Runware providers use custom REST API clients via fetch() instead of the SDK. The dependency was listed in package.json but never imported.
- Veo31/Veo31Fast: Duration is fixed at 8 seconds (not 5 or 8) - Veo31/Veo31Fast: Remove unsupported 1:1 aspect ratio - Sora2Pro: Add missing 1792×1024 and 1024×1792 resolutions
Dead code from when client management was refactored to per-provider initialization.
Add two new Runware image generation providers: - Seedream 4.5 (bytedance:seedream@4.5): ByteDance's improved model with enhanced facial detail and text generation, 2K minimum resolution output - GPT Image 1 Mini (openai:1@2): Cost-efficient variant of GPT Image 1 with ~80% cost savings and same capabilities Both providers support text-to-image and image-to-image generation with full quick action support.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add comprehensive Runware API integration for AI image and video generation, providing access to multiple state-of-the-art models through a unified HTTP REST interface.
Key Features
Infrastructure
Image Providers (Text-to-Image & Image-to-Image)
Video Providers (Text-to-Video & Image-to-Video)
Quick Actions Support
editImage,swapBackground,styleTransfer,artistTransfer,createVariant,remixPagecombineImagesquick action enabled for models supporting multiple reference imagesimageToVideo,animateImageDocumentation
specs/providers/)partner-providers-runware)Changes
packages/plugin-ai-image-generation-web/src/runware/- Image providers and HTTP clientpackages/plugin-ai-video-generation-web/src/runware/- Video providers with async pollingspecs/providers/- Architecture docs, patterns, and Runware catalog.claude/skills/partner-providers-runware.md- Provider generation skillexamples/ai/src/- Provider partner switch and Runware configurationTest plan