Substrate

We're building Zo Computer, the world's first AI computer.Infrastructure for intelligent software.Substrate is the platform for agentic AI. Elegant abstractions. High-performance components: optimized models, vector database, code interpreter, model router. And the only compute engine optimized for multi-step AI workloads.Automatic workload tuningMaximum parallelismFewer roundtripsWe analyze your workload as a directed acyclic graph and optimize the graph — for example, merging nodes that can be run in a batch.Your codeOptimized workflowCompound AIModel libraryVector DBCode interpreterSimple abstractions for compound AISubstrate introduces simple, intuitive abstractions for building compound AI systems, making it easy to create complex workflows by connecting modular building blocks called nodes.PythonTypeScript# pip install substrate from substrate import Substrate, ComputeText, sb s = Substrate(api_key="SUBSTRATE_API_KEY") topic1 = "a magical forest" topic2 = "a futuristic city" story1 = ComputeText(prompt=f"Tell me a story about {topic1}") story2 = ComputeText(prompt=f"Tell me a story about {topic2}") summary = ComputeText(prompt=sb.format( "Summarize these stories:\nStory 1: {story1}\nStory 2: {story2}", story1=story1.future.text, story2=story2.future.text) ) response = s.run(summary)What people sayI LOVE this API paradigm.Finally, a principled agent framework.Your speeds are blazing fast rn.Sooo clean, kind of makes LangChain look bloated.I’m here because I checked out the SDK and really liked the abstraction choices.Mike CohenHead of AI & MLE at SubstackVery excited for others to start to see and understand how amazing this product is.Love it – was using a version of this internally, but this implementation is sick. Bullish on Substrate!Discovered Substrate. It’s really cool and useful.This is the first agents platform that’s very first principle IMO.I will tell you that at the very minimum you’ve gained a customer in me that will show other people this platform.I’ve been testing your API, it’s really amazing. And FAST!Shreyans BhansaliCTO at MavenSubstrate is the developer experience I always wanted. We tried LangChain, Pinecone, various inference APIs… Now we just use Substrate.I’ve been looking for a replacement to Hugging Face and OpenAI API for years. Glad I found ya.Just wanted to say Substrate looks very cool. Your demos make it look soooo easy.So, I finally got to check out Substrate, and I must admit it’s damn interesting.Love the futures concept […] absolutely love it.Having a ‘this is a product we ought to use’ moment.Love the naming conventions. […] Love seeing fellow ex-Stripes build new companies.I’m annoyed I haven’t been recommending your product to my clients in my independent AI consulting work, because it is awesome.A proper abstraction makes development feel natural, and honestly, joyful too. --- We're building Zo Computer, the world's first AI computer.PlansStarterFree with $50 starter creditsthen $30/month + usage8 node concurrency limitAccess to community SlackPro$500/month + usage64 node concurrency limitPrioritized input to our roadmapEnterpriseContact salesMaximum node concurrency limitCustom modelsDedicated instancesSupport via private SlackSelf-hostedContact salesSubstrate runs on your clusterCustom model deploymentsGenerated SDKsUsage pricingLanguage modelsPer 1M tokensMistral 7B$0.50Llama 3 8B$0.50FireLLaVA 13B$0.50Mixtral 8x7B$1.00Llama 3 70B$1.50Llama 3.1 405B$5.00Image modelsPer imageAll models$0.02Segmentation modelsPer imageAll models$0.02Audio modelsPer minuteTranscribeSpeech$0.01Per 1M charactersGenerateSpeech$5Embedding modelsPer 1M tokensJina v2$0.02CLIP$0.01Per 1k imagesCLIP$0.01Vector storesPer GB vector storage per day$0.10Code interpreterPer sessionCPU$0.03GPUContact sales --- We're building Zo Computer, the world's first AI computer.Language ModelsComputeTextMultiComputeTextBatchComputeTextComputeJSONMultiComputeJSONBatchComputeJSONMistral7BInstructMixtral8x7BInstructLlama3Instruct8BLlama3Instruct70BFirellava13BImage ModelsGenerateImageMultiGenerateImageInpaintImageMultiInpaintImageUpscaleImageEraseImageInterpolateFramesStableDiffusionXLLightningStableDiffusionXLInpaintStableDiffusionXLControlNetStableVideoDiffusionSegmentation ModelsRemoveBackgroundSegmentUnderPointSegmentAnythingVector EmbeddingsSplitDocumentEmbedTextMultiEmbedTextEmbedImageMultiEmbedImageJinaV2CLIPVector StoresFindOrCreateVectorStoreListVectorStoresDeleteVectorStoreQueryVectorStoreFetchVectorsUpdateVectorsDeleteVectorsAudio ModelsTranscribeSpeechGenerateSpeechUtility NodesBoxIfRunPython ↗Language ModelsComputeTextCompute text using a language model.InputOutputpromptstringInput prompt.image_urisarray[string]OptionalImage prompts.temperaturefloat[0..1]OptionalSampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.Default: 0.4max_tokensintegerOptionalMaximum number of tokens to generate.modelstringOptionalSelected model. Firellava13B is automatically selected when image_uris is provided.Options: Mixtral8x7BInstructLlama3Instruct8BLlama3Instruct70BLlama3Instruct405BFirellava13Bgpt-4ogpt-4o-miniclaude-3-5-sonnet-20240620Default: Llama3Instruct8B PythonTypeScriptComputeText( prompt="Who is Don Quixote?", temperature=0.4, max_tokens=800,) Output{ "text": "Don Quixote is a fictional character in the novel of the same name by Miguel de Cervantes."}MultiComputeTextGenerate multiple text choices using a language model.InputOutputpromptstringInput prompt.num_choicesinteger[1..8]Number of choices to generate.Default: 1temperaturefloat[0..1]OptionalSampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.Default: 0.4max_tokensintegerOptionalMaximum number of tokens to generate.modelstringOptionalSelected model.Options: Mixtral8x7BInstructLlama3Instruct8BLlama3Instruct70BDefault: Llama3Instruct8B PythonTypeScriptMultiComputeText( prompt="Who is Don Quixote?", num_choices=2, max_tokens=800,) Output{ "choices": [ { "text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..." }, { "text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..." } ]}BatchComputeTextCompute text for multiple prompts in batch using a language model.InputOutputpromptsarray[string]Batch input prompts.temperaturefloat[0..1]OptionalSampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.Default: 0.4max_tokensintegerOptionalMaximum number of tokens to generate.modelstringOptionalSelected model.Default: Llama3Instruct8B PythonTypeScriptBatchComputeText( prompts=[ "Who is Don Quixote?", "Who is Sancho Panza?", ], max_tokens=800,) Output{ "outputs": [ { "text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..." }, { "text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..." } ]}ComputeJSONCompute JSON using a language model.InputOutputpromptstringInput prompt.json_schemaobjectJSON schema to guide json_object response.temperaturefloat[0..1]OptionalSampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.Default: 0.4max_tokensintegerOptionalMaximum number of tokens to generate.modelstringOptionalSelected model.Options: Mixtral8x7BInstructLlama3Instruct8BLlama3Instruct70Bgpt-4oDefault: Llama3Instruct8B PythonTypeScriptComputeJSON( prompt="Who wrote Don Quixote?", json_schema={ "type": "object", "properties": { "name": { "type": "string", "description": "The name of the author.", }, "bio": { "type": "string", "description": "Concise biography of the author.", }, }, }, temperature=0.4, max_tokens=800,) Output{ "json_object": {}}MultiComputeJSONCompute multiple JSON choices using a language model.InputOutputpromptstringInput prompt.json_schemaobjectJSON schema to guide json_object response.num_choicesinteger[1..8]Number of choices to generate.Default: 2temperaturefloat[0..1]OptionalSampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.Default: 0.4max_tokensintegerOptionalMaximum number of tokens to generate.modelstringOptionalSelected model.Options: Mixtral8x7BInstructLlama3Instruct8BDefault: Llama3Instruct8B PythonTypeScriptMultiComputeJSON( prompt="Who wrote Don Quixote?", json_schema={ "type": "object", "properties": { "name": { "type": "string", "description": "The name of the author.", }, "bio": { "type": "string", "description": "Concise biography of the author.", }, }, }, num_choices=2, temperature=0.4, max_tokens=800,) Output{ "choices": [ { "json_object": {} }, { "json_object": {} } ]}BatchComputeJSONCompute JSON for multiple prompts in batch using a language model.InputOutputpromptsarray[string]Batch input prompts.json_schemaobjectJSON schema to guide json_object response.temperaturefloat[0..1]OptionalSampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.Default: 0.4max_tokensintegerOptionalMaximum number of tokens to generate.modelstringOptionalSelected model.Default: Llama3Instruct8B PythonTypeScriptBatchComputeJSON( prompts=[ "Who is Don Quixote?", "Who is Sancho Panza?", ], max_tokens=800, json_schema={ "type": "object", "properties": { "name": { "type": "string", "description": "The name of the character.", }, "bio": { "type": "string", "description": "Concise biography of the character.", }, }, },) Output{ "outputs": [ { "json_object": {} }, { "json_object": {} } ]}Mistral7BInstructCompute text using Mistral 7B Instruct.InputOutputpromptstringInput prompt.system_promptstringOptionalSystem prompt.num_choicesinteger[1..8]OptionalNumber of choices to generate.Default: 1json_schemaobjectOptionalJSON schema to guide response.temperaturefloat[0..1]OptionalHigher values make the output more random, lower values make the output more deterministic.frequency_penaltyfloat[-2..2]OptionalHigher values decrease the likelihood of repeating previous tokens.Default: 0repetition_penaltyfloat[-2..2]OptionalHigher values decrease the likelihood of repeated sequences.Default: 1presence_penaltyfloat[-2..2]OptionalHigher values increase the likelihood of new topics appearing.Default: 1.1top_pfloat[0..1]OptionalProbability below which less likely tokens are filtered out.Default: 0.95max_tokensintegerOptionalMaximum number of tokens to generate. PythonTypeScriptMistral7BInstruct( prompt="Who is Don Quixote?", num_choices=2, temperature=0.4, max_tokens=800,) Output{ "choices": [ { "text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..." }, { "text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..." } ]}Mixtral8x7BInstructCompute text using instruct-tuned Mixtral 8x7B.InputOutputpromptstringInput prompt.system_promptstringOptionalSystem prompt.num_choicesinteger[1..8]OptionalNumber of choices to generate.Default: 1json_schemaobjectOptionalJSON schema to guide response.temperaturefloat[0..1]OptionalHigher values make the output more random, lower values make the output more deterministic.frequency_penaltyfloat[-2..2]OptionalHigher values decrease the likelihood of repeating previous tokens.Default: 0repetition_penaltyfloat[-2..2]OptionalHigher values decrease the likelihood of repeated sequences.Default: 1presence_penaltyfloat[-2..2]OptionalHigher values increase the likelihood of new topics appearing.Default: 1.1top_pfloat[0..1]OptionalProbability below which less likely tokens are filtered out.Default: 0.95max_tokensintegerOptionalMaximum number of tokens to generate. PythonTypeScriptMixtral8x7BInstruct( prompt="Who is Don Quixote?", num_choices=2, temperature=0.4, max_tokens=800,) Output{ "choices": [ { "text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..." }, { "text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..." } ]}Llama3Instruct8BCompute text using instruct-tuned Llama 3 8B.InputOutputpromptstringInput prompt.system_promptstringOptionalSystem prompt.num_choicesinteger[1..8]OptionalNumber of choices to generate.Default: 1temperaturefloat[0..1]OptionalHigher values make the output more random, lower values make the output more deterministic.frequency_penaltyfloat[-2..2]OptionalHigher values decrease the likelihood of repeating previous tokens.Default: 0repetition_penaltyfloat[-2..2]OptionalHigher values decrease the likelihood of repeated sequences.Default: 1presence_penaltyfloat[-2..2]OptionalHigher values increase the likelihood of new topics appearing.Default: 1.1top_pfloat[0..1]OptionalProbability below which less likely tokens are filtered out.Default: 0.95max_tokensintegerOptionalMaximum number of tokens to generate.json_schemaobjectOptionalJSON schema to guide response. PythonTypeScriptLlama3Instruct8B( prompt="Who is Don Quixote?", num_choices=2, temperature=0.4, max_tokens=800,) Output{ "choices": [ { "text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..." }, { "text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..." } ]}Llama3Instruct70BCompute text using instruct-tuned Llama 3 70B.InputOutputpromptstringInput prompt.system_promptstringOptionalSystem prompt.num_choicesinteger[1..8]OptionalNumber of choices to generate.Default: 1temperaturefloat[0..1]OptionalHigher values make the output more random, lower values make the output more deterministic.frequency_penaltyfloat[-2..2]OptionalHigher values decrease the likelihood of repeating previous tokens.Default: 0repetition_penaltyfloat[-2..2]OptionalHigher values decrease the likelihood of repeated sequences.Default: 1presence_penaltyfloat[-2..2]OptionalHigher values increase the likelihood of new topics appearing.Default: 1.1top_pfloat[0..1]OptionalProbability below which less likely tokens are filtered out.Default: 0.95max_tokensintegerOptionalMaximum number of tokens to generate. PythonTypeScriptLlama3Instruct70B( prompt="Who is Don Quixote?", num_choices=2, temperature=0.4, max_tokens=800,) Output{ "choices": [ { "text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..." }, { "text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..." } ]}Firellava13BCompute text with image input using FireLLaVA 13B.InputOutputpromptstringText prompt.image_urisarray[string]Image prompts.max_tokensintegerOptionalMaximum number of tokens to generate. PythonTypeScriptFirellava13B( prompt="what are these paintings of and who made them?", image_uris=[ "https://media.substrate.run/docs-fuji-red.jpg", "https://media.substrate.run/docs-fuji-blue.jpg", ],) Output{ "text": "The artist who created these paintings is Hokusai Katsushika, a renowned Japanese artist known for his woodblock prints and paintings."}Image ModelsGenerateImageGenerate an image.InputOutputpromptstringText prompt.storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string. PythonTypeScriptGenerateImage( prompt="hokusai futuristic supercell spiral cloud with glowing core over turbulent ocean", store="hosted",) Output{ "image_uri": "https://assets.substrate.run/84848484.jpg"}MultiGenerateImageGenerate multiple images.InputOutputpromptstringText prompt.num_imagesinteger[1..8]Number of images to generate.Default: 2storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string. PythonTypeScriptMultiGenerateImage( prompt="hokusai futuristic supercell spiral cloud with glowing core over turbulent ocean", num_images=2, store="hosted",) Output{ "outputs": [ { "image_uri": "https://assets.substrate.run/84848484.jpg" }, { "image_uri": "https://assets.substrate.run/48484848.jpg" } ]}InpaintImageEdit an image using image generation inside part of the image or the full image.InputOutputimage_uristringOriginal image.promptstringText prompt.mask_image_uristringOptionalMask image that controls which pixels are inpainted. If unset, the entire image is edited (image-to-image).storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string. PythonTypeScriptInpaintImage( image_uri="https://media.substrate.run/docs-klimt-park.jpg", mask_image_uri="https://media.substrate.run/spiral-logo.jpeg", prompt="large tropical colorful bright anime birds in a dark jungle full of vines, high resolution", store="hosted",) Output{ "image_uri": "https://assets.substrate.run/84848484.jpg"}MultiInpaintImageEdit multiple images using image generation.InputOutputimage_uristringOriginal image.promptstringText prompt.mask_image_uristringOptionalMask image that controls which pixels are edited (inpainting). If unset, the entire image is edited (image-to-image).num_imagesinteger[1..8]Number of images to generate.Default: 2storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string. PythonTypeScriptMultiInpaintImage( image_uri="https://media.substrate.run/docs-klimt-park.jpg", mask_image_uri="https://media.substrate.run/spiral-logo.jpeg", prompt="large tropical colorful bright anime birds in a dark jungle full of vines, high resolution", num_images=2, store="hosted",) Output{ "outputs": [ { "image_uri": "https://assets.substrate.run/84848484.jpg" }, { "image_uri": "https://assets.substrate.run/48484848.jpg" } ]}UpscaleImageUpscale an image using image generation.InputOutputpromptstringOptionalPrompt to guide model on the content of image to upscale.image_uristringInput image.output_resolutioninteger[512..2048]OptionalResolution of the output image, in pixels.Default: 1024storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string. PythonTypeScriptUpscaleImage( prompt="high resolution detailed spiral shell", image_uri="https://media.substrate.run/docs-shell-emoji.jpg", store="hosted",) Output{ "image_uri": "https://assets.substrate.run/84848484.jpg"}EraseImageErase the masked part of an image, e.g. to remove an object by inpainting.InputOutputimage_uristringInput image.mask_image_uristringMask image that controls which pixels are inpainted.storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string. PythonTypeScriptEraseImage( image_uri="https://media.substrate.run/apple-forest.jpeg", mask_image_uri="https://media.substrate.run/apple-forest-mask.jpeg", store="hosted",) Output{ "image_uri": "https://assets.substrate.run/84848484.jpg"}InterpolateFramesGenerates a interpolation frames between each adjacent frames.InputOutputframe_urisarray[string]Frames.storestringOptionalUse "hosted" to return a video URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the video data will be returned as a base64-encoded string.output_formatstringOptionalOutput video format.Options: gifwebpmp4framesDefault: giffpsinteger[1..]OptionalFrames per second of the generated video. Ignored if output format is frames.Default: 7num_stepsinteger[1..]OptionalNumber of interpolation steps. Each step adds an interpolated frame between adjacent frames. For example, 2 steps over 2 frames produces 5 frames.Default: 2 PythonTypeScriptInterpolateFrames( frame_uris=[ "https://media.substrate.run/apple-forest2.jpeg", "https://media.substrate.run/apple-forest3.jpeg", ], store="hosted",) Output{ "video_uri": "https://assets.substrate.run/84848484.mp4"}StableDiffusionXLLightningGenerate an image using Stable Diffusion XL Lightning.InputOutputpromptstringText prompt.negative_promptstringOptionalNegative input prompt.num_imagesinteger[1..8]OptionalNumber of images to generate.Default: 1storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.heightinteger[256..1536]OptionalHeight of output image, in pixels.Default: 1024widthinteger[256..1536]OptionalWidth of output image, in pixels.Default: 1024seedsarray[integer]OptionalSeeds for deterministic generation. Default is a random seed. PythonTypeScriptStableDiffusionXLLightning( prompt="hokusai futuristic supercell spiral cloud with glowing core over turbulent ocean", negative_prompt="night, moon", num_images=2, seeds=[ 330699, 136464, ], store="hosted",) Output{ "outputs": [ { "image_uri": "https://assets.substrate.run/84848484.jpg", "seed": 330418 }, { "image_uri": "https://assets.substrate.run/48484848.jpg", "seed": 1364164 } ]}StableDiffusionXLInpaintEdit an image using Stable Diffusion XL. Supports inpainting (edit part of the image with a mask) and image-to-image (edit the full image).InputOutputimage_uristringOriginal image.promptstringText prompt.mask_image_uristringOptionalMask image that controls which pixels are edited (inpainting). If unset, the entire image is edited (image-to-image).num_imagesinteger[1..8]Number of images to generate.Default: 1output_resolutioninteger[512..2048]OptionalResolution of the output image, in pixels.Default: 1024negative_promptstringOptionalNegative input prompt.storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.strengthfloat[0..1]OptionalControls the strength of the generation process.Default: 0.8seedsarray[integer]OptionalRandom noise seeds. Default is random seeds for each generation. PythonTypeScriptStableDiffusionXLInpaint( image_uri="https://media.substrate.run/docs-klimt-park.jpg", mask_image_uri="https://media.substrate.run/spiral-logo.jpeg", prompt="large tropical colorful bright birds in a jungle, high resolution oil painting", negative_prompt="dark, cartoon, anime", strength=0.8, num_images=2, store="hosted", seeds=[ 1607280, 1720395, ],) Output{ "outputs": [ { "image_uri": "https://assets.substrate.run/84848484.jpg", "seed": 1607326 }, { "image_uri": "https://assets.substrate.run/48484848.jpg", "seed": 1720398 } ]}StableDiffusionXLControlNetGenerate an image with generation structured by an input image, using Stable Diffusion XL with ControlNet.InputOutputimage_uristringInput image.control_methodstringStrategy to control generation using the input image.Options: edgedepthillusiontilepromptstringText prompt.num_imagesinteger[1..8]Number of images to generate.Default: 1output_resolutioninteger[512..2048]OptionalResolution of the output image, in pixels.Default: 1024negative_promptstringOptionalNegative input prompt.storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.conditioning_scalefloat[0..1]OptionalControls the influence of the input image on the generated output.Default: 0.5strengthfloat[0..1]OptionalControls how much to transform the input image.Default: 0.5seedsarray[integer]OptionalRandom noise seeds. Default is random seeds for each generation. PythonTypeScriptStableDiffusionXLControlNet( image_uri="https://media.substrate.run/spiral-logo.jpeg", prompt="the futuristic solarpunk city of atlantis at sunset, cinematic bokeh HD", control_method="illusion", conditioning_scale=1.0, strength=1.0, store="hosted", num_images=2, seeds=[ 1607226, 1720395, ],) Output{ "outputs": [ { "image_uri": "https://assets.substrate.run/84848484.jpg", "seed": 1607266 }, { "image_uri": "https://assets.substrate.run/48484848.jpg", "seed": 1720398 } ]}StableVideoDiffusionGenerates a video using a still image as conditioning frame.InputOutputimage_uristringOriginal image.storestringOptionalUse "hosted" to return a video URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the video data will be returned as a base64-encoded string.output_formatstringOptionalOutput video format.Options: gifwebpmp4framesDefault: gifseedintegerOptionalSeed for deterministic generation. Default is a random seed.fpsinteger[1..]OptionalFrames per second of the generated video. Ignored if output format is frames.Default: 7motion_bucket_idintegerOptionalThe motion bucket id to use for the generated video. This can be used to control the motion of the generated video. Increasing the motion bucket id increases the motion of the generated video.Default: 180noisefloatOptionalThe amount of noise added to the conditioning image. The higher the values the less the video resembles the conditioning image. Increasing this value also increases the motion of the generated video.Default: 0.1 PythonTypeScriptStableVideoDiffusion( image_uri="https://media.substrate.run/apple-forest.jpeg", store="hosted",) Output{ "video_uri": "https://assets.substrate.run/84848484.mp4"}Segmentation ModelsRemoveBackgroundRemove the background from an image and return the foreground segment as a cut-out or a mask.InputOutputimage_uristringInput image.return_maskbooleanOptionalReturn a mask image instead of the original content.Default: falseinvert_maskbooleanOptionalInvert the mask image. Only takes effect if return_mask is true.Default: falsebackground_colorstringOptionalHex value background color. Transparent if unset.storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string. PythonTypeScriptRemoveBackground( image_uri="https://media.substrate.run/apple-forest.jpeg", store="hosted",) Output{ "image_uri": "https://assets.substrate.run/84848484.jpg"}SegmentUnderPointSegment an image under a point and return the segment.InputOutputimage_uristringInput image.pointPointPoint prompt.xintegerX position.yintegerY position.storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string. PythonTypeScriptSegmentUnderPoint( image_uri="https://media.substrate.run/docs-vg-bedroom.jpg", point={ "x": 189, "y": 537, }, store="hosted",) Output{ "mask_image_uri": "https://assets.substrate.run/84848484.jpg"}SegmentAnythingSegment an image using SegmentAnything.InputOutputimage_uristringInput image.point_promptsarray[Point]OptionalPoint prompts, to detect a segment under the point. One of point_prompts or box_prompts must be set.xintegerX position.yintegerY position.box_promptsarray[BoundingBox]OptionalBox prompts, to detect a segment within the bounding box. One of point_prompts or box_prompts must be set.x1floatTop left corner x.y1floatTop left corner y.x2floatBottom right corner x.y2floatBottom right corner y.storestringOptionalUse "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string. PythonTypeScriptSegmentAnything( image_uri="https://media.substrate.run/docs-vg-bedroom.jpg", point_prompts=[ { "x": 189, "y": 537, }, ], store="hosted",) Output{ "mask_image_uri": "https://assets.substrate.run/84848484.jpg"}Vector EmbeddingsSplitDocumentSplit document into text segments.InputOutputuristringURI of the document.doc_idstringOptionalDocument ID.metadataobjectOptionalDocument metadata.chunk_sizeinteger[1..]OptionalMaximum number of units per chunk. Defaults to 1024 tokens for text or 40 lines for code.chunk_overlapintegerOptionalNumber of units to overlap between chunks. Defaults to 200 tokens for text or 15 lines for code. PythonTypeScriptSplitDocument( doc_id="example_pdf", uri="https://arxiv.org/pdf/2405.07945", metadata={ "title": "GRASS II: Simulations of Potential Granulation Noise Mitigation Methods", },) Output{ "items": [ { "text": "This is the first chunk of the pdf", "metadata": { "title": "GRASS II: Simulations of Potential Granulation Noise Mitigation Methods", "chunk_id": "chk_asd897asdhnad0j8qd8qnd98" }, "doc_id": "example_pdf" }, { "text": "This is the second chunk of the pdf", "metadata": { "title": "GRASS II: Simulations of Potential Granulation Noise Mitigation Methods", "chunk_id": "chk_nvsiusd89adsy89dahd9abs8" }, "doc_id": "example_pdf" } ]}EmbedTextGenerate embedding for a text document.InputOutputtextstringText to embed.collection_namestringOptionalVector store name.metadataobjectOptionalMetadata that can be used to query the vector store. Ignored if collection_name is unset.embedded_metadata_keysarray[string]OptionalChoose keys from metadata to embed with text.doc_idstringOptionalVector store document ID. Ignored if store is unset.modelstringOptionalSelected embedding model.Options: jina-v2clipDefault: jina-v2 PythonTypeScriptEmbedText( text="Argon is the third most abundant gas in Earth's atmosphere, at 0.934% (9340 ppmv). It is more than twice as abundant as water vapor.", model="jina-v2", collection_name="smoke_tests", metadata={ "group": "18", }, embedded_metadata_keys=[ "group", ],) Output{ "embedding": { "vector": [ -0.035030052065849304, -0.04128379374742508, 0.05782046541571617 ], "doc_id": "c9de81fb98804ce0afb2b8ac17c0799b", "metadata": { "group": "18", "doc_id": "c9de81fb98804ce0afb2b8ac17c0799b", "doc": "group: 18\n\nArgon is the third most abundant gas in Earth's atmosphere, at 0.934% (9340 ppmv). It is more than twice as abundant as water vapor." } }}MultiEmbedTextGenerate embeddings for multiple text documents.InputOutputitemsarray[EmbedTextItem]Items to embed.textstringText to embed.metadataobjectOptionalMetadata that can be used to query the vector store. Ignored if collection_name is unset.doc_idstringOptionalVector store document ID. Ignored if collection_name is unset.collection_namestringOptionalVector store name.embedded_metadata_keysarray[string]OptionalChoose keys from metadata to embed with text.modelstringOptionalSelected embedding model.Options: jina-v2clipDefault: jina-v2 PythonTypeScriptMultiEmbedText( model="jina-v2", items=[ { "text": "Osmium is the densest naturally occurring element. When experimentally measured using X-ray crystallography, it has a density of 22.59 g/cm3. Manufacturers use its alloys with platinum, iridium, and other platinum-group metals to make fountain pen nib tipping, electrical contacts, and in other applications that require extreme durability and hardness.", "metadata": { "group": "8", }, }, { "text": "Despite its abundant presence in the universe and Solar System—ranking fifth in cosmic abundance following hydrogen, helium, oxygen, and carbon—neon is comparatively scarce on Earth.", "metadata": { "group": "18", }, }, ], collection_name="smoke_tests", embedded_metadata_keys=[ "group", ],) Output{ "embeddings": [ { "vector": [ -0.035030052065849304, -0.04128379374742508, 0.05782046541571617 ], "doc_id": "c9de81fb98804ce0afb2b8ac17c0799b", "metadata": { "group": "8", "doc_id": "c9de81fb98804ce0afb2b8ac17c0799b", "doc": "group: 8\n\nOsmium is the densest naturally occurring element. When experimentally measured using X-ray crystallography, it has a density of 22.59 g/cm3. Manufacturers use its alloys with platinum, iridium, and other platinum-group metals to make fountain pen nib tipping, electrical contacts, and in other applications that require extreme durability and hardness." } }, { "vector": [ 0.0003024724137503654, -0.025219274684786797, -0.009984994307160378 ], "doc_id": "c4464f69c93946a896925589681d38b4", "metadata": { "group": "18", "doc_id": "c4464f69c93946a896925589681d38b4", "doc": "group: 18\n\nDespite its abundant presence in the universe and Solar System\u2014ranking fifth in cosmic abundance following hydrogen, helium, oxygen, and carbon\u2014neon is comparatively scarce on Earth." } } ]}EmbedImageGenerate embedding for an image.InputOutputimage_uristringImage to embed.collection_namestringOptionalVector store name.doc_idstringOptionalVector store document ID. Ignored if collection_name is unset.modelstringOptionalSelected embedding model.Default: clip PythonTypeScriptEmbedImage( image_uri="https://media.substrate.run/docs-fuji-red.jpg", collection_name="smoke_tests",) Output{ "embedding": { "vector": [ 0.0003024724137503654, -0.025219274684786797, -0.009984994307160378 ], "doc_id": "c4464f69c93946a896925589681d38b4" }}MultiEmbedImageGenerate embeddings for multiple images.InputOutputitemsarray[EmbedImageItem]Items to embed.image_uristringImage to embed.doc_idstringOptionalVector store document ID. Ignored if collection_name is unset.collection_namestringOptionalVector store name.modelstringOptionalSelected embedding model.Default: clip PythonTypeScriptMultiEmbedImage( items=[ { "image_uri": "https://media.substrate.run/docs-fuji-red.jpg", }, { "image_uri": "https://media.substrate.run/docs-fuji-blue.jpg", }, ], collection_name="smoke_tests",) Output{ "embeddings": [ { "vector": [ -0.035030052065849304, -0.04128379374742508, 0.05782046541571617 ], "doc_id": "c9de81fb98804ce0afb2b8ac17c0799b" }, { "vector": [ 0.0003024724137503654, -0.025219274684786797, -0.009984994307160378 ], "doc_id": "c4464f69c93946a896925589681d38b4" } ]}JinaV2Generate embeddings for multiple text documents using Jina Embeddings 2.InputOutputitemsarray[EmbedTextItem]Items to embed.textstringText to embed.metadataobjectOptionalMetadata that can be used to query the vector store. Ignored if collection_name is unset.doc_idstringOptionalVector store document ID. Ignored if collection_name is unset.collection_namestringOptionalVector store name.embedded_metadata_keysarray[string]OptionalChoose keys from metadata to embed with text. PythonTypeScriptJinaV2( items=[ { "text": "Hassium is a superheavy element; it has been produced in a laboratory only in very small quantities by fusing heavy nuclei with lighter ones. Natural occurrences of the element have been hypothesised but never found.", "metadata": { "group": "8", }, }, { "text": "Xenon is also used to search for hypothetical weakly interacting massive particles and as a propellant for ion thrusters in spacecraft.", "metadata": { "group": "18", }, }, ], collection_name="smoke_tests", embedded_metadata_keys=[ "group", ],) Output{ "embeddings": [ { "vector": [ -0.035030052065849304, -0.04128379374742508, 0.05782046541571617 ], "doc_id": "c9de81fb98804ce0afb2b8ac17c0799b", "metadata": { "group": "8", "doc_id": "c9de81fb98804ce0afb2b8ac17c0799b", "doc": "group: 8\n\nHassium is a superheavy element; it has been produced in a laboratory only in very small quantities by fusing heavy nuclei with lighter ones. Natural occurrences of the element have been hypothesised but never found." } }, { "vector": [ 0.0003024724137503654, -0.025219274684786797, -0.009984994307160378 ], "doc_id": "c4464f69c93946a896925589681d38b4", "metadata": { "group": "18", "doc_id": "c4464f69c93946a896925589681d38b4", "doc": "group: 18\n\nXenon is also used to search for hypothetical weakly interacting massive particles and as a propellant for ion thrusters in spacecraft." } } ]}CLIPGenerate embeddings for text or images using CLIP.InputOutputitemsarray[EmbedTextOrImageItem]Items to embed.image_uristringOptionalImage to embed.textstringOptionalText to embed.metadataobjectOptionalMetadata that can be used to query the vector store. Ignored if collection_name is unset.doc_idstringOptionalVector store document ID. Ignored if collection_name is unset.collection_namestringOptionalVector store name.embedded_metadata_keysarray[string]OptionalChoose keys from metadata to embed with text. Only applies to text items. PythonTypeScriptCLIP( items=[ { "image_uri": "https://media.substrate.run/docs-fuji-red.jpg", }, { "image_uri": "https://media.substrate.run/docs-fuji-blue.jpg", }, ], collection_name="smoke_tests",) Output{ "embeddings": [ { "vector": [ -0.035030052065849304, -0.04128379374742508, 0.05782046541571617 ], "doc_id": "c9de81fb98804ce0afb2b8ac17c0799b" }, { "vector": [ 0.0003024724137503654, -0.025219274684786797, -0.009984994307160378 ], "doc_id": "c4464f69c93946a896925589681d38b4" } ]}Vector StoresFindOrCreateVectorStoreFind a vector store matching the given collection name, or create a new vector store.InputOutputcollection_namestringVector store name.modelstringSelected embedding model.Options: jina-v2clip PythonTypeScriptFindOrCreateVectorStore( collection_name="smoke_tests", model="jina-v2",) Output{ "collection_name": "smoke_tests", "model": "jina-v2"}ListVectorStoresList all vector stores.InputOutput PythonTypeScriptListVectorStores() Output{ "items": [ { "collection_name": "comments", "model": "jina-v2" }, { "collection_name": "images", "model": "jina-v2" } ]}DeleteVectorStoreDelete a vector store.InputOutputcollection_namestringVector store name.modelstringSelected embedding model.Options: jina-v2clip PythonTypeScriptDeleteVectorStore( collection_name="fake_store", model="jina-v2",) Output{ "collection_name": "comments", "model": "jina-v2"}QueryVectorStoreQuery a vector store for similar vectors.InputOutputcollection_namestringVector store to query against.modelstringSelected embedding model.Options: jina-v2clipquery_stringsarray[string]OptionalTexts to embed and use for the query.query_image_urisarray[string]OptionalImage URIs to embed and use for the query.query_vectorsarray[array]OptionalVectors to use for the query.query_idsarray[string]OptionalDocument IDs to use for the query.top_kinteger[1..1000]OptionalNumber of results to return.Default: 10ef_searchinteger[1..1000]OptionalThe size of the dynamic candidate list for searching the index graph.Default: 40num_leaves_to_searchinteger[1..1000]OptionalThe number of leaves in the index tree to search.Default: 40include_valuesbooleanOptionalInclude the values of the vectors in the response.Default: falseinclude_metadatabooleanOptionalInclude the metadata of the vectors in the response.Default: falsefiltersobjectOptionalFilter metadata by key-value pairs. PythonTypeScriptQueryVectorStore( collection_name="smoke_tests", model="jina-v2", query_strings=[ "gas", "metal", ], top_k=1, include_metadata=True,) Output{ "results": [ [ { "id": "483e75021c9d4ad69c3d78ace76da2ea", "distance": -0.78324556350708, "metadata": { "doc": "group: 18\n\nArgon is the third most abundant gas in Earth's atmosphere, at 0.934% (9340 ppmv). It is more than twice as abundant as water vapor.", "group": "18", "doc_id": "483e75021c9d4ad69c3d78ace76da2ea" } } ], [ { "id": "dd8f3774e05d42caa53cfbaa7389c08f", "distance": -0.74278724193573, "metadata": { "doc": "group: 8\n\nOsmium is the densest naturally occurring element. When experimentally measured using X-ray crystallography, it has a density of 22.59 g/cm3. Manufacturers use its alloys with platinum, iridium, and other platinum-group metals to make fountain pen nib tipping, electrical contacts, and in other applications that require extreme durability and hardness.", "group": "8", "doc_id": "dd8f3774e05d42caa53cfbaa7389c08f" } } ] ], "collection_name": "comments", "model": "jina-v2"}FetchVectorsFetch vectors from a vector store.InputOutputcollection_namestringVector store name.modelstringSelected embedding model.Options: jina-v2clipidsarray[string]Document IDs to retrieve. PythonTypeScriptFetchVectors( collection_name="smoke_tests", model="jina-v2", ids=[ "dd8f3774e05d42caa53cfbaa7389c08f", ],) Output{ "vectors": [ { "id": "dd8f3774e05d42caa53cfbaa7389c08f", "vector": [ 0.036658343, -0.0066040196, 0.028221145 ], "metadata": { "doc": "group: 8\n\nOsmium is the densest naturally occurring element. When experimentally measured using X-ray crystallography, it has a density of 22.59 g/cm3. Manufacturers use its alloys with platinum, iridium, and other platinum-group metals to make fountain pen nib tipping, electrical contacts, and in other applications that require extreme durability and hardness.", "group": "8", "doc_id": "dd8f3774e05d42caa53cfbaa7389c08f" } } ]}UpdateVectorsUpdate vectors in a vector store.InputOutputcollection_namestringVector store name.modelstringSelected embedding model.Options: jina-v2clipvectorsarray[UpdateVectorParams]Vectors to upsert.idstringDocument ID.vectorarray[number]OptionalEmbedding vector.metadataobjectOptionalDocument metadata. PythonTypeScriptUpdateVectors( collection_name="smoke_tests", model="jina-v2", vectors=[ { "id": "dd8f3774e05d42caa53cfbaa7389c08f", "metadata": { "appearance": "silvery, blue cast", }, }, ],) Output{ "count": 1}DeleteVectorsDelete vectors in a vector store.InputOutputcollection_namestringVector store name.modelstringSelected embedding model.Options: jina-v2clipidsarray[string]Document IDs to delete. PythonTypeScriptDeleteVectors( collection_name="smoke_tests", model="jina-v2", ids=[ "ac32b9a133dd4e3689004f6e8f0fd6cd", "629df177c7644062a68bceeff223cefa", ],) Output{ "count": 2}Audio ModelsTranscribeSpeechTranscribe speech in an audio or video file.InputOutputaudio_uristringInput audio.promptstringOptionalPrompt to guide model on the content and context of input audio.languagestringOptionalLanguage of input audio in ISO-639-1 format. Use auto to automatically detect the language.Default: ensegmentbooleanOptional(Deprecated) Segment the text into sentences with approximate timestamps.Default: falsealignbooleanOptionalAlign transcription to produce more accurate sentence-level timestamps and word-level timestamps. An array of word segments will be included in each sentence segment.Default: falsediarizebooleanOptionalIdentify speakers for each segment. Speaker IDs will be included in each segment.Default: falsesuggest_chaptersbooleanOptionalSuggest automatic chapter markers.Default: false PythonTypeScriptTranscribeSpeech( audio_uri="https://media.substrate.run/dfw-clip.m4a", prompt="David Foster Wallace interviewed about US culture, and Infinite Jest", segment=True, align=True, diarize=True, suggest_chapters=True,) Output{ "text": "language like that, the wounded inner child, the inner pain, is part of a kind of pop psychological movement in the United States that is a sort of popular Freudianism that ...", "segments": [ { "start": 0.874, "end": 15.353, "speaker": "SPEAKER_00", "text": "language like that, the wounded inner child, the inner pain, is part of a kind of pop psychological movement in the United States that is a sort of popular Freudianism that", "words": [ { "word": "language", "start": 0.874, "end": 1.275, "speaker": "SPEAKER_00" }, { "word": "like", "start": 1.295, "end": 1.455, "speaker": "SPEAKER_00" } ] } ], "chapters": [ { "title": "Introduction to the Wounded Inner Child and Popular Psychology in US", "start": 0.794 }, { "title": "The Paradox of Popular Psychology and Anger in America", "start": 16.186 } ]}GenerateSpeechGenerate speech from text.InputOutputtextstringInput text.storestringOptionalUse "hosted" to return an audio URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the audio data will be returned as a base64-encoded string. PythonTypeScriptGenerateSpeech( text="Substrate: an underlying substance or layer.", store="hosted",) Output{ "audio_uri": "https://assets.substrate.run/84848484.wav"}Utility NodesBoxCombine multiple values into a single output.InputOutputvalueAnyValues to box. PythonTypeScriptBox( value={ "a": "b", "c": { "d": [ 1, 2, 3, ], }, },) Output{ "value": { "a": "b", "c": { "d": [ 1, 2, 3 ] } }}IfReturn one of two options based on a condition.InputOutputconditionbooleanCondition.value_if_trueAnyResult when condition is true.value_if_falseAnyOptionalResult when condition is false. PythonTypeScriptIf( condition=True, value_if_true="yes", value_if_false="no",) Output{ "result": "yes"}