Generating Visual Blocks pipelines with human instructions and LLMs

Pipeline generation from instructions

We implement InstructPipe with a two-stage LLM refinement prompting strategy, followed by a pseudocode interpretation step to render a pipeline. The figure below illustrates the high-level workflow of the InstructPipe implementation. InstructPipe leverages two LLM modules (highlighted in red) — a Node Selector, and a Code Writer. Given a user instruction and a pipeline tag (e.g., a multimodal pipeline), we first devise the Node Selector to identify a list of potential nodes that would be used according to the instructions. In the Node Selector, we prompt the LLM with a very brief description of each node, aiming to filter out unrelated nodes for a target pipeline. The selected nodes and the original user input (the prompt and the tag) are then fed into the Code Writer, which generates pseudocode (i.e., a succinct code format that defines the selections and connections of the essential nodes) for the desired pipeline. In Code Writer, we provide the LLM with detailed descriptions and examples of each selected node to ensure the LLM has extensive context for each candidate node. Finally, we employ a Code Interpreter to parse the pseudocode and render a visual programming pipeline with which the user may interact.