Solution Generation Workflow

Last updated at 2026-07-30 AEST.

A walkthrough of how the platform turns a "generate solution" action into a structured, versioned solution model — the architecture design, current-state narrative, rendered architecture diagrams, solution overview, and priced commercial model that describe how the deal will be solved, without yet assembling any document. The workflow lives at solution-generation-workflow.ts and is composed of eight Mastra steps.

At a glance

A handful of numbers capture the shape of the workflow — how it's structured, what it calls, and how it's bounded:

Overview

Solution generation produces the platform's solution model: a deterministic, structured architecture design (components, integrations, diagrams, ADRs, security controls, compliance mappings, professional-services work packages) plus two rendered narrative sections — a current-state analysis and a solution overview — with architecture diagrams rendered to durable images and embedded into the overview, and a priced commercial model for the finalized version. It is the upstream design stage: it runs before, and independently of, every document-producing workflow, so a user can review, iterate on, and approve the solution before committing to a proposal. The characteristics below describe what the code actually does.

The remainder of this document walks the flow end to end, from the triggering mutation through to where the finished solution model is stored.

1. Where it fits

Solution generation sits between two neighbours in the deal lifecycle:

• Upstream — deal context & solution strategy. It reads the latest DealContext (the structured, versioned understanding of the deal — see Deal Context Generation) and the deal's usable approved SolutionStrategy — approved and non-stale (see Solution Strategy). A stale approved strategy is not valid grounding, so it counts as no strategy at all — and because a usable strategy is a hard precondition, such a deal is refused rather than generating on bad inputs.
• Downstream — three consumers of the persisted output. The proposal-generation workflow and the solution-document workflow both load the persisted SolutionModel — with its solution-overview section, its current-state section, and its pre-rendered diagram assets — through a shared loader step, and assemble a document around it. The solution builder UI reads the same rows to review, edit, and approve a version.

This is why the heavy architecture work lives here: solution generation builds and persists the design once, and every downstream surface reuses that persisted output rather than regenerating it — so the solution never diverges between surfaces. Approval is also the gate: a proposal can only be generated against an approved, non-stale SolutionModel version.

Two sections live here, not in the proposal. currentState and solutionOverview are solution-owned proposal sections: they are generated once by this workflow, persisted onto the SolutionModel, and consumed verbatim by the document workflows. The proposal's refinement loop keeps them visible to its judge (so conformance can be scored) but treats findings against them as non-routable — a HIGH finding on solution content means "regenerate the solution", not "rewrite the proposal".

2. Entry point

The user triggers solution generation from the web app, which calls a single GraphQL mutation on the API:

• Mutation: initiateSolutionGeneration

This is a fire-and-forget mutation: it returns almost immediately with a job handle (AsyncProcessingJob) and the actual work runs asynchronously. The input carries a dealId, optional free-text customInstructions, and an optional shouldForceRegeneration flag. The resolver:

1. Validates the deal exists and belongs to the caller's organization.
2. Validates the deal has requirements — a solution can't be designed against an empty deal.
3. Resolves the latest deal context for the deal, failing with a user-readable BadRequestException when there is none.
4. Creates an AsyncProcessingJob of type GenerateSolution and entity type Deal, linked to that deal context, starting in Pending.
5. Publishes the Pub/Sub message (section 3) and returns the job to the UI.

A usable approved solution strategy — approved and non-stale — is a hard precondition here: a deal without one is refused with a message naming the missing grounding, rather than generating an un-actionable solution version that nothing commercial hangs off. The same rule is re-applied on every later delivery of the job, so an automatic retry cannot quietly resume a run whose strategy has since changed. As with every workflow on the platform, identity is taken from the authenticated request — the resolver never trusts a caller-supplied user or organization ID.

3. The Pub/Sub message

The resolver publishes a small message to a Google Cloud Pub/Sub topic — the hand-off from the synchronous request to the in-process background listener.

• Topic: <environment>.deal-output.generate-solution (the literal name is environment-prefixed and resolved per environment via env vars; e.g. api-1ke-local-au.deal-output.generate-solution).
• Schema: solutionGenerationPubSubMessageDataPayloadSchema

Payload:

{
  "correlationId": "<asyncProcessingJobId>",
  "payload": {
    "asyncProcessingJobId": "...",      // required
    "dealId": "...",                    // required
    "userId": "...",                    // who triggered the run
    "dealContextId": "...",             // optional — latest is resolved if absent
    "customInstructions": "...",        // optional free-text steering
    "shouldForceRegeneration": false    // optional — bypass the cross-run cache
  }
}

The message carries only pointer IDs and small flags — the runner loads everything else from the database. No business data is on the wire.

4. How the message is executed

There are two execution paths, selected per workflow by the WORKFLOW_PUSH_JOBS_ENABLED env var. Solution generation is in that set in dev and both production regions, so the push path is the deployed reality; the pull path is what runs locally.

• Push path (deployed). A push subscription POSTs the message to the OIDC-gated workflow-dispatcher Cloud Run service, which claims the job with a compare-and-swap (Pending → InProgress) before starting anything, then launches a dedicated Cloud Run Job execution (2Gi, a 1-hour task timeout, up to 3 task retries), stamping the workflow type and the verbatim base64 message data as container-override env vars. The Job's entrypoint boots a slim Nest application context — runner services only, no HTTP, no GraphQL, no subscribers, no recovery sweeps — re-validates the payload through the same Zod schema, publishes the InProgress event (the dispatcher has no Redis), and drives the shared runner.
• Pull path (local). An in-process streaming-pull listener inside the API claims the job — with a DLQ attached it claims Pending or Failed so a dead-letter replay can re-claim; without one, only Pending, so a failed job can't loop forever — and drives the same shared runner. Its flow-control window is env-bounded (a 10-minute max ack deadline, a 60-minute max lease extension, and a small concurrent-message cap per instance).

What the shared runner does

The workflow body lives in SolutionGenerationRunnerService and is driven identically by both paths:

1. Re-read the job pointers — customer, deal name, organization — and validate the deal has requirements; an empty deal fails here.
2. Resolve the grounding deal context (the version pinned by id, else the latest) and the deal's usable approved SolutionStrategy — approved and non-stale. A stale one yields null, and the delivery is refused rather than run: the same grounding rule the resolver applied is re-asserted here, so a redelivery days later cannot resume against a strategy that has moved on. The refusal is deterministic, so the message is acknowledged and the job goes terminal with the reason on it, instead of redelivering against a condition that cannot clear by itself.
3. Initialize or restore WorkflowProgress with one WorkflowStep row per step, walking the canonical runtime order derived by solutionGenerationWorkflowStepOrder() (currently eight). On retry, the not-yet-completed steps are reset and the isRetry flag is set.
4. Compile the commercial proposal context off that usable strategy and stamp its snapshot id onto WorkflowProgress. This is what lets the solution-model agent present priced, committed scope — without it every component is classified non-committed. The compile is idempotent on a content-hash match (no version bump, no event publish on the reuse branch), so re-compiling on every regeneration is cheap. It is skipped only when the strategy carries no solution-context snapshot, and never throws — the strategy itself is guaranteed present by the grounding gate above.
5. Seed the Mastra request context with the pointers and service handles the steps need (Google Cloud Storage, Redis pub/sub, the commercial-model builder and publisher, the workflow-progress / step-id map, the isRetry and shouldForceRegeneration flags, the resolved dealContextId / solutionStrategyId / proposalContextSnapshotId / solutionModelId). Mastra steps can't read Nest DI directly, so they read these from the request context at run time.
6. Create and start the workflow run (section 5), surfacing its AbortController to a run registry so an external cancel — or a Job SIGTERM — can interrupt an in-flight step.
7. On success, finalize the diagram assets, log the deal activity, flip the job to Completed (unless it was cancelled out from under the run), then publish the real-time events (section 9).

Shutdown semantics differ by path, deliberately. The pull subscriber registers a handler that reverts an in-flight job to Pending on SIGTERM. The Job path does the opposite: it aborts the run and exits non-zero, leaving the row InProgress so Cloud Run task-retry resumes from the per-step WorkflowStep checkpoints.

Why the request-context keys matter. Mastra steps can't read Nest DI directly, so each step resolves its inputs from the request context by a fixed set of key names. This workflow's requestContext constants populate exactly those keys, so every step observes the contract it depends on — cache discrimination, solution-model version allocation, and step-progress publishing all work as intended.

5. The workflow

The workflow is a Mastra workflow (solutionGenerationWorkflow) of eight steps. It is mostly a sequential spine, with a single parallel fan-out of two branches in the middle: once the structured solution model exists, the current-state narrative and the diagram specs can be produced independently.

The .map(async () => ({})) between the parallel block and the next step is a deliberate join-and-reset: the two branches write their results into the workflow's shared state (the current-state section and the diagram items), so the merged tuple that .parallel pipes forward is discarded — every downstream step reads what it needs from state, not from the piped value.

The eight steps

6. What each step does

1 · Fetch deal context — loads the deal's latest DealContext version (or the explicitly supplied one), validates it against dealContextSchema, and writes it into workflow state. Pure data fetching, no LLM.

2 · Generate solution model — the heart of the workflow. A single frontier agent reads the deal context, requirements, the optional compiled proposal context, and any custom instructions, and emits the structured SolutionModel JSON: components, integrations, diagrams, architecture decision records, non-functional-requirements mapping, security controls, compliance mappings, and professional-services work packages. The result is Zod-validated, then run through two deterministic guards before anything is persisted:

• Service-offering reconciliation. The agent stamps each priced component's serviceOfferingId so the commercial builder can join the designed quantity to its pricing line by exact id. Because LLM output can't be trusted to stay inside the provided offering set, every stamp is reconciled against the live in-scope offerings and any unrecognized id is dropped (and all stamps are dropped when there is no proposal context at all).
• Fixed-footprint sizing smell-check. A committed component whose delivery model implies a small, fixed footprint (cloud tenants, hosted instances, data centres, a single project) but which is sized at population scale is almost certainly mis-sized against the raw demand population. Rather than shipping a silently-inflated number, its status is escalated to RequiresCommercialValidation for human review.

The validated model is what gets persisted — as the Generation audit row's text and as a new SolutionModel row whose version number is allocated under a Postgres advisory lock inside an interactive transaction (a 20-second timeout giving headroom for lock contention + retry backoff). The row also captures the institutional-overlay, commercial-profile, ontology-release and solution-context-snapshot versions it was compiled against. This one structured artifact is the design brain every later step is grounded in.

3 · Generate current state (parallel branch A) — a frontier section agent writes the "Current State" narrative: the customer's existing landscape, technology profile, gaps, pain points, and key integrations, structured into a fixed sub-section outline.

4 · Generate diagram specs (parallel branch B) — a deterministic step with no LLM call and no repair loop. For each diagram specification the solution-model agent embedded in the model it:

1. Self-heals locally-cosmetic authoring defects — drops edges pointing at undeclared nodes and collapses duplicate node ids — so a single dangling edge no longer discards an entire already-completed solution model.
2. Validates the sanitized spec against the strict diagramSpecSchema (unique node ids, every edge endpoint resolving to a declared node).

A spec still invalid after pruning (e.g. zero declared nodes), or missing entirely, is a deterministic failure. The set is capped at 6 diagrams per run as a soft slice at the consumption site — over-generation truncates rather than hard-failing the upstream agent's parse.

How many diagrams should exist is decided upstream, in the prompt, by a shared diagram-count policy module that is the single source of truth for both the prompt text and the quality gate's floor. Count scales with the effective solution depth level (L1: 0–1, L2: 1+, L3: 2+, L4: 2–3) and with declarative engagement-mix rules — e.g. a Professional Services mix requires at least one target-design diagram; a Managed Services mix at L3 or deeper requires a service-operations view; a Resell-only engagement above a SKU threshold requires a product-integration view. The 6-diagram cap here is a cost backstop, not the target.

5 · Render diagram images — not an LLM call. Each validated spec is POSTed to the self-hosted graph-diagram-renderer HTTP service (a separate Cloud Run service: elkjs layout + hand-authored SVG, resvg for the PNG), which returns both SVG and PNG in a single call. Both images are streamed to durable, private GCS objects and the embedded reference is rewritten to the SVG object path (section 7).

6 · Generate solution overview — a frontier section agent writes the "Solution Overview" narrative — a walkthrough of the architecture diagrams, components, integrations, and how the solution maps to requirements — then a post-generation transform injects the rendered diagram images (with headings, links, and a scope disclaimer) into the section.

7 · Persist solution — the native step that commits the rendered sections to the SolutionModel (section 9).

8 · Build commercial model — prices the finalized solution version into its own 1:1 CommercialModel, grounded on the deal-context version and solution strategy the persist step just stamped, and publishes commercialModelChanged. The build is idempotent on a content-hash match, and a build failure is non-fatal — it never fails the already-persisted solution.

Agents and their model tiers

Each agent runs on a fallback chain — providers are tried in order until one succeeds. All three reasoning-heavy agents use the frontier chain.

Agent	Step	Tier	Fallback order
Solution model generation	2	Frontier	gpt-5.5 → claude-opus-4-8 → gemini-3.1-pro-preview
Current state generation	3	Frontier	gpt-5.5 → claude-opus-4-8 → gemini-3.1-pro-preview
Solution overview generation	6	Frontier	gpt-5.5 → claude-opus-4-8 → gemini-3.1-pro-preview

Every agent call is bounded by a 5-minute per-call timeout (SECTION_AGENT_CALL_TIMEOUT_MS, shared with the proposal's section steps) inside a stall guard whose absolute wall-clock ceiling across all in-process tier retries is 13 minutes, with a 4-minute idle-liveness window for high-reasoning structured output. Each call writes a Generation audit row (system prompt, user prompt, model id, token usage, timing, response snapshot). Steps 4 and 5 (diagram spec validation and rendering) are deterministic — no model call — so they are not in this table.

7. The diagram sub-pipeline

Steps 4 → 5 form a two-stage sub-pipeline that turns each diagram specification in the solution model into a durable, embeddable image. Two properties are crucial: rendering happens in a separate self-hosted service (no third-party render SaaS ever sees customer data), and the persisted reference is a durable GCS object path — never an ephemeral signed URL — so the image embedded in the solution overview never decays.

The render step (step 5) iterates the validated diagram specs and, for each one:

• Re-validates the spec client-side against the strict renderable-spec schema — the same schema the service re-validates with, so there is zero drift — before serializing it. A parse failure deterministic-fails the job and saves a round trip.
• POSTs it to the graph-diagram-renderer service, which returns both the SVG and the base64 PNG in one response. The service URL and request-timeout budget are read from env at the boundary (the Mastra step has no Nest DI at the call site) and are both required at boot.
• Streams both images to private GCS objects under the solution-diagrams path prefix, keyed by organization, deal, solution model, and diagram; the canonical SVG is uploaded last, so a failed sibling upload throws before any .svg object exists.
• Rewrites the embedded image reference to the durable SVG GCS object path, so the solution-overview section (generated next) embeds the durable path, not a signed URL. The PNG sibling is derived from the SVG object key by extension swap at the document-render boundary.
• Records an upload descriptor (object path, title, description, type, file size, sort order) in state for the runner's finalize transaction to turn into Asset + SolutionModelDiagram rows.

Because the render is now a real network hop, its failures are classified by HTTP status class — and that classification is load-bearing for the job's retry behaviour:

Outcome	Classification	Effect
Non-retryable 4xx — 422 invalid spec, 413 payload too large, 404/405 misroute, 400	Deterministic	Reproduces on replay, so the job fails terminally rather than nack-looping and burning the agent ceiling.
429, any 5xx, transport reject, abort-timeout	Transient	One deadline-bounded retry inside the client for a momentary blip; if it still fails, the job retries from its step checkpoints.

Signed URLs are minted only at read time, in the SolutionModelDiagram.imageUrl resolver — never persisted.

8. Caching, retries & failure handling

Every step writes a WorkflowStep row with started / completed / failed timestamps, which powers the live progress UI and underpins resumption.

• Retry-resume. When isRetry is set, a step that already completed in the current run reuses its prior result instead of re-running — so a redelivery picks up where it left off rather than regenerating everything.
• Cross-run cache. The generation steps match a prior run's completed WorkflowStep on the tuple of grounding inputs (deal-context id, solution-strategy id, custom-instructions hash, step id). On a hit, the prior result is reused — an unchanged solution skips the frontier agent calls entirely. shouldForceRegeneration bypasses this.

Failures are classified the same way as every async-processing job on the platform — deterministic (a schema/contract violation surfaced as DeterministicWorkflowError, which reproduces on replay) versus transient — but the two execution paths route them oppositely, because "retry" means something different in each:

• Deterministic failures mark the job Failed and stop — acked on pull, exit 0 on push.
• Transient failures retry. On pull the job reverts to Pending and the message nacks (Pub/Sub redelivers up to the retry budget, then dead-letters). On push the row stays InProgress and the process exits non-zero so Cloud Run task-retry resumes from the WorkflowStep checkpoints.
• The diagram finalize transaction is compensated. Diagram images are already durable GCS objects by the time the runner writes their rows, so a rollback of that transaction would orphan them: the compensation path best-effort deletes both the SVG and its derived PNG sibling before rethrowing. The post-commit event publish sits deliberately outside that try — once committed, the SolutionModel is the source of truth and a dropped Redis publish must never tear down uploaded files.
• The commercial-model build is non-fatal. A failure there never fails the already-persisted solution.
• Cancellation is checked after the run and at step boundaries; a cancelled job exits cleanly without flipping to failed. An external cancel — or a Job SIGTERM — aborts an in-flight step through the registered AbortController.
• A recovery sweep runs on startup and every 15 minutes, failing jobs stuck past their stale thresholds. On the push path the stale-InProgress threshold widens to 270 minutes so it can't reap a Job mid task-retry.

9. Where the result is stored & real-time updates

The output is structured rows in Postgres plus diagram images in GCS — not a document file. Persistence happens in two native steps:

After the run completes, the runner finalizes the diagram assets transactionally from the render step's upload descriptors: it creates a private, system-generated Asset row per diagram and a SolutionModelDiagram row linking the asset to the solution model. This is idempotent on retry — a prior run's diagram rows are deleted before the new ones are created — and any orphaned GCS objects are cleaned up if finalization fails.

During the persist step, the SolutionModel row (created back in step 2) is updated with:

• currentStateContent — the rendered current-state section (JSON). Downstream document workflows consume this verbatim rather than regenerating it, so all documents share one source of truth.
• overviewContent — the rendered solution-overview section (JSON), with diagram images embedded as durable GCS object paths.
• solutionStrategyId — the usable approved strategy this solution was built against. Written when the row is created, not three steps later at persist time, so a solution version is attributable to its grounding for its whole life; a run that somehow reached persistence without one fails loudly rather than writing a strategy-less version.
• capturedDealContextVersion — the deal-context version the solution was grounded on, so the UI — and the downstream loaders' hard staleness gate — can later detect that newer deal context has drifted past this solution.

Finally, the runner records a "Generated solution model version N" DealActivity (idempotent on the solution-model id), flips the AsyncProcessingJob to Completed, and publishes the real-time events:

• solutionModelChanged over Redis pub/sub, scoped to the organization, so every open client tab viewing the deal sees the new solution model immediately. This publish is fire-and-forget — a transient Redis failure must not fail a completed run, and the view self-heals off the job-status event.
• asyncProcessingJobStatusChanged, so the UI's job indicator transitions to complete.

The commercial-model step (step 8) additionally publishes commercialModelChanged as it runs, so any open Commercials view re-prices in real time.

Approval and the quality gate

A generated version is not yet usable for a proposal. Two more pieces of state hang off SolutionModel:

• isApproved — at most one approved version per deal (an exclusive approval, indexed on (dealId, isApproved)). This, not strategy approval, is the proposal-generation gate.
• qualityGateReview — a 1:1 review row computed either on demand from the UI or pre-flight by proposal generation. Its isPassed verdict is what stamps a generated proposal as a draft; it never blocks generation.

10. Security

The workflow consumes customer deal data and renders its own diagram images, so the controls apply at the API boundary, on the wire, at the dispatch hop, at the render hop, and at the storage edge.

• Identity is never caller-supplied. The triggering mutation derives the logged-in user — and therefore the organization — from the authenticated request. Every read and write in the runner and the workflow steps is scoped to that resolved organizationId, including the SolutionModel.update in the persist step.
• The wire carries only pointers. The Pub/Sub message is pointer IDs and small flags; no business data leaves the process boundary. The Cloud Run Job container override carries that same pointer-only payload verbatim.
• The dispatch hop is authenticated and minimally privileged. Every dispatcher route is gated by GoogleOidcGuard against a fixed audience, so only the allowlisted Pub/Sub push service account can invoke it, and the dispatcher holds only the database-connection secret — never the API's full secret set.
• Diagram rendering is self-hosted. Architecture diagrams are rendered by an internal graph-diagram-renderer Cloud Run service from typed, schema-validated specs and streamed straight to GCS as buffers — no third-party render SaaS receives customer data, and no external image URL is fetched. The spec is validated against the same strict schema on both sides of the hop.
• Diagram images live in a private bucket. Persisted references are durable GCS object paths, and the bucket is private; signed URLs are minted only at read time in the diagram resolver, never persisted.
• Agent output is validated, then re-checked. Every agent result is Zod-validated against its output schema, and code — not the model — owns the authoritative IDs and version numbers written to the database. Beyond validation, the service-offering reconciliation and fixed-footprint sizing guards (section 6) treat the model's own stamps and quantities as untrusted and correct or flag them before persistence.