llm-multiverse/llm-multiverse
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'feat: set up proto project structure and build tooling (#7)' (#100) from feature/issue-7-proto-structure into main

Browse Source 
feat: set up proto project structure and build tooling (#7)
This commit was merged in pull request #100.
This commit is contained in:
shahondin1624
2026-03-08 21:43:04 +01:00
parent 0616dd3d58 722d275124
commit ac61144553
14 changed files with 324 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
implementation-plans/_index.md
View File

@@ -4,6 +4,7 @@
| Issue | Title | Milestone | Status | Language | Plan |
|---|---|---|---|---|---|
| #7 | Set up proto project structure and build tooling | Phase 1 | `COMPLETED` | Protobuf | [issue-007.md](issue-007.md) |
## Status Legend
@@ -37,7 +38,7 @@ _(Plans affecting the tool broker)_
_(Plans affecting the orchestrator)_
### Proto Definitions
_(Plans affecting proto definitions and code generation)_
- [issue-007.md](issue-007.md) — Proto project structure and build tooling
### Infrastructure / DevOps
_(Plans affecting Docker, deployment, CI/CD)_

139
implementation-plans/issue-007.md Normal file
View File

@@ -0,0 +1,139 @@
# Implementation Plan — Issue #7: Set up proto project structure and build tooling
## Metadata
| Field | Value |
|---|---|
| Issue | [#7](https://git.shahondin1624.de/llm-multiverse/llm-multiverse/issues/7) |
| Title | Set up proto project structure and build tooling |
| Milestone | Phase 1: Proto Definitions |
| Labels | `type:infrastructure`, `priority:critical`, `lang:protobuf`, `cat:devops` |
| Status | `COMPLETED` |
| Language | Protobuf / Shell |
| Related Plans | None (first plan) |
| Blocked by | None |
## Acceptance Criteria
- [ ] `proto/` directory created with organized subdirectories
- [ ] Build tooling configured (buf.yaml or Makefile with protoc)
- [ ] Lint configuration for proto files
- [ ] CI-ready build script that compiles all protos
- [ ] README with proto development workflow
## Architecture Analysis
### Tool Choice: buf
`buf` is the modern standard for protobuf development. It provides:
- Linting with configurable rules
- Breaking change detection
- Code generation orchestration
- Module management
This is preferred over raw `protoc` + Makefile for maintainability.
### Directory Structure
```
proto/
├── buf.yaml # Module configuration + lint rules
├── buf.gen.yaml # Code generation config (Rust + Python)
├── buf.lock # Dependency lock file
├── README.md # Proto development workflow
├── llm_multiverse/
│ └── v1/
│ ├── common.proto # Shared types (SessionContext, AgentLineage)
│ ├── audit.proto # AuditService
│ ├── secrets.proto # SecretsService
│ ├── memory.proto # MemoryService
│ ├── model_gateway.proto # ModelGatewayService
│ ├── search.proto # SearchService
│ ├── tool_broker.proto # ToolBrokerService
│ └── orchestrator.proto # OrchestratorService
└── scripts/
└── generate.sh # CI-ready generation script
```
The `llm_multiverse/v1/` package path follows buf conventions for versioned APIs.
### Proto Package Convention
All protos use package `llm_multiverse.v1`. This gives clean generated code paths in both Rust and Python.
## Implementation Steps
### 1. Install buf (prerequisites documentation)
Document in README that `buf` CLI must be installed:
```bash
# Linux
curl -sSL https://github.com/bufbuild/buf/releases/latest/download/buf-Linux-x86_64 -o /usr/local/bin/buf && chmod +x /usr/local/bin/buf
```
### 2. Create directory structure
Create `proto/llm_multiverse/v1/` directory and placeholder `.proto` files (empty but valid — just the package declaration and syntax). These will be filled in by issues #8-#15.
### 3. Configure buf.yaml
```yaml
version: v2
modules:
- path: .
lint:
use:
- STANDARD
except:
- PACKAGE_VERSION_SUFFIX
breaking:
use:
- FILE
```
### 4. Configure buf.gen.yaml
Set up generation targets for both Rust (prost/tonic) and Python (grpcio-tools). These won't run until issues #16-#17 but the config should be ready.
### 5. Create generation script
`proto/scripts/generate.sh` — runs `buf lint`, `buf build`, and `buf generate`. CI-ready (exit on error, clear output).
### 6. Create README
Document:
- Prerequisites (buf CLI)
- Directory structure explanation
- How to add a new proto file
- How to run lint/build/generate
- Package naming conventions
## Files to Create
| File | Action | Purpose |
|---|---|---|
| `proto/buf.yaml` | Create | Module config + lint rules |
| `proto/buf.gen.yaml` | Create | Code generation targets (Rust + Python) |
| `proto/llm_multiverse/v1/common.proto` | Create | Placeholder with package declaration |
| `proto/llm_multiverse/v1/audit.proto` | Create | Placeholder |
| `proto/llm_multiverse/v1/secrets.proto` | Create | Placeholder |
| `proto/llm_multiverse/v1/memory.proto` | Create | Placeholder |
| `proto/llm_multiverse/v1/model_gateway.proto` | Create | Placeholder |
| `proto/llm_multiverse/v1/search.proto` | Create | Placeholder |
| `proto/llm_multiverse/v1/tool_broker.proto` | Create | Placeholder |
| `proto/llm_multiverse/v1/orchestrator.proto` | Create | Placeholder |
| `proto/scripts/generate.sh` | Create | CI-ready build/lint/generate script |
| `proto/README.md` | Create | Proto development workflow docs |
## Risks and Edge Cases
- **buf version compatibility:** Pin to a specific version in CI to avoid breakage
- **Placeholder protos must be valid:** Even empty protos need `syntax` and `package` declarations to pass lint
- **Generation targets:** buf.gen.yaml references plugins that won't be installed until #16/#17 — the script should handle this gracefully
## Deviation Log
_(Filled during implementation if deviations from plan occur)_
| Deviation | Reason |
|---|---|

79
proto/README.md Normal file
View File

@@ -0,0 +1,79 @@
# Proto Definitions
This directory contains all Protocol Buffer definitions for the llm-multiverse service mesh.
## Prerequisites
Install the `buf` CLI:
```bash
# Linux (x86_64)
curl -sSL https://github.com/bufbuild/buf/releases/latest/download/buf-Linux-x86_64 \
-o /usr/local/bin/buf && chmod +x /usr/local/bin/buf
# Or via Go
go install github.com/bufbuild/buf/cmd/buf@latest
```
## Directory Structure
```
proto/
├── buf.yaml # Module config + lint rules
├── buf.gen.yaml # Code generation targets (Rust + Python)
├── llm_multiverse/v1/ # All proto definitions (versioned API)
│ ├── common.proto # Shared types: SessionContext, AgentLineage
│ ├── audit.proto # AuditService (write-only append log)
│ ├── secrets.proto # SecretsService (GetSecret only)
│ ├── memory.proto # MemoryService (Write, Query, GetCorrelated)
│ ├── model_gateway.proto # ModelGatewayService (Inference, Embedding)
│ ├── search.proto # SearchService (Search)
│ ├── tool_broker.proto # ToolBrokerService (Execute, Validate, Discover)
│ └── orchestrator.proto # OrchestratorService (subagent return schema)
└── scripts/
└── generate.sh # CI-ready lint + build + generate script
```
## Usage
### Lint all protos
```bash
cd proto && buf lint
```
### Build (compile check)
```bash
cd proto && buf build
```
### Generate stubs
```bash
cd proto && bash scripts/generate.sh
```
Generated code is output to `gen/rust/src/` and `gen/python/` (at repo root).
### Breaking change detection
```bash
cd proto && buf breaking --against '.git#subdir=proto'
```
## Conventions
- **Package:** All protos use `llm_multiverse.v1`
- **Session context:** Every RPC request message must include a `SessionContext` field (defined in `common.proto`) for audit trail and broker enforcement
- **Streaming:** Use server-streaming RPCs for large/progressive output (token generation, corpus retrieval). Use unary RPCs for simple operations (cache hits, writes, validation)
- **Naming:** Service names match the service they define (e.g., `AuditService` in `audit.proto`)
## Adding a New Proto
1. Create `proto/llm_multiverse/v1/<service>.proto`
2. Set `syntax = "proto3";` and `package llm_multiverse.v1;`
3. Import `common.proto` for shared types: `import "llm_multiverse/v1/common.proto";`
4. Run `buf lint` to validate
5. Run `buf build` to compile-check
6. Update `buf.gen.yaml` if new generation targets are needed

17
proto/buf.gen.yaml Normal file
View File

@@ -0,0 +1,17 @@
version: v2
managed:
enabled: true
plugins:
# Rust: prost for types, tonic for gRPC
- remote: buf.build/community/neoeinstein-prost
out: ../gen/rust/src
opt:
- compile_well_known_types
- remote: buf.build/community/neoeinstein-tonic
out: ../gen/rust/src
# Python: grpcio-tools for types and gRPC
- remote: buf.build/protocolbuffers/python
out: ../gen/python
- remote: buf.build/grpc/python
out: ../gen/python

11
proto/buf.yaml Normal file
View File

@@ -0,0 +1,11 @@
version: v2
modules:
- path: .
lint:
use:
- STANDARD
except:
- PACKAGE_VERSION_SUFFIX
breaking:
use:
- FILE

6
proto/llm_multiverse/v1/audit.proto Normal file
View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package llm_multiverse.v1;
// Audit service interface.
// Full definitions added in issue #9.

6
proto/llm_multiverse/v1/common.proto Normal file
View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package llm_multiverse.v1;
// Shared types used across all services.
// Full definitions added in issue #8.

6
proto/llm_multiverse/v1/memory.proto Normal file
View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package llm_multiverse.v1;
// Memory service interface.
// Full definitions added in issue #11.

6
proto/llm_multiverse/v1/model_gateway.proto Normal file
View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package llm_multiverse.v1;
// Model Gateway service interface.
// Full definitions added in issue #12.

6
proto/llm_multiverse/v1/orchestrator.proto Normal file
View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package llm_multiverse.v1;
// Orchestrator service interface and subagent return schema.
// Full definitions added in issue #15.

6
proto/llm_multiverse/v1/search.proto Normal file
View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package llm_multiverse.v1;
// Search service interface.
// Full definitions added in issue #13.

6
proto/llm_multiverse/v1/secrets.proto Normal file
View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package llm_multiverse.v1;
// Secrets service interface.
// Full definitions added in issue #10.

6
proto/llm_multiverse/v1/tool_broker.proto Normal file
View File

@@ -0,0 +1,6 @@
syntax = "proto3";
package llm_multiverse.v1;
// Tool Broker service interface.
// Full definitions added in issue #14.

28
proto/scripts/generate.sh Executable file
View File

@@ -0,0 +1,28 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROTO_DIR="$(dirname "$SCRIPT_DIR")"
cd "$PROTO_DIR"
echo "=== buf lint ==="
buf lint
echo "Lint passed."
echo ""
echo "=== buf build ==="
buf build
echo "Build passed."
echo ""
echo "=== buf generate ==="
if buf generate 2>/dev/null; then
echo "Generation complete."
else
echo "Generation skipped (plugins not yet configured or unavailable)."
echo "This is expected until issues #16/#17 set up Rust and Python stub generation."
fi
echo ""
echo "All checks passed."