A methodology for building executable specifications that supercharge AI-driven development

Based on comprehensive research into existing BDD frameworks, AI optimization patterns, technical implementation strategies, cross-language approaches, and practical examples, this report presents a unified methodology for creating living documentation through specifications specifically optimized for AI-driven Elixir development.

The Problem with Traditional BDD in AI Workflows

Traditional BDD tools like Cucumber were designed for human collaboration, not AI assistance. Research reveals critical limitations:

Verbosity and Token Inefficiency: Gherkin's natural language approach consumes significant context window space with phrases like "Given I am on the registration page" instead of structured data. Token analysis shows switching from natural language to structured formats like YAML can save 190 tokens per request on average.

Ambiguous References: Steps like "click the first result" or "the user should see appropriate content" lack the precision AI needs for reliable code generation. AI models struggle with implicit context and require explicit, unambiguous specifications.

Complex State Management: The Given-When-Then structure enforces procedural thinking with complex state passing between steps, making it difficult for AI to understand the complete context and generate correct implementations.

Framework Fragmentation: The Elixir BDD ecosystem is fragmented (ESpec, Cabbage, White Bread) with no clear winner. Most have maintenance issues, limited adoption, and poor AI tool integration. This creates additional complexity for AI assistants trying to generate appropriate test code.

The AI-Optimized Specification Approach

Core Innovation: Structured Living Documentation

The methodology combines three key innovations:

1. Property-Based Specifications (inspired by Clojure.spec and QuickCheck)
2. Contract-Driven Development (adapted from Design by Contract)
3. AI-Optimized Formats (following llms.txt principles)

Implementation Architecture

defmodule SpecFramework do
  @moduledoc """
  Living documentation framework optimized for AI-driven development.
  Combines property-based testing, contracts, and structured specifications.
  """
  
  defmacro __using__(_opts) do
    quote do
      import SpecFramework.DSL
      use ExUnit.Case, async: true
      
      @before_compile SpecFramework.Compiler
    end
  end
end

defmodule SpecFramework.DSL do
  defmacro specification(name, opts \\ [], do: block) do
    quote do
      @spec_metadata %{
        name: unquote(name),
        context: unquote(opts[:context]) || %{},
        examples: unquote(opts[:examples]) || [],
        properties: unquote(opts[:properties]) || []
      }
      
      # Generate both test and documentation
      test "Spec: #{unquote(name)}" do
        unquote(block)
      end
      
      # Generate AI-readable documentation
      @doc """
      ## Specification: #{unquote(name)}
      
      #{unquote(opts[:description])}
      
      ### Context
      #{inspect(unquote(opts[:context]), pretty: true)}
      
      ### Examples
      #{format_examples(unquote(opts[:examples]))}
      """
      def unquote(:"spec_#{name |> String.replace(" ", "_")}")() do
        unquote(block)
      end
    end
  end
end

Practical Specification Format

1. YAML-Based Specifications (For Complex Systems)

# specs/user_authentication.yaml
specification:
  name: "User Authentication System"
  version: "1.0.0"
  elixir_context:
    framework: "Phoenix"
    version: "1.7+"
    patterns: ["changeset_validation", "guardian_jwt"]
    
  contracts:
    - name: "register_user"
      input:
        type: "map"
        required: ["email", "password"]
        properties:
          email:
            type: "string"
            format: "email"
            constraints: ["unique", "lowercase"]
          password:
            type: "string"
            min_length: 8
      output:
        success: 
          pattern: "{:ok, %User{}}"
          fields: ["id", "email", "inserted_at"]
        error:
          pattern: "{:error, %Ecto.Changeset{}}"
          
  properties:
    - description: "All registered emails are unique"
      generator: "email_generator()"
      assertion: |
        forall email <- email_generator() do
          register_user(%{email: email, password: "valid_pass"})
          case register_user(%{email: email, password: "other_pass"}) do
            {:error, changeset} -> 
              assert {:email, {"has already been taken", _}} in changeset.errors
            _ -> false
          end
        end
        
  examples:
    - name: "Successful registration"
      input: {email: "test@example.com", password: "secure123"}
      output: {:ok, %User{email: "test@example.com"}}
    - name: "Duplicate email"
      setup: "register_user(%{email: 'existing@example.com', password: 'pass123'})"
      input: {email: "existing@example.com", password: "newpass123"}
      output: {:error, %Ecto.Changeset{errors: [email: {"has already been taken", []}]}}

2. Markdown + Code (For Simpler Specifications)

# Payment Processing Specification

> AI Context: Phoenix 1.7+, Stripe integration, idempotency required

## Contract

@spec process_payment(map()) :: {:ok, Payment.t()} | {:error, atom() | Changeset.t()} def process_payment(%{ "amount" => amount, "currency" => currency, "source" => source, "idempotency_key" => key }) when is_integer(amount) and amount > 0

## Properties

1. **Idempotency**: Same payment request with same key returns same result
2. **Amount validation**: Only positive integers accepted
3. **Currency support**: Only USD, EUR, GBP supported

## Examples

### Success Case

process_payment(%{ "amount" => 1000, # $10.00 "currency" => "USD", "source" => "tok_visa", "idempotency_key" => "order_123" })

=> {:ok, %Payment{id: "pay_xyz", amount: 1000, status: "succeeded"}}

### Duplicate Request

First call

= process_payment(%{"idempotency_key" => "order_123", ...})

Second call with same key

= process_payment(%{"idempotency_key" => "order_123", ...})

assert payment1.id == payment2.id # Same payment returned

Integration with Mix Test

Custom Mix Task

# lib/mix/tasks/spec.ex
defmodule Mix.Tasks.Spec do
  use Mix.Task
  
  @shortdoc "Runs specifications with AI-optimized reporting"
  
  def run(args) do
    Mix.Task.run("app.start")
    
    # Configure for specification mode
    ExUnit.configure(
      formatters: [SpecFramework.Formatter],
      include: [spec: true]
    )
    
    # Load specifications
    load_yaml_specs()
    load_markdown_specs()
    
    # Run tests
    ExUnit.run()
    
    # Generate living documentation
    SpecFramework.DocGenerator.generate()
  end
  
  defp load_yaml_specs do
    Path.wildcard("specs/**/*.yaml")
    |> Enum.each(&SpecFramework.YamlLoader.load/1)
  end
end

Living Documentation Generation

defmodule SpecFramework.DocGenerator do
  def generate do
    specs = collect_all_specs()
    
    # Generate llms.txt for AI tools
    generate_llms_txt(specs)
    
    # Generate ExDoc compatible documentation
    generate_exdoc_pages(specs)
    
    # Validate specification coverage
    validate_implementation_coverage(specs)
  end
  
  defp generate_llms_txt(specs) do
    content = """
    # #{Mix.Project.config()[:app]} Specifications
    
    > Living documentation for AI-driven development
    
    ## Specifications
    #{Enum.map(specs, &format_spec_link/1) |> Enum.join("\n")}
    
    ## Context
    - Elixir #{System.version()}
    - Phoenix #{Application.spec(:phoenix, :vsn)}
    - Property-based testing with StreamData
    - Contract validation with compile-time checks
    """
    
    File.write!("llms.txt", content)
  end
end

Framework vs. Methodology Decision

The research strongly suggests starting with a methodology rather than a full framework:

Why Methodology First

1. Lower Adoption Barrier: Teams can gradually adopt practices without changing their entire test infrastructure
2. ExUnit Compatibility: Leverages existing Elixir testing tools rather than replacing them
3. Flexibility: Allows teams to customize the approach for their specific needs
4. AI Tool Evolution: As AI tools rapidly evolve, a methodology can adapt more easily than a rigid framework

Core Methodology Components

1. Specification Structure

• Use YAML/JSON for complex data structures and API contracts
• Use Markdown with code blocks for behavioral specifications
• Include explicit AI context blocks with framework versions and patterns
• Organize specifications alongside code in specs/ directory

2. Property-Based Contracts

defmodule UserSpec do
  use ExUnitProperties
  
  property "email uniqueness is enforced" do
    check all email <- StreamData.string(:alphanumeric, min_length: 5) do
      email_with_domain = "#{email}@test.com"
      
      # First registration should succeed
      assert {:ok, _} = register_user(%{email: email_with_domain, password: "pass123"})
      
      # Second registration should fail
      assert {:error, changeset} = register_user(%{email: email_with_domain, password: "pass456"})
      assert "has already been taken" in errors_on(changeset).email
    end
  end
end

3. Living Documentation Validation

# test/support/spec_validator.ex
defmodule SpecValidator do
  def validate_module_specs(module, spec_file) do
    specs = parse_spec_file(spec_file)
    actual_functions = module.__info__(:functions)
    
    Enum.each(specs.contracts, fn contract ->
      assert {contract.name, contract.arity} in actual_functions,
        "Missing implementation for #{contract.name}/#{contract.arity}"
        
      # Validate examples actually work
      Enum.each(contract.examples, fn example ->
        result = apply(module, contract.name, example.input)
        assert match?(example.output_pattern, result),
          "Example failed: expected #{inspect(example.output_pattern)}, got #{inspect(result)}"
      end)
    end)
  end
end

Best Practices for AI-Driven Development

1. Context-Rich Specifications

Always include an AI context block:

ai_context:
  elixir_version: "1.15+"
  framework: "Phoenix LiveView"
  key_patterns:
    - "Use pattern matching for all function definitions"
    - "Return {:ok, result} | {:error, reason} tuples"
    - "Use with statements for complex error handling"
  avoid:
    - "Don't use exceptions for control flow"
    - "Don't create deeply nested case statements"

2. Executable Examples Over Abstract Descriptions

Instead of: "The function should validate user input"

Write:

examples:
  - input: %{email: "test@example.com", age: 25}
    output: {:ok, %User{email: "test@example.com", age: 25}}
  - input: %{email: "invalid", age: 25}
    output: {:error, %{email: ["invalid format"]}}
  - input: %{email: "test@example.com", age: -1}
    output: {:error, %{age: ["must be greater than 0"]}}

3. Property Definitions for Edge Cases

properties:
  - name: "handles all valid email formats"
    generator: "valid_email_generator()"
    assertion: "validate_email(email) == {:ok, normalize_email(email)}"
    
  - name: "rejects emails over 255 characters"
    generator: "string(:alphanumeric, min_length: 256)"
    assertion: "match?({:error, _}, validate_email(email <> \"@test.com\"))"

Comparison with Traditional Approaches

Implementation Roadmap

Phase 1: Adopt the Methodology (Month 1)

1. Create specs/ directory structure
2. Write first specifications in YAML/Markdown
3. Add property-based tests for core functions
4. Generate llms.txt file for AI tools

Phase 2: Build Supporting Tools (Months 2-3)

1. Create specification parser for YAML/Markdown
2. Build ExUnit integration helpers
3. Implement specification validator
4. Add mix spec task

Phase 3: Expand Coverage (Months 3-6)

1. Convert existing tests to specifications
2. Train team on specification writing
3. Integrate with CI/CD pipeline
4. Generate comprehensive documentation

Conclusion

The future of AI-driven development in Elixir lies not in adapting human-centric BDD tools, but in creating a new methodology that serves both human understanding and machine processing. By combining property-based testing, contract specifications, and AI-optimized formats, teams can create living documentation that:

• Reduces development time by providing clear, unambiguous specifications for AI assistants
• Improves code quality through property-based testing and contract validation
• Maintains synchronization between documentation and implementation automatically
• Scales efficiently with the codebase while remaining maintainable

This methodology represents a fundamental shift from "writing tests for humans to read" to "writing specifications that both humans and AI can understand and execute." The key is starting with simple practices and gradually building more sophisticated tooling as the approach proves its value.