Skip to main content

Semantic Kernel Memory & Connectors

Memory and connectors are the backbone of any knowledge‑driven AI application. Memory gives agents the ability to store and retrieve facts, documents, and past interactions. Connectors link the kernel to external systems—vector databases, search services, APIs, and enterprise data sources. Together they enable retrieval‑augmented generation (RAG), personalised experiences, and grounded, trustworthy responses.

This handbook article is a practical, implementation‑focused guide to building memory and connector layers in Semantic Kernel. You’ll learn how to configure semantic memory, choose a vector store, implement RAG patterns, secure connectors, and optimise retrieval performance. All code examples are in Python with notes for .NET/Java developers.

What Are Memory and Connectors in Semantic Kernel

Memory in Semantic Kernel is a persistent, searchable knowledge layer. It is not the same as the short‑term conversation history that an agent maintains during a single dialogue turn. Memory stores information as embeddings in a vector database, enabling semantic search across large collections of text. When a user asks a question, the kernel can retrieve the most relevant pieces of stored knowledge and inject them into the prompt, grounding the LLM’s answer in real data.

Connectors are the integration layer. They allow the kernel to communicate with external services—anything from an Azure AI Search index to a Postgres database, a REST API, or a Microsoft Graph endpoint. Connectors abstract away the complexity of authentication, networking, and data serialisation so that your plugins and agents can access external data sources with minimal code.

A simple example: a support agent that remembers the user’s preferred contact method (stored in memory) and uses a connector to look up order details from a SQL database.

# Conceptual snippet – actual code will vary by provider
memory.save_information(collection="users", id="user123", text="Prefers email for notifications.")
# Later...
relevant_facts = await memory.search("users", "How does user123 want to be contacted?")
# Use facts in prompt

# Connector to get order details
order_data = await sql_connector.query("SELECT * FROM orders WHERE id = @id", {"id": order_id})

This combination—memory for knowledge retrieval, connectors for external data—makes Semantic Kernel applications context‑aware and deeply integrated with enterprise systems.

Why Memory Matters

  • Context retention – maintain facts across user sessions and long‑running workflows.
  • Knowledge grounding – provide the LLM with authoritative, up‑to‑date information from your documents.
  • Personalised experiences – store user preferences, past interactions, and decisions.
  • Reduced hallucinations – when the LLM is given relevant facts, it is less likely to invent answers.
  • Long‑term knowledge access – unlike a limited conversation window, memory persists indefinitely.

Memory transforms a stateless agent into one with institutional knowledge.

Why Connectors Matter

  • External system integration – connect to databases, search engines, CRM, ERP, and Microsoft 365.
  • Data access – read and write structured and unstructured data directly from AI workflows.
  • Knowledge retrieval – fetch live data that cannot be stored in a static vector index (e.g., real‑time inventory).
  • Enterprise connectivity – leverage Azure services, on‑premises systems, and third‑party APIs through built‑in connectors.
  • Tool enablement – connectors become the implementation layer of native functions, giving agents real‑world capabilities.

Connectors are the bridge between the AI model and the rest of the world.

Core Concepts Overview

ConceptPurpose
MemoryPersistent, searchable store of knowledge (facts, documents).
EmbeddingA numerical vector that represents the semantic meaning of a piece of text.
RetrievalThe process of querying the memory store for the most relevant information to a given input.
ConnectorA component that integrates an external service (database, API, vector store) with the kernel.
Vector StoreA database optimised for storing and searching embeddings (Azure AI Search, Pinecone, etc.).
Knowledge SourceThe origin of data—a file, a web page, a database record—that is ingested into memory.

Memory Architecture

The memory subsystem follows a classic ingest → embed → store → retrieve → augment pipeline:

  1. Ingest – raw text (documents, user facts) is split into chunks.
  2. Embed – each chunk is converted into a vector using an embedding model.
  3. Store – the vector and the original text are saved in a vector database.
  4. Retrieve – a user query is embedded, and the most similar chunks are fetched.
  5. Augment – the retrieved text is added to the LLM prompt as context.

This is the Retrieval‑Augmented Generation (RAG) pattern at its core.

Types of Memory

Semantic Kernel distinguishes several levels of memory, each with a different lifespan and purpose.

TypeDescriptionStorageExample
Working MemoryShort‑lived information held during a single turn or conversation.In‑memory variables, KernelArgumentsThe current user message, intermediate step results.
Persistent MemoryInformation that survives sessions and can be recalled later.Vector store, relational databaseUser preferences, knowledge base articles.
Semantic MemoryA subset of persistent memory that uses embeddings for semantic search.Vector store (Azure AI Search, Pinecone, etc.)“Company holidays are 25 Dec and 1 Jan.”
External MemoryData that lives in external systems and is accessed live via connectors, not pre‑indexed.CRM, ERP, REST APIsCurrent stock levels, live order status.

Semantic Memory is the most common use case and is what people usually refer to when they say “memory” in the context of SK.

How Memory Works

The lifecycle of a memory operation from ingestion to retrieval is:

  1. Information ingestion – text is loaded from a file, received from a user, or generated by an agent.
  2. Embedding generation – the text is sent to an embedding model (e.g., text-embedding-ada-002 or a local model). The model returns a vector.
  3. Storage – the vector, along with metadata (ID, collection name, original text), is saved into the configured MemoryStore.
  4. Retrieval – at query time, the question is embedded, and the memory store returns the top‑K most similar records.
  5. Context enrichment – the retrieved text snippets are inserted into the prompt or passed to a function.
  6. Response generation – the LLM uses the enriched context to generate a grounded answer.

Embeddings are the bridge between text and vector space. A good embedding model maps semantically similar phrases to nearby vectors. Semantic Kernel abstracts the embedding generation behind ITextEmbeddingGenerationService (in .NET) or the equivalent in Python.

Generating embeddings manually:

from semantic_kernel.connectors.ai.open_ai import OpenAITextEmbedding
embedding_service = OpenAITextEmbedding(service_id="embed", api_key="...")
vector = await embedding_service.generate_embeddings(["User prefers dark mode"])

However, you rarely call this directly. Instead, you use the SemanticTextMemory class, which handles embedding generation, storage, and retrieval transparently.

Semantic search is performed by calling memory.search(collection, query). The memory store finds the vectors closest to the query vector and returns the corresponding texts. This is the engine behind RAG.

Vector Stores

Semantic Kernel supports a wide range of vector stores through connectors. The choice depends on your infrastructure and scalability needs.

StoreTypeStrengthsTypical use
Azure AI SearchManaged cloud serviceEnterprise‑grade, hybrid (keyword + vector), integrated with AzureLarge‑scale enterprise RAG, Microsoft ecosystem
PostgreSQL + pgvectorRelational + vector extensionFamiliar SQL interface, combines structured and vector dataApplications already on Postgres, moderate scale
PineconeManaged vector databaseFully serverless, high performance, easy scalingStartups and cloud‑native apps, purely vector workloads
WeaviateOpen‑source vector DBRich schema, built‑in vectorisation modules, hybrid searchSelf‑hosted or cloud, complex object storage
QdrantOpen‑source vector DBHigh performance, written in Rust, good filteringOn‑premises or cloud, high throughput
ChromaOpen‑source embedded DBSimple, Python‑native, great for prototypingDevelopment, small‑scale deployments
RedisIn‑memory with vector searchExtremely fast, also a cacheReal‑time applications, caching + search

All of these are accessible via IMemoryStore implementations. You simply pass the store instance when creating the SemanticTextMemory.

Example: using Azure AI Search as a memory store.

from semantic_kernel.connectors.memory.azure_cognitive_search import AzureCognitiveSearchMemoryStore

store = AzureCognitiveSearchMemoryStore(
vector_size=1536,
search_endpoint=os.environ["AZURE_SEARCH_ENDPOINT"],
admin_key=os.environ["AZURE_SEARCH_KEY"]
)

memory = SemanticTextMemory(
storage=store,
embeddings_generator=kernel.get_service("embed")
)

What Are Connectors

Connectors are pre‑built or custom components that enable the kernel to interact with external services. They handle authentication, network communication, and data mapping so that your functions don’t have to. Connectors are the implementation layer of many native plugins.

Connectors can be:

  • AI service connectors – to models like OpenAI, Azure OpenAI, Hugging Face.
  • Memory connectors – to vector stores as described above.
  • Data connectors – to databases (SQL Server, Postgres), Microsoft Graph, SharePoint, OneDrive.
  • Search connectors – to Bing, Azure AI Search (as a search engine, not just a vector store).
  • API connectors – generic HTTP clients or OpenAPI‑defined services.

Semantic Kernel ships with many connectors out of the box, and you can create your own by implementing the required interfaces.

Types of Connectors

1. AI Service Connectors

Provide access to language models and embeddings. Examples: OpenAIChatCompletion, AzureOpenAIChatCompletion, HuggingFaceTextCompletion.

2. Database Connectors

Allow plugins to execute queries and commands. The kernel doesn’t have built‑in SQL connectors in the same way as memory connectors, but you typically write a native function that uses a database client (e.g., asyncpg, sqlalchemy, Microsoft.Data.SqlClient). For .NET, Entity Framework Core can be injected directly.

3. Vector Store Connectors

Implement the IMemoryStore interface. Already listed: Azure AI Search, Pinecone, Weaviate, Qdrant, Chroma, Redis, etc.

4. Search Connectors

Connect to external search services. For example, a BingSearchConnector can perform web searches and return structured results.

5. API Connectors

Generic connectors that wrap REST APIs. The OpenApiKernelExtension in .NET can import an entire OpenAPI specification as a plugin. In Python, you can use httpx inside a native function.

6. Enterprise System Connectors

Pre‑built connectors for Microsoft 365, Dynamics 365, or other LOB systems. These leverage the Microsoft Graph API and are primarily available in the .NET ecosystem.

Connector Lifecycle

A typical connector interaction during a function call:

The connector abstracts away the transport and authentication. Functions remain clean and testable.

Memory Retrieval Patterns

1. Retrieval‑Augmented Generation (RAG)

The standard pattern: embed the query, retrieve top‑K documents, stuff them into the prompt, generate answer. Semantic Kernel’s SemanticTextMemory and the kernel’s prompt functions make this easy.

2. Semantic Search Pattern

Used when you only need to find relevant documents without generating a summary. Implemented via memory.search().

3. Knowledge Base Pattern

A curated collection of documents (e.g., HR policies) ingested once and updated periodically. The agent always searches this base before answering.

4. Personal Memory Pattern

Storing and retrieving user‑specific facts (e.g., preferences, past conversations). Often uses a separate collection per user ID.

Example: simple RAG step in a function.

async def rag_function(kernel: Kernel, query: str) -> str:
memory = kernel.get_service("memory") # or injected
facts = await memory.search("policies", query, limit=3)
context = "\n".join([f.text for f in facts])
prompt = f"Context:\n{context}\n\nQuestion: {query}\nAnswer:"
result = await kernel.invoke_prompt(prompt)
return str(result)

Connector Design Patterns

PatternDescriptionExample
Data Access PatternEncapsulate database or API calls in a connector that a native function uses.OrderConnector with get_order(id) method, used by OrderPlugin.
Search PatternA connector that performs a search and returns structured results.BingSearchConnector in a ResearchPlugin.
Synchronization PatternA connector that periodically syncs external data into the memory store.A background job that reads SharePoint files and updates the Azure AI Search index.
Knowledge Integration PatternCombine multiple connectors to enrich context.Retrieve from both a vector store and a live CRM API, then merge into the prompt.

These patterns help you build maintainable, testable integrations.

Memory and RAG

RAG is the primary use case for memory in Semantic Kernel. The framework supports the full RAG pipeline:

  1. Retrievalmemory.search() or a custom connector fetches relevant documents.
  2. Ranking – the vector store returns results ordered by similarity. You can apply additional re‑ranking (e.g., using a cross‑encoder) inside a native function.
  3. Context injection – the retrieved texts are inserted into a prompt template using Handlebars placeholders ({{$context}}).
  4. Response generation – the LLM generates an answer grounded in the provided context.

Semantic Kernel’s flexibility allows you to implement advanced RAG strategies like query rewriting, hybrid search (combining vector and keyword), and multi‑hop retrieval.

Context Enrichment

When you inject retrieved memory into the prompt, the LLM’s answer becomes significantly more accurate and specific. This is context enrichment. Key benefits:

  • Relevance – the model focuses on the provided facts rather than its general knowledge.
  • Precision – factual errors are drastically reduced when the source documents are present.
  • Grounding – you can trace the answer back to the original document chunk.
  • Hallucination reduction – the model is less likely to invent information when it has clear data to refer to.

Always include a citation or source identifier in the enriched context so users (or downstream logic) can verify the information.

Performance Optimization

RAG and memory retrieval can introduce latency. Optimise with:

  • Efficient embeddings – use the smallest embedding model that meets your accuracy needs. Cache embeddings for static documents.
  • Retrieval optimization – set appropriate top_k (3–5 is often enough). Use server‑side filtering to reduce the search space.
  • Caching – cache frequent queries and their results. Semantic Kernel’s memory store does not cache by default, but you can wrap it with a caching decorator.
  • Connector efficiency – use connection pooling, async I/O, and batch APIs where available.
  • Context size control – limit the length of retrieved chunks. Truncate or summarise before injecting into the prompt.
  • Optimisation checklist:
    • Pre‑warm connections to vector stores.
    • Chunk documents intelligently (overlap, meaningful boundaries).
    • Use hybrid search (vector + keyword) if supported for better recall.
    • Monitor retrieval latency and token usage.

Security Considerations

Memory and connectors handle sensitive data. Implement these safeguards:

  • Data privacy – store only necessary information. Anonymise PII before embedding.
  • Connector authentication – use managed identities, API keys from a vault, and least‑privilege service accounts.
  • Access control – filter retrieved documents based on the user’s permissions. This may require custom logic that checks claims before returning results.
  • Sensitive information protection – redact PII in logs and prompts. Avoid storing secrets in memory.
  • Secure retrieval – ensure that one user cannot search another user’s private collection unless authorised.

Error Handling

  • Missing knowledge – if no relevant documents are found, the agent should inform the user rather than hallucinate. Return a controlled message or trigger a clarification.
  • Connector failures – wrap connector calls in try/except with retry logic (using tenacity). Return a meaningful error that the planner/agent can handle.
  • Retrieval failures – catch exceptions from the memory store and fall back to a generic “I don’t know” response.
  • Embedding errors – if the embedding service is down, the agent cannot perform retrieval. Have a circuit breaker and a fallback mode that skips enrichment.
  • Recovery strategies – design agents to operate in a “degraded” mode without memory if the store is unavailable, rather than failing completely.

Common Beginner Mistakes

  1. Treating memory as chat history – using memory to store every conversation turn, bloating the vector store and causing noise.
  2. Storing excessive information – adding large, unsplit documents leads to poor retrieval quality and high cost.
  3. Weak retrieval strategies – not tuning top_k or using a single embedding model for all tasks.
  4. Poor connector management – hard‑coding connection strings, using admin credentials, not handling timeouts.
  5. Ignoring security controls – not filtering search results by user permissions.
  6. Forgetting to chunk and clean data – ingesting raw HTML or PDFs without pre‑processing.

Best Practices

  • Store only valuable knowledge – be selective about what goes into memory.
  • Optimise embeddings – choose the right model and batch embeddings when possible.
  • Monitor retrieval quality – track the relevance of retrieved documents (user feedback or automated evaluation).
  • Secure external connections – use managed identities, rotate keys, and limit network exposure.
  • Minimise unnecessary context – keep injected context concise; prune irrelevant chunks.
  • Validate retrieved information – cross‑check critical facts before using them in decisions.
  • Version your data – when you update your knowledge base, re‑ingest and test retrieval.

Practical Example: Enterprise Knowledge Assistant

We’ll build an assistant that answers HR questions by searching an Azure AI Search index of policy documents and also looks up live employee data from a SQL database via a connector.

1. Configure the kernel, embedding service, and memory store

from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion, OpenAITextEmbedding
from semantic_kernel.memory import SemanticTextMemory
from semantic_kernel.connectors.memory.azure_cognitive_search import AzureCognitiveSearchMemoryStore

kernel = Kernel()
kernel.add_service(OpenAIChatCompletion(service_id="gpt", api_key="...", deployment_name="gpt-4o"))
embedding_service = OpenAITextEmbedding(service_id="embed", api_key="...", deployment_name="text-embedding-ada-002")
kernel.add_service(embedding_service)

store = AzureCognitiveSearchMemoryStore(vector_size=1536, ...)
memory = SemanticTextMemory(storage=store, embeddings_generator=embedding_service)
kernel.add_memory(memory)

2. Populate the memory with policy documents (offline script)

# Load documents, chunk them, and save
policy_chunks = chunk_documents("hr_policies.pdf", max_tokens=500)
for i, chunk in enumerate(policy_chunks):
await memory.save_information(collection="policies", id=f"policy-{i}", text=chunk)

3. Create a native plugin with a SQL connector for employee data

from semantic_kernel.functions import kernel_function
import asyncpg

class EmployeePlugin:
def __init__(self, conn_string):
self.conn_string = conn_string

@kernel_function(description="Get employee details by ID")
async def get_employee(self, employee_id: str) -> str:
conn = await asyncpg.connect(self.conn_string)
row = await conn.fetchrow("SELECT name, department, location FROM employees WHERE id=$1", employee_id)
await conn.close()
if row:
return f"Name: {row['name']}, Dept: {row['department']}, Location: {row['location']}"
return "Employee not found."

kernel.add_plugin(EmployeePlugin(os.environ["DB_CONNECTION"]), "Employee")

4. Create an agent that combines memory and the connector

from semantic_kernel.agents import ChatCompletionAgent

class HRAssistant:
def __init__(self, kernel):
self.kernel = kernel

async def answer(self, query: str, employee_id: str = None) -> str:
# 1. Retrieve policy context
policy_chunks = await self.kernel.memory.search("policies", query, limit=3)
context = "\n".join([c.text for c in policy_chunks])

# 2. If employee_id provided, fetch live data
emp_data = ""
if employee_id:
emp_data = await self.kernel.invoke(plugin_name="Employee", function_name="get_employee", arguments=KernelArguments(employee_id=employee_id))

# 3. Build prompt and invoke
prompt = f"""You are an HR assistant. Use the following policy information and employee data to answer the question.

Policies:
{context}

Employee Info: {emp_data}

Question: {query}
Answer:"""
result = await self.kernel.invoke_prompt(prompt)
return str(result)

5. Use the assistant

assistant = HRAssistant(kernel)
answer = await assistant.answer("What is the remote work policy?", employee_id="E123")
print(answer)

This example demonstrates a full RAG pipeline with a live data connector. The assistant retrieves relevant policies from the vector store, fetches the employee’s department and location, and produces a grounded, personalised response.

Memory & Connectors in Production Systems

  • Reliability – use highly available vector stores (Azure AI Search, Pinecone with replication). Implement retries on connector calls.
  • Monitoring – log retrieval latency, memory hit rates, and connector errors. Integrate with Application Insights or Prometheus.
  • Scalability – partition collections by tenant or user. Use serverless vector stores that scale automatically.
  • Cost Management – embedding and vector storage costs can add up. Set data retention policies. Cache frequent queries.
  • Data Governance – classify data sensitivity. Ensure memory stores comply with GDPR, HIPAA, etc. Delete user data on request.

Memory & Connectors and Other Semantic Kernel Concepts

  • Core Concepts – The kernel, AI services, and plugins are the foundation. Memory and connectors are services registered in the kernel.
  • Plugins – Native plugins often encapsulate connector logic. Prompt plugins can use memory directly via the kernel.
  • Planners – Planners can dynamically decide to retrieve memory or call a connector as part of a plan.
  • Production – Production deployment requires secure, monitored, and scalable memory and connectors.
  • MCP – MCP servers can be integrated as connectors, allowing the kernel to use standardised tools from any MCP server.

Semantic Kernel Memory vs Other Frameworks

FrameworkMemory ModelConnector EcosystemKey Strengths
Semantic KernelSemanticTextMemory + IMemoryStore pluggable backends; full RAG pipelineRich built‑in connectors for Azure, OpenAI, vector DBs; extensible via custom pluginsEnterprise integration, strong .NET support, native Azure AI Search & Microsoft Graph
LangGraphCheckpointer for graph state; memory is externalised; can integrate with vector stores as nodesTools and toolkits; large community ecosystem; any Python library can be a toolUltimate control over state and retrieval logic; suitable for complex RAG with custom logic
CrewAIAgent memory (short‑term and long‑term) backed by vector store; knowledge sources for documentsTools (crewai_tools), MCP integration; more focused on agent tooling than data connectorsSimple, quick setup for multi‑agent teams; memory is easy to enable
AutoGenConversation history + vector memory via Memory abstraction; supports external storesTools via FunctionTool; can integrate with any Python function; growing ecosystemEvent‑driven, scalable; flexible memory and tool integration
OpenAI Agents SDKSessions store conversation state; no built‑in vector memory; RAG implemented via tools (e.g., FileSearchTool)Hosted tools (web search, file search); simple API; open for custom toolsVery lightweight; easy to add custom RAG as a tool

Semantic Kernel’s tight integration with Azure and its structured plugin model make it the go‑to choice for enterprises already in the Microsoft ecosystem. Its memory and connector architecture is designed for production from day one, with clear separation of concerns and robust dependency injection.

FAQ

1. What is Semantic Kernel Memory?

It’s a persistent, searchable knowledge layer that uses embeddings and a vector store to retrieve relevant information for grounding AI responses.

2. How does semantic retrieval work?

Text is converted to vectors; a query is also embedded, and the vector store returns the most similar items based on distance metrics.

3. What are Connectors?

Connectors are components that integrate external services (databases, APIs, vector stores) with the kernel, handling authentication and communication.

4. Which vector stores are supported?

Azure AI Search, PostgreSQL (pgvector), Pinecone, Weaviate, Qdrant, Chroma, Redis, and any store implementing IMemoryStore.

5. How does RAG work in Semantic Kernel?

You search memory for relevant context, inject it into the prompt, and let the LLM generate a grounded answer. This can be done manually or via a planner.

6. How are embeddings used?

An embedding model (e.g., Ada‑002) generates vectors from text. These vectors are stored and used for similarity search.

7. Is memory production‑ready?

Yes, when backed by a robust vector store (Azure AI Search, Pinecone), with proper error handling and security.

8. Can I use my own embedding model?

Yes, any model implementing ITextEmbeddingGenerationService can be plugged in. This includes local models via Ollama or Hugging Face.

9. How do I secure connector credentials?

Use environment variables, Azure Key Vault, or managed identities. Never hard‑code secrets.

10. How do I handle large document collections?

Chunk documents, use async ingestion, and consider an indexing pipeline (e.g., Azure Functions) separate from the request path.

11. Can memory be shared across multiple kernels or services?

Yes, the vector store is a shared resource. Multiple kernels can read/write the same collections, provided they are consistent.

12. How do I delete or update stored memories?

Use memory.remove(collection, id) or the underlying store’s API. For updates, you re‑save with the same ID.

13. What is the difference between a memory store and a search connector?

A memory store is a vector store used for semantic similarity. A search connector might perform keyword or web searches and return results that are not pre‑embedded.

If your vector store supports it (e.g., Azure AI Search, Weaviate), you can enable hybrid ranking in the store configuration. Otherwise, combine results from both vector and keyword searches in a custom function.

15. Can I use Semantic Kernel memory with non‑OpenAI models?

Yes, as long as you have an embedding service that implements the required interface. Many local models are supported via the Hugging Face connector or custom adapters.

16. Is there a limit on the number of documents I can store?

That depends on the vector store, not Semantic Kernel. Plan capacity accordingly.

Conclusion

Semantic Kernel’s memory and connector layers provide everything you need to build knowledge‑grounded, enterprise‑connected AI applications. By combining semantic memory for RAG with a rich set of connectors for external data, you can create agents that are not only intelligent but also deeply integrated into your existing infrastructure.

Key takeaways:

  • Memory is a vector‑based knowledge store that enables RAG and personalisation.
  • Connectors are the bridge to external services—databases, search engines, and APIs.
  • The pipeline of ingest → embed → store → retrieve → augment is the backbone of memory operations.
  • Choose the right vector store based on your scale and ecosystem.
  • Production demands security, monitoring, and performance tuning.
  • Semantic Kernel’s model is plugin‑centric: memory and connectors become services that plugins consume.

Now, deepen your production knowledge:

For cross‑framework comparisons, explore the LangGraph, CrewAI, AutoGen, and OpenAI Agents SDK guides. The framework comparison can help you choose. To standardise tool and memory integration, see the MCP Guide.

Now, connect your agents to the world—and let them remember.