Skip to content

Commit 8f998ed

Browse files
pamelafoxpamelafox
authored
Private endpoint support for container apps (#2322)
* Configure Azure Developer Pipeline * Private endpoints draft * Conditional for SPL * private endpoint for ACA * Add P2S VPN gateway and other improvements * Private endpoint almost working * Usving avm for the subnets * Connected app to vnet * Feedback from Matt * Move resources into modules * Bring back unneeded changes * Update prepdocs with ping and update docs * Fix VPN client link * Add App Service private endpoint for deployment * Revert unneeded bicep changes * Revert unneeded changes * Remove unneeded NSG rules * Address feedback from Copilot * Address feedback from Copilot * Address Copilot feedback * Update NSG, container registry * Update NSG, container registry * Address feedback * Bring back workload profile name
1 parent a57cf7a commit 8f998ed

13 files changed

+610
-144
lines changed

‎app/backend/prepdocs.py

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
import os
55
from typing import Optional, Union
66

7+
import aiohttp
78
from azure.core.credentials import AzureKeyCredential
89
from azure.core.credentials_async import AsyncTokenCredential
910
from azure.identity.aio import AzureDeveloperCliCredential, get_bearer_token_provider
@@ -45,6 +46,19 @@ def clean_key_if_exists(key: Union[str, None]) -> Union[str, None]:
4546
return None
4647

4748

49+
async def check_search_service_connectivity(search_service: str) -> bool:
50+
"""Check if the search service is accessible by hitting the /ping endpoint."""
51+
ping_url = f"https://{search_service}.search.windows.net/ping"
52+
53+
try:
54+
async with aiohttp.ClientSession() as session:
55+
async with session.get(ping_url, timeout=aiohttp.ClientTimeout(total=10)) as response:
56+
return response.status == 200
57+
except Exception as e:
58+
logger.debug(f"Search service ping failed: {e}")
59+
return False
60+
61+
4862
async def setup_search_info(
4963
search_service: str,
5064
index_name: str,
@@ -323,7 +337,10 @@ async def main(strategy: Strategy, setup_index: bool = True):
323337

324338
load_azd_env()
325339

326-
if os.getenv("AZURE_PUBLIC_NETWORK_ACCESS") == "Disabled":
340+
if (
341+
os.getenv("AZURE_PUBLIC_NETWORK_ACCESS") == "Disabled"
342+
and os.getenv("AZURE_USE_VPN_GATEWAY", "").lower() != "true"
343+
):
327344
logger.error("AZURE_PUBLIC_NETWORK_ACCESS is set to Disabled. Exiting.")
328345
exit(0)
329346

@@ -372,6 +389,23 @@ async def main(strategy: Strategy, setup_index: bool = True):
372389
search_key=clean_key_if_exists(args.searchkey),
373390
)
374391
)
392+
393+
# Check search service connectivity
394+
search_service = os.environ["AZURE_SEARCH_SERVICE"]
395+
is_connected = loop.run_until_complete(check_search_service_connectivity(search_service))
396+
397+
if not is_connected:
398+
if os.getenv("AZURE_USE_PRIVATE_ENDPOINT"):
399+
logger.error(
400+
"Unable to connect to Azure AI Search service, which indicates either a network issue or a misconfiguration. You have AZURE_USE_PRIVATE_ENDPOINT enabled. Perhaps you're not yet connected to the VPN? Download the VPN configuration from the Azure portal here: %s",
401+
os.getenv("AZURE_VPN_CONFIG_DOWNLOAD_LINK"),
402+
)
403+
else:
404+
logger.error(
405+
"Unable to connect to Azure AI Search service, which indicates either a network issue or a misconfiguration."
406+
)
407+
exit(1)
408+
375409
blob_manager = setup_blob_manager(
376410
azure_credential=azd_credential,
377411
storage_account=os.environ["AZURE_STORAGE_ACCOUNT"],

‎docs/deploy_private.md

Lines changed: 72 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,75 @@ If you want to disable public access when deploying the Chat App, you can do so
3131

3232
Deploying with public access disabled adds additional cost to your deployment. Please see pricing for the following products:
3333

34-
1. [Private Endpoints](https://azure.microsoft.com/pricing/details/private-link/)
35-
1. The exact number of private endpoints created depends on the [optional features](./deploy_features.md) used.
36-
1. [Private DNS Zones](https://azure.microsoft.com/pricing/details/dns/)
34+
* [Azure Container Registry](https://azure.microsoft.com/pricing/details/container-registry/): Premium tier is used when virtual network is added (required for private links), which incurs additional costs.
35+
* [Azure Container Apps](https://azure.microsoft.com/pricing/details/container-apps/): Workload profiles environment is used when virtual network is added (required for private links), which incurs additional costs. Additionally, min replica count is set to 1, so you will be charged for at least one instance.
36+
* [VPN Gateway](https://azure.microsoft.com/pricing/details/vpn-gateway/): VpnGw2 SKU. Pricing includes a base monthly cost plus an hourly cost based on the number of connections.
37+
* [Virtual Network](https://azure.microsoft.com/pricing/details/virtual-network/): Pay-as-you-go tier. Costs based on data processed.
38+
39+
The pricing for the following features depends on the [optional features](./deploy_features.md) used. Most deployments will have at least 5 private endpoints (Azure OpenAI, Azure Cognitive Services, Azure AI Search, Azure Blob Storage, and either Azure App Service or Azure Container Apps).
40+
41+
* [Azure Private Endpoints](https://azure.microsoft.com/pricing/details/private-link/): Pricing is per hour per endpoint.
42+
* [Private DNS Zones](https://azure.microsoft.com/pricing/details/dns/): Pricing is per month and zones.
43+
* [Azure Private DNS Resolver](https://azure.microsoft.com/pricing/details/dns/): Pricing is per month and zones.
44+
45+
⚠️ To avoid unnecessary costs, remember to take down your app if it's no longer in use,
46+
either by deleting the resource group in the Portal or running `azd down`.
47+
You might also decide to delete the VPN Gateway when not in use.
48+
49+
## Recommended deployment strategy for private access
50+
51+
1. Configure the azd environment variables to use private endpoints and a VPN gateway, with public network access disabled. This will allow you to connect to the chat app from inside the virtual network, but not from the public Internet.
52+
53+
```shell
54+
azd env set AZURE_USE_PRIVATE_ENDPOINT true
55+
azd env set AZURE_USE_VPN_GATEWAY true
56+
azd env set AZURE_PUBLIC_NETWORK_ACCESS Disabled
57+
azd up
58+
```
59+
60+
2. Provision all the Azure resources:
61+
62+
```bash
63+
azd provision
64+
```
65+
66+
3. Once provisioning is complete, you will see an error when it tries to run the data ingestion script, because you are not yet connected to the VPN. That message should provide a URL for the VPN configuration file download. If you don't see that URL, run this command:
67+
68+
```bash
69+
azd env get-value AZURE_VPN_CONFIG_DOWNLOAD_LINK
70+
```
71+
72+
Open that link in your browser. Select "Download VPN client" to download a ZIP file containing the VPN configuration.
73+
74+
4. Open `AzureVPN/azurevpnconfig.xml`, and replace the `<clientconfig>` empty tag with the following:
75+
76+
```xml
77+
<clientconfig>
78+
<dnsservers>
79+
<dnsserver>10.0.11.4</dnsserver>
80+
</dnsservers>
81+
</clientconfig>
82+
```
83+
84+
> **Note:** We use the IP address `10.0.11.4` since it is the first available IP in the `dns-resolver-subnet`(10.0.11.0/28) from the provisioned virtual network, as Azure reserves the first four IP addresses in each subnet. Adding this DNS server allows your VPN client to resolve private DNS names for Azure services accessed through private endpoints. See the network configuration in [network-isolation.bicep](../infra/network-isolation.bicep) for details.
85+
86+
5. Install the [Azure VPN Client](https://learn.microsoft.com/azure/vpn-gateway/azure-vpn-client-versions).
87+
88+
6. Open the Azure VPN Client and select "Import" button. Select the `azurevpnconfig.xml` file you just downloaded and modified.
89+
90+
7. Select "Connect" and the new VPN connection. You will be prompted to select your Microsoft account and login.
91+
92+
8. Once you're successfully connected to VPN, you can run the data ingestion script:
93+
94+
```bash
95+
azd hooks run postprovision
96+
```
97+
98+
9. Finally, you can deploy the app:
99+
100+
```bash
101+
azd deploy
102+
```
37103

38104
## Environment variables controlling private access
39105

@@ -43,23 +109,8 @@ Deploying with public access disabled adds additional cost to your deployment. P
43109
1. `AZURE_USE_PRIVATE_ENDPOINT`: Controls deployment of [private endpoints](https://learn.microsoft.com/azure/private-link/private-endpoint-overview) which connect Azure resources to the virtual network.
44110
1. When set to 'true', ensures private endpoints are deployed for connectivity even when `AZURE_PUBLIC_NETWORK_ACCESS` is 'Disabled'.
45111
1. Note that private endpoints do not make the chat app accessible from the internet. Connections must be initiated from inside the virtual network.
112+
1. `AZURE_USE_VPN_GATEWAY`: Controls deployment of a VPN gateway for the virtual network. If you do not use this and public access is disabled, you will need a different way to connect to the virtual network.
46113

47-
## Recommended deployment strategy for private access
48-
49-
1. Deploy the app with private endpoints enabled and public access enabled.
50-
51-
```shell
52-
azd env set AZURE_USE_PRIVATE_ENDPOINT true
53-
azd env set AZURE_PUBLIC_NETWORK_ACCESS Enabled
54-
azd up
55-
```
56-
57-
1. Validate that you can connect to the chat app and it's working as expected from the internet.
58-
1. Re-provision the app with public access disabled.
59-
60-
```shell
61-
azd env set AZURE_PUBLIC_NETWORK_ACCESS Disabled
62-
azd provision
63-
```
114+
## Compatibility with other features
64115

65-
1. Log into your network using a tool like [Azure VPN Gateway](https://azure.microsoft.com/services/vpn-gateway/) and validate that you can connect to the chat app from inside the network.
116+
* **GitHub Actions / Azure DevOps**: The private access deployment is not compatible with the built-in CI/CD pipelines, as it requires a VPN connection to deploy the app. You could modify the pipeline to only do provisioning, and set up a different deployment strategy for the app.

‎infra/abbreviations.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,7 @@
110110
"privateEndpoint": "pe-",
111111
"privateLink": "pl-",
112112
"purviewAccounts": "pview-",
113+
"privateDnsResolver": "pdr-",
113114
"recoveryServicesVaults": "rsv-",
114115
"resourcesResourceGroups": "rg-",
115116
"searchSearchServices": "srch-",

‎infra/core/host/container-app-upsert.bicep

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -78,9 +78,6 @@ param serviceBinds array = []
7878
@description('The target port for the container')
7979
param targetPort int = 80
8080

81-
@allowed(['Consumption', 'D4', 'D8', 'D16', 'D32', 'E4', 'E8', 'E16', 'E32', 'NC24-A100', 'NC48-A100', 'NC96-A100'])
82-
param workloadProfile string = 'Consumption'
83-
8481
param allowedOrigins array = []
8582

8683
resource existingApp 'Microsoft.App/containerApps@2023-05-02-preview' existing = if (exists) {
@@ -98,7 +95,6 @@ module app 'container-app.bicep' = {
9895
name: '${deployment().name}-update'
9996
params: {
10097
name: name
101-
workloadProfile: workloadProfile
10298
location: location
10399
tags: tags
104100
identityType: identityType

‎infra/core/host/container-app.bicep

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -79,8 +79,6 @@ param serviceType string = ''
7979
@description('The target port for the container')
8080
param targetPort int = 80
8181

82-
param workloadProfile string = 'Consumption'
83-
8482
resource userIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' existing = if (!empty(identityName)) {
8583
name: identityName
8684
}
@@ -125,7 +123,6 @@ resource app 'Microsoft.App/containerApps@2023-05-02-preview' = {
125123
}
126124
properties: {
127125
managedEnvironmentId: containerAppsEnvironment.id
128-
workloadProfileName: workloadProfile
129126
configuration: {
130127
activeRevisionsMode: revisionMode
131128
ingress: ingressEnabled ? {

‎infra/core/host/container-apps-environment.bicep

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
param name string
2+
param location string = resourceGroup().location
3+
param tags object = {}
4+
5+
param daprEnabled bool = false
6+
param logAnalyticsWorkspaceName string = ''
7+
param applicationInsightsName string = ''
8+
9+
param subnetResourceId string
10+
11+
param usePrivateIngress bool = true
12+
13+
@allowed(['Consumption', 'D4', 'D8', 'D16', 'D32', 'E4', 'E8', 'E16', 'E32', 'NC24-A100', 'NC48-A100', 'NC96-A100'])
14+
param workloadProfile string
15+
16+
// Make sure that we are using a non-consumption workload profile for private endpoints
17+
var finalWorkloadProfile = (usePrivateIngress && workloadProfile == 'Consumption') ? 'D4' : workloadProfile
18+
19+
var minimumCount = usePrivateIngress ? 1 : 0
20+
var maximumCount = usePrivateIngress ? 3 : 2
21+
22+
resource containerAppsEnvironment 'Microsoft.App/managedEnvironments@2025-02-02-preview' = {
23+
name: name
24+
location: location
25+
tags: tags
26+
properties: {
27+
// We can't use a conditional here due to an issue with the Container Apps ARM parsing
28+
appLogsConfiguration: {
29+
destination: 'log-analytics'
30+
logAnalyticsConfiguration: {
31+
customerId: logAnalyticsWorkspace.properties.customerId
32+
sharedKey: logAnalyticsWorkspace.listKeys().primarySharedKey
33+
}
34+
}
35+
daprAIInstrumentationKey: daprEnabled && !empty(applicationInsightsName) ? applicationInsights.properties.InstrumentationKey : ''
36+
publicNetworkAccess: usePrivateIngress ? 'Disabled' : 'Enabled'
37+
vnetConfiguration: usePrivateIngress ? {
38+
infrastructureSubnetId: subnetResourceId
39+
internal: true
40+
} : null
41+
workloadProfiles: usePrivateIngress
42+
? [
43+
{
44+
name: 'Consumption'
45+
workloadProfileType: 'Consumption'
46+
}
47+
{
48+
name: 'Warm'
49+
workloadProfileType: finalWorkloadProfile
50+
minimumCount: minimumCount
51+
maximumCount: maximumCount
52+
}
53+
]
54+
: [
55+
{
56+
name: 'Consumption'
57+
workloadProfileType: 'Consumption'
58+
}
59+
]
60+
}
61+
}
62+
63+
resource logAnalyticsWorkspace 'Microsoft.OperationalInsights/workspaces@2022-10-01' existing = if (!empty(logAnalyticsWorkspaceName)) {
64+
name: logAnalyticsWorkspaceName
65+
}
66+
67+
resource applicationInsights 'Microsoft.Insights/components@2020-02-02' existing = if (daprEnabled && !empty(applicationInsightsName)){
68+
name: applicationInsightsName
69+
}
70+
71+
output defaultDomain string = containerAppsEnvironment.properties.defaultDomain
72+
output name string = containerAppsEnvironment.name
73+
output resourceId string = containerAppsEnvironment.id

‎infra/core/host/container-apps.bicep

Lines changed: 16 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,78 +1,48 @@
1-
metadata description = 'Creates an Azure Container Registry and an Azure Container Apps environment.'
21
param name string
32
param location string = resourceGroup().location
43
param tags object = {}
54

65
param containerAppsEnvironmentName string
76
param containerRegistryName string
8-
param containerRegistryResourceGroupName string = ''
9-
param containerRegistryAdminUserEnabled bool = false
10-
param logAnalyticsWorkspaceResourceId string
11-
param applicationInsightsName string = '' // Not used here, was used for DAPR
12-
param virtualNetworkSubnetId string = ''
7+
param logAnalyticsWorkspaceName string = ''
8+
param applicationInsightsName string = ''
9+
10+
@description('Virtual network name for container apps environment.')
11+
param vnetName string = ''
12+
1313
@allowed(['Consumption', 'D4', 'D8', 'D16', 'D32', 'E4', 'E8', 'E16', 'E32', 'NC24-A100', 'NC48-A100', 'NC96-A100'])
1414
param workloadProfile string
1515

16-
var workloadProfiles = workloadProfile == 'Consumption'
17-
? [
18-
{
19-
name: 'Consumption'
20-
workloadProfileType: 'Consumption'
21-
}
22-
]
23-
: [
24-
{
25-
name: 'Consumption'
26-
workloadProfileType: 'Consumption'
27-
}
28-
{
29-
minimumCount: 0
30-
maximumCount: 2
31-
name: workloadProfile
32-
workloadProfileType: workloadProfile
33-
}
34-
]
16+
param subnetResourceId string = ''
3517

36-
@description('Optional user assigned identity IDs to assign to the resource')
37-
param userAssignedIdentityResourceIds array = []
18+
param usePrivateIngress bool = true
3819

39-
module containerAppsEnvironment 'br/public:avm/res/app/managed-environment:0.8.0' = {
20+
module containerAppsEnvironment 'container-apps-environment.bicep' = {
4021
name: '${name}-container-apps-environment'
4122
params: {
42-
// Required parameters
43-
logAnalyticsWorkspaceResourceId: logAnalyticsWorkspaceResourceId
44-
45-
managedIdentities: empty(userAssignedIdentityResourceIds) ? {
46-
systemAssigned: true
47-
} : {
48-
userAssignedResourceIds: userAssignedIdentityResourceIds
49-
}
50-
5123
name: containerAppsEnvironmentName
52-
// Non-required parameters
53-
infrastructureResourceGroupName: containerRegistryResourceGroupName
54-
infrastructureSubnetId: virtualNetworkSubnetId
5524
location: location
5625
tags: tags
57-
zoneRedundant: false
58-
workloadProfiles: workloadProfiles
26+
logAnalyticsWorkspaceName: logAnalyticsWorkspaceName
27+
applicationInsightsName: applicationInsightsName
28+
usePrivateIngress: usePrivateIngress
29+
subnetResourceId: subnetResourceId
30+
workloadProfile: workloadProfile
5931
}
6032
}
6133

62-
module containerRegistry 'br/public:avm/res/container-registry/registry:0.5.1' = {
34+
module containerRegistry 'container-registry.bicep' = {
6335
name: '${name}-container-registry'
64-
scope: resourceGroup(!empty(containerRegistryResourceGroupName) ? containerRegistryResourceGroupName : resourceGroup().name)
6536
params: {
6637
name: containerRegistryName
6738
location: location
68-
acrAdminUserEnabled: containerRegistryAdminUserEnabled
6939
tags: tags
40+
useVnet: !empty(vnetName)
7041
}
7142
}
7243

7344
output defaultDomain string = containerAppsEnvironment.outputs.defaultDomain
7445
output environmentName string = containerAppsEnvironment.outputs.name
7546
output environmentId string = containerAppsEnvironment.outputs.resourceId
76-
7747
output registryLoginServer string = containerRegistry.outputs.loginServer
7848
output registryName string = containerRegistry.outputs.name

0 commit comments

Comments
 (0)