问题描述
我基本上正在编写一个程序来对给定表增长下 PostgreSQL 的插入性能进行基准测试，并且我想确保当我使用 Marten 插入数据时，数据库已完全准备好接受插入.
I am basically writing a little bit of a program to benchmark the insertion performance of PostgreSQL over a given table growth, and I would like to make sure that when I am using Marten to insert data, the database is fully ready to accept insertions.
我正在使用 Docker.DotNet 生成一个运行最新 PostgreSQL 映像的新容器，但即使容器处于 running 状态，有时也不是这种情况在该容器内运行的 Postgre.
I am using Docker.DotNet to spawn a new container running the latest PostgreSQL image but even if the container is in a running state it is sometimes not the case for the Postgre running inside that container.
当然，我可以添加一个 Thread.睡眠 但这有点随机，所以我更喜欢确定性的东西来确定数据库何时准备好接受插入?
Of course, I could have added a Thread. Sleep but this is a bit random so I would rather like something deterministic to figure out when the database is ready to accept insertions?
public static class Program
{
public static async Task Main(params string[] args)
{
const string imageName = "postgres:latest";
const string containerName = "MyProgreSQL";
var client = new DockerClientConfiguration(Docker.DefaultLocalApiUri).CreateClient();
var containers = await client.Containers.SearchByNameAsync(containerName);
var container = containers.SingleOrDefault();
if (container != null)
{
await client.Containers.StopAndRemoveContainerAsync(container);
}
var createdContainer = await client.Containers.RunContainerAsync(new CreateContainerParameters
{
Image = imageName,
HostConfig = new HostConfig
{
PortBindings = new Dictionary<string, IList<PortBinding>>
{
{"5432/tcp", new List<PortBinding>
{
new PortBinding
{
HostPort = "5432"
}
}}
},
PublishAllPorts = true
},
Env = new List<string>
{
"POSTGRES_PASSWORD=root",
"POSTGRES_USER=root"
},
Name = containerName
});
var containerState = string.Empty;
while (containerState != "running")
{
containers = await client.Containers.SearchByNameAsync(containerName);
container = containers.Single();
containerState = container.State;
}
var store = DocumentStore.For("host=localhost;database=postgres;password=root;username=root");
var stopwatch = new Stopwatch();
using (var session = store.LightweightSession())
{
var orders = OrderHelpers.FakeOrders(10000);
session.StoreObjects(orders);
stopwatch.Start();
await session.SaveChangesAsync();
var elapsed = stopwatch.Elapsed;
// Whatever else needs to be done...
}
}
}
仅供参考，如果我正在运行上面的程序而没有在该行之前暂停 await session.SaveChangesAsync(); 我遇到了以下异常:
FYI, the if I am running the program above without pausing before the line await session.SaveChangesAsync(); I am running than into the following exception:
Unhandled Exception: Npgsql.NpgsqlException: Exception while reading from stream ---> System.IO.EndOfStreamException: Attempted to read past the end of the streams.
at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass31_0.<<Ensure>g__EnsureLong|0>d.MoveNext() in C:projects
pgsqlsrcNpgsqlNpgsqlReadBuffer.cs:line 154
--- End of inner exception stack trace ---
at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass31_0.<<Ensure>g__EnsureLong|0>d.MoveNext() in C:projects
pgsqlsrcNpgsqlNpgsqlReadBuffer.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext() in C:projects
pgsqlsrcNpgsqlNpgsqlConnector.cs:l
ine 955
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.NpgsqlConnector.Authenticate(String username, NpgsqlTimeout timeout, Boolean async) in C:projects
pgsqlsrcNpgsqlNpgsqlConnector.Auth.cs
:line 26
at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken) in C:projects
pgsqlsrcNpgsqlNpgsqlConne
ctor.cs:line 425
at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken) in C:projects
npgsqlsrcNpgsqlConnectorPool.cs:line 246
at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext() in C:projects
pgsqlsrcNpgsqlNpgsqlConnection.cs:line 300
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.NpgsqlConnection.Open() in C:projects
pgsqlsrcNpgsqlNpgsqlConnection.cs:line 153
at Marten.Storage.Tenant.generateOrUpdateFeature(Type featureType, IFeatureSchema feature)
at Marten.Storage.Tenant.ensureStorageExists(IList`1 types, Type featureType)
at Marten.Storage.Tenant.ensureStorageExists(IList`1 types, Type featureType)
at Marten.Storage.Tenant.StorageFor(Type documentType)
at Marten.DocumentSession.Store[T](T[] entities)
at Baseline.GenericEnumerableExtensions.Each[T](IEnumerable`1 values, Action`1 eachAction)
at Marten.DocumentSession.StoreObjects(IEnumerable`1 documents)
at Benchmark.Program.Main(String[] args) in C:UserseperretDesktopBenchmarkBenchmarkProgram.cs:line 117
at Benchmark.Program.<Main>(String[] args)
我接受了一个答案，但由于 Docker.DotNet 中有关健康参数等效性的错误，我无法利用答案中给出的解决方案(我仍然认为该 docker 命令的正确翻译在 .NET 客户端中，如果可能的话)将是最好的解决方案.与此同时，这就是我解决问题的方法，我基本上轮询了预期在运行状况检查中运行的命令，直到结果正常:
I accepted an answer but due to a bug about health parameters equivalence in the Docker.DotNet I could not leverage the solution given in the answer (I still think that a proper translation of that docker command in the .NET client, if actually possible) would be the best solution. In the meanwhile this is how I solved my problem, I basically poll the command that was expected to run in the health check aside until the result is ok:
Program.cs，代码的真正内容:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Benchmark.DockerClient;
using Benchmark.Domain;
using Benchmark.Utils;
using Docker.DotNet;
using Docker.DotNet.Models;
using Marten;
using Microsoft.Extensions.Configuration;
namespace Benchmark
{
public static class Program
{
public static async Task Main(params string[] args)
{
var configurationBuilder = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json");
var configuration = new Configuration();
configurationBuilder.Build().Bind(configuration);
var client = new DockerClientConfiguration(DockerClient.Docker.DefaultLocalApiUri).CreateClient();
var containers = await client.Containers.SearchByNameAsync(configuration.ContainerName);
var container = containers.SingleOrDefault();
if (container != null)
{
await client.Containers.StopAndRemoveContainerAsync(container.ID);
}
var createdContainer = await client.Containers.RunContainerAsync(new CreateContainerParameters
{
Image = configuration.ImageName,
HostConfig = new HostConfig
{
PortBindings = new Dictionary<string, IList<PortBinding>>
{
{$@"{configuration.ContainerPort}/{configuration.ContainerPortProtocol}", new List<PortBinding>
{
new PortBinding
{
HostPort = configuration.HostPort
}
}}
},
PublishAllPorts = true
},
Env = new List<string>
{
$"POSTGRES_USER={configuration.Username}",
$"POSTGRES_PASSWORD={configuration.Password}"
},
Name = configuration.ContainerName
});
var isContainerReady = false;
while (!isContainerReady)
{
var result = await client.Containers.RunCommandInContainerAsync(createdContainer.ID, "pg_isready -U postgres");
if (result.stdout.TrimEnd('
') == $"/var/run/postgresql:{configuration.ContainerPort} - accepting connections")
{
isContainerReady = true;
}
}
var store = DocumentStore.For($"host=localhost;" +
$"database={configuration.DatabaseName};" +
$"username={configuration.Username};" +
$"password={configuration.Password}");
// Whatever else needs to be done...
}
}
在ContainerOperationsExtensions.cs中定义的扩展:
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Docker.DotNet;
using Docker.DotNet.Models;
namespace Benchmark.DockerClient
{
public static class ContainerOperationsExtensions
{
public static async Task<IList<ContainerListResponse>> SearchByNameAsync(this IContainerOperations source, string name, bool all = true)
{
return await source.ListContainersAsync(new ContainersListParameters
{
All = all,
Filters = new Dictionary<string, IDictionary<string, bool>>
{
{"name", new Dictionary<string, bool>
{
{name, true}
}
}
}
});
}
public static async Task StopAndRemoveContainerAsync(this IContainerOperations source, string containerId)
{
await source.StopContainerAsync(containerId, new ContainerStopParameters());
await source.RemoveContainerAsync(containerId, new ContainerRemoveParameters());
}
public static async Task<CreateContainerResponse> RunContainerAsync(this IContainerOperations source, CreateContainerParameters parameters)
{
var createdContainer = await source.CreateContainerAsync(parameters);
await source.StartContainerAsync(createdContainer.ID, new ContainerStartParameters());
return createdContainer;
}
public static async Task<(string stdout, string stderr)> RunCommandInContainerAsync(this IContainerOperations source, string containerId, string command)
{
var commandTokens = command.Split(" ", StringSplitOptions.RemoveEmptyEntries);
var createdExec = await source.ExecCreateContainerAsync(containerId, new ContainerExecCreateParameters
{
AttachStderr = true,
AttachStdout = true,
Cmd = commandTokens
});
var multiplexedStream = await source.StartAndAttachContainerExecAsync(createdExec.ID, false);
return await multiplexedStream.ReadOutputToEndAsync(CancellationToken.None);
}
}
}
Docker.cs 获取本地docker api uri:
Docker.cs to get the local docker api uri:
using System;
using System.Runtime.InteropServices;
namespace Benchmark.DockerClient
{
public static class Docker
{
static Docker()
{
DefaultLocalApiUri = RuntimeInformation.IsOSPlatform(OSPlatform.Windows)
? new Uri("npipe://./pipe/docker_engine")
: new Uri("unix:/var/run/docker.sock");
}
public static Uri DefaultLocalApiUri { get; }
}
}
推荐答案
我建议你使用自定义的 healtcheck 检查数据库是否准备好接受连接.
I suggest you to use a custom healtcheck to check if the database is ready to accept connections.
我不熟悉 Docker 的 .NET 客户端，但以下 docker run 命令显示了您应该尝试的内容:
I am not familiar with the .NET client of Docker, but the following docker run command shows what you should try :
docker run --name postgres
--health-cmd='pg_isready -U postgres'
--health-interval='10s'
--health-timeout='5s'
--health-start-period='10s'
postgres:latest
时间参数应根据您的需要进行相应更新.
Time parameters should be updated accordingly to your needs.
配置此健康检查后，您的应用程序必须等待容器处于健康"状态，然后再尝试连接到数据库.在这种特殊情况下，状态healthy"意味着命令 pg_isready -U postgres 已成功(因此数据库已准备好接受连接).
With this healtcheck configured, your application must wait for the container to be in state "healthy" before trying to connect to the database. The status "healthy", in that particular case, means that the command pg_isready -U postgres have succeeded (so the database is ready to accept connections).
可以通过以下方式检索容器的状态:
The status of your container can be retrieved with :
docker inspect --format "{{json .State.Health.Status }}" postgres
这篇关于如何确定运行在 Docker 容器中的 PostgreSQL 是否已使用 Marten 完成初始化?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！