Four failures of one deployment

The task was simple: a small service collecting measurements from devices, .NET 10, Entity Framework, a database in Azure SQL, hosting on App Service. A few endpoints, two tables. I assumed an hour.

The code wrote itself in a quarter of an hour. Everything else went into the deployment, and that turned out to be the real lesson. It fell over four times, each time for a different reason and in a different layer — from the archive format to the way the application starts.

I describe them in order, because each of these will catch anyone who deploys from Windows to Linux. Three of the four shared one trait: the message pointed somewhere completely different from the cause.

Starting point

The application is trivial. A device register, measurements, two endpoints for writing and reading. All in a minimal API, no controllers.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddDbContext<AppDbContext>(o =>
    o.UseSqlServer(
        builder.Configuration.GetConnectionString("Default"),
        sql => sql.EnableRetryOnFailure(
            maxRetryCount: 5,
            maxRetryDelay: TimeSpan.FromSeconds(30),
            errorNumbersToAdd: null)));

var app = builder.Build();

app.MapGet("/devices", async (AppDbContext db) =>
    await db.Devices
        .AsNoTracking()
        .Select(d => new DeviceDto(d.Serial, d.Customer, d.Location, d.Measurements.Count))
        .ToListAsync());

app.Run();

Ordinary entities, two classes. One thing worth noting for someone coming back to .NET after a long break: properties marked as required. The compiler enforces setting them when the object is created, so you no longer have to promise it with an exclamation mark that someone will fill them in one day.

public class Device
{
    public int Id { get; set; }
    public required string Serial { get; set; }
    public required string Customer { get; set; }
    public string? Location { get; set; }

    public List<Measurement> Measurements { get; set; } = [];
}

First: slashes in the archive

I built the package, sent it, and got a deployment error with no meaningful message — just “Deployment failed” and code 400.

PS C:\dev\MyDeviceHub> dotnet publish -c Release -o publish
  MyDeviceHub net10.0 powodzenie (4,1s) -> publish\

PS C:\dev\MyDeviceHub> Compress-Archive -Path publish\* -DestinationPath publish.zip
PS C:\dev\MyDeviceHub> az webapp deploy --src-path publish.zip --type zip
Initiating deployment
Deploying from local path: publish.zip
Deployment failed.
Error Code  : DeploymentFailed
Kudu Status : 400

Code 400 says nothing. The truth only shows up in the deployment log on the server side — and there, dozens of lines of the same pattern had spilled out.

$ az webapp log deployment show -n <app> -g <group>

rsync: [generator] recv_generator: failed to stat
  "/home/site/wwwroot/runtimes\linux-x64\native\libe_sqlite3.so": Invalid argument (22)
rsync: [generator] recv_generator: failed to stat
  "/home/site/wwwroot/runtimes\win-x64\native\Microsoft.Data.SqlClient.SNI.dll": Invalid argument (22)
rsync: [generator] recv_generator: failed to stat
  "/home/site/wwwroot/pl\Microsoft.Data.SqlClient.resources.dll": Invalid argument (22)

rsync error: some files/attrs were not transferred (code 23) at main.c(1338)
Deployment Failed. deployer = OneDeploy

Look at the paths. Backslashes in the middle, where forward slashes should be. The ZIP standard requires forward slashes in internal paths — so Linux sees not a path but a single file name containing characters a name cannot have. Hence “Invalid argument”.

Windows PowerShell 5.1 sits on the old .NET Framework, which does not follow this and writes the system separator. Files from the root of the package go through; everything from subdirectories fails. And a .NET publish has plenty of subdirectories, because the SQL Server driver pulls in native libraries for every platform in the world.

Switching to the class from the System.IO.Compression namespace did nothing — the same library underneath, the same result. Only one of two things works:

# 1. tar from Windows - writes according to the standard
tar.exe -a -c -f publish.zip -C publish .

# 2. or PowerShell 7, which runs on modern .NET
winget install Microsoft.PowerShell
pwsh -c 'Compress-Archive -Path publish\* -DestinationPath publish.zip'

Second: the CPU quota on the free plan

After fixing the package, the deployment started returning 403 with a “Web App - Unavailable” page. One command explained everything.

PS C:\dev\MyDeviceHub> az webapp show -n <app> -g <group> --query state -o tsv
QuotaExceeded

The F1 plan gives sixty minutes of CPU time per day. That sounds like a lot until you notice that unpacking the package and starting the application also draw on it — and every failed attempt starts from zero. Three failed deployments and the quota was gone. The application only comes back after midnight UTC.

The conclusion is not that the free plan is bad. It is that the free plan is not fit for iterating. It is enough to look around the portal, not to work.

az appservice plan update --name <plan> --resource-group <group> --sku B1

Third: an incomplete directory after a failed rsync

The third attempt gave an error that had not been there before. The application came up and immediately fell over — on a library I had certainly been sending.

Running the command: dotnet "MyDeviceHub.dll"

Unhandled exception. System.TypeInitializationException:
  The type initializer for 'SqlServerTypeMappingSource' threw an exception.
 ---> System.IO.FileNotFoundException: Could not load file or assembly
      'Microsoft.Data.SqlClient, Version=6.0.0.0'. The system cannot find the file.

   at MyDeviceHub.AppDbContext.get_Devices() in AppDbContext.cs:line 11
   at Program.<Main>$(String[] args) in Program.cs:line 28

/opt/startup/startup.sh: line 20: Aborted (core dumped) dotnet "MyDeviceHub.dll"
Container has finished running with exit code: 134.

The cause was left over from the first failure. Rsync copied what it could — the files from the root — and stopped at the subdirectories, leaving the target directory in a mixed state. Later deployments added to the mess, because by default they do not clean the directory, they only synchronise the differences.

The cure is one flag:

az webapp deploy --resource-group <group> --name <app> \
  --src-path publish.zip --type zip \
  --clean true --restart true

It is worth knowing about in advance, because the symptom — a missing library — points suspicion at packages and dependencies, that is, in completely the wrong direction. This one cost me the most time of the four.

Fourth: the database at application startup

The last one is the most interesting, because it is not a tooling fault but my own design mistake.

The code had a block seeding startup data: on start, the application checked whether there were any devices in the database and, if not, added two examples. Convenient locally.

using var scope = app.Services.CreateScope();
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();

if (!await db.Devices.AnyAsync())      // database unavailable => exception
{                                      // => the process falls over
    db.Devices.AddRange(/* ... */);    // => the container never comes up
    await db.SaveChangesAsync();
}

In the cloud this means an unavailable database turns into an application that does not work. And a database is unavailable perfectly normally: Azure SQL in the serverless tier goes to sleep after an hour of inactivity and takes a dozen or so seconds to wake.

PS C:\dev\MyDeviceHub> dotnet ef database update

System.InvalidOperationException: An exception has been raised that is likely
due to a transient failure. Consider enabling transient error resiliency by
adding 'EnableRetryOnFailure' to the 'UseSqlServer' call.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904):
      Database 'devicehub' is not currently available. Please retry later.

Numer bledu: 40613, stan: 1, klasa: 20

Two fixes. The first is wrapping the seeding in error handling, so that a database failure goes to the logs instead of bringing the process down. The second is retrying on the provider side — the mechanism knows a list of errors considered transient, and 40613 is on it.

try
{
    using var scope = app.Services.CreateScope();
    var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
    if (!await db.Devices.AnyAsync())
    {
        db.Devices.AddRange(/* ... */);
        await db.SaveChangesAsync();
    }
}
catch (Exception ex)
{
    app.Logger.LogError(ex, "Failed to seed startup data");
}

One trap with retrying that comes up in interviews: with the strategy enabled you must not open a transaction directly. Since the operation may be repeated, the strategy has to know exactly what to repeat — and a transaction started by hand does not tell it.

var strategy = db.Database.CreateExecutionStrategy();

await strategy.ExecuteAsync(async () =>
{
    await using var tx = await db.Database.BeginTransactionAsync();
    // ...
    await tx.CommitAsync();
});

What remains

Three of the four failures shared a trait: the message pointed somewhere completely different from the cause. The missing library was an uncleaned directory. Code 403 was the CPU quota. The failed unpacking was the path separator in an archive built on a different system.

None of them could have been solved by guessing, and repeating the same action would not have helped with any of them. Every time I had to go down to the logs — to the deployment log, to the state of the resource, to the container log — and read what the system was really saying.

That is, in fact, the only skill that really counts in deployments. The rest is knowing specific flags, and nobody remembers those anyway.

Oh, and one more thing. The first publication of this text returned 404, because I had set the date five minutes into the future, and the engine does not show entries whose time has not yet come. A fifth failure, in a text about four.