Failed jobs do not become expired to allow you to re-queue them without any
time pressure. You should re-queue or delete them manually, or apply AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete)
attribute to delete them automatically.
The maximum number of retries (5) was exceeded while executing database operations with 'NpgsqlRetryingExecutionStrategy'. See the inner exception for the most recent failure.
Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException: The maximum number of retries (5) was exceeded while executing database operations with 'NpgsqlRetryingExecutionStrategy'. See the inner exception for the most recent failure. ---> Npgsql.PostgresException: 53100: could not write to file "base/pgsql_tmp/pgsql_tmp606891.0": No space left on device
at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Booleanasync, DataRowLoadingModedataRowLoadingMode, BooleanreadingNotifications, BooleanisReadingPrependedMessage)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16token)
at Npgsql.NpgsqlDataReader.NextResult(Booleanasync, BooleanisConsuming, CancellationTokencancellationToken)
at Npgsql.NpgsqlDataReader.NextResult(Booleanasync, BooleanisConsuming, CancellationTokencancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Booleanasync, CommandBehaviorbehavior, CancellationTokencancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Booleanasync, CommandBehaviorbehavior, CancellationTokencancellationToken)
at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehaviorbehavior, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObjectparameterObject, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObjectparameterObject, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.<PopulateSplitIncludeCollectionAsync>g__InitializeReaderAsync|26_0[TIncludingEntity,TIncludedEntity](RelationalQueryContextqueryContext, RelationalCommandCacherelationalCommandCache, IReadOnlyList`1readerColumns, BooleandetailedErrorsEnabled, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass30_0`2.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4operation, Func`4verifySucceeded, TStatestate, CancellationTokencancellationToken)
--- End of inner exception stack trace ---
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4operation, Func`4verifySucceeded, TStatestate, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TStatestate, Func`4operation, Func`4verifySucceeded, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.PopulateSplitIncludeCollectionAsync[TIncludingEntity,TIncludedEntity](Int32collectionId, RelationalQueryContextqueryContext, IExecutionStrategyexecutionStrategy, RelationalCommandCacherelationalCommandCache, IReadOnlyList`1readerColumns, BooleandetailedErrorsEnabled, SplitQueryResultCoordinatorresultCoordinator, Func`3childIdentifier, IReadOnlyList`1identifierValueComparers, Func`5innerShaper, Func`4relatedDataLoaders, INavigationBaseinverseNavigation, Action`2fixup, BooleantrackingQuery)
at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.TaskAwaiter(Func`1[]taskFactories)
at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1asyncEnumerable, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1asyncEnumerable, CancellationTokencancellationToken)
at EnergyDataHub.Data.Services.GenerationService.AggregateOutputAndConsumption(GuidaggregatedGenerationGuid, List`1generationIds, AppDbContextcontext)
at EnergyDataHub.Data.Services.BiddingZoneService.AggregateGenerationOutputAndConsumption(AppDbContextcontext)
at EnergyDataHub.Data.Hangfire.Jobs.GenerationSyncJob.AggregateGenerationOutputAndConsumption()
at EnergyDataHub.Data.Hangfire.Jobs.GenerationSyncJob.Run()
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Objectobj, BindingFlagsinvokeAttr)
An exception occurred during performance of the job. More details...
07/16/2025 14:13:56
System.Net.Http.HttpRequestException
Response status code does not indicate success: 429 (Too Many Requests).
System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessagerequest, CancellationTokencancellationToken)
at EnergyDataHub.Data.Services.XWeatherClient.FetchData(NameValueCollectionqueryParams, Stringpath)
at EnergyDataHub.Data.Services.XWeatherClient.GetWeatherForcasts(Doublelatitude, Doublelongitude, Nullable`1start, XWeatherIntervalinteval, Int32periodCount)
at EnergyDataHub.Data.Hangfire.Jobs.XWeatherSyncJob.SyncLocationWeatherData(Locationlocation, DataSourcedataSource, DateTimetaskStartHour)
at EnergyDataHub.Data.Hangfire.Jobs.XWeatherSyncJob.Run()
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Objectobj, BindingFlagsinvokeAttr)
An exception occurred during performance of the job. More details...
07/14/2025 20:45:05
System.NullReferenceException
Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at EnergyDataHub.Data.Hangfire.Jobs.PowerExchangeSyncJob.AddPowerExchangeData(HopsPowerExchangeDtohopsData, StringsourceName, DataSourcedataSource, DateTimetimestamp, Int64timeseriesId)
at EnergyDataHub.Data.Hangfire.Jobs.PowerExchangeSyncJob.Run()
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Objectobj, BindingFlagsinvokeAttr)
The maximum number of retries (5) was exceeded while executing database operations with 'NpgsqlRetryingExecutionStrategy'. See the inner exception for the most recent failure.
Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException: The maximum number of retries (5) was exceeded while executing database operations with 'NpgsqlRetryingExecutionStrategy'. See the inner exception for the most recent failure. ---> Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details. ---> Npgsql.PostgresException: 53100: could not extend file "base/18813/19274.8": No space left on device
at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Booleanasync, DataRowLoadingModedataRowLoadingMode, BooleanreadingNotifications, BooleanisReadingPrependedMessage)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16token)
at Npgsql.NpgsqlDataReader.NextResult(Booleanasync, BooleanisConsuming, CancellationTokencancellationToken)
at Npgsql.NpgsqlDataReader.NextResult(Booleanasync, BooleanisConsuming, CancellationTokencancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Booleanasync, CommandBehaviorbehavior, CancellationTokencancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Booleanasync, CommandBehaviorbehavior, CancellationTokencancellationToken)
at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehaviorbehavior, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObjectparameterObject, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObjectparameterObject, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnectionconnection, CancellationTokencancellationToken)
--- End of inner exception stack trace ---
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnectionconnection, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1commandBatches, IRelationalConnectionconnection, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1commandBatches, IRelationalConnectionconnection, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1commandBatches, IRelationalConnectionconnection, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1entriesToSave, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManagerstateManager, BooleanacceptAllChangesOnSuccess, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass30_0`2.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4operation, Func`4verifySucceeded, TStatestate, CancellationTokencancellationToken)
--- End of inner exception stack trace ---
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4operation, Func`4verifySucceeded, TStatestate, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TStatestate, Func`4operation, Func`4verifySucceeded, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(BooleanacceptAllChangesOnSuccess, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(BooleanacceptAllChangesOnSuccess, CancellationTokencancellationToken)
at EnergyDataHub.Data.Services.GenerationService.AggregateOutputAndConsumption(GuidaggregatedGenerationGuid, List`1generationIds, AppDbContextcontext)
at EnergyDataHub.Data.Services.BiddingZoneService.AggregateGenerationOutputAndConsumption(AppDbContextcontext)
at EnergyDataHub.Data.Hangfire.Jobs.GenerationSyncJob.AggregateGenerationOutputAndConsumption()
at EnergyDataHub.Data.Hangfire.Jobs.GenerationSyncJob.Run()
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Objectobj, BindingFlagsinvokeAttr)