Failed Jobs

Enqueued 0 / 1 Scheduled 0 Processing 2 Succeeded 140,665 Failed 3,500 Deleted 84 Awaiting 0

Failed jobs do not become expired to allow you to re-queue them without any time pressure. You should re-queue or delete them manually, or apply AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete) attribute to delete them automatically.

Id	Job	Failed
#4020	IGenerationSyncJob.Run An exception occurred during performance of the job. Fewer details...	05/13/2025 13:28:08
#4020	Npgsql.NpgsqlException Exception while writing to stream Npgsql.NpgsqlException: Exception while writing to stream ---> System.TimeoutException: Timeout during writing attempt --- End of inner exception stack trace --- at Npgsql.Internal.NpgsqlWriteBuffer.Flush(Boolean async, CancellationToken cancellationToken) at Npgsql.Internal.NpgsqlBufferWriter.Flush(TimeSpan timeout) at Npgsql.Internal.PgWriter.Flush(Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlParameter.Write(Boolean async, PgWriter writer, CancellationToken cancellationToken) at Npgsql.NpgsqlBinaryImporter.<Write>g__Core\|26_0[T](Boolean async, T value, Nullable`1 npgsqlDbType, String dataTypeName, CancellationToken cancellationToken) at EnergyDataHub.Data.Services.TimeseriesBulkInserter.BulkInsertAsync(IEnumerable`1 values) at EnergyDataHub.Data.Services.TimeseriesBulkInserter.BulkInsertAsync(IEnumerable`1 values) at EnergyDataHub.Data.Services.TimeseriesBulkInserter.BulkInsertAsync(IEnumerable`1 values) at EnergyDataHub.Data.Services.GenerationService.UpdateOutputAndConsumption(List`1 generationIds, List`1 data, AppDbContext context, DataSource dataSource) at EnergyDataHub.Data.Services.ControlAreaService.MapActualGeneration(List`1 data, AppDbContext context, DataSource dataSource) at EnergyDataHub.Data.Services.SftpService.MapDataToEachArea[DataClass,CsvMapper](String remoteDirectory, List`1 dataMethods, Int32 retryCount) at EnergyDataHub.Data.Services.SftpService.MapDataToEachArea[DataClass,CsvMapper](String remoteDirectory, List`1 dataMethods) at EnergyDataHub.Data.Hangfire.Jobs.GenerationSyncJob.Run() at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
#4015	IJaoSyncJob.Run An exception occurred during performance of the job. More details...	05/13/2025 12:09:19
#4015	System.Net.Http.HttpRequestException Response status code does not indicate success: 500 (Internal Server Error). System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error). at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at EnergyDataHub.Data.Services.FinalComputationClient.GetFinalComputation(DateTime from, Nullable`1 to, Int32 skip, Int32 take, FinalComputationFilter filters) at EnergyDataHub.Data.Hangfire.Jobs.JaoSyncJob.GetHourlyData(DateTime start) at EnergyDataHub.Data.Hangfire.Jobs.JaoSyncJob.Run() at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
#4014	IGenerationSyncJob.Run An exception occurred during performance of the job. More details...	05/13/2025 13:28:08
#4014	Npgsql.NpgsqlException Exception while writing to stream Npgsql.NpgsqlException: Exception while writing to stream ---> System.TimeoutException: Timeout during writing attempt --- End of inner exception stack trace --- at Npgsql.Internal.NpgsqlWriteBuffer.Flush(Boolean async, CancellationToken cancellationToken) at Npgsql.Internal.NpgsqlBufferWriter.Flush(TimeSpan timeout) at Npgsql.Internal.PgWriter.Flush(Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlParameter.Write(Boolean async, PgWriter writer, CancellationToken cancellationToken) at Npgsql.NpgsqlBinaryImporter.<Write>g__Core\|26_0[T](Boolean async, T value, Nullable`1 npgsqlDbType, String dataTypeName, CancellationToken cancellationToken) at EnergyDataHub.Data.Services.TimeseriesBulkInserter.BulkInsertAsync(IEnumerable`1 values) at EnergyDataHub.Data.Services.TimeseriesBulkInserter.BulkInsertAsync(IEnumerable`1 values) at EnergyDataHub.Data.Services.TimeseriesBulkInserter.BulkInsertAsync(IEnumerable`1 values) at EnergyDataHub.Data.Services.GenerationService.UpdateOutputAndConsumption(List`1 generationIds, List`1 data, AppDbContext context, DataSource dataSource) at EnergyDataHub.Data.Services.BiddingZoneService.MapActualGeneration(List`1 data, AppDbContext context, DataSource dataSource) at EnergyDataHub.Data.Services.SftpService.MapDataToEachArea[DataClass,CsvMapper](String remoteDirectory, List`1 dataMethods, Int32 retryCount) at EnergyDataHub.Data.Services.SftpService.MapDataToEachArea[DataClass,CsvMapper](String remoteDirectory, List`1 dataMethods) at EnergyDataHub.Data.Hangfire.Jobs.GenerationSyncJob.Run() at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
#3879	Can not find the target method. An exception occurred during performance of the job. More details...	05/12/2025 13:05:22
#3879	System.Exception Forecast script failed with exit code 143 `System.Exception: Forecast script failed with exit code 143 at EnergyDataHub.Data.Services.ProductionForecastService.RunForecastScript(ForecastLocationSettings forecastSettings, String predictorsPath) at EnergyDataHub.Data.Services.ProductionForecastService.ForecastGeneration(Location location, VisualCrossingDataDto weatherForecastDto) at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)`
#3585	IJaoSyncJob.Run An exception occurred during performance of the job. More details...	05/10/2025 12:07:09
#3585	System.Net.Http.HttpRequestException Response status code does not indicate success: 500 (Internal Server Error). System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error). at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at EnergyDataHub.Data.Services.FinalComputationClient.GetFinalComputation(DateTime from, Nullable`1 to, Int32 skip, Int32 take, FinalComputationFilter filters) at EnergyDataHub.Data.Hangfire.Jobs.JaoSyncJob.GetHourlyData(DateTime start) at EnergyDataHub.Data.Hangfire.Jobs.JaoSyncJob.Run() at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
#3442	IJaoSyncJob.Run An exception occurred during performance of the job. More details...	05/09/2025 12:04:21
#3442	System.Net.Http.HttpRequestException Response status code does not indicate success: 500 (Internal Server Error). System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error). at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at EnergyDataHub.Data.Services.FinalComputationClient.GetFinalComputation(DateTime from, Nullable`1 to, Int32 skip, Int32 take, FinalComputationFilter filters) at EnergyDataHub.Data.Hangfire.Jobs.JaoSyncJob.GetHourlyData(DateTime start) at EnergyDataHub.Data.Hangfire.Jobs.JaoSyncJob.Run() at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
#3273	ILoadSyncJob.Run An exception occurred during performance of the job. More details...	05/08/2025 07:15:44
#3273	Renci.SshNet.Common.SshOperationTimeoutException Session operation has timed out Renci.SshNet.Common.SshOperationTimeoutException: Session operation has timed out at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout) at Renci.SshNet.PasswordAuthenticationMethod.Authenticate(Session session) at Renci.SshNet.ClientAuthentication.TryAuthenticate(ISession session, AuthenticationState authenticationState, String[] allowedAuthenticationMethods, SshAuthenticationException& authenticationException) at Renci.SshNet.ClientAuthentication.Authenticate(IConnectionInfoInternal connectionInfo, ISession session) at Renci.SshNet.Session.Connect() at Renci.SshNet.BaseClient.CreateAndConnectSession() at Renci.SshNet.BaseClient.Connect() at EnergyDataHub.Data.Services.SftpService.MapDataToEachArea[DataClass,CsvMapper](String remoteDirectory, List`1 dataMethods, Int32 retryCount) at EnergyDataHub.Data.Services.SftpService.MapDataToEachArea[DataClass,CsvMapper](String remoteDirectory, List`1 dataMethods) at EnergyDataHub.Data.Hangfire.Jobs.LoadSyncJob.Run() at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
#2447	IJaoSyncJob.Run An exception occurred during performance of the job. More details...	05/02/2025 12:01:45
#2447	System.Threading.Tasks.TaskCanceledException The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing. System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing. ---> System.TimeoutException: The operation was canceled. ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException: Operation canceled --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token) at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core\|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken) at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core\|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken) at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) --- End of inner exception stack trace --- --- End of inner exception stack trace --- at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) at System.Net.Http.HttpClient.GetStringAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at EnergyDataHub.Data.Services.FinalComputationClient.GetFinalComputation(DateTime from, Nullable`1 to, Int32 skip, Int32 take, FinalComputationFilter filters) at EnergyDataHub.Data.Hangfire.Jobs.JaoSyncJob.GetHourlyData(DateTime start) at EnergyDataHub.Data.Hangfire.Jobs.JaoSyncJob.Run() at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
#1760	IScheduledDayAheadTransferSyncJob.Run An exception occurred during performance of the job. More details...	04/26/2025 20:52:40
#1760	Npgsql.PostgresException 53100: could not extend file "base/18813/19276.11": No space left on device Npgsql.PostgresException: 53100: could not extend file "base/18813/19276.11": No space left on device at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at Npgsql.NpgsqlBinaryImporter.Complete(Boolean async, CancellationToken cancellationToken) at EnergyDataHub.Data.Services.TimeseriesBulkInserter.BulkInsertAsync(IEnumerable`1 values) at EnergyDataHub.Data.Services.TimeseriesBulkInserter.BulkInsertAsync(IEnumerable`1 values) at EnergyDataHub.Data.Services.TimeseriesBulkInserter.BulkInsertAsync(IEnumerable`1 values) at EnergyDataHub.Data.Services.TransferCapacityService.UpdateScheduled(Guid guid, List`1 data, AppDbContext context, DataSource dataSource) at EnergyDataHub.Data.Services.BiddingZoneService.MapTransferCapacity(List`1 data, AppDbContext context, DataSource dataSource) at EnergyDataHub.Data.Services.SftpService.MapDataToEachArea[DataClass,CsvMapper](String remoteDirectory, List`1 dataMethods, Int32 retryCount) at EnergyDataHub.Data.Services.SftpService.MapDataToEachArea[DataClass,CsvMapper](String remoteDirectory, List`1 dataMethods) at EnergyDataHub.Data.Hangfire.Jobs.TransmissionCapacitySyncJob.Run() at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
#1755	ILoadSyncJob.Run An exception occurred during performance of the job. More details...	04/26/2025 19:18:43
#1755	Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException The maximum number of retries (5) was exceeded while executing database operations with 'NpgsqlRetryingExecutionStrategy'. See the inner exception for the most recent failure. Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException: The maximum number of retries (5) was exceeded while executing database operations with 'NpgsqlRetryingExecutionStrategy'. See the inner exception for the most recent failure. ---> Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details. ---> Npgsql.PostgresException: 53100: could not extend file "base/18813/19276.11": No space left on device at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken) at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Update.Internal.NpgsqlModificationCommandBatch.Consume(RelationalDataReader reader, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at Npgsql.EntityFrameworkCore.PostgreSQL.Update.Internal.NpgsqlModificationCommandBatch.Consume(RelationalDataReader reader, Boolean async, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass30_0`2.<<ExecuteAsync>b__0>d.MoveNext() --- End of stack trace from previous location --- at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken) --- End of inner exception stack trace --- at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at EnergyDataHub.Data.Services.LoadService.UpdateActualLoadAsync(Guid loadGuid, List`1 data, AppDbContext context, DataSource dataSource) at EnergyDataHub.Data.Services.ControlAreaService.MapLoads(List`1 data, AppDbContext context, DataSource dataSource) at EnergyDataHub.Data.Services.SftpService.MapDataToEachArea[DataClass,CsvMapper](String remoteDirectory, List`1 dataMethods, Int32 retryCount) at EnergyDataHub.Data.Services.SftpService.MapDataToEachArea[DataClass,CsvMapper](String remoteDirectory, List`1 dataMethods) at EnergyDataHub.Data.Hangfire.Jobs.LoadSyncJob.Run() at InvokeStub_TaskAwaiter.GetResult(Object, Object, IntPtr*) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Npgsql.NpgsqlException

System.Net.Http.HttpRequestException

Npgsql.NpgsqlException

System.Exception

System.Net.Http.HttpRequestException

System.Net.Http.HttpRequestException

Renci.SshNet.Common.SshOperationTimeoutException

System.Threading.Tasks.TaskCanceledException

Npgsql.PostgresException

Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException