alibabasglab commited on
Commit
d588fc2
·
verified ·
1 Parent(s): 4459443

Upload 8 files

Browse files
checkpoints/.DS_Store ADDED
Binary file (6.15 kB). View file
 
checkpoints/log_LRS2_lip_tfgridnet_3spk/config.yaml ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## Config file
2
+
3
+ # Log
4
+ seed: 777
5
+ use_cuda: 1 # 1 for True, 0 for False
6
+
7
+ # dataset
8
+ speaker_no: 3
9
+ mix_lst_path: ./data/LRS2/mixture_data_list_3mix.csv
10
+ audio_direc: /mnt/nas_sg/wulanchabu/zexu.pan/datasets/LRS2/audio_clean/
11
+ reference_direc: /mnt/nas_sg/wulanchabu/zexu.pan/datasets/LRS2/mvlrs_v1/
12
+ audio_sr: 16000
13
+ ref_sr: 25
14
+
15
+ # dataloader
16
+ num_workers: 4
17
+ batch_size: 1 # 8-GPU training with a total effective batch size of 8
18
+ accu_grad: 0
19
+ effec_batch_size: 4 # per GPU, only used if accu_grad is set to 1, must be multiple times of batch size
20
+ max_length: 3 # truncate the utterances in dataloader, in seconds
21
+
22
+ # network settings
23
+ init_from: None # 'None' or a log name 'log_2024-07-22(18:12:13)'
24
+ causal: 0 # 1 for True, 0 for False
25
+ network_reference:
26
+ cue: lip # lip or speech or gesture or EEG
27
+ backbone: resnet18 # resnet18 or shufflenetV2 or blazenet64
28
+ emb_size: 256 # resnet18:256
29
+ network_audio:
30
+ backbone: av_tfgridnet
31
+ n_fft: 256
32
+ stride: 128
33
+ window: "hann"
34
+ use_builtin_complex: False
35
+ n_srcs: 1
36
+ n_imics: 1
37
+ n_layers: 6
38
+ lstm_hidden_units: 192
39
+ attn_n_head: 4
40
+ attn_qk_output_channel: 4
41
+ emb_dim: 48
42
+ emb_ks: 4
43
+ emb_hs: 1
44
+ activation: "prelu"
45
+
46
+ # optimizer
47
+ loss_type: sisdr # "snr", "sisdr", "hybrid"
48
+ init_learning_rate: 0.001
49
+ max_epoch: 150
50
+ clip_grad_norm: 5
checkpoints/log_LRS2_lip_tfgridnet_3spk/last_best_checkpoint.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:623f54eb793683defc579c2ff0d9ba9fa26100e016fe82278d274d57a57fc5bd
3
+ size 160433102
checkpoints/log_LRS2_lip_tfgridnet_3spk/last_checkpoint.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:360ac77f05d11f21539dda902fd2d8a7857ece03e6bd4946c818a436d776b45b
3
+ size 160425982
checkpoints/log_LRS2_lip_tfgridnet_3spk/log_2024-10-01(15:53:50).txt ADDED
@@ -0,0 +1,503 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## Config file
2
+
3
+ # Log
4
+ seed: 777
5
+ use_cuda: 1 # 1 for True, 0 for False
6
+
7
+ # dataset
8
+ speaker_no: 3
9
+ mix_lst_path: ./data/LRS2/mixture_data_list_3mix.csv
10
+ audio_direc: /mnt/nas_sg/mit_sg/zexu.pan/datasets/LRS2/audio_clean/
11
+ reference_direc: /mnt/nas_sg/mit_sg/zexu.pan/datasets/LRS2/mvlrs_v1/
12
+ audio_sr: 16000
13
+ ref_sr: 25
14
+
15
+ # dataloader
16
+ num_workers: 4
17
+ batch_size: 1 # 8-GPU training with a total effective batch size of 8
18
+ accu_grad: 0
19
+ effec_batch_size: 4 # per GPU, only used if accu_grad is set to 1, must be multiple times of batch size
20
+ max_length: 3 # truncate the utterances in dataloader, in seconds
21
+
22
+ # network settings
23
+ init_from: None # 'None' or a log name 'log_2024-07-22(18:12:13)'
24
+ causal: 0 # 1 for True, 0 for False
25
+ network_reference:
26
+ cue: lip # lip or speech or gesture or EEG
27
+ backbone: resnet18 # resnet18 or shufflenetV2 or blazenet64
28
+ emb_size: 256 # resnet18:256
29
+ network_audio:
30
+ backbone: tfgridnet
31
+ n_fft: 256
32
+ stride: 128
33
+ window: "hann"
34
+ use_builtin_complex: False
35
+ n_srcs: 1
36
+ n_imics: 1
37
+ n_layers: 6
38
+ lstm_hidden_units: 192
39
+ attn_n_head: 4
40
+ attn_qk_output_channel: 4
41
+ emb_dim: 48
42
+ emb_ks: 4
43
+ emb_hs: 1
44
+ activation: "prelu"
45
+
46
+ # optimizer
47
+ loss_type: sisdr # "snr", "sisdr", "hybrid"
48
+ init_learning_rate: 0.001
49
+ max_epoch: 150
50
+ clip_grad_norm: 5
51
+ W1001 15:53:53.420768 139891169711936 torch/distributed/run.py:757]
52
+ W1001 15:53:53.420768 139891169711936 torch/distributed/run.py:757] *****************************************
53
+ W1001 15:53:53.420768 139891169711936 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
54
+ W1001 15:53:53.420768 139891169711936 torch/distributed/run.py:757] *****************************************
55
+ started on checkpoints/log_2024-10-01(15:53:50)
56
+
57
+ namespace(seed=777, use_cuda=1, config=[<yamlargparse.Path object at 0x7f20217b4e80>], checkpoint_dir='checkpoints/log_2024-10-01(15:53:50)', train_from_last_checkpoint=0, loss_type='sisdr', init_learning_rate=0.001, lr_warmup=0, max_epoch=150, clip_grad_norm=5.0, batch_size=1, accu_grad=0, effec_batch_size=4, max_length=3, num_workers=4, causal=0, network_reference=namespace(cue='lip', backbone='resnet18', emb_size=256), network_audio=namespace(backbone='tfgridnet', n_fft=256, stride=128, window='hann', use_builtin_complex=False, n_srcs=1, n_imics=1, n_layers=6, lstm_hidden_units=192, attn_n_head=4, attn_qk_output_channel=4, emb_dim=48, emb_ks=4, emb_hs=1, activation='prelu'), init_from='None', mix_lst_path='./data/LRS2/mixture_data_list_3mix.csv', audio_direc='/mnt/nas_sg/mit_sg/zexu.pan/datasets/LRS2/audio_clean/', reference_direc='/mnt/nas_sg/mit_sg/zexu.pan/datasets/LRS2/mvlrs_v1/', speaker_no=3, audio_sr=16000, ref_sr=25, local_rank=0, distributed=True, world_size=8, device=device(type='cuda'))
58
+ network_wrapper(
59
+ (sep_network): TFGridNetV3(
60
+ (enc): STFTEncoder(
61
+ (stft): Stft(n_fft=256, win_length=256, hop_length=128, center=True, normalized=False, onesided=True)
62
+ )
63
+ (dec): STFTDecoder(
64
+ (stft): Stft(n_fft=256, win_length=256, hop_length=128, center=True, normalized=False, onesided=True)
65
+ )
66
+ (conv): Sequential(
67
+ (0): Conv2d(2, 48, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
68
+ (1): GroupNorm(1, 48, eps=1e-05, affine=True)
69
+ )
70
+ (blocks): ModuleList(
71
+ (0-5): 6 x GridNetV3Block(
72
+ (intra_norm): LayerNorm((48,), eps=1e-05, elementwise_affine=True)
73
+ (intra_rnn): LSTM(192, 192, batch_first=True, bidirectional=True)
74
+ (intra_linear): ConvTranspose1d(384, 48, kernel_size=(4,), stride=(1,))
75
+ (inter_norm): LayerNorm((48,), eps=1e-05, elementwise_affine=True)
76
+ (inter_rnn): LSTM(192, 192, batch_first=True, bidirectional=True)
77
+ (inter_linear): ConvTranspose1d(384, 48, kernel_size=(4,), stride=(1,))
78
+ (attn_conv_Q): Conv2d(48, 16, kernel_size=(1, 1), stride=(1, 1))
79
+ (attn_norm_Q): AllHeadPReLULayerNormalization4DC(
80
+ (act): PReLU(num_parameters=4)
81
+ )
82
+ (attn_conv_K): Conv2d(48, 16, kernel_size=(1, 1), stride=(1, 1))
83
+ (attn_norm_K): AllHeadPReLULayerNormalization4DC(
84
+ (act): PReLU(num_parameters=4)
85
+ )
86
+ (attn_conv_V): Conv2d(48, 48, kernel_size=(1, 1), stride=(1, 1))
87
+ (attn_norm_V): AllHeadPReLULayerNormalization4DC(
88
+ (act): PReLU(num_parameters=4)
89
+ )
90
+ (attn_concat_proj): Sequential(
91
+ (0): Conv2d(48, 48, kernel_size=(1, 1), stride=(1, 1))
92
+ (1): PReLU(num_parameters=1)
93
+ (2): LayerNormalization()
94
+ )
95
+ )
96
+ )
97
+ (deconv): ConvTranspose2d(48, 2, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
98
+ (av_conv): ModuleList(
99
+ (0-5): 6 x Linear(in_features=304, out_features=48, bias=True)
100
+ )
101
+ )
102
+ (ref_encoder): Visual_encoder(
103
+ (v_frontend): VisualFrontend(
104
+ (frontend3D): Sequential(
105
+ (0): Conv3d(1, 64, kernel_size=(5, 7, 7), stride=(1, 2, 2), padding=(2, 3, 3), bias=False)
106
+ (1): SyncBatchNorm(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
107
+ (2): ReLU()
108
+ (3): MaxPool3d(kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), dilation=1, ceil_mode=False)
109
+ )
110
+ (resnet): ResNet(
111
+ (layer1): ResNetLayer(
112
+ (conv1a): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
113
+ (bn1a): SyncBatchNorm(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
114
+ (conv2a): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
115
+ (downsample): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
116
+ (outbna): SyncBatchNorm(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
117
+ (conv1b): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
118
+ (bn1b): SyncBatchNorm(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
119
+ (conv2b): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
120
+ (outbnb): SyncBatchNorm(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
121
+ )
122
+ (layer2): ResNetLayer(
123
+ (conv1a): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
124
+ (bn1a): SyncBatchNorm(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
125
+ (conv2a): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
126
+ (downsample): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
127
+ (outbna): SyncBatchNorm(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
128
+ (conv1b): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
129
+ (bn1b): SyncBatchNorm(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
130
+ (conv2b): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
131
+ (outbnb): SyncBatchNorm(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
132
+ )
133
+ (layer3): ResNetLayer(
134
+ (conv1a): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
135
+ (bn1a): SyncBatchNorm(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
136
+ (conv2a): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
137
+ (downsample): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
138
+ (outbna): SyncBatchNorm(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
139
+ (conv1b): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
140
+ (bn1b): SyncBatchNorm(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
141
+ (conv2b): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
142
+ (outbnb): SyncBatchNorm(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
143
+ )
144
+ (layer4): ResNetLayer(
145
+ (conv1a): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
146
+ (bn1a): SyncBatchNorm(512, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
147
+ (conv2a): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
148
+ (downsample): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
149
+ (outbna): SyncBatchNorm(512, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
150
+ (conv1b): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
151
+ (bn1b): SyncBatchNorm(512, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
152
+ (conv2b): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
153
+ (outbnb): SyncBatchNorm(512, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
154
+ )
155
+ (avgpool): AvgPool2d(kernel_size=(4, 4), stride=(1, 1), padding=0)
156
+ )
157
+ )
158
+ (v_ds): Conv1d(512, 256, kernel_size=(1,), stride=(1,), bias=False)
159
+ (visual_conv): Sequential(
160
+ (0): VisualConv1D(
161
+ (relu_0): ReLU()
162
+ (norm_0): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
163
+ (conv1x1): Conv1d(256, 512, kernel_size=(1,), stride=(1,), bias=False)
164
+ (relu): ReLU()
165
+ (norm_1): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
166
+ (dsconv): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(1,), groups=512)
167
+ (prelu): PReLU(num_parameters=1)
168
+ (norm_2): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
169
+ (pw_conv): Conv1d(512, 256, kernel_size=(1,), stride=(1,), bias=False)
170
+ )
171
+ (1): VisualConv1D(
172
+ (relu_0): ReLU()
173
+ (norm_0): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
174
+ (conv1x1): Conv1d(256, 512, kernel_size=(1,), stride=(1,), bias=False)
175
+ (relu): ReLU()
176
+ (norm_1): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
177
+ (dsconv): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(1,), groups=512)
178
+ (prelu): PReLU(num_parameters=1)
179
+ (norm_2): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
180
+ (pw_conv): Conv1d(512, 256, kernel_size=(1,), stride=(1,), bias=False)
181
+ )
182
+ (2): VisualConv1D(
183
+ (relu_0): ReLU()
184
+ (norm_0): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
185
+ (conv1x1): Conv1d(256, 512, kernel_size=(1,), stride=(1,), bias=False)
186
+ (relu): ReLU()
187
+ (norm_1): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
188
+ (dsconv): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(1,), groups=512)
189
+ (prelu): PReLU(num_parameters=1)
190
+ (norm_2): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
191
+ (pw_conv): Conv1d(512, 256, kernel_size=(1,), stride=(1,), bias=False)
192
+ )
193
+ (3): VisualConv1D(
194
+ (relu_0): ReLU()
195
+ (norm_0): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
196
+ (conv1x1): Conv1d(256, 512, kernel_size=(1,), stride=(1,), bias=False)
197
+ (relu): ReLU()
198
+ (norm_1): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
199
+ (dsconv): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(1,), groups=512)
200
+ (prelu): PReLU(num_parameters=1)
201
+ (norm_2): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
202
+ (pw_conv): Conv1d(512, 256, kernel_size=(1,), stride=(1,), bias=False)
203
+ )
204
+ (4): VisualConv1D(
205
+ (relu_0): ReLU()
206
+ (norm_0): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
207
+ (conv1x1): Conv1d(256, 512, kernel_size=(1,), stride=(1,), bias=False)
208
+ (relu): ReLU()
209
+ (norm_1): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
210
+ (dsconv): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(1,), groups=512)
211
+ (prelu): PReLU(num_parameters=1)
212
+ (norm_2): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
213
+ (pw_conv): Conv1d(512, 256, kernel_size=(1,), stride=(1,), bias=False)
214
+ )
215
+ )
216
+ )
217
+ )
218
+
219
+ Total number of parameters: 20780101
220
+
221
+
222
+ Total number of trainable parameters: 9595013
223
+
224
+ Start new training from scratch
225
+ [rank1]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
226
+ [rank3]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
227
+ [rank5]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
228
+ [rank2]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
229
+ [rank6]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
230
+ [rank4]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
231
+ [rank7]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
232
+ [rank0]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
233
+ Train Summary | End of Epoch 1 | Time 10648.35s | Train Loss 0.798
234
+ Valid Summary | End of Epoch 1 | Time 611.94s | Valid Loss -0.670
235
+ Test Summary | End of Epoch 1 | Time 303.72s | Test Loss -0.847
236
+ Fund new best model, dict saved
237
+ Train Summary | End of Epoch 2 | Time 10657.50s | Train Loss -1.751
238
+ Valid Summary | End of Epoch 2 | Time 612.09s | Valid Loss -2.574
239
+ Test Summary | End of Epoch 2 | Time 304.01s | Test Loss -2.965
240
+ Fund new best model, dict saved
241
+ Train Summary | End of Epoch 3 | Time 10659.67s | Train Loss -3.420
242
+ Valid Summary | End of Epoch 3 | Time 612.06s | Valid Loss -4.035
243
+ Test Summary | End of Epoch 3 | Time 303.92s | Test Loss -4.392
244
+ Fund new best model, dict saved
245
+ Train Summary | End of Epoch 4 | Time 10657.45s | Train Loss -4.505
246
+ Valid Summary | End of Epoch 4 | Time 612.09s | Valid Loss -4.946
247
+ Test Summary | End of Epoch 4 | Time 304.13s | Test Loss -5.418
248
+ Fund new best model, dict saved
249
+ Train Summary | End of Epoch 5 | Time 10660.00s | Train Loss -5.172
250
+ Valid Summary | End of Epoch 5 | Time 612.05s | Valid Loss -5.432
251
+ Test Summary | End of Epoch 5 | Time 304.03s | Test Loss -5.920
252
+ Fund new best model, dict saved
253
+ Train Summary | End of Epoch 6 | Time 10658.66s | Train Loss -5.674
254
+ Valid Summary | End of Epoch 6 | Time 612.21s | Valid Loss -5.829
255
+ Test Summary | End of Epoch 6 | Time 303.88s | Test Loss -6.409
256
+ Fund new best model, dict saved
257
+ Train Summary | End of Epoch 7 | Time 10658.47s | Train Loss -6.028
258
+ Valid Summary | End of Epoch 7 | Time 611.67s | Valid Loss -6.065
259
+ Test Summary | End of Epoch 7 | Time 303.84s | Test Loss -6.727
260
+ Fund new best model, dict saved
261
+ Train Summary | End of Epoch 8 | Time 10658.61s | Train Loss -6.365
262
+ Valid Summary | End of Epoch 8 | Time 612.25s | Valid Loss -6.345
263
+ Test Summary | End of Epoch 8 | Time 303.94s | Test Loss -7.031
264
+ Fund new best model, dict saved
265
+ Train Summary | End of Epoch 9 | Time 10656.89s | Train Loss -6.661
266
+ Valid Summary | End of Epoch 9 | Time 611.95s | Valid Loss -6.569
267
+ Test Summary | End of Epoch 9 | Time 304.14s | Test Loss -7.234
268
+ Fund new best model, dict saved
269
+ Train Summary | End of Epoch 10 | Time 10657.06s | Train Loss -6.882
270
+ Valid Summary | End of Epoch 10 | Time 612.12s | Valid Loss -6.834
271
+ Test Summary | End of Epoch 10 | Time 304.11s | Test Loss -7.553
272
+ Fund new best model, dict saved
273
+ Train Summary | End of Epoch 11 | Time 10655.25s | Train Loss -7.082
274
+ Valid Summary | End of Epoch 11 | Time 611.78s | Valid Loss -7.065
275
+ Test Summary | End of Epoch 11 | Time 304.19s | Test Loss -7.669
276
+ Fund new best model, dict saved
277
+ Train Summary | End of Epoch 12 | Time 10657.47s | Train Loss -7.274
278
+ Valid Summary | End of Epoch 12 | Time 612.17s | Valid Loss -7.168
279
+ Test Summary | End of Epoch 12 | Time 304.15s | Test Loss -7.859
280
+ Fund new best model, dict saved
281
+ Train Summary | End of Epoch 13 | Time 10657.14s | Train Loss -7.426
282
+ Valid Summary | End of Epoch 13 | Time 611.98s | Valid Loss -7.124
283
+ Test Summary | End of Epoch 13 | Time 304.07s | Test Loss -7.863
284
+ Train Summary | End of Epoch 14 | Time 10657.35s | Train Loss -7.545
285
+ Valid Summary | End of Epoch 14 | Time 612.10s | Valid Loss -7.389
286
+ Test Summary | End of Epoch 14 | Time 303.98s | Test Loss -8.066
287
+ Fund new best model, dict saved
288
+ Train Summary | End of Epoch 15 | Time 10655.98s | Train Loss -7.667
289
+ Valid Summary | End of Epoch 15 | Time 611.84s | Valid Loss -7.454
290
+ Test Summary | End of Epoch 15 | Time 303.98s | Test Loss -8.181
291
+ Fund new best model, dict saved
292
+ Train Summary | End of Epoch 16 | Time 10656.82s | Train Loss -7.756
293
+ Valid Summary | End of Epoch 16 | Time 611.77s | Valid Loss -7.408
294
+ Test Summary | End of Epoch 16 | Time 304.19s | Test Loss -8.045
295
+ Train Summary | End of Epoch 17 | Time 10657.17s | Train Loss -7.853
296
+ Valid Summary | End of Epoch 17 | Time 611.99s | Valid Loss -7.559
297
+ Test Summary | End of Epoch 17 | Time 304.10s | Test Loss -8.310
298
+ Fund new best model, dict saved
299
+ Train Summary | End of Epoch 18 | Time 10656.45s | Train Loss -7.899
300
+ Valid Summary | End of Epoch 18 | Time 611.96s | Valid Loss -7.622
301
+ Test Summary | End of Epoch 18 | Time 304.08s | Test Loss -8.364
302
+ Fund new best model, dict saved
303
+ Train Summary | End of Epoch 19 | Time 10657.14s | Train Loss -7.992
304
+ Valid Summary | End of Epoch 19 | Time 612.15s | Valid Loss -7.821
305
+ Test Summary | End of Epoch 19 | Time 304.16s | Test Loss -8.521
306
+ Fund new best model, dict saved
307
+ Train Summary | End of Epoch 20 | Time 10656.52s | Train Loss -8.046
308
+ Valid Summary | End of Epoch 20 | Time 611.83s | Valid Loss -7.770
309
+ Test Summary | End of Epoch 20 | Time 303.92s | Test Loss -8.574
310
+ Train Summary | End of Epoch 21 | Time 10655.12s | Train Loss -7.968
311
+ Valid Summary | End of Epoch 21 | Time 611.92s | Valid Loss -7.635
312
+ Test Summary | End of Epoch 21 | Time 304.04s | Test Loss -8.250
313
+ Train Summary | End of Epoch 22 | Time 10657.13s | Train Loss -8.084
314
+ Valid Summary | End of Epoch 22 | Time 611.90s | Valid Loss -7.788
315
+ Test Summary | End of Epoch 22 | Time 304.11s | Test Loss -8.518
316
+ Train Summary | End of Epoch 23 | Time 10655.92s | Train Loss -8.129
317
+ Valid Summary | End of Epoch 23 | Time 612.01s | Valid Loss -7.663
318
+ Test Summary | End of Epoch 23 | Time 304.23s | Test Loss -8.161
319
+ Train Summary | End of Epoch 24 | Time 10657.57s | Train Loss -8.117
320
+ Valid Summary | End of Epoch 24 | Time 611.89s | Valid Loss -7.801
321
+ Test Summary | End of Epoch 24 | Time 304.07s | Test Loss -8.504
322
+ reload weights and optimizer from last best checkpoint
323
+ Learning rate adjusted to: 0.000500
324
+ Train Summary | End of Epoch 25 | Time 10662.95s | Train Loss -8.577
325
+ Valid Summary | End of Epoch 25 | Time 611.80s | Valid Loss -8.209
326
+ Test Summary | End of Epoch 25 | Time 304.21s | Test Loss -9.011
327
+ Fund new best model, dict saved
328
+ Train Summary | End of Epoch 26 | Time 10665.89s | Train Loss -8.824
329
+ Valid Summary | End of Epoch 26 | Time 611.63s | Valid Loss -8.406
330
+ Test Summary | End of Epoch 26 | Time 304.24s | Test Loss -9.215
331
+ Fund new best model, dict saved
332
+ Train Summary | End of Epoch 27 | Time 10663.44s | Train Loss -8.947
333
+ Valid Summary | End of Epoch 27 | Time 611.91s | Valid Loss -8.379
334
+ Test Summary | End of Epoch 27 | Time 304.43s | Test Loss -9.244
335
+ Train Summary | End of Epoch 28 | Time 10663.82s | Train Loss -9.064
336
+ Valid Summary | End of Epoch 28 | Time 611.97s | Valid Loss -8.507
337
+ Test Summary | End of Epoch 28 | Time 304.21s | Test Loss -9.352
338
+ Fund new best model, dict saved
339
+ Train Summary | End of Epoch 29 | Time 10662.40s | Train Loss -9.153
340
+ Valid Summary | End of Epoch 29 | Time 611.80s | Valid Loss -8.476
341
+ Test Summary | End of Epoch 29 | Time 304.05s | Test Loss -9.340
342
+ Train Summary | End of Epoch 30 | Time 10663.15s | Train Loss -9.249
343
+ Valid Summary | End of Epoch 30 | Time 612.03s | Valid Loss -8.474
344
+ Test Summary | End of Epoch 30 | Time 304.14s | Test Loss -9.398
345
+ Train Summary | End of Epoch 31 | Time 10662.70s | Train Loss -9.324
346
+ Valid Summary | End of Epoch 31 | Time 611.83s | Valid Loss -8.623
347
+ Test Summary | End of Epoch 31 | Time 304.21s | Test Loss -9.531
348
+ Fund new best model, dict saved
349
+ Train Summary | End of Epoch 32 | Time 10665.05s | Train Loss -9.391
350
+ Valid Summary | End of Epoch 32 | Time 611.73s | Valid Loss -8.651
351
+ Test Summary | End of Epoch 32 | Time 303.92s | Test Loss -9.543
352
+ Fund new best model, dict saved
353
+ Train Summary | End of Epoch 33 | Time 10662.53s | Train Loss -9.476
354
+ Valid Summary | End of Epoch 33 | Time 611.92s | Valid Loss -8.705
355
+ Test Summary | End of Epoch 33 | Time 304.13s | Test Loss -9.607
356
+ Fund new best model, dict saved
357
+ Train Summary | End of Epoch 34 | Time 10661.27s | Train Loss -9.530
358
+ Valid Summary | End of Epoch 34 | Time 611.97s | Valid Loss -8.828
359
+ Test Summary | End of Epoch 34 | Time 304.28s | Test Loss -9.661
360
+ Fund new best model, dict saved
361
+ Train Summary | End of Epoch 35 | Time 10661.89s | Train Loss -9.607
362
+ Valid Summary | End of Epoch 35 | Time 611.96s | Valid Loss -8.786
363
+ Test Summary | End of Epoch 35 | Time 304.18s | Test Loss -9.703
364
+ Train Summary | End of Epoch 36 | Time 10662.46s | Train Loss -9.655
365
+ Valid Summary | End of Epoch 36 | Time 611.76s | Valid Loss -8.833
366
+ Test Summary | End of Epoch 36 | Time 304.26s | Test Loss -9.735
367
+ Fund new best model, dict saved
368
+ Train Summary | End of Epoch 37 | Time 10662.54s | Train Loss -9.721
369
+ Valid Summary | End of Epoch 37 | Time 612.18s | Valid Loss -8.914
370
+ Test Summary | End of Epoch 37 | Time 304.19s | Test Loss -9.810
371
+ Fund new best model, dict saved
372
+ Train Summary | End of Epoch 38 | Time 10663.13s | Train Loss -9.772
373
+ Valid Summary | End of Epoch 38 | Time 612.06s | Valid Loss -8.902
374
+ Test Summary | End of Epoch 38 | Time 304.36s | Test Loss -9.759
375
+ Train Summary | End of Epoch 39 | Time 10661.24s | Train Loss -9.825
376
+ Valid Summary | End of Epoch 39 | Time 611.75s | Valid Loss -8.974
377
+ Test Summary | End of Epoch 39 | Time 304.21s | Test Loss -9.806
378
+ Fund new best model, dict saved
379
+ Train Summary | End of Epoch 40 | Time 10662.98s | Train Loss -9.878
380
+ Valid Summary | End of Epoch 40 | Time 612.25s | Valid Loss -8.918
381
+ Test Summary | End of Epoch 40 | Time 304.14s | Test Loss -9.836
382
+ Train Summary | End of Epoch 41 | Time 10662.94s | Train Loss -9.919
383
+ Valid Summary | End of Epoch 41 | Time 612.15s | Valid Loss -8.889
384
+ Test Summary | End of Epoch 41 | Time 304.30s | Test Loss -9.874
385
+ Train Summary | End of Epoch 42 | Time 10665.59s | Train Loss -9.980
386
+ Valid Summary | End of Epoch 42 | Time 612.08s | Valid Loss -8.965
387
+ Test Summary | End of Epoch 42 | Time 304.16s | Test Loss -9.857
388
+ Train Summary | End of Epoch 43 | Time 10665.85s | Train Loss -10.028
389
+ Valid Summary | End of Epoch 43 | Time 611.91s | Valid Loss -8.998
390
+ Test Summary | End of Epoch 43 | Time 304.24s | Test Loss -9.924
391
+ Fund new best model, dict saved
392
+ Train Summary | End of Epoch 44 | Time 10424.67s | Train Loss -10.076
393
+ Valid Summary | End of Epoch 44 | Time 610.11s | Valid Loss -8.930
394
+ Test Summary | End of Epoch 44 | Time 303.99s | Test Loss -9.907
395
+ Train Summary | End of Epoch 45 | Time 10422.39s | Train Loss -10.107
396
+ Valid Summary | End of Epoch 45 | Time 609.72s | Valid Loss -8.909
397
+ Test Summary | End of Epoch 45 | Time 303.88s | Test Loss -9.927
398
+ Train Summary | End of Epoch 46 | Time 10421.71s | Train Loss -10.158
399
+ Valid Summary | End of Epoch 46 | Time 610.10s | Valid Loss -9.059
400
+ Test Summary | End of Epoch 46 | Time 304.16s | Test Loss -10.024
401
+ Fund new best model, dict saved
402
+ Train Summary | End of Epoch 47 | Time 10419.35s | Train Loss -10.200
403
+ Valid Summary | End of Epoch 47 | Time 610.06s | Valid Loss -9.029
404
+ Test Summary | End of Epoch 47 | Time 303.96s | Test Loss -10.083
405
+ Train Summary | End of Epoch 48 | Time 10439.37s | Train Loss -10.240
406
+ Valid Summary | End of Epoch 48 | Time 610.23s | Valid Loss -9.088
407
+ Test Summary | End of Epoch 48 | Time 303.97s | Test Loss -10.094
408
+ Fund new best model, dict saved
409
+ Train Summary | End of Epoch 49 | Time 10449.91s | Train Loss -10.280
410
+ Valid Summary | End of Epoch 49 | Time 609.81s | Valid Loss -9.062
411
+ Test Summary | End of Epoch 49 | Time 303.80s | Test Loss -10.063
412
+ Train Summary | End of Epoch 50 | Time 10446.89s | Train Loss -10.315
413
+ Valid Summary | End of Epoch 50 | Time 610.27s | Valid Loss -9.149
414
+ Test Summary | End of Epoch 50 | Time 303.96s | Test Loss -10.108
415
+ Fund new best model, dict saved
416
+ Train Summary | End of Epoch 51 | Time 10451.47s | Train Loss -10.340
417
+ Valid Summary | End of Epoch 51 | Time 609.86s | Valid Loss -9.030
418
+ Test Summary | End of Epoch 51 | Time 303.94s | Test Loss -10.115
419
+ Train Summary | End of Epoch 52 | Time 10450.57s | Train Loss -10.388
420
+ Valid Summary | End of Epoch 52 | Time 610.34s | Valid Loss -9.155
421
+ Test Summary | End of Epoch 52 | Time 303.96s | Test Loss -10.158
422
+ Fund new best model, dict saved
423
+ Train Summary | End of Epoch 53 | Time 10449.92s | Train Loss -10.428
424
+ Valid Summary | End of Epoch 53 | Time 610.19s | Valid Loss -9.224
425
+ Test Summary | End of Epoch 53 | Time 304.09s | Test Loss -10.183
426
+ Fund new best model, dict saved
427
+ Train Summary | End of Epoch 54 | Time 10447.21s | Train Loss -10.393
428
+ Valid Summary | End of Epoch 54 | Time 610.14s | Valid Loss -9.138
429
+ Test Summary | End of Epoch 54 | Time 303.98s | Test Loss -9.971
430
+ Train Summary | End of Epoch 55 | Time 10445.48s | Train Loss -10.387
431
+ Valid Summary | End of Epoch 55 | Time 610.25s | Valid Loss -9.124
432
+ Test Summary | End of Epoch 55 | Time 303.95s | Test Loss -10.155
433
+ Train Summary | End of Epoch 56 | Time 10445.60s | Train Loss -10.454
434
+ Valid Summary | End of Epoch 56 | Time 610.28s | Valid Loss -9.191
435
+ Test Summary | End of Epoch 56 | Time 304.00s | Test Loss -10.171
436
+ Train Summary | End of Epoch 57 | Time 10448.32s | Train Loss -10.513
437
+ Valid Summary | End of Epoch 57 | Time 610.21s | Valid Loss -9.220
438
+ Test Summary | End of Epoch 57 | Time 304.01s | Test Loss -10.244
439
+ Train Summary | End of Epoch 58 | Time 10447.46s | Train Loss -10.552
440
+ Valid Summary | End of Epoch 58 | Time 610.24s | Valid Loss -9.194
441
+ Test Summary | End of Epoch 58 | Time 303.98s | Test Loss -10.193
442
+ reload weights and optimizer from last best checkpoint
443
+ Learning rate adjusted to: 0.000250
444
+ Train Summary | End of Epoch 59 | Time 10452.01s | Train Loss -10.697
445
+ Valid Summary | End of Epoch 59 | Time 610.06s | Valid Loss -9.388
446
+ Test Summary | End of Epoch 59 | Time 304.16s | Test Loss -10.339
447
+ Fund new best model, dict saved
448
+ Train Summary | End of Epoch 60 | Time 10449.84s | Train Loss -10.795
449
+ Valid Summary | End of Epoch 60 | Time 609.93s | Valid Loss -9.292
450
+ Test Summary | End of Epoch 60 | Time 303.94s | Test Loss -10.374
451
+ Train Summary | End of Epoch 61 | Time 10447.76s | Train Loss -10.856
452
+ Valid Summary | End of Epoch 61 | Time 610.40s | Valid Loss -9.381
453
+ Test Summary | End of Epoch 61 | Time 304.09s | Test Loss -10.382
454
+ Train Summary | End of Epoch 62 | Time 10448.63s | Train Loss -10.906
455
+ Valid Summary | End of Epoch 62 | Time 610.24s | Valid Loss -9.351
456
+ Test Summary | End of Epoch 62 | Time 304.06s | Test Loss -10.372
457
+ Train Summary | End of Epoch 63 | Time 10450.52s | Train Loss -10.956
458
+ Valid Summary | End of Epoch 63 | Time 610.42s | Valid Loss -9.362
459
+ Test Summary | End of Epoch 63 | Time 304.03s | Test Loss -10.362
460
+ Train Summary | End of Epoch 64 | Time 10450.50s | Train Loss -10.990
461
+ Valid Summary | End of Epoch 64 | Time 610.12s | Valid Loss -9.318
462
+ Test Summary | End of Epoch 64 | Time 304.12s | Test Loss -10.371
463
+ reload weights and optimizer from last best checkpoint
464
+ Learning rate adjusted to: 0.000125
465
+ Train Summary | End of Epoch 65 | Time 10450.03s | Train Loss -10.885
466
+ Valid Summary | End of Epoch 65 | Time 609.95s | Valid Loss -9.396
467
+ Test Summary | End of Epoch 65 | Time 303.90s | Test Loss -10.418
468
+ Fund new best model, dict saved
469
+ Train Summary | End of Epoch 66 | Time 10449.01s | Train Loss -10.948
470
+ Valid Summary | End of Epoch 66 | Time 610.45s | Valid Loss -9.409
471
+ Test Summary | End of Epoch 66 | Time 303.96s | Test Loss -10.412
472
+ Fund new best model, dict saved
473
+ Train Summary | End of Epoch 67 | Time 10448.14s | Train Loss -11.002
474
+ Valid Summary | End of Epoch 67 | Time 610.08s | Valid Loss -9.367
475
+ Test Summary | End of Epoch 67 | Time 304.11s | Test Loss -10.416
476
+ Train Summary | End of Epoch 68 | Time 10452.36s | Train Loss -11.047
477
+ Valid Summary | End of Epoch 68 | Time 610.20s | Valid Loss -9.337
478
+ Test Summary | End of Epoch 68 | Time 304.06s | Test Loss -10.386
479
+ Train Summary | End of Epoch 69 | Time 10450.39s | Train Loss -11.080
480
+ Valid Summary | End of Epoch 69 | Time 610.41s | Valid Loss -9.371
481
+ Test Summary | End of Epoch 69 | Time 304.09s | Test Loss -10.388
482
+ Train Summary | End of Epoch 70 | Time 10453.44s | Train Loss -11.110
483
+ Valid Summary | End of Epoch 70 | Time 610.19s | Valid Loss -9.376
484
+ Test Summary | End of Epoch 70 | Time 303.94s | Test Loss -10.408
485
+ Train Summary | End of Epoch 71 | Time 10448.80s | Train Loss -11.143
486
+ Valid Summary | End of Epoch 71 | Time 610.22s | Valid Loss -9.392
487
+ Test Summary | End of Epoch 71 | Time 304.20s | Test Loss -10.402
488
+ reload weights and optimizer from last best checkpoint
489
+ Learning rate adjusted to: 0.000063
490
+ Train Summary | End of Epoch 72 | Time 10449.24s | Train Loss -11.037
491
+ Valid Summary | End of Epoch 72 | Time 610.13s | Valid Loss -9.400
492
+ Test Summary | End of Epoch 72 | Time 304.13s | Test Loss -10.421
493
+ Train Summary | End of Epoch 73 | Time 10454.82s | Train Loss -11.073
494
+ Valid Summary | End of Epoch 73 | Time 609.99s | Valid Loss -9.402
495
+ Test Summary | End of Epoch 73 | Time 303.91s | Test Loss -10.436
496
+ Train Summary | End of Epoch 74 | Time 10455.29s | Train Loss -11.108
497
+ Valid Summary | End of Epoch 74 | Time 610.10s | Valid Loss -9.418
498
+ Test Summary | End of Epoch 74 | Time 303.96s | Test Loss -10.421
499
+ Fund new best model, dict saved
500
+ Start evaluation
501
+ Avg SISNR:i tensor([14.9690], device='cuda:0')
502
+ Avg SNRi: 15.427827702003633
503
+ Avg STOIi: 0.3833540046412457
checkpoints/log_LRS2_lip_tfgridnet_3spk/tensorboard/events.out.tfevents.1727769243.bach-gpu011017044238.na61.52651.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4375c5c910c8025b8c5aaad54636192b2df2bfd150f1ca6f1e18734170ffbedd
3
+ size 6452
checkpoints/log_LRS2_lip_tfgridnet_3spk/tensorboard/events.out.tfevents.1728269430.nls-dev-servers011167134195.na63.96142.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d6d3f458b99a5612988dc77a21ad6e5ef8e1c63cc0149dc1072eafc32b8d64d6
3
+ size 4676
checkpoints/log_LRS2_lip_tfgridnet_3spk/tensorboard/events.out.tfevents.1728630190.dsw-111903-69597c964d-glc75.604215.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8cb2350eead52fb6f716685cbed7ea6f23578dd1a129dac94fa57c1b2bd1e0a2
3
+ size 88