File size: 10,416 Bytes
5ea8129
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
layer,module,loss,samples,damp,time
0,self_attn.k_proj,1.1475261450,0.01000,2.468
0,self_attn.v_proj,0.0107328082,0.01000,1.111
0,self_attn.q_proj,1.6792541742,0.01000,1.125
0,self_attn.o_proj,0.0007965644,0.01000,1.147
0,mlp.gate_proj,0.8748422861,0.01000,1.140
0,mlp.up_proj,0.7267805338,0.01000,1.149
0,mlp.down_proj,0.0032293317,0.01000,4.942
1,self_attn.k_proj,0.7058779001,0.01000,1.090
1,self_attn.v_proj,0.0272232741,0.01000,1.098
1,self_attn.q_proj,1.2407879829,0.01000,1.137
1,self_attn.o_proj,0.0022073006,0.01000,1.112
1,mlp.gate_proj,1.6644470692,0.01000,1.144
1,mlp.up_proj,1.4123177528,0.01000,1.154
1,mlp.down_proj,0.4668061137,0.01000,4.930
2,self_attn.k_proj,3.9154675007,0.01000,1.146
2,self_attn.v_proj,0.1671302617,0.01000,1.160
2,self_attn.q_proj,5.8791418076,0.01000,1.169
2,self_attn.o_proj,0.0026952710,0.01000,1.174
2,mlp.gate_proj,3.0024461746,0.01000,1.149
2,mlp.up_proj,2.4086780548,0.01000,1.161
2,mlp.down_proj,0.0158377104,0.01000,4.979
3,self_attn.k_proj,3.3933033943,0.01000,1.103
3,self_attn.v_proj,0.3081016541,0.01000,1.092
3,self_attn.q_proj,5.9866695404,0.01000,1.135
3,self_attn.o_proj,0.0066352515,0.01000,1.167
3,mlp.gate_proj,4.7357149124,0.01000,1.203
3,mlp.up_proj,3.3190371990,0.01000,1.182
3,mlp.down_proj,0.0302925445,0.01000,5.019
4,self_attn.k_proj,3.1403093338,0.01000,1.084
4,self_attn.v_proj,0.2942028642,0.01000,1.087
4,self_attn.q_proj,5.4390058517,0.01000,1.107
4,self_attn.o_proj,0.0120283859,0.01000,1.101
4,mlp.gate_proj,6.6809277534,0.01000,1.139
4,mlp.up_proj,4.1057472229,0.01000,1.138
4,mlp.down_proj,0.0517220125,0.01000,4.826
5,self_attn.k_proj,5.0449981689,0.01000,1.074
5,self_attn.v_proj,0.2839890718,0.01000,1.079
5,self_attn.q_proj,7.8570251465,0.01000,1.141
5,self_attn.o_proj,0.0153575996,0.01000,1.122
5,mlp.gate_proj,8.0337257385,0.01000,1.129
5,mlp.up_proj,5.0011324883,0.01000,1.122
5,mlp.down_proj,0.0737693906,0.01000,4.802
6,self_attn.k_proj,4.0896453857,0.01000,1.072
6,self_attn.v_proj,0.3205337226,0.01000,1.074
6,self_attn.q_proj,7.2172508240,0.01000,1.101
6,self_attn.o_proj,0.0274211578,0.01000,1.095
6,mlp.gate_proj,8.8662490845,0.01000,1.119
6,mlp.up_proj,5.4516530037,0.01000,1.134
6,mlp.down_proj,0.0911406577,0.01000,4.811
7,self_attn.k_proj,4.0856785774,0.01000,1.086
7,self_attn.v_proj,0.3383469582,0.01000,1.125
7,self_attn.q_proj,6.7441992760,0.01000,1.125
7,self_attn.o_proj,0.0383612439,0.01000,1.103
7,mlp.gate_proj,8.9464616776,0.01000,1.137
7,mlp.up_proj,5.8620467186,0.01000,1.152
7,mlp.down_proj,0.1056507826,0.01000,4.914
8,self_attn.k_proj,5.2506866455,0.01000,1.072
8,self_attn.v_proj,0.4422910810,0.01000,1.090
8,self_attn.q_proj,8.4763450623,0.01000,1.106
8,self_attn.o_proj,0.0497262478,0.01000,1.209
8,mlp.gate_proj,9.5526027679,0.01000,1.122
8,mlp.up_proj,6.1954283714,0.01000,1.103
8,mlp.down_proj,0.1141102761,0.01000,4.991
9,self_attn.k_proj,5.0246934891,0.01000,1.072
9,self_attn.v_proj,0.6303853989,0.01000,1.085
9,self_attn.q_proj,8.4155311584,0.01000,1.117
9,self_attn.o_proj,0.0606980324,0.01000,1.097
9,mlp.gate_proj,9.9771251678,0.01000,1.133
9,mlp.up_proj,6.4280290604,0.01000,1.135
9,mlp.down_proj,0.1241965592,0.01000,4.805
10,self_attn.k_proj,6.0160880089,0.01000,1.081
10,self_attn.v_proj,0.4758896232,0.01000,1.087
10,self_attn.q_proj,9.7882556915,0.01000,1.111
10,self_attn.o_proj,0.0556933545,0.01000,1.106
10,mlp.gate_proj,9.6824035645,0.01000,1.116
10,mlp.up_proj,6.7106361389,0.01000,1.123
10,mlp.down_proj,0.1305213124,0.01000,4.822
11,self_attn.k_proj,5.4834017754,0.01000,1.072
11,self_attn.v_proj,0.5022003651,0.01000,1.084
11,self_attn.q_proj,8.5779695511,0.01000,1.099
11,self_attn.o_proj,0.0583102107,0.01000,1.102
11,mlp.gate_proj,9.8283424377,0.01000,1.146
11,mlp.up_proj,7.0486173630,0.01000,1.216
11,mlp.down_proj,0.1389759183,0.01000,4.970
12,self_attn.k_proj,4.3386583328,0.01000,1.085
12,self_attn.v_proj,0.6001272798,0.01000,1.087
12,self_attn.q_proj,7.5542769432,0.01000,1.127
12,self_attn.o_proj,0.0756879896,0.01000,1.112
12,mlp.gate_proj,9.5294275284,0.01000,1.138
12,mlp.up_proj,7.2102746964,0.01000,1.134
12,mlp.down_proj,0.1575263739,0.01000,4.883
13,self_attn.k_proj,6.3363122940,0.01000,1.083
13,self_attn.v_proj,0.6461595893,0.01000,1.085
13,self_attn.q_proj,9.6406240463,0.01000,1.136
13,self_attn.o_proj,0.0848969668,0.01000,1.107
13,mlp.gate_proj,9.9299240112,0.01000,1.128
13,mlp.up_proj,7.5109939575,0.01000,1.130
13,mlp.down_proj,0.1766805649,0.01000,4.849
14,self_attn.k_proj,6.6320075989,0.01000,1.111
14,self_attn.v_proj,0.6643830538,0.01000,1.088
14,self_attn.q_proj,9.5511035919,0.01000,1.105
14,self_attn.o_proj,0.0965307355,0.01000,1.113
14,mlp.gate_proj,11.6048107147,0.01000,1.137
14,mlp.up_proj,8.2849397659,0.01000,1.121
14,mlp.down_proj,0.2228597105,0.01000,4.831
15,self_attn.k_proj,6.2356123924,0.01000,1.065
15,self_attn.v_proj,0.8308242559,0.01000,1.111
15,self_attn.q_proj,11.5919666290,0.01000,1.135
15,self_attn.o_proj,0.1115171313,0.01000,1.115
15,mlp.gate_proj,12.9646749496,0.01000,1.138
15,mlp.up_proj,8.7790040970,0.01000,1.147
15,mlp.down_proj,0.2702335119,0.01000,4.878
16,self_attn.k_proj,6.4466848373,0.01000,1.082
16,self_attn.v_proj,0.7520785332,0.01000,1.088
16,self_attn.q_proj,10.5518913269,0.01000,1.116
16,self_attn.o_proj,0.0855024755,0.01000,1.118
16,mlp.gate_proj,14.8053245544,0.01000,1.132
16,mlp.up_proj,9.4714431763,0.01000,1.136
16,mlp.down_proj,0.2961856127,0.01000,4.867
17,self_attn.k_proj,7.0454621315,0.01000,1.086
17,self_attn.v_proj,0.8348115683,0.01000,1.095
17,self_attn.q_proj,11.2236194611,0.01000,1.116
17,self_attn.o_proj,0.0778950602,0.01000,1.105
17,mlp.gate_proj,15.9026432037,0.01000,1.166
17,mlp.up_proj,9.9591217041,0.01000,1.141
17,mlp.down_proj,0.3455633223,0.01000,4.874
18,self_attn.k_proj,7.7922224998,0.01000,1.091
18,self_attn.v_proj,0.8560650349,0.01000,1.095
18,self_attn.q_proj,11.4546375275,0.01000,1.127
18,self_attn.o_proj,0.0532253161,0.01000,1.105
18,mlp.gate_proj,17.1779804230,0.01000,1.132
18,mlp.up_proj,10.6506500244,0.01000,1.142
18,mlp.down_proj,0.3454488516,0.01000,4.812
19,self_attn.k_proj,7.1313219070,0.01000,1.086
19,self_attn.v_proj,0.9581958055,0.01000,1.082
19,self_attn.q_proj,11.9034891129,0.01000,1.109
19,self_attn.o_proj,0.0492735319,0.01000,1.086
19,mlp.gate_proj,18.4003639221,0.01000,1.125
19,mlp.up_proj,11.2164115906,0.01000,1.123
19,mlp.down_proj,0.3670601249,0.01000,4.813
20,self_attn.k_proj,7.7340869904,0.01000,1.067
20,self_attn.v_proj,1.0284407139,0.01000,1.070
20,self_attn.q_proj,12.2217864990,0.01000,1.101
20,self_attn.o_proj,0.0547249839,0.01000,1.089
20,mlp.gate_proj,19.7666130066,0.01000,1.113
20,mlp.up_proj,12.1536312103,0.01000,1.124
20,mlp.down_proj,0.4030991197,0.01000,4.900
21,self_attn.k_proj,7.6493363380,0.01000,1.082
21,self_attn.v_proj,1.1354986429,0.01000,1.087
21,self_attn.q_proj,11.7636947632,0.01000,1.115
21,self_attn.o_proj,0.0786665529,0.01000,1.124
21,mlp.gate_proj,21.1686744690,0.01000,1.133
21,mlp.up_proj,12.9163112640,0.01000,1.138
21,mlp.down_proj,0.4641517401,0.01000,4.814
22,self_attn.k_proj,8.0380859375,0.01000,1.064
22,self_attn.v_proj,1.3097620010,0.01000,1.061
22,self_attn.q_proj,11.8790740967,0.01000,1.091
22,self_attn.o_proj,0.0659603477,0.01000,1.091
22,mlp.gate_proj,22.1032676697,0.01000,1.115
22,mlp.up_proj,13.5992584229,0.01000,1.114
22,mlp.down_proj,0.4817797542,0.01000,4.806
23,self_attn.k_proj,7.8610706329,0.01000,1.086
23,self_attn.v_proj,1.4520299435,0.01000,1.101
23,self_attn.q_proj,12.4829196930,0.01000,1.116
23,self_attn.o_proj,0.0676236451,0.01000,1.106
23,mlp.gate_proj,23.1737003326,0.01000,1.137
23,mlp.up_proj,14.3504924774,0.01000,1.134
23,mlp.down_proj,0.5118615627,0.01000,5.056
24,self_attn.k_proj,7.7259273529,0.01000,1.087
24,self_attn.v_proj,1.8023245335,0.01000,1.102
24,self_attn.q_proj,12.4955329895,0.01000,1.151
24,self_attn.o_proj,0.0688903779,0.01000,1.107
24,mlp.gate_proj,24.7712249756,0.01000,1.123
24,mlp.up_proj,15.3326759338,0.01000,1.128
24,mlp.down_proj,0.5504493713,0.01000,4.816
25,self_attn.k_proj,7.3518638611,0.01000,1.146
25,self_attn.v_proj,1.8994591236,0.01000,1.079
25,self_attn.q_proj,12.5168266296,0.01000,1.112
25,self_attn.o_proj,0.0802244395,0.01000,1.096
25,mlp.gate_proj,26.4093017578,0.01000,1.132
25,mlp.up_proj,16.3634262085,0.01000,1.133
25,mlp.down_proj,0.6082845926,0.01000,4.908
26,self_attn.k_proj,7.7135314941,0.01000,1.086
26,self_attn.v_proj,1.7557444572,0.01000,1.087
26,self_attn.q_proj,12.0845994949,0.01000,1.133
26,self_attn.o_proj,0.1087998450,0.01000,1.099
26,mlp.gate_proj,28.4340057373,0.01000,1.134
26,mlp.up_proj,17.5897064209,0.01000,1.136
26,mlp.down_proj,0.6887663603,0.01000,4.893
27,self_attn.k_proj,8.4675750732,0.01000,1.078
27,self_attn.v_proj,2.5324683189,0.01000,1.082
27,self_attn.q_proj,12.6983404160,0.01000,1.119
27,self_attn.o_proj,0.1327282339,0.01000,1.108
27,mlp.gate_proj,30.9341011047,0.01000,1.126
27,mlp.up_proj,19.2698402405,0.01000,1.134
27,mlp.down_proj,0.8203021288,0.01000,4.828
28,self_attn.k_proj,6.7639026642,0.01000,1.072
28,self_attn.v_proj,2.2498350143,0.01000,1.091
28,self_attn.q_proj,11.8598346710,0.01000,1.106
28,self_attn.o_proj,0.2453110516,0.01000,1.175
28,mlp.gate_proj,32.7573089600,0.01000,1.155
28,mlp.up_proj,21.3828105927,0.01000,1.332
28,mlp.down_proj,1.0597875118,0.01000,4.883
29,self_attn.k_proj,7.1089892387,0.01000,1.073
29,self_attn.v_proj,2.7313909531,0.01000,1.087
29,self_attn.q_proj,12.5505723953,0.01000,1.112
29,self_attn.o_proj,0.1926715672,0.01000,1.105
29,mlp.gate_proj,33.3567161560,0.01000,1.130
29,mlp.up_proj,22.8287887573,0.01000,1.142
29,mlp.down_proj,1.4321247339,0.01000,4.903
30,self_attn.k_proj,6.9233446121,0.01000,1.075
30,self_attn.v_proj,3.6408839226,0.01000,1.073
30,self_attn.q_proj,11.4353094101,0.01000,1.103
30,self_attn.o_proj,0.3697905540,0.01000,1.094
30,mlp.gate_proj,35.7464675903,0.01000,1.127
30,mlp.up_proj,23.8753814697,0.01000,1.131
30,mlp.down_proj,2.2603495121,0.01000,4.849
31,self_attn.k_proj,5.1827154160,0.01000,1.071
31,self_attn.v_proj,2.2259736061,0.01000,1.092
31,self_attn.q_proj,10.0211391449,0.01000,1.098
31,self_attn.o_proj,0.7152889967,0.01000,1.090
31,mlp.gate_proj,31.6050128937,0.01000,1.130
31,mlp.up_proj,21.6463909149,0.01000,1.131
31,mlp.down_proj,5.6123828888,0.01000,4.841