You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-17 - 18:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Jun 17, 2024 00:44:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13158839911524,21cyclictest0-21swapper/223:54:052
1623991140,99rtkit-daemon0-21swapper/019:08:510
13158789911123,16cyclictest880726-21firefox23:19:191
13158839910825,20cyclictest0-21swapper/223:41:152
13158839910223,22cyclictest1378434-21/usr/sbin/munin22:35:322
1315495210151,41sleep10-21swapper/119:07:061
13717352960,8sleep2881251-21Softwar~cThread22:15:162
131588799942,7cyclictest0-21swapper/323:54:343
1315887999420,11cyclictest1366682-21DNS3
162399930,8rtkit-daemon1622-21rtkit-daemon23:01:102
131588799932,7cyclictest0-21swapper/300:34:123
131588799922,6cyclictest0-21swapper/322:27:453
131588799922,6cyclictest0-21swapper/300:26:213
131588799914,84cyclictest0-21swapper/319:17:343
131548729156,28sleep30-21swapper/319:07:013
131588799902,85cyclictest0-21swapper/319:52:423
131588799902,85cyclictest0-21swapper/319:52:423
131588799902,6cyclictest0-21swapper/319:26:003
131588799902,6cyclictest0-21swapper/319:26:003
1315887998982,4cyclictest0-21swapper/320:51:463
131588799892,28cyclictest1753-21gnome-shell22:36:343
131588799872,7cyclictest0-21swapper/323:58:203
131588799872,6cyclictest0-21swapper/320:01:143
1315874998767,5cyclictest141rcu_preempt20:49:450
1315887998679,4cyclictest0-21swapper/323:24:483
1315887998629,52cyclictest42-21ksoftirqd/320:20:253
131515328670,9sleep20-21swapper/219:05:352
1315883998562,15cyclictest34-21ksoftirqd/223:23:222
1315887998417,14cyclictest881251-21Softwar~cThread21:19:293
1315883998461,14cyclictest34-21ksoftirqd/220:50:392
131588799832,22cyclictest1753-21gnome-shell21:46:063
1315887998315,18cyclictest42-21ksoftirqd/300:00:373
131588799831,19cyclictest1753-21gnome-shell22:40:353
1315878998365,6cyclictest141rcu_preempt23:54:111
1315878998363,5cyclictest26-21ksoftirqd/123:40:391
1315874998365,6cyclictest13-21ksoftirqd/022:42:190
1315874998363,6cyclictest13-21ksoftirqd/023:55:330
1315874998363,6cyclictest13-21ksoftirqd/021:30:330
1315874998363,6cyclictest13-21ksoftirqd/021:30:320
131588799821,22cyclictest1753-21gnome-shell23:25:103
1315883998259,14cyclictest34-21ksoftirqd/221:48:052
1315883998259,14cyclictest34-21ksoftirqd/219:10:312
1315883998240,24cyclictest34-21ksoftirqd/220:23:192
1315883998233,44cyclictest1753-21gnome-shell00:05:312
1315883998220,19cyclictest34-21ksoftirqd/219:25:232
1315883998220,19cyclictest34-21ksoftirqd/219:25:232
1315874998270,8cyclictest1753-21gnome-shell22:49:140
1315874998270,8cyclictest1753-21gnome-shell22:49:130
1315874998228,43cyclictest13-21ksoftirqd/021:43:560
1315874998225,8cyclictest13-21ksoftirqd/023:32:590
1315887998122,16cyclictest1374714-21cron22:25:013
1315887998122,16cyclictest1374714-21cron22:25:003
131588799812,19cyclictest1753-21gnome-shell21:30:313
131588799812,19cyclictest1753-21gnome-shell21:30:303
1315887998112,8cyclictest141rcu_preempt00:05:203
1315883998158,14cyclictest34-21ksoftirqd/221:35:182
1315874998163,7cyclictest13-21ksoftirqd/000:21:290
1315874998123,9cyclictest13-21ksoftirqd/019:59:530
131588799803,11cyclictest880726-21firefox22:51:453
131588799802,12cyclictest1362842-40lsb_release21:27:193
1315887998010,8cyclictest141rcu_preempt19:35:393
1315883998058,14cyclictest34-21ksoftirqd/219:50:062
1315883998058,14cyclictest34-21ksoftirqd/219:50:062
131588399802,26cyclictest1753-21gnome-shell00:34:552
1315878998056,9cyclictest141rcu_preempt19:55:511
1315874998021,9cyclictest13-21ksoftirqd/019:27:170
1315874998021,9cyclictest13-21ksoftirqd/019:27:170
131588799792,56cyclictest1753-21gnome-shell19:31:023
131588799792,13cyclictest880726-21firefox20:49:563
1315887997911,8cyclictest141rcu_preempt19:12:203
1315883997958,13cyclictest34-21ksoftirqd/221:20:442
1315883997957,14cyclictest34-21ksoftirqd/200:37:022
1315883997956,14cyclictest34-21ksoftirqd/200:02:072
1315883997938,23cyclictest34-21ksoftirqd/223:45:322
1315874997923,45cyclictest13-21ksoftirqd/023:41:090
131588799782,19cyclictest1753-21gnome-shell23:42:483
1315887997810,8cyclictest141rcu_preempt00:20:153
1315883997856,13cyclictest34-21ksoftirqd/200:21:312
131588399782,60cyclictest1753-21gnome-shell19:34:422
1315883997821,43cyclictest34-21ksoftirqd/220:00:272
1315874997861,6cyclictest13-21ksoftirqd/022:24:520
1315874997861,6cyclictest13-21ksoftirqd/022:24:520
131588799772,14cyclictest0-21swapper/323:48:533
131588799772,11cyclictest1753-21gnome-shell23:30:273
1315883997722,51cyclictest34-21ksoftirqd/219:23:432
1315883997719,17cyclictest34-21ksoftirqd/223:34:562
1315878997766,7cyclictest1753-21gnome-shell00:20:201
1315878997726,46cyclictest881251-21Softwar~cThread21:43:571
1315874997761,5cyclictest13-21ksoftirqd/023:36:430
1315874997761,5cyclictest13-21ksoftirqd/023:36:430
1315874997759,6cyclictest13-21ksoftirqd/020:53:450
1315874997759,5cyclictest13-21ksoftirqd/022:28:460
1315874997758,7cyclictest13-21ksoftirqd/019:15:490
1315874997726,46cyclictest1753-21gnome-shell20:55:290
1315874997726,41cyclictest1753-21gnome-shell23:46:330
1315874997721,45cyclictest13-21ksoftirqd/000:19:310
1315874997716,17cyclictest1362886-21gdbus21:27:280
1315874997715,9cyclictest13-21ksoftirqd/023:29:020
131588799768,52cyclictest141rcu_preempt21:35:223
131588799762,17cyclictest1362272-21runrttasks21:24:263
1315883997621,51cyclictest34-21ksoftirqd/222:26:522
1315874997661,5cyclictest13-21ksoftirqd/019:30:510
1315874997660,6cyclictest13-21ksoftirqd/020:01:450
1315874997659,6cyclictest13-21ksoftirqd/021:36:270
1315874997659,5cyclictest13-21ksoftirqd/000:38:570
162399750,5rtkit-daemon0-21swapper/320:19:243
162399750,5rtkit-daemon0-21swapper/320:19:243
131588799753,9cyclictest1382678-21ntp_states22:45:263
131588799753,9cyclictest1382678-21ntp_states22:45:263
131588799753,10cyclictest1321635-21munin-run19:25:013
1315887997512,50cyclictest141rcu_preempt20:55:363
1315874997558,13cyclictest13-21ksoftirqd/023:19:410
131588799742,7cyclictest0-21swapper/321:42:203
131588799742,67cyclictest1753-21gnome-shell23:35:043
131588799742,67cyclictest1753-21gnome-shell23:35:043
131588799742,47cyclictest1412427-21ps00:10:343
1315883997463,6cyclictest34-21ksoftirqd/219:59:392
1315883997462,7cyclictest1753-21gnome-shell23:29:032
1315883997424,22cyclictest880726-21firefox19:18:192
1315883997418,52cyclictest34-21ksoftirqd/222:34:362
131588799736,7cyclictest141rcu_preempt19:55:493
131588799732,7cyclictest0-21swapper/323:19:453
131588799732,7cyclictest0-21swapper/320:25:083
131588799732,7cyclictest0-21swapper/300:19:533
131588799732,10cyclictest0-21swapper/319:49:373
131588399732,48cyclictest1753-21gnome-shell00:15:352
1315874997358,11cyclictest13-21ksoftirqd/021:24:300
1315874997357,5cyclictest13-21ksoftirqd/019:38:360
1315874997343,20cyclictest13-21ksoftirqd/023:51:010
162399720,8rtkit-daemon1785-21gmain20:43:203
162399720,8rtkit-daemon1785-21gmain20:43:203
131588799729,8cyclictest141rcu_preempt00:35:243
131588799723,11cyclictest1377208-21irqstats22:30:223
131588799722,12cyclictest1753-21gnome-shell22:19:253
131588799722,11cyclictest1327513-21hddtemp_smartct19:40:243
1315883997262,5cyclictest34-21ksoftirqd/223:56:232
1315878997264,5cyclictest0-21swapper/100:25:391
1315878997215,51cyclictest881251-21Softwar~cThread19:50:031
1315878997215,51cyclictest881251-21Softwar~cThread19:50:031
1315874997255,6cyclictest13-21ksoftirqd/022:38:530
1315883997160,8cyclictest1753-21gnome-shell21:51:412
1315883997154,13cyclictest34-21ksoftirqd/222:46:292
1315883997154,13cyclictest34-21ksoftirqd/222:46:292
1315883997153,5cyclictest34-21ksoftirqd/200:25:192
1315878997163,5cyclictest0-21swapper/100:31:281
1315878997154,12cyclictest881251-21Softwar~cThread23:25:141
1315878997116,50cyclictest1412090-21cat00:10:231
1315874997123,44cyclictest13-21ksoftirqd/000:05:490
1315874997122,7cyclictest13-21ksoftirqd/021:49:550
1315874997122,7cyclictest13-21ksoftirqd/019:40:210
1315883997059,6cyclictest34-21ksoftirqd/222:54:152
1315883997059,6cyclictest34-21ksoftirqd/200:13:372
1315883997053,6cyclictest34-21ksoftirqd/222:40:312
1315883997052,13cyclictest34-21ksoftirqd/220:55:452
1315883997024,22cyclictest0-21swapper/219:40:222
131588399701,43cyclictest854-21avahi-daemon21:28:342
1315878997042,17cyclictest26-21ksoftirqd/100:00:391
1315878997019,47cyclictest1317559-21http19:15:011
1315878997014,52cyclictest26-21ksoftirqd/100:15:591
131587499707,7cyclictest141rcu_preempt19:50:200
131587499707,7cyclictest141rcu_preempt19:50:200
1315874997061,5cyclictest13-21ksoftirqd/000:33:060
13594482690,7sleep30-21swapper/321:10:403
131588399693,59cyclictest1753-21gnome-shell20:19:122
131588399693,59cyclictest1753-21gnome-shell20:19:122
1315883996925,21cyclictest1365869-21pool-udisksd21:41:552
1315878996952,14cyclictest26-21ksoftirqd/122:28:491
1315878996924,19cyclictest26-21ksoftirqd/121:48:591
1315878996918,14cyclictest881251-21Softwar~cThread22:33:521
131587499692,8cyclictest141rcu_preempt22:50:350
1315874996920,7cyclictest13-21ksoftirqd/022:32:290
162399680,9rtkit-daemon1622-21rtkit-daemon21:19:022
1315883996850,14cyclictest34-21ksoftirqd/220:49:512
1315878996851,13cyclictest26-21ksoftirqd/120:55:371
1315878996832,17cyclictest26-21ksoftirqd/121:50:491
1315878996814,49cyclictest881369-21Timer23:30:171
1315878996814,48cyclictest881251-21Softwar~cThread00:35:031
1315878996812,52cyclictest26-21ksoftirqd/121:34:251
1315878996812,52cyclictest26-21ksoftirqd/121:34:251
131587499682,25cyclictest1315804-21kworker/0:119:15:010
131588399671,42cyclictest141rcu_preempt23:35:212
131588399671,42cyclictest141rcu_preempt23:35:202
1315878996759,5cyclictest0-21swapper/119:33:041
1315878996757,6cyclictest1362328-21latency_hist21:25:021
1315878996747,12cyclictest26-21ksoftirqd/121:28:301
1315874996714,11cyclictest881369-21Timer00:27:000
1315874996713,10cyclictest880726-21firefox21:54:100
131588399662,22cyclictest1753-21gnome-shell22:21:042
131588399662,22cyclictest1753-21gnome-shell22:21:042
1315878996659,4cyclictest0-21swapper/122:44:301
1315878996651,12cyclictest26-21ksoftirqd/122:24:471
1315878996651,12cyclictest26-21ksoftirqd/122:24:471
1315878996651,10cyclictest881251-21Softwar~cThread23:38:101
1315878996651,10cyclictest881251-21Softwar~cThread23:38:101
1315878996632,9cyclictest26-21ksoftirqd/119:15:101
131587899662,58cyclictest1753-21gnome-shell20:50:131
1315874996657,5cyclictest13-21ksoftirqd/019:23:200
131587499662,59cyclictest1753-21gnome-shell20:20:360
1315878996554,6cyclictest1327474-21sh19:40:161
1315878996546,12cyclictest26-21ksoftirqd/120:20:491
1315874996555,6cyclictest13-21ksoftirqd/000:03:550
1315874996543,8cyclictest13-21ksoftirqd/019:49:510
131588399642,20cyclictest1753-21gnome-shell21:30:502
131588399642,20cyclictest1753-21gnome-shell21:30:502
131588399642,20cyclictest1753-21gnome-shell19:36:542
1315878996451,9cyclictest1334668-21missed_timers20:00:281
1315878996439,8cyclictest26-21ksoftirqd/119:20:211
1315878996410,49cyclictest438-20systemd-journal21:20:001
131587499642,20cyclictest1412023-21runrttasks00:10:220
1315874996414,8cyclictest13-21ksoftirqd/022:19:220
131588399632,19cyclictest1753-21gnome-shell20:25:102
1315878996350,9cyclictest1383275-21sshd22:48:411
1315878996350,9cyclictest1383275-21sshd22:48:411
1315878996347,12cyclictest26-21ksoftirqd/100:05:521
1315878996342,7cyclictest26-21ksoftirqd/119:35:241
131587499631,19cyclictest1753-21gnome-shell20:25:200
131588399622,19cyclictest1753-21gnome-shell19:49:362
1315878996213,7cyclictest0-21swapper/121:36:021
1315878996211,46cyclictest0-21swapper/121:55:071
1315878996211,46cyclictest0-21swapper/121:55:071
1315874996243,14cyclictest881251-21Softwar~cThread20:20:020
1315874996243,14cyclictest881251-21Softwar~cThread20:20:020
131588399612,55cyclictest1753-21gnome-shell23:19:522
1315878996144,12cyclictest881251-21Softwar~cThread23:46:211
1315878996140,15cyclictest881251-21Softwar~cThread19:25:111
1315878996140,15cyclictest881251-21Softwar~cThread19:25:101
1315878996111,8cyclictest13940482chrt23:20:031
131587499613,53cyclictest1753-21gnome-shell21:55:040
131587499613,53cyclictest1753-21gnome-shell21:55:030
162399600,8rtkit-daemon1622-21rtkit-daemon20:07:122
162399590,9rtkit-daemon1622-21rtkit-daemon23:00:401
162399590,7rtkit-daemon1622-21rtkit-daemon20:44:252
162399590,7rtkit-daemon1622-21rtkit-daemon20:44:252
13452112590,7chrt0-21swapper/020:30:290
1315878995945,11cyclictest26-21ksoftirqd/122:20:011
131587899593,12cyclictest880726-21firefox23:55:541
131587899592,52cyclictest1343221-21irqstats20:25:281
131587899592,15cyclictest1380003-21cat22:40:011
131587499583,4cyclictest141rcu_preempt23:20:340
131587499583,11cyclictest1753-21gnome-shell21:19:200
131587899572,51cyclictest1340660-21chrt20:19:231
131587899572,51cyclictest1340660-21chrt20:19:221
131587899572,13cyclictest1383907-21cstates22:50:141
131587899563,47cyclictest0-21swapper/119:49:121
131587899562,50cyclictest1753-21gnome-shell20:49:131
1315887995518,11cyclictest1368747-21cat21:55:363
1315887995518,11cyclictest1368747-21cat21:55:353
162399530,45rtkit-daemon0-21swapper/220:39:372
1315887995318,13cyclictest1389923-21cat23:05:403
1315887995022,8cyclictest1753-21gnome-shell20:37:023
131588799502,25cyclictest1345342-21python320:30:333
131588799502,13cyclictest1358114-21taskset21:08:483
131588799502,13cyclictest1358114-21taskset21:08:483
131588399500,46cyclictest1622-21rtkit-daemon21:08:432
131588399500,46cyclictest1622-21rtkit-daemon21:08:432
131588799482,26cyclictest1753-21gnome-shell23:04:083
1315887994822,10cyclictest42-21ksoftirqd/322:11:123
1315887994821,8cyclictest1753-21gnome-shell20:07:453
131588799472,8cyclictest1753-21gnome-shell20:12:013
131588399472,5cyclictest0-21swapper/221:55:052
131588399472,5cyclictest0-21swapper/221:55:052
1315883994717,9cyclictest34-21ksoftirqd/222:13:052
1315883994712,19cyclictest880726-21firefox23:06:062
131588799462,8cyclictest1753-21gnome-shell23:13:243
131588799462,8cyclictest1753-21gnome-shell23:13:233
1315887994616,8cyclictest141rcu_preempt22:07:233
131588399462,21cyclictest1345216-21ntp_states20:30:282
131588799451,25cyclictest1753-21gnome-shell22:56:443
1315878994511,30cyclictest1370445-21unin-run22:10:011
1315874994526,7cyclictest13-21ksoftirqd/023:05:260
131587499442,8cyclictest1347669-21turbostat20:40:010
13573512430,5sleep0161rcuc/021:05:260
13573512430,5sleep0161rcuc/021:05:250
131588799432,25cyclictest1753-21gnome-shell21:02:003
1315883994325,13cyclictest438-20systemd-journal21:15:002
1315883994312,12cyclictest141rcu_preempt20:10:232
131588399431,17cyclictest0-21swapper/222:01:492
1315874994313,6cyclictest141rcu_preempt20:43:260
1315874994313,6cyclictest141rcu_preempt20:43:260
13446682420,8sleep10-21swapper/120:30:141
131588399422,22cyclictest1392085-21munin-run23:15:012
131588399422,22cyclictest1392085-21munin-run23:15:002
1315874994220,14cyclictest13-21ksoftirqd/023:00:000
131588399412,21cyclictest1753-21gnome-shell22:57:222
131588399412,19cyclictest1355357-21diskstats21:00:182
1315878994122,7cyclictest26-21ksoftirqd/122:59:061
131587899411,35cyclictest881251-21Softwar~cThread21:00:421
1315874994121,6cyclictest13-21ksoftirqd/022:10:210
131587899401,14cyclictest1371213-21idleruntime-cro22:15:001
1315874994021,6cyclictest13-21ksoftirqd/023:00:060
1315874994019,6cyclictest13-21ksoftirqd/023:12:100
1315874994019,6cyclictest13-21ksoftirqd/023:12:100
131588799392,15cyclictest1753-21gnome-shell22:01:193
131588399391,20cyclictest1753-21gnome-shell22:05:172
1315878993922,13cyclictest1357497-21ntpq21:05:301
1315878993922,13cyclictest1357497-21ntpq21:05:291
131587899391,5cyclictest141rcu_preempt20:44:251
131587899391,5cyclictest141rcu_preempt20:44:241
1315878993820,6cyclictest26-21ksoftirqd/120:38:391
1315878993818,15cyclictest26-21ksoftirqd/120:05:151
1315874993826,7cyclictest13-21ksoftirqd/020:05:290
131587499382,31cyclictest880726-21firefox20:12:420
131587499382,18cyclictest881251-21Softwar~cThread22:09:500
1315878993720,13cyclictest1358859-21df21:10:191
1315874993720,12cyclictest1355395-21sh21:00:220
1315878993614,17cyclictest881251-21Softwar~cThread23:11:211
1315878993614,17cyclictest881251-21Softwar~cThread23:11:211
1315878993520,11cyclictest1389600-21ntpq23:05:311
1315878993514,16cyclictest881356-21Isolated14:31:441
1315874993519,11cyclictest1359579-21sed21:10:420
131587899348,6cyclictest26-21ksoftirqd/122:01:391
131587499321,28cyclictest0-21swapper/022:00:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional