You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 15:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Thu Dec 04, 2025 12:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12065832940,5chrt0-21swapper/310:21:163
12886492730,2chrt451irq_work/311:46:123
1012069996859,7cyclictest282-21systemd-journal07:20:012
1012064996357,4cyclictest1058545-21kworker/u8:1+events_unbound08:25:010
1012064996256,4cyclictest1173690-21kworker/u8:1+events_unbound10:00:010
1012064996155,4cyclictest1311999-21kworker/u8:2+events_unbound12:16:070
1012064996055,3cyclictest1093898-21kworker/u8:0+events_unbound10:20:010
1012069995849,6cyclictest282-21systemd-journal09:16:072
1012064995751,5cyclictest187351-21kworker/u8:3+events_unbound07:21:240
1012064995751,4cyclictest1093898-21kworker/u8:0+flush-8:009:21:220
1012064995551,3cyclictest1232811-21kworker/u8:2+flush-8:010:51:220
1012064995550,3cyclictest1093898-21kworker/u8:0+flush-8:009:41:080
1012067995446,6cyclictest282-21systemd-journal11:00:001
1012064995343,7cyclictest1282771-21latency_hist11:40:010
1012067995244,6cyclictest187351-21kworker/u8:3+flush-8:007:26:251
1012064995244,5cyclictest1011906-21latency_hist07:10:010
11923022510,3chrt481ktimers/310:06:163
1012067995143,6cyclictest282-21systemd-journal09:16:081
1012064995145,4cyclictest1286888-21kworker/u8:3+flush-8:012:36:230
10884262500,2sleep20-21swapper/208:21:232
1012064995046,3cyclictest1218166-21kworker/u8:3+flush-8:011:11:230
1012064995044,5cyclictest971518-21kworker/u8:2+flush-8:007:15:060
1012064995043,5cyclictest1093898-21kworker/u8:0+flush-8:008:41:210
1012064994945,3cyclictest1083980-21kworker/u8:3+flush-8:009:26:250
1012064994939,8cyclictest282-21systemd-journal11:35:000
1012064994843,4cyclictest1083980-21kworker/u8:3+flush-8:008:56:300
1012064994842,4cyclictest971518-21kworker/u8:2+flush-8:007:11:160
1012064994842,4cyclictest1173690-21kworker/u8:1+events_unbound10:46:220
1012064994839,6cyclictest1188564-21latency_hist10:05:010
1012069994737,7cyclictest282-21systemd-journal08:30:002
1012064994742,4cyclictest1043366-21kworker/u8:2+flush-8:007:41:180
1012064994741,4cyclictest1043366-21kworker/u8:2+flush-8:008:01:230
1012064994739,6cyclictest1093898-21kworker/u8:0+flush-8:009:36:210
1012067994637,7cyclictest282-21systemd-journal11:45:001
1012067994637,6cyclictest1292954-21/usr/sbin/munin11:51:151
1012064994641,4cyclictest1093898-21kworker/u8:0+flush-8:009:16:220
1012064994639,5cyclictest1083980-21kworker/u8:3+events_unbound08:35:000
1012064994637,7cyclictest282-21systemd-journal12:35:000
1012064994637,7cyclictest282-21systemd-journal12:34:590
1012064994633,11cyclictest1093898-21kworker/u8:0+flush-8:011:06:170
10461762450,6sleep20-21swapper/207:41:152
1012069994536,7cyclictest1158579-21apt-get09:35:002
1012067994538,5cyclictest1093898-21kworker/u8:0+flush-8:008:41:241
1012064994540,3cyclictest1265412-21kworker/u8:1+events_unbound11:56:190
1012064994540,3cyclictest1073810-21kworker/u8:0+events_unbound08:10:160
1012064994540,3cyclictest1043366-21kworker/u8:2+flush-8:008:06:190
1012064994534,9cyclictest282-21systemd-journal10:56:170
1012067994440,3cyclictest1083980-21kworker/u8:3+events_unbound10:11:231
1012067994437,5cyclictest1058855-21apt-get07:56:031
1012064994439,4cyclictest1265412-21kworker/u8:1+flush-8:011:26:220
1012064994438,5cyclictest1043366-21kworker/u8:2+flush-8:007:46:170
1012064994438,4cyclictest1173690-21kworker/u8:1+events_unbound10:26:170
1012067994337,4cyclictest1198758-21apt-get10:16:041
1012067994335,6cyclictest1149006-21apt-get09:26:051
1012067994334,7cyclictest282-21systemd-journal09:46:171
1012067994333,8cyclictest282-21systemd-journal10:55:001
1012067994333,8cyclictest282-21systemd-journal09:56:161
1012064994339,3cyclictest1173690-21kworker/u8:1+flush-8:010:41:210
1012064994336,5cyclictest1173690-21kworker/u8:1+flush-8:010:06:200
11457622420,2sleep3586-21kworker/3:2+events09:21:133
1012067994236,4cyclictest1232811-21kworker/u8:2+flush-8:011:06:221
1012064994238,3cyclictest1218166-21kworker/u8:3+events_unbound11:16:230
1012064994236,4cyclictest1093898-21kworker/u8:0+events_unbound09:46:190
1012067994133,6cyclictest282-21systemd-journal12:30:171
1012067994133,6cyclictest282-21systemd-journal12:30:171
1012064994137,3cyclictest1301720-21kworker/u8:0+flush-8:012:01:240
1012064994137,3cyclictest1138157-21kworker/u8:1+events_unbound09:31:180
1012064994130,8cyclictest1127682-21ntpq09:01:160
1012069994029,9cyclictest282-21systemd-journal12:24:592
1012067994035,3cyclictest1118438-21gpgv08:54:591
1012064993935,3cyclictest1218166-21kworker/u8:3+events_unbound10:36:240
1012064993934,4cyclictest1058545-21kworker/u8:1+events_unbound08:26:150
1012064993933,4cyclictest1093898-21kworker/u8:0+events_unbound10:21:210
1012064993933,4cyclictest1019163-21kworker/u8:0+flush-8:007:31:180
1012067993832,5cyclictest1173690-21kworker/u8:1+events_unbound10:41:171
1012067993827,8cyclictest1048684-21latency_hist07:45:011
1012064993833,4cyclictest187351-21kworker/u8:3+events_unbound07:51:250
1012064993833,4cyclictest1093898-21kworker/u8:0+flush-8:009:11:200
1012064993832,4cyclictest187351-21kworker/u8:3+events_unbound07:36:120
1012067993732,4cyclictest1084016-21kworker/u8:4+events_unbound08:21:081
1012067993731,5cyclictest1093898-21kworker/u8:0+flush-8:011:21:231
1012067993730,5cyclictest1253142-21apt-get11:11:041
1012067993730,5cyclictest1093898-21kworker/u8:0+flush-8:009:51:181
1012067993729,6cyclictest1034136-21apt-get07:31:031
1012064993732,3cyclictest187351-21kworker/u8:3+flush-8:007:26:140
1012064993731,5cyclictest1108801-21kworker/u8:1+flush-8:008:51:200
10116992374,28sleep30-21swapper/307:07:383
1012064993632,3cyclictest1232811-21kworker/u8:2+flush-8:011:21:270
1012064993631,4cyclictest1173690-21kworker/u8:1+flush-8:010:31:220
1012064993630,4cyclictest1286888-21kworker/u8:3+flush-8:011:51:130
1012064993624,4cyclictest1265412-21kworker/u8:1+events_unbound11:41:080
10118312363,9sleep10-21swapper/107:09:211
1012067993531,3cyclictest1265412-21kworker/u8:1+events_unbound11:39:591
1012067993529,5cyclictest1311999-21kworker/u8:2+events_unbound12:26:091
1012067993529,5cyclictest1083980-21kworker/u8:3+flush-8:008:36:201
1012067993528,5cyclictest282-21systemd-journal07:15:001
1012067993526,6cyclictest282-21systemd-journal08:00:131
1012064993530,4cyclictest1093898-21kworker/u8:0+flush-8:009:51:200
1012064993529,4cyclictest1218166-21kworker/u8:3+events_unbound11:01:170
1012064993529,4cyclictest1073810-21kworker/u8:0+events_unbound08:16:170
1012067993425,7cyclictest1322328-21apt-get12:19:591
1012064993429,4cyclictest1083980-21kworker/u8:3+events_unbound10:11:160
1012064993428,5cyclictest1286888-21kworker/u8:3+events_unbound11:46:190
1012067993329,2cyclictest1302559-21systemd-run12:00:001
1012067993327,5cyclictest187351-21kworker/u8:3+events_unbound07:51:061
1012067993327,5cyclictest1301720-21kworker/u8:0+flush-8:012:06:191
1012067993327,5cyclictest1301720-21kworker/u8:0+flush-8:012:06:181
1012067993327,4cyclictest1265412-21kworker/u8:1+events_unbound12:11:081
1012067993327,4cyclictest1265412-21kworker/u8:1+events_unbound12:11:081
1012067993327,4cyclictest1118899-21kworker/u8:2+events_unbound09:01:101
1012067993326,5cyclictest1058545-21kworker/u8:1+flush-8:008:16:121
1012064993325,6cyclictest1317410-21cron12:15:000
1012064993325,6cyclictest1317410-21cron12:15:000
101206999324,15cyclictest1317742-21apt-get12:16:052
1012069993224,6cyclictest282-21systemd-journal07:25:002
1012067993226,5cyclictest1083980-21kworker/u8:3+events_unbound08:56:151
1012064993227,3cyclictest1311999-21kworker/u8:2+events_unbound12:10:000
1012064993227,3cyclictest1311999-21kworker/u8:2+events_unbound12:10:000
1012064993223,6cyclictest1113483-21latency_hist08:50:000
1012069993120,9cyclictest282-21systemd-journal07:57:142
1012067993126,4cyclictest1218166-21kworker/u8:3+flush-8:011:16:111
1012067993126,3cyclictest1-21systemd07:20:011
1012067993125,5cyclictest1093898-21kworker/u8:0+flush-8:009:11:221
1012067993125,5cyclictest1083980-21kworker/u8:3+flush-8:009:41:171
1012064993123,6cyclictest1058830-21/usr/sbin/munin07:56:210
101093623126,1sleep20-21swapper/207:06:162
101206999303,10cyclictest11854332kthreadcore10:01:112
1012069993024,4cyclictest1143446-21apt-get09:20:032
1012067993025,4cyclictest187351-21kworker/u8:3+events_unbound07:21:091
1012067993024,5cyclictest1265412-21kworker/u8:1+flush-8:012:21:221
1012067993024,5cyclictest1173690-21kworker/u8:1+events_unbound10:26:221
1012067993024,5cyclictest1058545-21kworker/u8:1+flush-8:008:11:081
1012067993024,4cyclictest1218166-21kworker/u8:3+flush-8:011:01:181
1012067993023,5cyclictest282-21systemd-journal10:35:001
1012067993023,5cyclictest1093898-21kworker/u8:0+flush-8:009:21:481
1012064993024,4cyclictest1084016-21kworker/u8:4+flush-8:008:36:110
101206999295,5cyclictest1029179-21apt-get07:26:052
1012069992922,5cyclictest282-21systemd-journal10:30:012
1012069992913,10cyclictest1218545-21apt-get10:36:082
1012064992925,3cyclictest1132585-21needreboot09:06:210
101206999283,10cyclictest1203105-21sshd10:16:502
1012069992820,6cyclictest282-21systemd-journal08:55:002
1012069992819,7cyclictest282-21systemd-journal10:25:002
1012069992817,8cyclictest1119237-21diskstats08:56:152
1012067992824,3cyclictest1138157-21kworker/u8:1+flush-8:009:31:241
1012067992822,5cyclictest1043366-21kworker/u8:2+events_unbound07:46:091
1012067992819,6cyclictest1093972-21/usr/sbin/munin08:31:161
101206999274,5cyclictest1227465-21systemctl10:41:212
1012069992724,2cyclictest1153245-21unixbench_multi09:26:242
1012069992723,2cyclictest1303185-21fschecks_time12:01:082
1012069992723,2cyclictest1178286-21unixbench_singl09:51:252
1012069992723,2cyclictest1083530-21basename08:20:002
1012067992722,4cyclictest1301720-21kworker/u8:0+events_unbound12:01:201
1012067992722,4cyclictest1058545-21kworker/u8:1+events_unbound08:26:211
1012067992721,3cyclictest0-21swapper/111:25:031
1012067992719,6cyclictest1203715-21apt-get10:21:051
1012067992718,7cyclictest1043691-21latency_hist07:40:011
1012067992716,8cyclictest282-21systemd-journal08:08:131
1012064992722,4cyclictest1265412-21kworker/u8:1+flush-8:012:21:190
101206999263,14cyclictest0-21swapper/209:41:172
101206999262,9cyclictest1248029-21sed11:05:012
1012069992623,2cyclictest1262303-21unixbench_multi11:16:272
1012067992621,3cyclictest1286888-21kworker/u8:3+flush-8:011:47:101
1012064992621,3cyclictest1331530-21needreboot12:26:170
101206999253,11cyclictest1112718-21ntp_states08:46:232
1012069992522,2cyclictest1133888-21aten2_r5power_e09:11:072
1012069992521,3cyclictest1248262-21aten2_r5power_c11:06:052
1012069992521,3cyclictest1063891-21aten2_r5power_e08:01:072
1012069992517,6cyclictest1054232-21cut07:51:082
1012069992516,4cyclictest1233411-21apt-get10:51:042
1012069992515,5cyclictest1123956-21apt-get09:01:052
1012067992519,5cyclictest1218166-21kworker/u8:3+events_unbound10:46:061
1012069992420,2cyclictest1069062-21aten2_r5power_p08:06:072
1012069992419,4cyclictest0-21swapper/208:15:002
1012069992415,4cyclictest1327845-21apt-get12:26:062
1012069992414,5cyclictest1312764-21apt-get12:11:052
1012069992414,5cyclictest1312764-21apt-get12:11:052
1012067992420,2cyclictest0-21swapper/112:36:101
1012067992420,2cyclictest0-21swapper/110:36:241
101206999237,6cyclictest0-21swapper/211:36:252
101206999232,5cyclictest1108527-21munin-run08:45:002
1012069992319,2cyclictest1179182-21cpuspeed_turbos09:56:082
1012069992314,4cyclictest1256878-21ntpq11:11:172
1012069992314,4cyclictest1012290-21apt-get07:11:042
1012069992312,9cyclictest0-21swapper/211:31:062
1012067992319,2cyclictest0-21swapper/110:01:121
1012067992319,2cyclictest0-21swapper/109:06:151
1012067992317,5cyclictest1093898-21kworker/u8:0+flush-8:011:31:141
101206999225,5cyclictest0-21swapper/210:59:072
1012069992219,2cyclictest1194111-21fschecks_count10:11:092
1012069992218,2cyclictest1337066-21unixbench_singl12:31:262
1012069992218,2cyclictest1337066-21unixbench_singl12:31:252
1012069992218,2cyclictest1053288-21unixbench_singl07:46:222
101206999221,6cyclictest1043370-21users07:36:242
1012069992214,3cyclictest1172847-21ntpq09:46:192
1012069992213,5cyclictest1292981-21apt-get11:51:042
1012069992212,8cyclictest282-21systemd-journal11:24:162
1012069992210,8cyclictest1311601-21ntpq12:06:182
1012069992210,8cyclictest1311601-21ntpq12:06:182
101206999220,7cyclictest39-21ksoftirqd/211:41:112
1012067992218,2cyclictest0-21swapper/110:06:051
1012067992217,3cyclictest1112409-21latency08:46:141
1012067992215,5cyclictest1163455-21unixbench_singl09:36:271
101206999217,11cyclictest1297799-21taskset11:55:092
101206999216,6cyclictest0-21swapper/211:46:122
101206999216,4cyclictest0-21swapper/209:35:572
101206999204,7cyclictest0-21swapper/211:27:492
101206999204,6cyclictest0-21swapper/208:34:302
1012069992017,2cyclictest1188951-21aten2_r5power_e10:06:072
1012069992012,4cyclictest1132625-21ntpq09:06:222
101206999196,11cyclictest1342308-21taskset12:39:072
101206999195,5cyclictest0-21swapper/208:35:122
101206999195,4cyclictest0-21swapper/210:26:252
101206999195,4cyclictest0-21swapper/207:30:142
101206999194,4cyclictest0-21swapper/210:46:082
13278432152,8sleep30-21swapper/312:26:063
10860802154,8sleep30-21swapper/308:21:143
10936542144,5sleep30-21swapper/308:30:013
13374332132,6sleep30-21swapper/312:35:013
12778842132,6sleep30-21swapper/311:35:013
12617682132,7sleep30-21swapper/311:16:193
12522472134,5sleep30-21swapper/311:06:243
12281432133,6sleep30-21swapper/310:45:013
11777972132,7sleep30-21swapper/309:51:203
11598452132,7sleep30-21swapper/309:36:153
11544492132,7sleep30-21swapper/309:31:133
10478712132,7sleep30-21swapper/307:41:213
10338242132,6sleep30-21swapper/307:30:023
13272092122,6sleep30-21swapper/312:21:253
13208572122,6sleep30-21swapper/312:16:153
13124512122,6sleep30-21swapper/312:10:013
13124512122,6sleep30-21swapper/312:10:003
13083172122,6sleep30-21swapper/312:06:123
13083172122,6sleep30-21swapper/312:06:123
13032482122,6sleep30-21swapper/312:01:103
12964812122,6sleep30-21swapper/311:51:153
12876832122,5sleep30-21swapper/311:45:003
12753852122,6sleep30-21swapper/311:31:163
12630462122,6sleep30-21swapper/311:21:093
12528372122,6sleep30-21swapper/311:10:013
12471492122,6sleep30-21swapper/311:01:233
12426552122,6sleep30-21swapper/310:56:293
12333832122,6sleep30-21swapper/310:51:023
12144692122,6sleep30-21swapper/310:31:133
12031172122,6sleep30-21swapper/310:17:003
11976872122,6sleep30-21swapper/310:11:223
11882972123,5sleep30-21swapper/310:01:463
11687852122,5sleep30-21swapper/309:45:013
11529542123,6sleep30-21swapper/309:26:243
11323052122,6sleep30-21swapper/309:06:213
11279382123,5sleep30-21swapper/309:01:563
11025672122,6sleep30-21swapper/308:36:153
10955422123,6sleep30-21swapper/308:31:133
10827152122,6sleep30-21swapper/308:16:213
10743472122,6sleep30-21swapper/308:11:113
10698602123,5sleep30-21swapper/308:06:123
10390212122,6sleep30-21swapper/307:36:033
10296752122,6sleep30-21swapper/307:26:123
10285502122,6sleep30-21swapper/307:21:273
10174852122,6sleep30-21swapper/307:15:053
10125352122,6sleep30-21swapper/307:11:073
13327862112,5sleep30-21swapper/312:31:083
13327862112,5sleep30-21swapper/312:31:083
12981422112,5sleep30-21swapper/311:56:083
12682372112,6sleep30-21swapper/311:26:063
12244142112,5sleep30-21swapper/310:41:123
12190682112,5sleep30-21swapper/310:36:133
12086532112,5sleep30-21swapper/310:26:073
11794832112,6sleep30-21swapper/309:56:133
11424252112,5sleep30-21swapper/309:16:163
11341992112,5sleep30-21swapper/309:11:113
11188862112,5sleep30-21swapper/308:55:123
11164792112,5sleep30-21swapper/308:51:163
11092892112,5sleep30-21swapper/308:46:103
11043342112,6sleep30-21swapper/308:41:133
10637752112,6sleep30-21swapper/308:00:403
10628682112,5sleep30-21swapper/307:56:223
10582872112,5sleep30-21swapper/307:52:043
10498952112,5sleep30-21swapper/307:46:113
11685062102,5sleep30-21swapper/309:41:533
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional