You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 21:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Fri Nov 21, 2025 12:39:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1281334995932,24cyclictest1374732-21ssh10:35:021
1281335995243,5cyclictest1461778-21tr12:40:012
1281333995143,5cyclictest1370035-21/usr/sbin/munin10:29:050
1281333995135,12cyclictest0-21swapper/009:49:230
1281335995041,6cyclictest474-21in:imuxsock09:30:012
1281335995029,18cyclictest0-21swapper/211:50:012
1281335995027,2cyclictest0-21swapper/209:40:012
1281333995043,4cyclictest1407797-21systemd-run11:25:010
1281335994924,22cyclictest1450417-21latency_hist12:25:022
1281335994921,6cyclictest1300823-21systemd-run08:15:022
128133499491,24cyclictest1353421-21ssh10:04:141
1281335994825,3cyclictest0-21swapper/211:30:012
1281335994825,3cyclictest0-21swapper/211:30:012
1281335994724,3cyclictest0-21swapper/211:44:182
1281334994739,5cyclictest1309836-21systemd-run08:45:021
1281335994624,3cyclictest0-21swapper/210:44:252
1281335994622,3cyclictest0-21swapper/209:34:242
1281334994637,5cyclictest1391458-21systemd-run11:00:011
1281334994622,20cyclictest1414510-21latency_hist11:35:021
1281334994621,22cyclictest421-21avahi-daemon08:59:181
1281334994620,23cyclictest1317308-21diskmemload09:59:141
1281334994615,3cyclictest0-21swapper/109:19:031
1281335994519,7cyclictest1387587-21switchtime10:54:232
1281334994538,4cyclictest266-21systemd-journal12:40:011
1281334994537,5cyclictest1450330-21grep12:25:011
1281334994526,16cyclictest1315409-21sshd09:04:201
1281334994520,21cyclictest0-21swapper/111:44:181
1281335994421,4cyclictest959-21mta-sts-daemon11:16:262
1281335994421,4cyclictest959-21mta-sts-daemon11:16:262
1281334994418,23cyclictest1297744-21unixbench_multi08:04:231
1281334994334,6cyclictest1431167-21/usr/sbin/munin11:59:211
1281334994320,20cyclictest1303772-21unixbench_singl08:24:271
1281334994319,20cyclictest1330242-21unixbench_multi09:29:231
128133499429,7cyclictest422-21dbus-daemon11:30:011
128133499429,7cyclictest422-21dbus-daemon11:30:011
128133499423,7cyclictest1407049-21ssh11:24:201
1281334994220,19cyclictest0-21swapper/110:49:131
1281334994218,19cyclictest0-21swapper/109:34:351
128133499421,14cyclictest1453053-21ntpq12:29:121
1281334994116,3cyclictest1351927-21kworker/u8:14-events_unbound10:24:011
1281334994116,2cyclictest0-21swapper/111:09:221
1281334994116,2cyclictest0-21swapper/111:09:221
1281334994115,3cyclictest0-21swapper/111:55:011
1281334994114,2cyclictest0-21swapper/107:14:141
1281334994113,3cyclictest0-21swapper/107:49:141
128133499411,13cyclictest1300211-21ntpq08:14:091
1281335994032,5cyclictest1431712-21latency_hist12:00:012
1281335994031,6cyclictest491530-21cups-browsed12:29:042
1281335994015,14cyclictest130829750latency_hist08:40:012
1281334994019,17cyclictest0-21swapper/110:39:181
1281334994019,17cyclictest0-21swapper/110:39:181
1281334994018,2cyclictest0-21swapper/111:50:011
1281334994017,20cyclictest0-21swapper/110:09:141
1281334994016,2cyclictest0-21swapper/110:14:111
1281334994015,3cyclictest0-21swapper/109:44:061
1281334994014,2cyclictest0-21swapper/110:29:491
1281334994013,6cyclictest1319189-21aten_r2power_cu09:13:581
1281334994013,3cyclictest0-21swapper/112:09:241
1281334994013,2cyclictest0-21swapper/111:03:531
1281334994013,2cyclictest0-21swapper/111:03:531
1281333994034,4cyclictest1417804-21munin-run11:40:010
1281333994033,4cyclictest957-21mta-sts-daemon09:09:170
1281335993933,3cyclictest0-21swapper/211:10:012
1281335993933,3cyclictest0-21swapper/211:10:012
1281335993929,7cyclictest698-21cron10:30:012
1281335993929,7cyclictest1446571-21cron12:20:012
1281334993918,18cyclictest1454477-21ssh12:30:331
1281334993918,17cyclictest1342524-21fschecks_count09:49:031
1281334993915,2cyclictest0-21swapper/112:04:191
1281334993914,3cyclictest1283042-21kworker/u8:13-xprtiod08:09:201
1281334993914,3cyclictest0-21swapper/109:51:351
1281334993914,3cyclictest0-21swapper/109:51:351
1281334993911,6cyclictest1317308-21diskmemload12:12:481
128133499391,14cyclictest1403512-21sh11:19:161
128133499391,14cyclictest1403512-21sh11:19:161
1281333993932,5cyclictest955-21mta-sts-daemon10:24:130
1281333993931,5cyclictest1-21systemd07:50:010
1281333993929,7cyclictest12450irq/32-eth010:04:120
1281335993830,5cyclictest1357332-21unixbench-2d10:09:252
1281335993830,5cyclictest1292355-21fschecks_count07:49:022
1281335993830,4cyclictest0-21swapper/210:04:132
1281334993830,4cyclictest955-21mta-sts-daemon12:17:541
1281334993815,20cyclictest0-21swapper/107:55:011
1281334993815,20cyclictest0-21swapper/107:55:011
1281334993813,4cyclictest0-21swapper/111:39:231
1281333993832,3cyclictest0-21swapper/012:15:010
1281333993830,5cyclictest422-21dbus-daemon10:40:020
1281333993830,5cyclictest422-21dbus-daemon10:40:020
1281333993821,7cyclictest1431760-21sh12:00:010
1281335993729,5cyclictest955-21mta-sts-daemon09:14:102
1281335993728,5cyclictest1394390-21vmstat11:04:362
1281335993728,5cyclictest1394390-21vmstat11:04:362
1281334993728,6cyclictest491-21gdbus07:35:011
1281334993714,3cyclictest0-21swapper/108:20:011
1281333993732,2cyclictest0-21swapper/010:59:170
1281333993730,4cyclictest491530-21cups-browsed12:21:200
1281335993631,2cyclictest0-21swapper/212:35:012
1281335993628,5cyclictest1442778-21cron12:15:002
1281335993627,6cyclictest1347035-21unixbench_multi09:54:232
1281335993627,6cyclictest1347035-21unixbench_multi09:54:232
1281335993623,7cyclictest684-21munin-node10:12:412
1281334993613,3cyclictest0-21swapper/110:19:531
128133499361,15cyclictest1336135-21ssh09:39:121
1281333993629,4cyclictest1337165-21latency_hist09:40:020
1281333993628,5cyclictest491530-21cups-browsed09:42:470
1281335993526,6cyclictest491530-21cups-browsed10:32:232
1281335993524,8cyclictest1304738-21ntp_kernel_pll_08:29:132
128133499353,9cyclictest437-21mta-sts-daemon10:53:281
128133499353,8cyclictest1380493-21ntpq10:44:161
1281334993526,6cyclictest1285847-21systemd-run07:25:021
1281334993515,2cyclictest0-21swapper/111:14:141
1281334993515,2cyclictest0-21swapper/111:14:141
1281333993529,4cyclictest1452976-21ntp_kernel_err12:29:090
1281333993529,4cyclictest1326901-21threads09:24:210
1281333993528,5cyclictest1424512-21systemd-run11:50:010
1281333993526,6cyclictest491530-21cups-browsed08:44:000
1281333993523,10cyclictest956-21mta-sts-daemon12:36:460
1281335993427,3cyclictest0-21swapper/211:24:132
1281335993426,5cyclictest1294829-21systemd-run07:55:012
1281335993426,5cyclictest1294829-21systemd-run07:55:012
1281335993426,5cyclictest1287297-21head07:29:202
1281335993425,6cyclictest1378107-21idleruntime-cro10:40:012
1281335993425,6cyclictest1378107-21idleruntime-cro10:40:012
128133499343,8cyclictest421-21avahi-daemon08:49:101
1281334993426,5cyclictest1317312-21munin-plugin-st09:10:011
1281334993426,5cyclictest1306802-21sh08:35:001
1281334993426,5cyclictest1287281-21head07:29:201
1281333993427,4cyclictest1411350-21latency_hist11:30:020
1281333993427,4cyclictest1411350-21latency_hist11:30:020
1281333993427,4cyclictest1378158-21kworker/u8:1-rpciod11:32:560
1281333993426,5cyclictest1427452-21switchtime11:54:220
1281333993421,10cyclictest0-21swapper/010:34:260
1281333993327,3cyclictest0-21swapper/011:09:170
1281333993327,3cyclictest0-21swapper/011:09:170
1281333993327,3cyclictest0-21swapper/009:29:160
1281333993323,7cyclictest1446300-21unixbench_singl12:19:250
1281335993227,2cyclictest0-21swapper/211:15:012
1281335993227,2cyclictest0-21swapper/211:15:012
1281335993226,3cyclictest0-21swapper/209:25:032
1281335993225,4cyclictest0-21swapper/211:00:012
1281335993224,5cyclictest1302226-21tail08:19:242
1281335993223,7cyclictest1350738-21munin-plugin-st10:00:022
1281335993223,3cyclictest0-21swapper/207:14:142
1281335993221,8cyclictest1416676-21latency11:39:072
128133599321,24cyclictest1438392-21ssh12:09:192
1281334993221,8cyclictest956-21mta-sts-daemon07:10:011
1281334993221,8cyclictest956-21mta-sts-daemon07:10:011
128133499321,14cyclictest1326066-21df09:24:021
1281333993226,4cyclictest1347009-21sed09:54:230
1281333993226,4cyclictest1347009-21sed09:54:230
1281333993226,3cyclictest0-21swapper/010:40:040
1281333993223,6cyclictest1360810-21unixbench_singl10:14:230
128133399321,28cyclictest1403440-21ssh11:19:130
128133399321,28cyclictest1403440-21ssh11:19:130
1281335993125,3cyclictest0-21swapper/209:49:122
1281335993124,5cyclictest1281332-21cyclictest08:00:012
1281335993123,5cyclictest1367439-21grep10:24:242
1281335993123,5cyclictest1297811-21cron08:05:002
1281334993123,5cyclictest1305311-21cron08:30:011
1281334993123,5cyclictest1290310-21sh07:40:011
128133399313,23cyclictest1420498-21ssh11:44:130
1281333993125,3cyclictest1421411-21kworker/u8:14-xprtiod12:35:010
1281333993124,4cyclictest422-21dbus-daemon10:20:010
1281335993024,3cyclictest0-21swapper/210:49:172
1281335993024,3cyclictest0-21swapper/210:20:012
1281335993024,3cyclictest0-21swapper/207:25:022
1281335993023,4cyclictest956-21mta-sts-daemon09:16:232
1281335993022,5cyclictest1290309-21idleruntime-cro07:40:012
1281335993020,6cyclictest1426442-21basename11:54:012
1281333993024,4cyclictest1281347-21kworker/u8:12+xprtiod07:15:000
1281333993022,6cyclictest1302302-21cron08:20:010
128133399302,25cyclictest1400078-21ssh11:14:090
128133399302,25cyclictest1400078-21ssh11:14:090
1281333993020,3cyclictest1320823-21cut09:15:010
1281335992923,3cyclictest0-21swapper/209:44:142
1281335992923,3cyclictest0-21swapper/209:10:022
1281335992923,3cyclictest0-21swapper/209:00:012
1281335992922,3cyclictest0-21swapper/212:04:112
1281335992921,5cyclictest591-21sshd07:43:092
1281335992919,7cyclictest1306200-21ntp_kernel_pll_08:34:132
1281334992921,5cyclictest1284319-21munin-plugin-st07:20:011
1281333992923,4cyclictest1281332-21cyclictest10:07:560
1281333992916,10cyclictest0-21swapper/008:39:230
128133399291,23cyclictest491530-21cups-browsed12:05:460
1281335992821,5cyclictest1412787-21/usr/sbin/munin11:34:022
1281335992820,5cyclictest1312822-21systemd-run08:55:012
1281335992820,5cyclictest1311320-21systemd-run08:50:012
1281333992823,3cyclictest151ktimers/008:24:190
1281333992822,4cyclictest1393765-21ntp_kernel_pll_11:04:240
1281333992822,4cyclictest1393765-21ntp_kernel_pll_11:04:240
1281333992822,3cyclictest0-21swapper/012:04:030
128133399282,21cyclictest1383883-21ssh10:49:140
1281333992821,5cyclictest0-21swapper/007:40:010
1281333992821,4cyclictest491530-21cups-browsed08:57:280
1281333992821,4cyclictest1350135-21kworker/u8:9+xprtiod10:55:010
1281333992820,6cyclictest0-21swapper/009:19:020
128133399281,23cyclictest1332432-21/usr/sbin/munin09:34:200
1281333992722,3cyclictest957-21mta-sts-daemon09:57:510
1281333992721,3cyclictest0-21swapper/007:10:010
1281333992721,3cyclictest0-21swapper/007:10:010
1281333992719,6cyclictest1297042-21latency08:04:070
1281335992620,3cyclictest0-21swapper/208:09:202
1281335992620,3cyclictest0-21swapper/207:34:082
1281335992620,3cyclictest0-21swapper/207:19:132
1281335992619,4cyclictest959-21mta-sts-daemon09:00:202
1281334992620,3cyclictest0-21swapper/108:40:011
1281333992620,4cyclictest1296292-21cron08:00:000
1281333992619,4cyclictest1305118-21timerwakeupswit08:29:230
1281334992517,5cyclictest1312871-21latency_hist08:55:021
1281333992518,5cyclictest1284887-21fschecks_time07:24:050
1281335992418,3cyclictest0-21swapper/207:10:012
1281335992418,3cyclictest0-21swapper/207:10:012
128133399242,16cyclictest1288283-21ntpq07:34:120
1281333992420,2cyclictest0-21swapper/007:29:200
1281333992417,4cyclictest491530-21cups-browsed07:42:150
1281333992416,5cyclictest474-21in:imuxsock08:10:000
1281333992413,7cyclictest1300838-21latency_hist08:15:020
128133399241,11cyclictest1314674-21aten_r2power_fr09:04:020
1281335992316,5cyclictest1308874-21gltestperf08:44:052
1281333992317,4cyclictest266-21systemd-journal07:55:010
1281333992317,4cyclictest266-21systemd-journal07:55:010
1281333992316,3cyclictest151ktimers/008:34:160
1281334992216,3cyclictest957-21mta-sts-daemon07:44:181
1281334992216,3cyclictest0-21swapper/107:59:091
128133399221,9cyclictest1310844-21proc_pri08:49:180
1281333992214,6cyclictest1283607-21missed_timers07:19:080
128133699212,9cyclictest0-21swapper/309:34:323
1281335992115,3cyclictest0-21swapper/208:23:252
128133399211,15cyclictest1312344-21ps08:54:170
128133699192,9cyclictest0-21swapper/311:59:103
128133699192,9cyclictest0-21swapper/310:04:153
128133699192,12cyclictest0-21swapper/309:39:173
128133699192,10cyclictest0-21swapper/309:49:143
128133699191,11cyclictest0-21swapper/312:39:103
128133699191,11cyclictest0-21swapper/310:33:563
128133699182,8cyclictest0-21swapper/311:44:113
128133699181,9cyclictest0-21swapper/312:04:023
128133699173,8cyclictest0-21swapper/309:54:093
128133699173,8cyclictest0-21swapper/309:54:093
128133699173,7cyclictest0-21swapper/312:14:153
128133699172,9cyclictest0-21swapper/311:34:103
128133699172,9cyclictest0-21swapper/310:14:103
128133699172,9cyclictest0-21swapper/309:14:153
128133699171,9cyclictest0-21swapper/311:19:213
128133699171,9cyclictest0-21swapper/311:19:213
128133699171,9cyclictest0-21swapper/310:39:123
128133699171,9cyclictest0-21swapper/310:39:123
128133699171,9cyclictest0-21swapper/310:09:193
128133699171,8cyclictest0-21swapper/312:19:143
128133699171,8cyclictest0-21swapper/307:54:303
128133699171,8cyclictest0-21swapper/307:54:303
128133699171,10cyclictest0-21swapper/308:24:123
128133699164,7cyclictest0-21swapper/312:24:123
128133699164,6cyclictest0-21swapper/311:24:153
128133699162,9cyclictest0-21swapper/311:39:163
128133699162,9cyclictest0-21swapper/311:29:073
128133699162,9cyclictest0-21swapper/311:29:073
128133699162,9cyclictest0-21swapper/311:09:183
128133699162,9cyclictest0-21swapper/311:09:183
128133699162,8cyclictest0-21swapper/310:19:133
128133699162,8cyclictest0-21swapper/309:24:113
128133699161,8cyclictest0-21swapper/308:49:103
128133699154,4cyclictest0-21swapper/308:34:173
128133699153,7cyclictest0-21swapper/311:49:143
128133699153,7cyclictest0-21swapper/310:29:163
128133699153,6cyclictest0-21swapper/311:04:033
128133699153,6cyclictest0-21swapper/311:04:033
128133699152,8cyclictest0-21swapper/310:24:003
128133699152,8cyclictest0-21swapper/309:59:223
128133699151,9cyclictest0-21swapper/310:49:113
128133699151,8cyclictest0-21swapper/309:29:093
128133699151,8cyclictest0-21swapper/309:19:103
128133699151,6cyclictest0-21swapper/309:09:163
128133699151,6cyclictest0-21swapper/307:44:153
128133699151,10cyclictest0-21swapper/311:14:113
128133699151,10cyclictest0-21swapper/311:14:113
128133699143,6cyclictest0-21swapper/312:34:123
128133699142,8cyclictest0-21swapper/311:54:103
128133699142,7cyclictest0-21swapper/310:59:133
128133699142,7cyclictest0-21swapper/309:44:133
128133699142,7cyclictest0-21swapper/308:40:023
128133699142,6cyclictest0-21swapper/308:04:113
128133699141,8cyclictest0-21swapper/312:09:023
128133699141,8cyclictest0-21swapper/310:54:023
128133699141,8cyclictest0-21swapper/310:44:033
128133699141,8cyclictest0-21swapper/307:59:113
128133699132,7cyclictest0-21swapper/309:00:023
128133699132,6cyclictest0-21swapper/308:54:113
128133699132,6cyclictest0-21swapper/308:15:013
128133699132,6cyclictest0-21swapper/307:24:153
128133699131,8cyclictest0-21swapper/312:29:053
128133699131,8cyclictest0-21swapper/309:04:163
128133699131,6cyclictest0-21swapper/308:44:133
128133699131,6cyclictest0-21swapper/308:09:103
128133699131,6cyclictest0-21swapper/307:14:123
128133699125,5cyclictest0-21swapper/307:39:103
128133699122,6cyclictest0-21swapper/307:19:133
128133699111,6cyclictest0-21swapper/308:19:133
128133699111,6cyclictest0-21swapper/307:49:023
128133699110,6cyclictest0-21swapper/308:29:113
128133699102,5cyclictest0-21swapper/307:34:113
128133699101,6cyclictest0-21swapper/307:28:053
12813369992,5cyclictest0-21swapper/307:10:013
12813369992,5cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional