You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-21 - 07:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Sat Mar 21, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
133983022230,8sleep20-21swapper/219:05:252
138484621270,4sleep33056-21PK-Backend21:14:553
13623512960,3sleep33056-21PK-Backend20:10:113
139099920,51rtkit-daemon0-21swapper/119:05:271
134031726645,14sleep00-21swapper/019:08:490
134009326444,14sleep30-21swapper/319:06:003
14257792520,4sleep10-21swapper/123:10:091
13979522440,4sleep10-21swapper/121:50:171
134054499390,27cyclictest1368289-21/usr/sbin/munin20:25:181
134054499381,23cyclictest1447966-21taskset00:10:341
134054499380,23cyclictest1379196-21seq20:57:271
134054499371,22cyclictest1446436-21seq00:08:101
134054499370,22cyclictest1364052-21taskset20:15:091
14053992360,4sleep01405393-21strings22:10:170
134054499362,25cyclictest1352010-21cat19:40:131
134054499361,24cyclictest1095-21runrttasks00:30:131
134054499361,16cyclictest0-21swapper/122:20:001
134054499360,27cyclictest1394787-21/usr/sbin/munin21:40:171
134054499360,23cyclictest1435703-21chrt23:37:281
134054499360,21cyclictest1381756-21ls21:05:131
134054499353,29cyclictest1433433-21ntpq23:30:181
134054499353,21cyclictest0-21swapper/123:46:541
134054499351,19cyclictest1344330-21runrttasks19:18:481
134054499351,19cyclictest1344330-21runrttasks19:18:471
134054499341,20cyclictest1384634-21runrttasks21:12:111
134054499340,25cyclictest1377400-21sshd20:51:381
134054499340,21cyclictest1341153-21basename19:10:131
134054499340,20cyclictest1397206-21taskset21:49:251
134054499340,19cyclictest1425239-21seq23:06:441
134054499340,19cyclictest1425239-21seq23:06:431
134054499340,19cyclictest1389956-21runrttasks21:25:221
134054499340,19cyclictest1372238-21taskset20:37:051
134054499331,30cyclictest1362081-21turbostat.cron20:10:001
134054499331,20cyclictest1423422-21taskset23:00:361
134054499331,20cyclictest0-21swapper/121:20:501
134054499331,19cyclictest0-21swapper/123:59:331
134054499331,19cyclictest0-21swapper/123:59:331
134054499330,20cyclictest1391871-21seq21:32:401
134054499330,18cyclictest0-21swapper/123:25:031
134054499321,17cyclictest1409323-21runrttasks22:20:551
134054499320,23cyclictest0-21swapper/122:34:051
134054499320,20cyclictest1386532-21taskset21:16:001
134054499320,19cyclictest1421475-21seq22:56:121
134054499320,19cyclictest1405589-21python322:10:201
134054499320,19cyclictest1367195-21seq20:24:071
134054499320,19cyclictest1354562-21taskset19:46:021
134054499320,19cyclictest1-21systemd00:05:011
134054499320,18cyclictest1448561-21seq00:15:031
134054499320,18cyclictest1375631-21seq20:46:131
134054499320,18cyclictest1095-21runrttasks00:26:101
134054499311,17cyclictest0-21swapper/121:55:541
134054499310,19cyclictest0-21swapper/122:02:431
134054499310,18cyclictest1451065-21ntpq00:20:171
134054499310,18cyclictest0-21swapper/123:16:091
134054499309,19cyclictest1437255-21awk23:40:221
134054499309,19cyclictest1417629-21awk22:45:181
134054499307,21cyclictest436-20systemd-journal22:51:431
134054499301,27cyclictest1456326-21grep00:35:181
134054499301,17cyclictest1095-21runrttasks19:58:211
134054499300,16cyclictest1359625-21ntp_states20:00:191
134054499290,16cyclictest1346751-21expr19:25:121
134054499282,21cyclictest1348226-21apt19:30:101
134054499280,25cyclictest436-20systemd-journal20:31:341
134054499280,16cyclictest0-21swapper/122:35:061
134054499277,18cyclictest1355607-21grep19:50:121
134054499276,19cyclictest1373016-21expr20:40:141
14014112260,9chrt34-21ksoftirqd/222:00:142
134054499261,23cyclictest1-21systemd23:25:001
1340541992613,11cyclictest1455295-21unin-run00:35:000
139099250,3rtkit-daemon1389-21rtkit-daemon23:32:093
134054999259,14cyclictest1443151-40updatedb.mlocat00:00:012
134054999259,14cyclictest1443151-40updatedb.mlocat00:00:002
134054499251,21cyclictest1404470-21idleruntime-cro22:10:011
134054499251,21cyclictest0-21swapper/120:10:111
139099240,3rtkit-daemon1389-21rtkit-daemon00:20:143
134054499248,14cyclictest1350168-21cstates19:35:121
134054499247,15cyclictest1392742-21idleruntime21:35:171
134054499241,21cyclictest1379883-21dpkg21:00:011
134054499241,14cyclictest1095-21runrttasks23:51:451
1340541992410,6cyclictest13-21ksoftirqd/000:00:000
1340541992410,6cyclictest13-21ksoftirqd/000:00:000
14434032230,3chrt3056-21PK-Backend00:00:113
134054499237,14cyclictest1344912-21cstates19:20:111
134054499230,21cyclictest1415486-21/usr/sbin/munin22:40:111
134054499230,21cyclictest1411408-21apt22:30:001
1340541992318,3cyclictest1375058-21memory20:45:170
139099220,16rtkit-daemon3056-21PK-Backend19:34:050
13656272220,7sleep234-21ksoftirqd/220:20:002
1340551992219,2cyclictest3056-21PK-Backend19:52:353
134054999222,2cyclictest0-21swapper/200:13:182
1340549992220,1cyclictest0-21swapper/222:44:022
1340549992220,1cyclictest0-21swapper/221:46:302
1340549992220,1cyclictest0-21swapper/220:54:392
1340549992220,1cyclictest0-21swapper/220:39:112
1340549992220,1cyclictest0-21swapper/219:12:262
1340549992220,1cyclictest0-21swapper/200:03:452
1340549992216,4cyclictest1346199-21systemd-run19:25:002
134054999220,21cyclictest0-21swapper/221:08:182
134054199220,2cyclictest1416622-21munin-run22:45:010
139099210,13rtkit-daemon0-21swapper/021:21:300
134055199214,15cyclictest3056-21PK-Backend22:15:453
1340551992113,6cyclictest1402733-21basename22:05:003
134055199211,18cyclictest1365551-21dump-pmu-power20:20:003
1340549992119,1cyclictest0-21swapper/223:45:272
1340549992119,1cyclictest0-21swapper/223:29:462
1340549992119,1cyclictest0-21swapper/223:02:142
1340549992119,1cyclictest0-21swapper/222:57:342
1340549992119,1cyclictest0-21swapper/222:51:192
1340549992119,1cyclictest0-21swapper/222:47:412
1340549992119,1cyclictest0-21swapper/222:08:062
1340549992119,1cyclictest0-21swapper/221:59:412
1340549992119,1cyclictest0-21swapper/221:30:092
1340549992119,1cyclictest0-21swapper/221:23:452
1340549992119,1cyclictest0-21swapper/219:51:522
1340549992119,1cyclictest0-21swapper/219:46:322
1340549992119,1cyclictest0-21swapper/219:32:202
1340549992119,1cyclictest0-21swapper/200:24:212
1340549992115,4cyclictest1360348-21idleruntime-cro20:05:002
134054999210,20cyclictest0-21swapper/223:15:052
134054999210,20cyclictest0-21swapper/219:19:012
134054999210,20cyclictest0-21swapper/219:19:012
134054999210,20cyclictest0-21swapper/200:25:222
134054999210,19cyclictest3056-21PK-Backend22:30:312
1340541992119,1cyclictest0-21swapper/022:06:480
134054199210,20cyclictest0-21swapper/022:16:150
134054199210,20cyclictest0-21swapper/021:01:520
134054199210,20cyclictest0-21swapper/019:12:010
14393982200,3sleep31439389-21awk23:50:013
139099200,13rtkit-daemon0-21swapper/020:10:010
139099200,13rtkit-daemon0-21swapper/019:15:090
139099200,13rtkit-daemon0-21swapper/019:15:090
1340551992013,5cyclictest883-21dbus-daemon21:25:013
1340551992012,6cyclictest1354890-21perf19:50:003
134055199200,1cyclictest3056-21PK-Backend00:13:053
134055199200,19cyclictest3056-21PK-Backend23:13:433
1340549992018,1cyclictest0-21swapper/223:53:112
1340549992018,1cyclictest0-21swapper/223:35:402
1340549992018,1cyclictest0-21swapper/223:24:522
1340549992018,1cyclictest0-21swapper/223:14:042
1340549992018,1cyclictest0-21swapper/223:08:092
1340549992018,1cyclictest0-21swapper/223:08:092
1340549992018,1cyclictest0-21swapper/222:35:582
1340549992018,1cyclictest0-21swapper/222:18:482
1340549992018,1cyclictest0-21swapper/221:53:062
1340549992018,1cyclictest0-21swapper/221:13:382
1340549992018,1cyclictest0-21swapper/220:21:262
1340549992018,1cyclictest0-21swapper/219:38:492
1340549992018,1cyclictest0-21swapper/200:37:112
1340549992018,1cyclictest0-21swapper/200:17:242
1340549992018,1cyclictest0-21swapper/200:07:372
134054999200,1cyclictest3056-21PK-Backend21:25:212
134054999200,19cyclictest0-21swapper/222:26:532
134054999200,19cyclictest0-21swapper/220:57:022
134054999200,19cyclictest0-21swapper/220:43:442
134054999200,19cyclictest0-21swapper/220:11:122
134054999200,19cyclictest0-21swapper/200:31:172
1340541992018,1cyclictest0-21swapper/019:23:380
1340541992018,1cyclictest0-21swapper/000:23:290
1340541992012,6cyclictest1423180-21python323:00:200
134054199200,1cyclictest3056-21PK-Backend23:36:380
134054199200,1cyclictest3056-21PK-Backend20:09:130
134054199200,1cyclictest0-21swapper/020:39:550
134054199200,19cyclictest3056-21PK-Backend19:43:110
134054199200,19cyclictest3056-21PK-Backend19:35:540
134054199200,19cyclictest0-21swapper/023:48:480
134054199200,19cyclictest0-21swapper/023:10:150
134054199200,19cyclictest0-21swapper/021:52:150
134054199200,19cyclictest0-21swapper/021:45:060
134054199200,19cyclictest0-21swapper/021:14:290
134054199200,19cyclictest0-21swapper/020:56:450
134054199200,19cyclictest0-21swapper/020:23:130
134054199200,19cyclictest0-21swapper/019:55:070
134054199200,19cyclictest0-21swapper/019:53:330
134054199200,19cyclictest0-21swapper/000:16:470
134054199200,19cyclictest0-21swapper/000:12:560
134055199198,9cyclictest42-21ksoftirqd/321:00:173
134055199197,3cyclictest141rcu_preempt21:35:003
134055199197,11cyclictest42-21ksoftirqd/322:31:533
134055199193,7cyclictest1372474-21sh20:40:003
1340551991917,1cyclictest3056-21PK-Backend20:45:063
1340551991917,1cyclictest0-21swapper/319:44:093
1340551991917,1cyclictest0-21swapper/319:35:233
1340551991914,4cyclictest42-21ksoftirqd/323:35:123
1340551991914,3cyclictest1445500-21df_inode00:05:133
1340551991912,5cyclictest883-21dbus-daemon19:25:003
1340551991911,6cyclictest436-20systemd-journal23:04:543
1340551991910,7cyclictest957-21gdbus19:27:443
134055199190,1cyclictest3056-21PK-Backend21:06:013
134055199190,1cyclictest0-21swapper/320:07:343
134055199190,18cyclictest3056-21PK-Backend00:36:373
134055199190,14cyclictest0-21swapper/319:11:103
1340549991917,1cyclictest0-21swapper/221:15:142
1340549991913,4cyclictest978-21snmpd19:25:282
134054999190,1cyclictest0-21swapper/223:43:242
134054999190,1cyclictest0-21swapper/223:30:472
134054999190,1cyclictest0-21swapper/222:23:142
134054999190,1cyclictest0-21swapper/222:12:322
134054999190,1cyclictest0-21swapper/221:40:422
134054999190,1cyclictest0-21swapper/220:28:432
134054999190,1cyclictest0-21swapper/219:41:382
1340541991917,1cyclictest0-21swapper/023:42:200
1340541991917,1cyclictest0-21swapper/023:16:590
1340541991917,1cyclictest0-21swapper/022:57:170
1340541991917,1cyclictest0-21swapper/022:37:100
1340541991917,1cyclictest0-21swapper/022:22:160
1340541991917,1cyclictest0-21swapper/020:41:310
134054199191,3cyclictest1411379-21cron22:30:000
1340541991912,5cyclictest1419100-21cut22:50:170
1340541991912,5cyclictest1385224-21date21:15:010
134054199191,16cyclictest1441102-21cron23:55:000
1340541991911,6cyclictest1401241-21cat22:00:110
1340541991911,6cyclictest1400743-21sh21:59:590
134054199190,1cyclictest3056-21PK-Backend23:30:090
134054199190,1cyclictest0-21swapper/023:09:140
134054199190,1cyclictest0-21swapper/023:09:140
134054199190,1cyclictest0-21swapper/022:47:390
134054199190,1cyclictest0-21swapper/019:46:020
134054199190,18cyclictest3056-21PK-Backend23:27:190
134054199190,18cyclictest3056-21PK-Backend20:03:390
134054199190,18cyclictest0-21swapper/021:30:450
134054199190,18cyclictest0-21swapper/021:05:370
134054199190,18cyclictest0-21swapper/020:15:020
134054199190,18cyclictest0-21swapper/000:08:150
134055199189,7cyclictest1400742-21cron21:59:593
1340551991815,2cyclictest3056-21PK-Backend23:09:453
1340551991815,2cyclictest3056-21PK-Backend23:09:453
1340551991815,2cyclictest3056-21PK-Backend22:06:173
1340551991813,3cyclictest1453529-21munin-plugin-st00:30:003
1340551991811,5cyclictest1374173-21sshd20:44:423
134055199180,6cyclictest3056-21PK-Backend23:45:003
134055199180,6cyclictest1358639-21chrt20:00:003
1340549991816,1cyclictest0-21swapper/220:46:552
1340549991816,1cyclictest0-21swapper/220:32:292
1340549991813,3cyclictest1362109-21unin-run20:10:002
1340541991816,1cyclictest3056-21PK-Backend23:22:390
1340541991816,1cyclictest0-21swapper/021:41:540
1340541991816,1cyclictest0-21swapper/021:36:200
1340541991816,1cyclictest0-21swapper/020:29:410
1340541991816,1cyclictest0-21swapper/000:27:210
1340541991815,2cyclictest3056-21PK-Backend00:04:370
1340541991811,5cyclictest436-20systemd-journal21:29:550
134055199172,13cyclictest1397512-21/usr/sbin/munin21:50:183
1340551991715,1cyclictest3056-21PK-Backend22:24:033
1340551991715,1cyclictest3056-21PK-Backend20:58:513
1340551991715,1cyclictest3056-21PK-Backend20:20:333
1340551991714,2cyclictest3056-21PK-Backend22:45:113
1340551991714,2cyclictest3056-21PK-Backend21:41:033
1340551991714,2cyclictest3056-21PK-Backend21:17:513
1340551991711,4cyclictest1388868-21apt21:25:103
1340541991715,1cyclictest0-21swapper/020:30:290
1340541991710,5cyclictest1412712-21sed22:30:200
134055199169,6cyclictest1428305-21grep23:15:193
134055199168,6cyclictest436-20systemd-journal23:30:003
134055199165,2cyclictest141rcu_preempt20:30:003
1340551991614,1cyclictest3056-21PK-Backend20:53:383
1340551991612,2cyclictest42-21ksoftirqd/323:58:213
1340551991612,2cyclictest42-21ksoftirqd/323:58:213
1340551991610,4cyclictest42-21ksoftirqd/323:50:133
134055199160,3cyclictest1370355-21users20:30:213
1340549991614,1cyclictest0-21swapper/219:55:052
1340549991611,3cyclictest1095-21runrttasks21:35:122
134054199169,6cyclictest1456082-21sed00:35:150
134054199169,5cyclictest1376803-21ls20:50:170
13939722150,3chrt3056-21PK-Backend21:40:013
134055199159,5cyclictest42-21ksoftirqd/322:35:113
134055199157,7cyclictest1342462-21switchtime19:15:023
134055199157,7cyclictest1342462-21switchtime19:15:023
1340551991513,1cyclictest3056-21PK-Backend19:32:123
1340551991511,2cyclictest42-21ksoftirqd/322:56:093
1340551991511,2cyclictest42-21ksoftirqd/322:40:183
1340551991511,2cyclictest42-21ksoftirqd/322:27:093
1340551991511,2cyclictest42-21ksoftirqd/300:30:093
134055199151,12cyclictest1404701-21/usr/sbin/munin22:10:213
1340551991510,3cyclictest42-21ksoftirqd/300:15:133
134055199150,5cyclictest3056-21PK-Backend22:50:153
134055199150,2cyclictest0-21swapper/320:01:503
134055199150,13cyclictest13972352chrt21:49:473
1340549991510,4cyclictest1380526-21grep21:00:182
1340541991513,1cyclictest3056-21PK-Backend19:26:150
134055199147,5cyclictest1009-21lldpd23:24:103
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional