You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-12 - 15:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Sun Apr 12, 2026 12:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
154361421120,4sleep33056-21PK-Backend10:20:173
14742662950,5sleep30-21swapper/307:05:243
139099840,74rtkit-daemon0-21swapper/107:09:411
15164882800,4sleep20-21swapper/209:05:142
147479826445,13sleep20-21swapper/207:09:272
147475026344,14sleep00-21swapper/007:08:520
15254152440,5sleep10-21swapper/109:30:121
147497699388,8cyclictest883-21dbus-daemon08:00:011
139099350,4rtkit-daemon1788643-21eadedCompositor10:43:390
147497699318,9cyclictest1519830-21taskset09:15:031
147497699316,13cyclictest1535372-21seq09:59:101
147497699304,10cyclictest1582356-21seq12:12:461
147497699297,12cyclictest1540531-21taskset10:13:371
1474972992911,3cyclictest13-21ksoftirqd/009:45:170
147497699286,10cyclictest0-21swapper/110:15:181
147497699285,13cyclictest0-21swapper/110:28:011
147497699285,10cyclictest1551196-21seq10:43:281
147497699285,10cyclictest1518965-21seq09:10:251
147497699284,9cyclictest1585544-21cat12:20:211
147497699276,8cyclictest1566948-21munin-run11:30:001
147497699276,8cyclictest1503076-21users08:25:221
147497699275,8cyclictest1557788-21sed11:00:231
147497699275,10cyclictest1587972-21runrttasks12:29:311
147497699274,8cyclictest1524764-21seq09:26:261
147497699274,8cyclictest1504976-21runrttasks08:32:211
147497699264,10cyclictest1501293-21taskset08:22:561
147497699263,13cyclictest0-21swapper/109:00:181
1474976992613,8cyclictest1487880-21apt07:45:131
1474976992612,12cyclictest1523443-21cat09:25:011
147497699258,15cyclictest1536709-21ls10:00:211
147497699256,9cyclictest0-21swapper/110:09:481
147497699256,9cyclictest0-21swapper/108:06:181
147497699255,8cyclictest1591408-21runrttasks12:39:041
147497699254,9cyclictest1485463-21taskset07:35:381
147497699254,8cyclictest1514122-21seq08:59:591
1474976992516,6cyclictest1-21systemd07:29:551
1474976992510,10cyclictest1508660-21apt-get08:45:021
1474972992510,3cyclictest13-21ksoftirqd/010:10:010
139099250,3rtkit-daemon1389-21rtkit-daemon07:23:340
147497699246,10cyclictest0-21swapper/111:42:591
147497699244,9cyclictest1095-21runrttasks12:32:401
147497699244,9cyclictest1095-21runrttasks12:32:391
147497699244,7cyclictest1569811-21sensors_temp11:35:211
147497699244,12cyclictest0-21swapper/110:48:491
1474976992415,7cyclictest436-20systemd-journal11:00:001
1474976992415,7cyclictest1561740-21perf11:15:001
1474976992415,5cyclictest1517609-21munin-plugin-st09:10:011
1474976992414,6cyclictest1574149-21apt11:50:121
1474972992422,1cyclictest0-21swapper/008:27:540
15613202230,4sleep3411rcuc/311:10:233
15115352230,3sleep00-21swapper/008:50:160
147497699236,7cyclictest0-21swapper/109:50:201
147497699235,3cyclictest1563272-21seq11:17:361
147497699234,6cyclictest1564323-21/usr/sbin/munin11:20:171
1474976992314,6cyclictest978-21snmpd07:15:021
1474976992312,8cyclictest1476046-21python307:10:211
147497299239,3cyclictest13-21ksoftirqd/007:18:410
1474972992321,1cyclictest0-21swapper/009:38:120
1474972992321,1cyclictest0-21swapper/008:17:280
139099230,3rtkit-daemon1389-21rtkit-daemon09:44:541
139099230,3rtkit-daemon1389-21rtkit-daemon09:44:541
15200352220,2sleep20-21swapper/209:15:132
147498299227,13cyclictest1591056-21sed12:35:243
1474982992220,1cyclictest3056-21PK-Backend11:53:583
1474982992220,1cyclictest3056-21PK-Backend08:54:393
1474982992220,1cyclictest3056-21PK-Backend08:30:323
1474982992219,2cyclictest3056-21PK-Backend07:52:203
1474982992214,6cyclictest1556528-21cron10:59:593
147498299220,21cyclictest3056-21PK-Backend09:24:463
147498299220,20cyclictest3056-21PK-Backend12:25:553
1474979992220,1cyclictest0-21swapper/211:32:032
1474979992220,1cyclictest0-21swapper/210:08:422
1474979992220,1cyclictest0-21swapper/207:58:172
1474979992220,1cyclictest0-21swapper/207:31:132
1474979992220,1cyclictest0-21swapper/207:26:402
1474979992217,3cyclictest1551970-21df_inode10:45:132
1474979992214,6cyclictest1546131-21/usr/sbin/munin10:30:012
147497999220,21cyclictest0-21swapper/207:46:532
147497699228,12cyclictest1558738-21cstates11:05:101
147497699223,7cyclictest0-21swapper/107:30:181
1474976992218,3cyclictest0-21swapper/108:17:091
1474976992213,7cyclictest1579096-21awk12:05:011
1474976992213,7cyclictest1553498-21cpuspeed_turbos10:50:131
1474976992213,7cyclictest1480599-21cron07:25:001
1474976992213,5cyclictest1507206-21apt08:40:121
1474976992212,6cyclictest1496542-21apt08:10:121
1474972992220,1cyclictest0-21swapper/007:56:530
1474972992220,1cyclictest0-21swapper/007:48:420
1474972992220,1cyclictest0-21swapper/007:42:190
1474972992217,4cyclictest13-21ksoftirqd/008:45:010
147497299220,21cyclictest0-21swapper/008:34:170
1474982992119,1cyclictest3056-21PK-Backend11:49:053
1474982992119,1cyclictest3056-21PK-Backend11:19:593
1474982992119,1cyclictest3056-21PK-Backend10:27:263
1474982992119,1cyclictest3056-21PK-Backend07:59:303
1474982992119,1cyclictest3056-21PK-Backend07:18:413
1474982992118,2cyclictest3056-21PK-Backend11:44:123
1474982992118,2cyclictest3056-21PK-Backend10:34:303
1474982992118,2cyclictest3056-21PK-Backend08:56:143
1474982992118,1cyclictest3056-21PK-Backend10:15:103
147498299210,20cyclictest3056-21PK-Backend12:32:393
147498299210,20cyclictest3056-21PK-Backend12:32:393
147498299210,20cyclictest3056-21PK-Backend07:45:103
147498299210,19cyclictest3056-21PK-Backend11:23:363
147498299210,19cyclictest3056-21PK-Backend08:40:133
147498299210,19cyclictest3056-21PK-Backend07:27:333
1474979992119,1cyclictest0-21swapper/212:23:412
1474979992119,1cyclictest0-21swapper/212:18:482
1474979992119,1cyclictest0-21swapper/212:13:552
1474979992119,1cyclictest0-21swapper/211:35:072
1474979992119,1cyclictest0-21swapper/211:08:172
1474979992119,1cyclictest0-21swapper/210:35:322
1474979992119,1cyclictest0-21swapper/210:10:322
1474979992119,1cyclictest0-21swapper/209:50:042
1474979992119,1cyclictest0-21swapper/209:43:282
1474979992119,1cyclictest0-21swapper/209:43:282
1474979992119,1cyclictest0-21swapper/209:10:502
1474979992119,1cyclictest0-21swapper/207:42:082
1474979992119,1cyclictest0-21swapper/207:14:562
147497999210,20cyclictest0-21swapper/211:53:062
147497999210,20cyclictest0-21swapper/211:21:292
147497999210,20cyclictest0-21swapper/210:18:232
147497999210,20cyclictest0-21swapper/208:43:242
147497999210,20cyclictest0-21swapper/207:20:452
147497699218,11cyclictest1505679-21cstates08:35:131
1474976992119,1cyclictest0-21swapper/112:08:271
147497699211,8cyclictest0-21swapper/107:40:161
1474976992114,5cyclictest1573281-21python311:45:201
1474976992113,5cyclictest1493937-21/usr/sbin/munin08:00:211
1474976992112,7cyclictest1548274-21cut10:35:131
147497699210,20cyclictest0-21swapper/107:51:341
147497299212,17cyclictest1521320-21cron09:20:010
1474972992119,1cyclictest0-21swapper/011:47:010
1474972992119,1cyclictest0-21swapper/011:32:090
1474972992119,1cyclictest0-21swapper/010:57:140
1474972992119,1cyclictest0-21swapper/010:24:080
1474972992119,1cyclictest0-21swapper/009:12:100
1474972992119,1cyclictest0-21swapper/009:02:500
1474972992119,1cyclictest0-21swapper/008:59:380
1474972992119,1cyclictest0-21swapper/007:32:590
1474972992119,1cyclictest0-21swapper/007:25:300
1474972992119,1cyclictest0-21swapper/007:11:100
1474972992117,3cyclictest13-21ksoftirqd/008:49:570
1474972992112,7cyclictest436-20systemd-journal10:34:590
147497299210,20cyclictest0-21swapper/011:23:220
147497299210,20cyclictest0-21swapper/010:17:520
147497299210,20cyclictest0-21swapper/008:38:030
147497299210,20cyclictest0-21swapper/008:14:510
1474982992018,1cyclictest3056-21PK-Backend10:41:333
1474982992018,1cyclictest3056-21PK-Backend09:28:453
1474982992018,1cyclictest3056-21PK-Backend09:08:453
1474982992018,1cyclictest3056-21PK-Backend08:47:433
1474982992018,1cyclictest3056-21PK-Backend08:39:253
1474982992018,1cyclictest3056-21PK-Backend08:21:543
1474982992017,2cyclictest3056-21PK-Backend12:03:463
1474982992017,2cyclictest3056-21PK-Backend11:39:193
1474982992017,2cyclictest3056-21PK-Backend11:26:273
1474982992017,2cyclictest3056-21PK-Backend10:49:243
147498299200,19cyclictest3056-21PK-Backend10:13:213
147498299200,19cyclictest3056-21PK-Backend10:08:073
147498299200,19cyclictest3056-21PK-Backend08:25:193
147498299200,19cyclictest3056-21PK-Backend07:42:463
147498299200,18cyclictest3056-21PK-Backend07:36:383
1474979992018,1cyclictest0-21swapper/212:38:222
1474979992018,1cyclictest0-21swapper/212:30:372
1474979992018,1cyclictest0-21swapper/212:30:372
1474979992018,1cyclictest0-21swapper/211:29:002
1474979992018,1cyclictest0-21swapper/211:02:362
1474979992018,1cyclictest0-21swapper/210:53:042
1474979992018,1cyclictest0-21swapper/208:55:562
1474979992018,1cyclictest0-21swapper/208:35:542
1474979992018,1cyclictest0-21swapper/208:27:572
1474979992018,1cyclictest0-21swapper/208:20:072
1474979992016,3cyclictest1510677-21turbostat.cron08:50:002
147497999200,19cyclictest0-21swapper/212:28:342
147497999200,19cyclictest0-21swapper/211:18:392
147497999200,19cyclictest0-21swapper/210:40:472
147497999200,19cyclictest0-21swapper/207:38:572
1474976992013,3cyclictest1567209-21apt11:30:101
1474976992012,6cyclictest1531990-21perf09:50:001
1474976992012,4cyclictest1546391-21apt10:30:121
1474976992011,6cyclictest1583014-21cpuspeed_turbos12:15:121
1474976992010,7cyclictest883-21dbus-daemon08:50:081
147497699200,19cyclictest0-21swapper/111:57:401
147497299207,10cyclictest436-20systemd-journal12:10:200
1474972992018,1cyclictest0-21swapper/011:40:060
1474972992018,1cyclictest0-21swapper/011:07:350
1474972992018,1cyclictest0-21swapper/010:02:590
1474972992018,1cyclictest0-21swapper/009:07:370
147497299200,19cyclictest0-21swapper/012:01:560
147497299200,19cyclictest0-21swapper/011:56:010
147497299200,19cyclictest0-21swapper/010:25:580
147497299200,19cyclictest0-21swapper/008:23:290
1474982991917,1cyclictest3056-21PK-Backend12:11:303
1474982991917,1cyclictest3056-21PK-Backend10:02:053
1474982991917,1cyclictest3056-21PK-Backend09:38:193
1474982991917,1cyclictest3056-21PK-Backend07:32:063
1474982991916,2cyclictest3056-21PK-Backend09:10:213
147498299190,1cyclictest3056-21PK-Backend12:24:543
147498299190,1cyclictest3056-21PK-Backend09:34:213
147498299190,1cyclictest0-21swapper/307:20:163
147498299190,18cyclictest3056-21PK-Backend10:52:003
1474979991917,1cyclictest0-21swapper/211:45:082
1474979991917,1cyclictest0-21swapper/210:56:542
1474979991917,1cyclictest0-21swapper/210:00:512
1474979991917,1cyclictest0-21swapper/209:58:142
1474979991917,1cyclictest0-21swapper/208:52:452
1474979991917,1cyclictest0-21swapper/208:32:102
1474979991917,1cyclictest0-21swapper/208:10:392
1474979991917,1cyclictest0-21swapper/207:18:212
1474979991915,2cyclictest34-21ksoftirqd/208:20:012
147497999190,5cyclictest1495676-21smartctl08:05:202
147497999190,1cyclictest0-21swapper/210:22:012
147497999190,1cyclictest0-21swapper/209:35:162
147497999190,1cyclictest0-21swapper/209:23:542
147497999190,18cyclictest0-21swapper/211:11:082
1474976991913,3cyclictest1542924-21apt10:20:121
1474972991917,1cyclictest0-21swapper/012:23:320
1474972991917,1cyclictest0-21swapper/012:08:520
1474972991917,1cyclictest0-21swapper/011:10:250
1474972991917,1cyclictest0-21swapper/010:50:440
1474972991917,1cyclictest0-21swapper/009:20:400
1474972991916,2cyclictest13-21ksoftirqd/008:00:200
1474972991914,4cyclictest13-21ksoftirqd/010:00:010
1474972991913,5cyclictest1583703-21awk12:15:200
1474972991911,6cyclictest1532263-21apt09:50:100
1474972991911,6cyclictest1529824-21sed09:40:210
1474972991911,6cyclictest1529824-21sed09:40:210
147497299190,18cyclictest0-21swapper/012:31:170
147497299190,18cyclictest0-21swapper/012:31:170
147497299190,18cyclictest0-21swapper/011:16:530
147497299190,18cyclictest0-21swapper/007:39:080
1474982991816,1cyclictest3056-21PK-Backend12:16:223
1474982991816,1cyclictest3056-21PK-Backend11:56:013
1474982991816,1cyclictest3056-21PK-Backend11:02:213
1474982991816,1cyclictest3056-21PK-Backend09:49:413
1474982991816,1cyclictest3056-21PK-Backend07:14:283
1474982991815,2cyclictest3056-21PK-Backend11:32:233
1474982991813,4cyclictest978-21snmpd09:01:083
1474982991812,4cyclictest1521341-21latency_hist09:20:013
1474979991816,1cyclictest0-21swapper/211:43:062
147497999181,4cyclictest141rcu_preempt10:30:122
1474976991810,6cyclictest1527123-21sort09:35:131
1474972991816,1cyclictest0-21swapper/011:50:060
1474972991816,1cyclictest0-21swapper/011:35:130
1474972991816,1cyclictest0-21swapper/010:36:390
1474972991816,1cyclictest0-21swapper/008:08:150
1474972991814,3cyclictest13-21ksoftirqd/009:35:010
1474972991812,5cyclictest13-21ksoftirqd/010:10:200
1474972991811,5cyclictest1552372-21pluginstate10:45:170
147497299180,1cyclictest0-21swapper/011:01:540
1474982991715,1cyclictest3056-21PK-Backend11:07:013
1474982991715,1cyclictest3056-21PK-Backend10:36:193
1474982991715,1cyclictest3056-21PK-Backend08:18:303
1474979991715,1cyclictest0-21swapper/212:09:002
1474979991715,1cyclictest0-21swapper/211:56:092
1474972991711,5cyclictest13-21ksoftirqd/012:25:160
1474982991614,1cyclictest3056-21PK-Backend12:08:393
1474982991614,1cyclictest3056-21PK-Backend08:09:173
147497999169,6cyclictest1491083-21perf07:55:002
1474979991614,1cyclictest0-21swapper/212:03:052
1474979991612,3cyclictest1515857-21cron09:04:592
147497299167,2cyclictest141rcu_preempt12:38:190
1474972991612,3cyclictest1524293-21netstat09:25:170
147498299159,5cyclictest1534983-21cat09:55:203
147498299152,9cyclictest42-21ksoftirqd/308:00:113
1474982991513,1cyclictest3056-21PK-Backend08:14:173
1474982991511,3cyclictest3056-21PK-Backend09:54:393
147497999158,5cyclictest1525816-21cut09:30:152
147497299159,5cyclictest1489896-21expr07:50:130
1474972991510,3cyclictest1565478-21cpu11:25:100
1474979991410,3cyclictest1523569-21/usr/sbin/munin09:25:202
147498299138,4cyclictest1529280-21cut09:40:163
147498299138,4cyclictest1529280-21cut09:40:163
147497999132,2cyclictest141rcu_preempt09:45:202
147497999121,9cyclictest1493075-21apt08:00:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional