You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-13 - 05:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat Jun 13, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1647318997368,3cyclictest150-21pr/ttyS223:05:010
1647328997110,5cyclictest131rcu_preempt23:45:013
1647323996713,7cyclictest131rcu_preempt22:55:021
164731899677,7cyclictest121ksoftirqd/000:20:160
1647318996714,11cyclictest12-21ksoftirqd/023:54:530
1647318996623,9cyclictest12-21ksoftirqd/020:10:020
1647318996623,9cyclictest12-21ksoftirqd/020:10:010
164731899653,5cyclictest131rcu_preempt22:34:590
164732599642,45cyclictest1709582-21sh23:15:012
164732399647,11cyclictest23-21ksoftirqd/100:20:011
1647323996430,10cyclictest23-21ksoftirqd/122:26:571
1647323996430,10cyclictest23-21ksoftirqd/122:26:571
164731899647,38cyclictest12-21ksoftirqd/022:43:570
1647318996422,21cyclictest0-21swapper/023:11:440
1647318996220,38cyclictest12-21ksoftirqd/022:55:000
1647318996216,7cyclictest12-21ksoftirqd/022:10:010
164731899621,4cyclictest131rcu_preempt22:15:000
1647325996117,8cyclictest0-21swapper/221:29:212
1647323996120,9cyclictest23-21ksoftirqd/121:15:001
1647318996127,17cyclictest1731858-21ssh00:05:000
164731899611,39cyclictest131rcu_preempt23:38:570
1647323996028,10cyclictest23-21ksoftirqd/121:40:011
164732399595,7cyclictest23-21ksoftirqd/122:50:021
164732399595,7cyclictest23-21ksoftirqd/122:50:021
1647323995925,12cyclictest23-21ksoftirqd/123:10:011
1647323995912,5cyclictest131rcu_preempt23:05:011
164731899596,6cyclictest131rcu_preempt23:10:020
1647318995949,6cyclictest1729615-21latency_hist00:00:030
1647318995949,6cyclictest1729615-21latency_hist00:00:020
164731899592,5cyclictest131rcu_preempt23:50:020
1647318995921,34cyclictest1648625-21latency_hist19:35:020
164732399588,7cyclictest23-21ksoftirqd/100:15:011
164732399587,36cyclictest0-21swapper/100:35:021
1647323995820,7cyclictest23-21ksoftirqd/121:43:391
1647323995812,6cyclictest131rcu_preempt21:29:191
1647318995823,12cyclictest12-21ksoftirqd/021:40:010
164732399577,9cyclictest23-21ksoftirqd/100:39:541
1647323995725,5cyclictest23-21ksoftirqd/121:22:401
1647323995710,5cyclictest131rcu_preempt00:10:011
1647318995747,8cyclictest12-21ksoftirqd/023:40:210
1647318995730,20cyclictest0-21swapper/022:49:350
1647318995730,20cyclictest0-21swapper/022:49:340
1647318995725,7cyclictest12-21ksoftirqd/020:45:000
164732599566,5cyclictest131rcu_preempt21:25:012
1647325995646,5cyclictest1639990-21kworker/2:122:15:012
1647325995638,13cyclictest1-21systemd00:05:012
164732399568,9cyclictest23-21ksoftirqd/120:35:001
164732399566,8cyclictest23-21ksoftirqd/122:15:011
1647323995616,13cyclictest23-21ksoftirqd/100:23:501
1647318995628,13cyclictest1676715-21ssh22:01:140
1647318995624,23cyclictest12-21ksoftirqd/023:30:020
164731899561,40cyclictest131rcu_preempt21:53:180
1647318995611,6cyclictest131rcu_preempt23:31:310
1647328995513,23cyclictest0-21swapper/323:59:153
1647328995513,23cyclictest0-21swapper/323:59:143
1647325995545,7cyclictest1707263-21kworker/2:200:25:352
164732399555,10cyclictest23-21ksoftirqd/122:55:221
164732399555,10cyclictest23-21ksoftirqd/122:55:221
164732399551,18cyclictest23-21ksoftirqd/100:00:011
164732399551,18cyclictest23-21ksoftirqd/100:00:001
1647318995528,12cyclictest1702140-21ssh22:58:210
1647318995528,12cyclictest1702140-21ssh22:58:210
164731899552,37cyclictest1655548-21ssh21:14:320
1647318995522,30cyclictest12-21ksoftirqd/021:48:430
1647318995519,4cyclictest131rcu_preempt00:15:330
1647318995513,37cyclictest12-21ksoftirqd/000:25:030
1647325995446,5cyclictest1707263-21kworker/2:223:19:102
1647325995446,5cyclictest1707263-21kworker/2:200:20:002
1647325995444,7cyclictest1707263-21kworker/2:200:39:562
164732399549,5cyclictest131rcu_preempt22:42:161
1647323995413,11cyclictest23-21ksoftirqd/121:52:281
1647323995411,5cyclictest131rcu_preempt21:57:461
1647323995411,5cyclictest131rcu_preempt21:48:391
1647318995432,4cyclictest131rcu_preempt20:20:040
1647318995420,9cyclictest12-21ksoftirqd/023:17:020
1647325995345,5cyclictest1641027-21kworker/2:323:43:282
1647325995345,5cyclictest1641027-21kworker/2:323:29:532
1647325995343,6cyclictest1639990-21kworker/2:122:10:032
164732399537,9cyclictest23-21ksoftirqd/123:52:571
164732399535,11cyclictest23-21ksoftirqd/119:20:011
1647323995312,12cyclictest131rcu_preempt22:25:021
164732399531,16cyclictest23-21ksoftirqd/121:31:421
1647323995311,5cyclictest131rcu_preempt23:31:191
164731899539,6cyclictest131rcu_preempt21:40:360
164731899532,14cyclictest1653489-21cut21:10:000
164731899532,10cyclictest12-21ksoftirqd/021:27:400
1647325995244,5cyclictest1641027-21kworker/2:322:33:052
1647325995243,5cyclictest1641027-21kworker/2:323:23:012
164732399529,11cyclictest23-21ksoftirqd/123:50:011
164732399528,7cyclictest23-21ksoftirqd/122:19:321
1647323995244,5cyclictest221rcuc/120:15:011
1647318995228,9cyclictest1688605-21ssh22:28:020
1647318995228,9cyclictest1688605-21ssh22:28:010
164732599516,32cyclictest30-21ksoftirqd/220:20:022
1647325995144,4cyclictest1641027-21kworker/2:322:22:362
1647325995143,5cyclictest1641027-21kworker/2:322:49:332
1647325995143,5cyclictest1641027-21kworker/2:322:49:322
1647325995143,5cyclictest1639990-21kworker/2:122:39:182
1647325995143,5cyclictest1639990-21kworker/2:122:39:172
1647325995142,5cyclictest1639990-21kworker/2:121:46:362
164732599511,10cyclictest1648447-21latency_hist19:30:022
164732399519,9cyclictest23-21ksoftirqd/122:30:501
164732399518,5cyclictest131rcu_preempt22:04:581
164732399517,8cyclictest23-21ksoftirqd/120:45:001
1647323995124,20cyclictest311-21rs:main1
1647323995114,5cyclictest131rcu_preempt00:26:511
164731899515,9cyclictest12-21ksoftirqd/020:30:000
164731899512,8cyclictest12-21ksoftirqd/000:30:100
1647318995125,21cyclictest1657962-21ssh21:20:000
1647318995125,21cyclictest1657962-21ssh21:20:000
1647318995116,5cyclictest131rcu_preempt21:21:520
164732899504,4cyclictest131rcu_preempt23:25:423
1647328995012,6cyclictest131rcu_preempt21:25:023
1647325995042,5cyclictest1707263-21kworker/2:200:12:452
1647325995042,5cyclictest1641027-21kworker/2:323:47:282
164732399509,9cyclictest23-21ksoftirqd/100:03:021
164732399509,5cyclictest131rcu_preempt23:13:151
164732399509,5cyclictest131rcu_preempt20:20:011
164732399509,5cyclictest131rcu_preempt19:55:011
164732399508,9cyclictest23-21ksoftirqd/123:22:521
164732399507,5cyclictest131rcu_preempt22:37:271
164732399507,5cyclictest131rcu_preempt22:37:271
164732399507,11cyclictest23-21ksoftirqd/123:40:031
1647323995012,5cyclictest131rcu_preempt20:30:021
164731899508,5cyclictest131rcu_preempt22:23:510
164731899507,6cyclictest131rcu_preempt22:40:010
164731899507,6cyclictest131rcu_preempt22:40:000
164731899505,5cyclictest131rcu_preempt20:20:000
1647318995024,11cyclictest1746549-21ssh00:37:570
164731899501,44cyclictest1736313-21latency_hist00:15:010
1647318995013,9cyclictest12-21ksoftirqd/019:20:020
1647325994941,5cyclictest1707263-21kworker/2:223:52:372
1647325994941,5cyclictest1641027-21kworker/2:323:02:172
1647325994941,5cyclictest1641027-21kworker/2:322:58:322
1647325994941,5cyclictest1641027-21kworker/2:322:58:312
1647325994941,5cyclictest1641027-21kworker/2:321:08:562
1647325994941,5cyclictest1639990-21kworker/2:121:52:062
1647325994941,5cyclictest1639990-21kworker/2:121:43:412
1647325994941,5cyclictest1639990-21kworker/2:121:33:242
1647325994932,13cyclictest1-21systemd19:50:022
164732599491,44cyclictest1676032-21munin-run22:00:012
164732399497,8cyclictest23-21ksoftirqd/121:20:001
164732399497,8cyclictest23-21ksoftirqd/121:20:001
164732399494,8cyclictest23-21ksoftirqd/121:07:261
1647323994914,5cyclictest131rcu_preempt19:25:031
164732399491,16cyclictest0-21swapper/120:21:271
164732399491,13cyclictest1650370-21systemd-run20:10:011
164732399491,13cyclictest1650370-21systemd-run20:10:011
1647318994932,13cyclictest3467602-21systemd-journal21:35:000
1647318994923,22cyclictest12-21ksoftirqd/023:21:170
164731899491,5cyclictest131rcu_preempt22:00:010
164732899484,4cyclictest131rcu_preempt22:21:393
164732899484,3cyclictest150-21pr/ttyS221:40:103
1647328994832,12cyclictest275-21dbus-daemon00:25:013
164732899482,42cyclictest37-21ksoftirqd/322:00:023
1647325994842,3cyclictest1639990-21kworker/2:122:00:132
1647325994842,3cyclictest1639990-21kworker/2:121:35:342
1647325994840,5cyclictest1707263-21kworker/2:200:31:222
1647325994840,5cyclictest1641027-21kworker/2:321:19:242
1647325994840,5cyclictest1641027-21kworker/2:321:19:242
1647325994832,13cyclictest1-21systemd20:00:022
1647325994832,13cyclictest1-21systemd20:00:012
164732399486,11cyclictest23-21ksoftirqd/121:05:001
164732399485,12cyclictest23-21ksoftirqd/123:42:011
1647323994813,6cyclictest131rcu_preempt23:26:311
1647323994810,6cyclictest131rcu_preempt22:06:561
164731899487,5cyclictest131rcu_preempt21:00:010
164731899487,5cyclictest131rcu_preempt21:00:010
1647318994841,4cyclictest16950irq/224-800000020:15:000
164731899483,40cyclictest1684931-21munin-run22:20:000
164731899483,21cyclictest131rcu_preempt19:20:030
164731899481,4cyclictest1650112-21/usr/sbin/munin20:05:010
164732899479,6cyclictest131rcu_preempt22:20:023
164732899474,3cyclictest150-21pr/ttyS222:51:133
1647328994730,11cyclictest1-21systemd21:30:023
1647328994721,17cyclictest0-21swapper/323:05:003
164732899471,5cyclictest131rcu_preempt22:37:183
164732899471,5cyclictest131rcu_preempt22:37:173
164732599478,6cyclictest131rcu_preempt22:20:032
164732599474,5cyclictest1720603-21modprobe23:40:002
1647325994740,5cyclictest1707263-21kworker/2:200:06:202
1647325994740,4cyclictest1641027-21kworker/2:322:54:322
1647325994739,5cyclictest1641027-21kworker/2:321:10:382
1647325994738,5cyclictest1639990-21kworker/2:122:43:332
164732599472,6cyclictest131rcu_preempt19:40:002
1647325994711,6cyclictest131rcu_preempt19:12:262
1647323994730,12cyclictest23-21ksoftirqd/119:40:001
164732399471,9cyclictest1648035-21modprobe19:25:001
1647323994712,8cyclictest131rcu_preempt23:16:451
164731899475,5cyclictest131rcu_preempt19:45:020
1647318994726,17cyclictest12-21ksoftirqd/000:05:440
164732899464,3cyclictest150-21pr/ttyS222:45:343
164732899464,3cyclictest150-21pr/ttyS222:45:333
1647328994629,13cyclictest275-21dbus-daemon23:40:013
164732599465,3cyclictest131rcu_preempt20:55:022
1647325994638,5cyclictest1707263-21kworker/2:200:24:252
1647325994638,5cyclictest1641027-21kworker/2:323:05:502
164732399465,7cyclictest23-21ksoftirqd/120:00:001
164732399465,7cyclictest23-21ksoftirqd/120:00:001
164732399461,8cyclictest1651946-21modprobe20:40:001
164732899459,6cyclictest131rcu_preempt00:40:033
164732899455,4cyclictest131rcu_preempt22:40:113
164732899454,3cyclictest131rcu_preempt20:40:023
164732899452,3cyclictest131rcu_preempt00:28:323
164732899452,37cyclictest275-21dbus-daemon20:20:013
1647328994510,9cyclictest0-21swapper/322:58:093
1647328994510,9cyclictest0-21swapper/322:58:083
1647325994538,4cyclictest1707263-21kworker/2:223:56:252
1647325994538,4cyclictest1707263-21kworker/2:223:56:242
1647325994537,5cyclictest1641027-21kworker/2:322:27:222
1647325994537,5cyclictest1641027-21kworker/2:322:27:222
164732399455,6cyclictest131rcu_preempt19:15:001
164732399454,10cyclictest23-21ksoftirqd/120:55:001
164731899455,6cyclictest131rcu_preempt19:15:010
164731899454,7cyclictest12-21ksoftirqd/020:00:000
164731899454,7cyclictest12-21ksoftirqd/020:00:000
164731899454,4cyclictest131rcu_preempt19:50:010
164731899453,6cyclictest131rcu_preempt20:35:010
164732899446,3cyclictest131rcu_preempt19:55:013
164732899445,4cyclictest131rcu_preempt21:40:023
164732899444,5cyclictest131rcu_preempt22:03:563
164732899443,3cyclictest150-21pr/ttyS221:45:373
164732899443,3cyclictest150-21pr/ttyS200:17:233
1647328994427,12cyclictest37-21ksoftirqd/300:13:433
164732899442,4cyclictest150-21pr/ttyS200:09:063
164732899442,4cyclictest131rcu_preempt00:04:043
164732899442,4cyclictest0-21swapper/300:33:423
164732899442,3cyclictest131rcu_preempt23:21:063
164732899441,3cyclictest131rcu_preempt23:47:573
1647325994438,3cyclictest1641027-21kworker/2:323:34:342
1647325994427,13cyclictest3467602-21systemd-journal20:41:322
164732599442,5cyclictest131rcu_preempt20:40:012
164732399441,8cyclictest0-21swapper/119:35:011
164732899435,4cyclictest131rcu_preempt21:31:343
164732899435,3cyclictest131rcu_preempt23:15:023
164732899434,3cyclictest150-21pr/ttyS221:11:533
164732899434,3cyclictest131rcu_preempt23:19:403
164732899434,3cyclictest131rcu_preempt20:25:023
164732899432,3cyclictest131rcu_preempt23:08:333
164732899431,4cyclictest131rcu_preempt22:08:153
164732899431,4cyclictest0-21swapper/321:54:293
164732599435,3cyclictest131rcu_preempt19:35:012
164732599434,3cyclictest131rcu_preempt19:20:002
1647325994330,10cyclictest1650769-21sed20:15:012
1647325994327,11cyclictest30-21ksoftirqd/220:10:012
1647325994327,11cyclictest30-21ksoftirqd/220:10:002
1647323994328,11cyclictest309-21in:imuxsock19:50:001
1647323994312,6cyclictest131rcu_preempt20:05:011
164731899436,6cyclictest131rcu_preempt19:30:010
164731899431,7cyclictest12-21ksoftirqd/019:40:010
1647318994317,5cyclictest131rcu_preempt20:40:020
1647318994310,6cyclictest131rcu_preempt21:01:270
164732899425,4cyclictest131rcu_preempt21:15:383
164732899425,4cyclictest131rcu_preempt21:15:373
164732899424,3cyclictest150-21pr/ttyS222:13:173
164732899423,3cyclictest131rcu_preempt23:32:293
1647328994231,8cyclictest275-21dbus-daemon20:50:013
164732899422,6cyclictest3467602-21systemd-journal22:34:593
164732899422,13cyclictest311-21rs:main3
1647328994213,6cyclictest131rcu_preempt21:00:033
1647328994213,6cyclictest131rcu_preempt21:00:023
164732599423,4cyclictest131rcu_preempt19:55:012
1647325994224,12cyclictest275-21dbus-daemon19:25:002
164732399427,8cyclictest23-21ksoftirqd/121:00:021
164732399427,8cyclictest23-21ksoftirqd/121:00:011
164731899429,7cyclictest131rcu_preempt20:55:020
1647318994210,4cyclictest131rcu_preempt20:50:010
164732899415,6cyclictest131rcu_preempt21:10:003
164732899414,4cyclictest131rcu_preempt23:53:503
164732899411,7cyclictest1687623-21ssh22:25:353
164732899411,7cyclictest1687623-21ssh22:25:353
1647328994030,8cyclictest37-21ksoftirqd/319:35:023
164732899401,4cyclictest131rcu_preempt19:27:093
164732599402,4cyclictest131rcu_preempt20:25:032
1647325994011,5cyclictest131rcu_preempt21:05:002
164731899404,5cyclictest131rcu_preempt19:55:010
164732599395,3cyclictest131rcu_preempt20:05:022
164732599393,3cyclictest131rcu_preempt20:35:022
164732399397,8cyclictest23-21ksoftirqd/119:10:021
164732399391,9cyclictest23-21ksoftirqd/120:46:331
1647318993915,19cyclictest131rcu_preempt19:10:010
164732899386,3cyclictest131rcu_preempt19:20:023
164732899385,4cyclictest131rcu_preempt20:15:023
1647328993819,14cyclictest275-21dbus-daemon20:35:013
164732599385,3cyclictest131rcu_preempt20:30:002
1647328993724,10cyclictest37-21ksoftirqd/319:15:023
164732899372,3cyclictest131rcu_preempt19:50:003
1647328993723,11cyclictest37-21ksoftirqd/320:05:013
164732899362,5cyclictest0-21swapper/319:45:023
1647328993620,10cyclictest275-21dbus-daemon20:00:013
1647328993620,10cyclictest275-21dbus-daemon20:00:013
1647328993613,6cyclictest131rcu_preempt19:25:013
164732599364,4cyclictest131rcu_preempt19:10:022
164732899356,4cyclictest131rcu_preempt20:08:353
164732899356,4cyclictest131rcu_preempt20:08:353
164732899354,4cyclictest131rcu_preempt20:41:273
1647328993528,4cyclictest3467562-21sshd21:01:263
1647328993518,12cyclictest1648817-21tr19:40:003
1647325993525,8cyclictest1652936-21cron21:00:002
1647325993525,8cyclictest1652936-21cron21:00:002
164732599341,3cyclictest131rcu_preempt19:45:002
164732899332,3cyclictest131rcu_preempt20:30:013
164732599321,3cyclictest131rcu_preempt20:50:002
1647328993119,9cyclictest1647417-21sed19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional