You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-23 - 05:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Sep 23, 2026 00:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
152355099916,69cyclictest1529763-21diskmemload22:40:041
1523544999082,3cyclictest0-21swapper/023:15:010
1523544998529,38cyclictest1532208-21latency_hist21:15:020
1523544998327,52cyclictest1571094-21ssh22:37:420
152354499813,8cyclictest1600607-21modprobe23:39:590
152354499791,72cyclictest309-21in:imuxsock00:30:010
1523544997910,64cyclictest0-21swapper/023:00:000
1523544997822,51cyclictest1533456-21ssh21:17:420
1523544997822,51cyclictest1533456-21ssh21:17:420
152354499781,11cyclictest1617566-21ssh00:15:230
152354499772,11cyclictest1574461-21modprobe22:45:000
152354499763,46cyclictest131rcu_preempt00:40:010
152354499762,8cyclictest1576843-21munin-run22:50:010
1523544997617,42cyclictest16850irq/134-800000022:50:440
152354499761,70cyclictest0-21swapper/022:22:250
152354499752,55cyclictest1541568-21latency_hist21:35:020
152354499751,6cyclictest1605280-21munin-run23:49:590
152354499751,57cyclictest1536112-21ssh21:23:340
1523544997514,7cyclictest16850irq/134-800000023:56:270
152354499751,2cyclictest0-21swapper/022:04:550
1523544997427,30cyclictest1589651-21ssh23:16:370
152354499742,67cyclictest48-21kcompactd021:26:150
1523544997417,52cyclictest1615238-21ssh00:10:330
1523544997416,41cyclictest1584833-21ssh23:06:250
1523544997415,14cyclictest1605703-21ssh23:50:280
1523544997415,14cyclictest1605703-21ssh23:50:270
1523544997343,26cyclictest16950irq/224-800000019:20:000
1523544997248,19cyclictest16950irq/224-800000000:34:590
1523544997227,28cyclictest1545495-21ssh21:43:240
152354499721,56cyclictest0-21swapper/021:36:510
1523544997126,29cyclictest1620545-21ssh00:21:480
1523544997124,29cyclictest1561833-21ssh22:18:070
152354499702,6cyclictest1529241-21munin-run21:00:000
1523544996933,18cyclictest1557014-21ssh22:07:550
1523544996933,18cyclictest1557014-21ssh22:07:540
1523544996827,33cyclictest16950irq/224-800000023:45:020
1523544996827,33cyclictest16950irq/224-800000023:45:020
152354499681,62cyclictest0-21swapper/023:20:290
152354499681,62cyclictest0-21swapper/023:20:280
152354499672,59cyclictest1614793-21cat00:10:010
1523544996720,30cyclictest1594524-21ssh23:26:490
152354499661,60cyclictest364-21snmpd21:50:560
152354499661,49cyclictest150-21pr/ttyS200:02:000
1523557996527,9cyclictest37-21ksoftirqd/323:20:023
152355099658,48cyclictest1558022-21ssh22:10:031
152355099658,48cyclictest1558022-21ssh22:10:021
152354499653,8cyclictest1526127-21systemctl19:59:590
1523557996413,6cyclictest131rcu_preempt23:10:023
1523544996431,30cyclictest16950irq/224-800000020:20:000
1523544996424,24cyclictest1565714-21ssh22:26:140
1523544996420,39cyclictest1527500-21cat20:25:030
1523544996419,40cyclictest1546760-21ssh21:45:590
1523544996418,42cyclictest12-21ksoftirqd/022:13:160
1523544996411,10cyclictest1567767-21ssh22:30:270
152354499634,13cyclictest1552488-21ssh21:58:340
152354499632,16cyclictest1528458-21sshd20:44:380
152354499632,15cyclictest1525118-21unin-run19:40:010
152354499632,15cyclictest1525118-21unin-run19:40:010
152354499625,47cyclictest1525603-21modprobe19:50:000
152354499622,12cyclictest1525876-21munin-run19:55:010
1523544996216,41cyclictest1-21systemd20:47:380
1523544996216,41cyclictest1-21systemd20:47:380
152354499621,57cyclictest0-21swapper/023:32:430
152355499618,12cyclictest30-21ksoftirqd/221:20:012
152355499618,12cyclictest30-21ksoftirqd/221:20:002
152355499611,8cyclictest30-21ksoftirqd/223:39:592
1523554996113,5cyclictest131rcu_preempt21:46:432
152354499612,14cyclictest1528979-21munin-run20:55:010
1523544996119,25cyclictest1529853-21ssh21:10:020
152354499611,13cyclictest364-21snmpd23:00:260
1523554996016,5cyclictest131rcu_preempt23:33:232
1523554996012,6cyclictest131rcu_preempt23:05:432
152354499601,13cyclictest364-21snmpd20:01:370
152354499601,13cyclictest1523546-21latency_hist19:10:010
152354499601,12cyclictest1528166-21sshd20:36:380
1523557995932,19cyclictest37-21ksoftirqd/319:31:043
1523557995910,6cyclictest131rcu_preempt23:40:033
1523557995910,6cyclictest131rcu_preempt23:40:023
152354499591,13cyclictest1529474-21sshd21:00:380
1523554995819,11cyclictest30-21ksoftirqd/221:40:012
1523554995811,6cyclictest131rcu_preempt22:15:572
1523557995713,6cyclictest131rcu_preempt00:26:133
152355499576,8cyclictest30-21ksoftirqd/223:54:412
152355499576,8cyclictest30-21ksoftirqd/223:54:402
1523554995715,5cyclictest131rcu_preempt00:10:302
1523554995712,5cyclictest131rcu_preempt22:25:012
152355099572,49cyclictest275-21dbus-daemon00:40:011
152354499572,10cyclictest284-21systemd-logind20:30:000
152354499571,13cyclictest364-21snmpd19:20:030
152354499571,12cyclictest1525337-21sshd19:43:390
152355499569,6cyclictest131rcu_preempt23:22:592
152355499569,6cyclictest131rcu_preempt23:22:582
152355499568,10cyclictest30-21ksoftirqd/200:30:022
152355499567,9cyclictest30-21ksoftirqd/222:11:442
1523554995621,5cyclictest131rcu_preempt21:27:432
1523554995617,11cyclictest30-21ksoftirqd/223:47:052
152355499561,6cyclictest30-21ksoftirqd/221:13:372
1523554995614,9cyclictest30-21ksoftirqd/219:20:012
1523554995613,5cyclictest131rcu_preempt23:03:002
1523554995613,5cyclictest131rcu_preempt21:52:512
1523554995612,6cyclictest131rcu_preempt22:59:502
1523554995612,6cyclictest131rcu_preempt21:20:482
1523554995612,5cyclictest131rcu_preempt22:54:102
1523554995611,6cyclictest131rcu_preempt22:28:012
1523554995610,6cyclictest131rcu_preempt21:41:232
1523554995610,6cyclictest131rcu_preempt00:10:002
1523557995511,6cyclictest131rcu_preempt00:11:563
152355499557,10cyclictest30-21ksoftirqd/200:24:442
152355499556,5cyclictest131rcu_preempt00:19:062
152355499555,9cyclictest30-21ksoftirqd/222:06:362
152355499555,9cyclictest30-21ksoftirqd/222:06:352
1523554995517,11cyclictest30-21ksoftirqd/222:02:362
1523554995514,5cyclictest131rcu_preempt00:33:172
1523554995512,6cyclictest131rcu_preempt23:11:002
1523554995510,5cyclictest131rcu_preempt00:05:012
152354499553,17cyclictest1-21systemd20:15:000
152355799547,11cyclictest37-21ksoftirqd/319:20:003
1523557995423,22cyclictest361rcuc/300:05:023
1523557995414,6cyclictest131rcu_preempt22:32:113
152355499549,6cyclictest131rcu_preempt00:36:092
1523554995415,8cyclictest30-21ksoftirqd/221:56:162
1523554995413,5cyclictest131rcu_preempt23:44:242
1523554995413,5cyclictest131rcu_preempt23:44:232
1523554995411,6cyclictest131rcu_preempt22:48:582
1523554995411,5cyclictest131rcu_preempt23:19:492
1523544995427,23cyclictest1526634-21cron20:10:010
1523544995427,23cyclictest1526634-21cron20:10:000
152355799537,8cyclictest37-21ksoftirqd/321:55:463
1523557995311,7cyclictest131rcu_preempt22:50:163
152355799531,13cyclictest37-21ksoftirqd/320:55:013
152355499539,5cyclictest131rcu_preempt23:58:182
152355499537,10cyclictest30-21ksoftirqd/222:40:372
152355499536,8cyclictest30-21ksoftirqd/222:39:112
1523554995311,6cyclictest131rcu_preempt19:25:012
1523550995333,15cyclictest275-21dbus-daemon19:40:011
1523550995333,15cyclictest275-21dbus-daemon19:40:011
152355799529,5cyclictest131rcu_preempt21:43:343
152355799526,25cyclictest0-21swapper/321:40:033
1523557995224,18cyclictest37-21ksoftirqd/320:03:533
1523557995221,21cyclictest0-21swapper/300:05:163
1523557995217,8cyclictest37-21ksoftirqd/323:57:343
1523557995214,8cyclictest131rcu_preempt22:14:513
1523557995213,6cyclictest131rcu_preempt21:29:013
1523557995213,5cyclictest131rcu_preempt21:31:573
1523557995211,6cyclictest131rcu_preempt23:34:503
152355499527,9cyclictest30-21ksoftirqd/221:30:082
152355499526,5cyclictest131rcu_preempt22:30:222
1523554995211,5cyclictest131rcu_preempt19:35:022
1523550995238,10cyclictest1553114-21systemctl22:00:001
152354499522,14cyclictest1528003-21cat20:35:030
152354499522,14cyclictest1528003-21cat20:35:020
152355799515,7cyclictest131rcu_preempt22:21:483
1523557995120,7cyclictest37-21ksoftirqd/321:55:023
1523557995113,10cyclictest37-21ksoftirqd/322:05:033
1523557995112,6cyclictest131rcu_preempt22:29:253
1523554995142,6cyclictest1526170-21kworker/2:220:10:532
152355499512,11cyclictest30-21ksoftirqd/219:29:592
1523550995131,14cyclictest275-21dbus-daemon23:59:591
152355799504,7cyclictest37-21ksoftirqd/300:20:013
1523557995030,8cyclictest37-21ksoftirqd/322:58:183
1523557995023,11cyclictest37-21ksoftirqd/321:47:163
1523557995011,6cyclictest131rcu_preempt23:38:363
1523557995011,6cyclictest131rcu_preempt00:39:333
152355499509,5cyclictest131rcu_preempt19:40:022
152355499509,5cyclictest131rcu_preempt19:40:022
152355499507,7cyclictest30-21ksoftirqd/223:26:032
152355499504,13cyclictest30-21ksoftirqd/220:40:002
152354499502,14cyclictest1523769-21cron19:15:000
152354499502,14cyclictest1523769-21cron19:15:000
152354499502,12cyclictest1524840-21systemd-run19:35:010
152355799496,7cyclictest37-21ksoftirqd/321:00:013
152355799495,6cyclictest37-21ksoftirqd/323:23:343
152355799495,6cyclictest37-21ksoftirqd/323:23:333
152355799495,12cyclictest37-21ksoftirqd/321:20:003
152355799495,12cyclictest37-21ksoftirqd/321:20:003
1523557994928,16cyclictest275-21dbus-daemon21:05:013
1523557994912,5cyclictest131rcu_preempt22:20:013
1523557994911,5cyclictest131rcu_preempt22:35:453
1523557994910,5cyclictest131rcu_preempt23:52:273
1523557994910,5cyclictest131rcu_preempt23:52:263
1523557994910,5cyclictest131rcu_preempt22:45:543
152355499499,13cyclictest30-21ksoftirqd/220:50:012
152355499499,13cyclictest30-21ksoftirqd/220:50:012
1523554994941,5cyclictest1525116-21kworker/2:121:00:162
1523554994941,5cyclictest1525116-21kworker/2:120:56:172
1523554994941,5cyclictest1525116-21kworker/2:120:20:022
1523554994941,5cyclictest1524586-21kworker/2:019:55:392
1523554994940,6cyclictest1525116-21kworker/2:120:09:232
1523554994940,6cyclictest1525116-21kworker/2:120:09:222
1523554994940,5cyclictest1524586-21kworker/2:019:50:302
1523554994910,5cyclictest131rcu_preempt20:55:012
1523554994910,5cyclictest131rcu_preempt19:50:002
1523550994932,13cyclictest3467602-21systemd-journal21:30:011
1523550994924,20cyclictest23-21ksoftirqd/122:00:281
152354499493,19cyclictest1524576-21grep19:29:590
152355799488,8cyclictest37-21ksoftirqd/321:24:013
152355799487,5cyclictest131rcu_preempt23:29:483
152355799487,4cyclictest131rcu_preempt00:33:133
152355799485,9cyclictest37-21ksoftirqd/321:15:013
152355799481,8cyclictest37-21ksoftirqd/323:49:003
1523557994812,5cyclictest131rcu_preempt22:42:303
1523554994840,5cyclictest1526170-21kworker/2:220:28:242
1523554994840,5cyclictest1525116-21kworker/2:121:05:402
1523554994839,5cyclictest1526170-21kworker/2:220:19:392
1523554994839,5cyclictest1524586-21kworker/2:019:41:422
1523554994813,6cyclictest131rcu_preempt19:10:392
1523554994813,6cyclictest131rcu_preempt19:10:392
1523550994831,12cyclictest275-21dbus-daemon21:00:011
152355799479,6cyclictest131rcu_preempt20:40:003
152355799479,6cyclictest131rcu_preempt19:40:023
152355799479,6cyclictest131rcu_preempt19:40:013
152355799477,9cyclictest37-21ksoftirqd/323:13:283
152355799477,5cyclictest131rcu_preempt20:50:023
152355799477,5cyclictest131rcu_preempt20:50:023
1523557994723,16cyclictest37-21ksoftirqd/320:34:453
1523557994723,16cyclictest37-21ksoftirqd/320:34:453
1523557994711,6cyclictest131rcu_preempt22:07:373
1523557994711,6cyclictest131rcu_preempt22:07:373
1523557994711,5cyclictest131rcu_preempt21:10:023
1523557994710,6cyclictest131rcu_preempt23:01:393
1523554994740,4cyclictest1525116-21kworker/2:120:40:452
1523554994739,5cyclictest1526170-21kworker/2:220:31:092
1523554994739,5cyclictest1526170-21kworker/2:220:31:092
1523554994739,5cyclictest1525116-21kworker/2:120:01:382
1523550994732,12cyclictest1-21systemd19:55:021
1523550994720,22cyclictest23-21ksoftirqd/100:00:031
1523550994710,9cyclictest131rcu_preempt00:17:171
1523557994632,10cyclictest37-21ksoftirqd/319:25:013
1523557994628,14cyclictest275-21dbus-daemon19:55:013
1523550994633,8cyclictest23-21ksoftirqd/123:15:001
1523550994629,13cyclictest275-21dbus-daemon00:15:001
1523550994628,14cyclictest1-21systemd21:50:011
1523550994628,12cyclictest23-21ksoftirqd/122:40:001
152355099462,5cyclictest0-21swapper/100:30:011
152355799456,6cyclictest131rcu_preempt00:20:403
1523557994531,11cyclictest37-21ksoftirqd/320:20:013
1523550994527,14cyclictest23-21ksoftirqd/121:55:001
1523550994519,20cyclictest23-21ksoftirqd/122:13:381
1523550994519,15cyclictest131rcu_preempt21:15:021
1523550994518,16cyclictest23-21ksoftirqd/122:26:291
1523550994515,21cyclictest23-21ksoftirqd/121:40:011
152355799444,7cyclictest37-21ksoftirqd/320:10:003
152355799444,7cyclictest37-21ksoftirqd/320:10:003
152355099449,5cyclictest131rcu_preempt00:24:091
1523550994429,12cyclictest3467602-21systemd-journal00:31:561
1523550994428,13cyclictest23-21ksoftirqd/121:32:151
1523550994425,9cyclictest23-21ksoftirqd/119:29:591
1523550994421,12cyclictest23-21ksoftirqd/121:21:491
152355099441,8cyclictest0-21swapper/123:04:501
1523550994410,6cyclictest131rcu_preempt21:18:181
1523550994410,6cyclictest131rcu_preempt21:18:171
1523550994333,7cyclictest1528458-21sshd20:44:381
1523550994319,20cyclictest23-21ksoftirqd/123:40:321
1523550994319,20cyclictest23-21ksoftirqd/123:40:311
1523550994319,19cyclictest23-21ksoftirqd/120:45:391
1523550994319,19cyclictest23-21ksoftirqd/120:45:381
1523550994312,21cyclictest1595470-21kworker/u8:023:40:021
152355799422,10cyclictest37-21ksoftirqd/319:15:003
152355799422,10cyclictest37-21ksoftirqd/319:15:003
1523557994212,15cyclictest37-21ksoftirqd/320:00:003
152355099425,5cyclictest131rcu_preempt21:44:581
152355099425,15cyclictest0-21swapper/119:15:031
152355099425,15cyclictest0-21swapper/119:15:021
1523550994227,11cyclictest1-21systemd23:30:001
1523550994226,12cyclictest1605289-21systemd-run23:50:011
1523550994225,4cyclictest23-21ksoftirqd/123:51:221
1523550994225,4cyclictest23-21ksoftirqd/123:51:211
152355099422,36cyclictest23-21ksoftirqd/122:30:031
1523550994219,9cyclictest23-21ksoftirqd/122:54:421
1523550994219,9cyclictest23-21ksoftirqd/100:05:071
152355099421,8cyclictest0-21swapper/123:30:141
1523550994218,20cyclictest23-21ksoftirqd/119:15:021
152355799415,6cyclictest37-21ksoftirqd/319:46:073
152355799411,7cyclictest0-21swapper/320:45:003
1523550994125,12cyclictest1591173-21systemd-run23:20:001
152355099411,6cyclictest0-21swapper/122:16:341
152355099411,5cyclictest0-21swapper/122:23:171
1523550994112,8cyclictest131rcu_preempt20:35:001
1523550994112,8cyclictest131rcu_preempt20:35:001
1523550994111,17cyclictest23-21ksoftirqd/123:09:581
152355099409,14cyclictest23-21ksoftirqd/123:22:311
152355099409,14cyclictest23-21ksoftirqd/123:22:301
1523550994027,8cyclictest23-21ksoftirqd/119:45:001
1523550994025,11cyclictest0-21swapper/120:15:021
1523550994013,5cyclictest131rcu_preempt20:20:001
1523550994013,16cyclictest23-21ksoftirqd/122:57:121
152355799399,5cyclictest131rcu_preempt20:10:393
152355799395,5cyclictest131rcu_preempt19:40:373
152355799393,5cyclictest131rcu_preempt19:10:023
152355099396,4cyclictest131rcu_preempt20:40:021
1523550993926,10cyclictest1524330-21systemd-run19:25:001
1523550993916,18cyclictest23-21ksoftirqd/122:45:381
152355799385,5cyclictest131rcu_preempt19:25:043
1523557993812,6cyclictest131rcu_preempt20:25:023
152355799375,7cyclictest131rcu_preempt20:29:203
152355099373,6cyclictest23-21ksoftirqd/119:34:381
1523550993727,7cyclictest1526647-21modprobe20:10:001
1523550993727,7cyclictest1526647-21modprobe20:10:001
1523550993723,10cyclictest275-21dbus-daemon20:00:001
152355099371,4cyclictest131rcu_preempt20:55:001
1523550993712,16cyclictest23-21ksoftirqd/121:01:481
1523550993626,7cyclictest3467602-21systemd-journal20:23:381
1523550993622,9cyclictest1529971-21sh21:10:021
1523550993612,15cyclictest23-21ksoftirqd/120:26:231
152355499357,5cyclictest131rcu_preempt19:10:032
152355099341,4cyclictest131rcu_preempt20:05:001
1523550993326,4cyclictest1525743-21latency_hist19:50:021
152355099321,5cyclictest0-21swapper/119:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional