You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 03:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Jun 27, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
214182240163,25sleep00-21swapper/019:09:140
212562240167,22sleep30-21swapper/319:07:093
213652239164,24sleep20-21swapper/219:08:332
213322214163,38sleep10-21swapper/119:08:071
3250221580,4sleep12156599cyclictest23:10:271
1767021550,4sleep22157199cyclictest21:25:172
21571991101,107cyclictest0-21swapper/222:25:292
21571991080,106cyclictest0-21swapper/223:47:122
159122660,7sleep081ktimersoftd/022:35:140
21571996359,3cyclictest0-21swapper/223:21:482
321612530,1sleep30-21swapper/300:25:173
228862530,1sleep20-21swapper/200:05:142
313492490,1sleep20-21swapper/221:55:232
277262490,2sleep00-21swapper/019:20:170
25902480,2sleep20-21swapper/220:50:182
2155999378,14cyclictest1129-21munin-run00:30:000
2155999378,13cyclictest10888-21munin-run22:25:020
21571993626,3cyclictest0-21swapper/221:47:212
2155999368,13cyclictest10273-21munin-run21:10:000
2157799331,14cyclictest0-21swapper/319:40:193
21559993310,10cyclictest12483-21perf21:15:010
2155999322,23cyclictest23985-21kworker/0:322:55:330
265072310,4sleep339-21rcuc/321:45:173
21571993117,7cyclictest0-21swapper/222:23:522
2157199311,21cyclictest0-21swapper/219:11:242
2157199311,13cyclictest0-21swapper/200:31:282
2157199311,13cyclictest0-21swapper/200:31:282
2157199301,14cyclictest0-21swapper/222:42:162
2157199300,20cyclictest0-21swapper/219:18:162
2155999301,11cyclictest4883-21kworker/0:023:36:080
21571992915,7cyclictest0-21swapper/223:12:562
2157199291,19cyclictest0-21swapper/221:03:532
2157199291,19cyclictest0-21swapper/219:50:562
2157199291,19cyclictest0-21swapper/219:50:562
2157199291,15cyclictest0-21swapper/222:33:522
2157199291,14cyclictest0-21swapper/223:44:192
2157199291,14cyclictest0-21swapper/222:02:492
2157199290,15cyclictest0-21swapper/222:45:362
2156599291,10cyclictest0-21swapper/120:50:151
2155999291,11cyclictest15706-21kworker/0:000:05:240
2155999291,11cyclictest13009-21kworker/0:122:30:410
2157799281,20cyclictest0-21swapper/322:54:363
2157199281,7cyclictest25702-21timerwakeupswit22:55:222
2157199281,12cyclictest15845-21df_inode22:35:122
2157199281,12cyclictest0-21swapper/222:08:292
2157199281,12cyclictest0-21swapper/200:19:532
2157199281,11cyclictest0-21swapper/223:09:522
2157199281,10cyclictest0-21swapper/221:05:252
2157199280,13cyclictest0-21swapper/223:31:472
2157199280,10cyclictest0-21swapper/223:18:282
2155999282,9cyclictest4883-21kworker/0:023:31:560
2155999282,16cyclictest11157-21kworker/0:123:57:200
2155999282,12cyclictest25054-21idleruntime-cro20:30:020
2155999282,11cyclictest31401-21kworker/0:320:51:530
2155999281,19cyclictest22898-21kworker/0:220:30:210
2157199278,3cyclictest0-21swapper/223:25:212
21571992716,3cyclictest0-21swapper/220:46:322
2157199271,24cyclictest0-21swapper/220:40:202
2157199271,14cyclictest4569-21hddtemp_smartct20:55:172
2157199271,10cyclictest0-21swapper/222:54:162
2157199271,10cyclictest0-21swapper/221:16:532
2156599271,19cyclictest0-21swapper/123:22:431
2156599271,17cyclictest32090-21latency19:30:171
263552260,3sleep30-21swapper/323:00:003
2157199268,3cyclictest0-21swapper/221:14:052
2157199267,3cyclictest0-21swapper/200:00:132
2157199261,9cyclictest16471-21/usr/sbin/munin20:10:202
21571992613,3cyclictest0-21swapper/222:14:092
2157199261,18cyclictest0-21swapper/219:44:322
2157199261,15cyclictest0-21swapper/220:17:002
2157199261,11cyclictest0-21swapper/219:38:132
2157199261,10cyclictest675-21taskset19:34:012
2157199261,10cyclictest0-21swapper/219:20:172
2155999262,17cyclictest17420-21kworker/0:121:28:400
2155999262,10cyclictest6192-21kworker/0:022:13:210
2155999261,14cyclictest3052-21munin-run19:40:000
2155999261,13cyclictest18447-21munin-run20:15:020
2157799254,14cyclictest13328-21head21:15:153
2157799252,12cyclictest2873-21sed23:20:013
2157199257,9cyclictest0-21swapper/219:55:522
2157199257,4cyclictest0-21swapper/200:35:042
2157199251,11cyclictest6755-21kworker/u8:123:36:532
2157199251,11cyclictest0-21swapper/221:22:532
2157199251,10cyclictest0-21swapper/220:22:162
2157199250,12cyclictest0-21swapper/223:51:212
2155999252,11cyclictest28821-21kworker/0:200:21:000
2155999252,11cyclictest26549-21kworker/0:100:15:320
2155999251,11cyclictest4909-21kworker/0:120:00:250
2155999250,12cyclictest22856-21basename20:25:020
2157799246,6cyclictest15755-40dnf23:48:243
2157799241,17cyclictest0-21swapper/321:10:223
2157799240,18cyclictest16718-21processes22:35:223
2157199248,3cyclictest0-21swapper/219:48:122
2157199246,3cyclictest0-21swapper/221:53:132
21571992413,3cyclictest0-21swapper/220:09:162
21571992413,3cyclictest0-21swapper/219:25:092
2157199241,13cyclictest0-21swapper/200:10:372
2157199241,10cyclictest0-21swapper/200:00:012
21571992410,3cyclictest0-21swapper/221:37:412
2156599247,5cyclictest9249-21df22:20:131
2156599243,14cyclictest87550irq/128-enp1s0-21:12:371
2156599240,16cyclictest26328-21sshd19:16:211
2155999242,9cyclictest3149-21kworker/0:300:34:210
2155999242,9cyclictest3149-21kworker/0:300:34:210
2155999242,9cyclictest26549-21kworker/0:100:12:400
2155999242,12cyclictest31356-21kworker/0:020:45:130
2155999242,11cyclictest8327-21kworker/0:322:28:450
2155999241,9cyclictest28271-21kworker/0:319:11:130
2155999241,9cyclictest18374-21kworker/0:120:15:250
2155999241,8cyclictest23164-21kworker/0:221:55:200
21559992411,3cyclictest28271-21kworker/0:319:15:210
2155999241,12cyclictest30344-21kworker/0:123:07:490
2155999241,10cyclictest21676-21kworker/0:222:52:530
2155999241,10cyclictest11157-21kworker/0:123:54:560
319432230,2sleep30-21swapper/323:10:203
2157799236,8cyclictest12050irq/123-ahci[0020:05:023
2157199237,3cyclictest0-21swapper/222:17:562
2157199237,3cyclictest0-21swapper/222:17:562
2157199236,8cyclictest0-21swapper/220:37:362
2157199236,3cyclictest0-21swapper/223:03:122
2157199232,9cyclictest87550irq/128-enp1s0-21:40:492
21571992312,3cyclictest0-21swapper/221:34:052
2157199231,10cyclictest0-21swapper/200:29:402
21571992310,3cyclictest0-21swapper/220:31:242
2156599232,11cyclictest5866-21munin-run21:00:011
21565992317,4cyclictest0-21swapper/120:45:201
2156599231,19cyclictest30009-21hddtemp_smartct00:20:141
2156599231,14cyclictest7780-21kworker/1:222:15:131
2156599231,14cyclictest7780-21kworker/1:222:15:131
2156599231,14cyclictest0-21swapper/121:38:231
2156599231,11cyclictest0-21swapper/123:28:381
2155999238,2cyclictest26370-21kworker/0:119:25:240
2155999232,8cyclictest29086-21kworker/0:120:37:370
2155999232,12cyclictest14610-21kworker/0:021:20:480
2155999232,10cyclictest17118-21kworker/0:022:41:210
2155999231,8cyclictest32222-21kworker/0:122:00:080
2155999231,8cyclictest30344-21kworker/0:123:17:330
2155999231,10cyclictest9282-21kworker/0:219:56:090
2155999230,10cyclictest3149-21kworker/0:300:38:290
2157799226,7cyclictest87350irq/126-enp1s0-00:30:193
2157799226,7cyclictest87350irq/126-enp1s0-00:30:193
2157799221,16cyclictest41-21ksoftirqd/321:30:133
2157799221,10cyclictest5475-21/usr/sbin/munin19:45:223
2157799221,10cyclictest0-21swapper/321:58:473
2157799220,13cyclictest0-21swapper/323:55:203
2157799220,10cyclictest24015-21df21:40:143
2157199227,3cyclictest0-21swapper/220:00:322
21571992210,3cyclictest0-21swapper/220:26:042
2156599229,4cyclictest7666-21grep21:00:261
2156599229,2cyclictest18503-21cat23:55:111
2156599227,10cyclictest0-21swapper/122:28:591
2156599221,5cyclictest0-21swapper/121:28:451
2156599221,3cyclictest277282chrt21:48:351
2156599221,15cyclictest0-21swapper/121:19:051
2156599221,14cyclictest0-21swapper/100:28:001
2156599221,12cyclictest0-21swapper/100:06:531
2156599221,11cyclictest0-21swapper/123:46:461
2156599221,11cyclictest0-21swapper/123:07:331
2155999229,3cyclictest26206-21kworker/0:023:03:010
2155999228,3cyclictest30344-21kworker/0:123:20:250
2155999222,12cyclictest5709-21kworker/0:021:04:240
2155999222,11cyclictest744-21kworker/0:319:42:400
2155999222,11cyclictest4883-21kworker/0:023:40:320
2155999221,11cyclictest23164-21kworker/0:221:53:520
2157799216,6cyclictest0-21swapper/300:18:413
2157799216,3cyclictest31009-21latency_hist19:30:013
2157799215,10cyclictest0-21swapper/320:12:073
2157799214,9cyclictest808-21snmpd20:19:363
2157799213,9cyclictest401ktimersoftd/322:40:253
2157799212,11cyclictest21661-40dnf22:47:243
2157799211,8cyclictest0-21swapper/321:25:233
2157799211,6cyclictest41-21ksoftirqd/323:00:223
2157799211,12cyclictest21478-21/usr/sbin/munin21:35:143
2157799211,12cyclictest0-21swapper/323:34:523
2157799211,10cyclictest0-21swapper/300:05:233
21577992110,2cyclictest87350irq/126-enp1s0-19:10:243
2156599219,3cyclictest1058-21df_inode23:15:131
2156599218,3cyclictest1377-21cpu00:30:111
2156599218,3cyclictest1377-21cpu00:30:111
2156599217,4cyclictest0-21swapper/122:04:321
2156599211,3cyclictest30728-21ntp_states20:40:221
21565992113,3cyclictest0-21swapper/100:01:571
2156599211,15cyclictest0-21swapper/119:24:231
2156599211,11cyclictest0-21swapper/122:42:171
2156599211,10cyclictest579-21systemd-journal19:30:001
2156599211,10cyclictest0-21swapper/122:33:171
21565992110,3cyclictest32241-21latency_hist22:00:001
2156599210,15cyclictest0-21swapper/100:20:001
2156599210,12cyclictest29665-21users21:50:251
2155999219,3cyclictest20938-21kworker/0:021:39:040
2155999216,3cyclictest3482-21kworker/0:120:55:200
2155999212,9cyclictest12393-21kworker/0:321:19:080
2155999212,8cyclictest3929-21kworker/0:222:07:130
2155999212,8cyclictest15706-21kworker/0:000:02:360
2155999212,10cyclictest4909-21kworker/0:119:50:530
2155999212,10cyclictest4909-21kworker/0:119:50:530
2155999211,8cyclictest25400-21kworker/0:121:49:120
2155999211,8cyclictest22898-21kworker/0:220:40:090
2157799209,2cyclictest12050irq/123-ahci[0021:00:013
2157799208,7cyclictest87350irq/126-enp1s0-22:25:363
2157799206,8cyclictest87650irq/129-enp1s0-23:05:193
2157799206,7cyclictest0-21swapper/322:30:463
2157799203,3cyclictest87350irq/126-enp1s0-23:35:123
2157799201,16cyclictest6228-21tr21:00:113
2157199201,12cyclictest0-21swapper/200:24:332
2156599208,3cyclictest0-21swapper/123:41:231
2156599208,3cyclictest0-21swapper/121:20:381
2156599207,3cyclictest19371-21basename20:15:161
2156599207,3cyclictest0-21swapper/122:50:381
2156599205,9cyclictest0-21swapper/121:32:531
2156599201,7cyclictest16911-21timerwakeupswit22:35:231
21565992016,3cyclictest0-21swapper/123:54:431
21565992015,3cyclictest27608-21ls23:00:231
21565992014,3cyclictest22575-21irqstats19:10:161
2156599201,3cyclictest8440-21pmu-power23:30:211
2156599201,3cyclictest25738-21latency_hist21:45:011
2156599201,17cyclictest0-21swapper/120:29:101
2156599201,14cyclictest0-21swapper/100:10:251
2156599201,11cyclictest7454-21awk19:50:001
2156599201,11cyclictest22297-21sed20:20:241
2156599201,10cyclictest0-21swapper/122:46:071
2156599201,10cyclictest0-21swapper/122:12:261
2156599200,6cyclictest0-21swapper/122:59:561
2156599200,18cyclictest0-21swapper/119:56:501
2156599200,14cyclictest0-21swapper/119:40:081
2155999208,2cyclictest30344-21kworker/0:123:12:050
2155999207,2cyclictest9282-21kworker/0:220:05:170
2155999202,8cyclictest30344-21kworker/0:123:28:370
2155999201,9cyclictest30818-21kworker/0:219:33:080
2155999201,6cyclictest0-21swapper/023:45:590
2155999201,6cyclictest0-21swapper/022:49:560
2155999201,4cyclictest24473-21if_err_enp2s021:40:170
2155999201,11cyclictest4909-21kworker/0:119:45:250
2157799199,2cyclictest12050irq/123-ahci[0020:49:553
2157799194,8cyclictest41-21ksoftirqd/322:05:163
2157799194,12cyclictest25796-21cut20:30:163
2157799193,10cyclictest0-21swapper/322:11:343
21577991912,4cyclictest7949-21fgrep22:15:223
21577991912,4cyclictest7949-21fgrep22:15:223
2157799191,11cyclictest0-21swapper/300:38:383
2157799191,10cyclictest6028-21sed23:25:193
21577991910,3cyclictest87350irq/126-enp1s0-00:10:193
2156599198,3cyclictest14612-21df_inode20:05:121
21565991915,3cyclictest0-21swapper/123:36:571
21565991915,3cyclictest0-21swapper/120:37:111
2156599191,3cyclictest0-21swapper/119:38:521
2156599191,14cyclictest0-21swapper/100:35:021
2156599191,10cyclictest26553-21wc20:30:271
2155999192,8cyclictest20938-21kworker/0:021:31:320
2155999191,5cyclictest0-21swapper/022:16:020
2155999191,5cyclictest0-21swapper/022:16:020
2157799188,3cyclictest29334-21/usr/sbin/munin00:20:133
2157799188,3cyclictest0-21swapper/323:20:063
2157799185,3cyclictest0-21swapper/320:45:003
2157799184,11cyclictest8433-21cpuspeed_turbos21:05:113
2157799183,2cyclictest87650irq/129-enp1s0-20:25:113
2157799182,11cyclictest0-21swapper/323:52:113
2157799181,8cyclictest0-21swapper/321:53:223
2157799181,7cyclictest2090-21sh22:05:013
2157799181,7cyclictest16248-21munin-run20:10:023
21577991814,3cyclictest0-21swapper/320:38:323
21577991813,3cyclictest22007-21grep20:20:213
2157799181,11cyclictest0-21swapper/321:24:163
2157799181,10cyclictest21805-21switchtime00:00:243
2157799181,10cyclictest0-21swapper/322:21:533
2157799180,9cyclictest87350irq/126-enp1s0-19:50:253
2157799180,9cyclictest87350irq/126-enp1s0-19:50:253
2156599188,3cyclictest0-21swapper/122:08:021
2156599182,7cyclictest16471-21/usr/sbin/munin20:10:131
2156599181,3cyclictest0-21swapper/121:06:111
2156599181,16cyclictest0-21swapper/119:51:011
2156599181,16cyclictest0-21swapper/119:51:011
2156599181,15cyclictest0-21swapper/120:00:061
2157799176,8cyclictest2343-21smart_sda19:35:243
2157799171,7cyclictest25421-21cat19:15:163
2157799171,13cyclictest11721-21/usr/sbin/munin23:40:113
2157799170,5cyclictest0-21swapper/319:22:403
2157799161,9cyclictest0-21swapper/319:55:233
21577991610,3cyclictest2053-21forks20:50:133
2157799150,2cyclictest87350irq/126-enp1s0-19:30:223
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional