You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 05:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sun Mar 22, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6882236160,24sleep00-21swapper/019:07:290
5532234163,59sleep10-21swapper/119:05:531
6152203166,25sleep30-21swapper/319:06:413
5342201164,25sleep20-21swapper/219:05:382
2211421710,4sleep0103999cyclictest22:20:250
1055998783,3cyclictest0-21swapper/320:34:153
41072560,4sleep011-21rcuc/020:25:270
167372560,1sleep10-21swapper/122:10:141
167372560,1sleep10-21swapper/122:10:141
279622530,3sleep081ktimersoftd/022:35:170
211162530,4sleep121-21rcuc/122:20:141
236752500,4sleep121-21rcuc/123:40:111
91912450,2sleep20-21swapper/200:20:132
103999349,12cyclictest21505-21munin-run19:55:020
103999349,12cyclictest21505-21munin-run19:55:020
103999346,13cyclictest22149-21perf21:10:010
103999331,14cyclictest8248-21cpu23:05:100
1039993311,11cyclictest0-21swapper/000:00:020
1055993228,3cyclictest0-21swapper/320:05:103
103999324,13cyclictest27770-21munin-run23:50:020
103999316,12cyclictest30305-21munin-run20:15:020
1039993114,15cyclictest14597-21perf23:20:000
69402300,4sleep1104499cyclictest21:45:261
103999292,12cyclictest22794-21ntp_states19:55:210
103999291,12cyclictest0-21swapper/023:40:190
103999281,2cyclictest0-21swapper/000:06:370
1039992811,14cyclictest6753-21awk20:35:010
105199276,3cyclictest12050irq/123-ahci[0022:45:022
105199274,11cyclictest25891-40dnf20:05:002
1039992712,3cyclictest11139-21munin-run20:45:000
103999271,13cyclictest21187-21awk23:35:020
103999271,12cyclictest4868-21ntp_states22:55:200
103999270,12cyclictest0-21swapper/021:44:140
1051992616,6cyclictest9922-40dnf23:06:512
104499261,18cyclictest0-21swapper/121:35:171
1044992611,3cyclictest26974-21systemd-journal22:32:591
103999261,13cyclictest0-21swapper/023:53:170
103999261,12cyclictest0-21swapper/022:52:320
103999261,12cyclictest0-21swapper/020:52:080
103999261,12cyclictest0-21swapper/020:19:370
103999261,12cyclictest0-21swapper/020:01:210
103999261,12cyclictest0-21swapper/000:33:530
103999261,12cyclictest0-21swapper/000:12:530
103999261,11cyclictest0-21swapper/021:55:540
103999261,11cyclictest0-21swapper/019:27:400
103999261,11cyclictest0-21swapper/019:12:250
103999260,11cyclictest0-21swapper/020:21:450
108842250,3sleep339-21rcuc/319:30:123
105599251,5cyclictest13027-21munin-run00:30:013
104499252,19cyclictest8634-21munin-run00:20:011
103999259,2cyclictest0-21swapper/022:10:150
103999259,2cyclictest0-21swapper/022:10:150
103999251,12cyclictest14923-21munin-run19:40:000
103999251,12cyclictest0-21swapper/023:20:160
103999251,12cyclictest0-21swapper/023:00:240
103999251,12cyclictest0-21swapper/020:38:280
198762240,4sleep230-21rcuc/222:15:242
190822240,7sleep230-21rcuc/223:30:012
190822240,7sleep230-21rcuc/223:30:012
105199247,3cyclictest12050irq/123-ahci[0019:20:012
104499243,3cyclictest0-21swapper/123:06:121
104499241,17cyclictest0-21swapper/100:10:411
104499240,18cyclictest0-21swapper/100:08:051
103999241,15cyclictest0-21swapper/022:05:190
103999241,13cyclictest0-21swapper/000:18:090
103999241,12cyclictest0-21swapper/023:13:480
103999241,12cyclictest0-21swapper/022:27:330
103999241,12cyclictest0-21swapper/000:29:570
103999241,11cyclictest0-21swapper/020:56:040
103999241,11cyclictest0-21swapper/000:02:370
103999241,10cyclictest0-21swapper/021:52:120
103999241,10cyclictest0-21swapper/019:18:410
103999240,11cyclictest0-21swapper/019:43:440
103999240,10cyclictest0-21swapper/019:21:400
23122230,5sleep3105599cyclictest22:50:173
105599237,4cyclictest0-21swapper/323:42:423
105599231,3cyclictest0-21swapper/321:30:243
105599231,12cyclictest812-21snmpd20:25:143
105199232,6cyclictest2349-21munin-run20:25:012
104499239,3cyclictest15535-21awk20:55:001
104499233,11cyclictest0-21swapper/121:17:051
1044992318,3cyclictest580-21systemd-journal21:15:011
1044992314,2cyclictest13111-21grep22:00:231
1044992312,9cyclictest31417-21kworker/u8:222:47:031
104499231,19cyclictest4167-21cpu19:15:121
104499231,17cyclictest0-21swapper/122:56:041
104499231,17cyclictest0-21swapper/122:43:441
104499231,16cyclictest0-21swapper/100:38:081
104499231,16cyclictest0-21swapper/100:38:081
104499231,16cyclictest0-21swapper/100:01:511
104499230,18cyclictest0-21swapper/122:27:021
104499230,15cyclictest0-21swapper/121:50:361
104499230,11cyclictest0-21swapper/123:51:041
103999238,3cyclictest0-21swapper/023:29:320
103999238,3cyclictest0-21swapper/023:29:320
103999236,2cyclictest0-21swapper/021:15:100
103999231,9cyclictest0-21swapper/019:49:010
103999231,11cyclictest0-21swapper/023:37:330
103999231,11cyclictest0-21swapper/000:20:410
103999230,16cyclictest0-21swapper/022:33:140
105599221,9cyclictest190832cat23:30:023
105599221,9cyclictest190832cat23:30:023
1055992215,4cyclictest25691-21tr21:15:213
105199229,3cyclictest2315-21ntp_offset19:10:202
105199229,2cyclictest11755-21awk22:00:002
105199228,4cyclictest14411-21seq23:18:042
105199228,3cyclictest86550irq/126-enp1s0-21:10:232
1051992215,4cyclictest23102-21tr22:25:132
1051992210,2cyclictest0-21swapper/220:39:412
104499222,7cyclictest86850irq/129-enp1s0-20:40:121
104499222,2cyclictest0-21swapper/122:19:521
1044992216,3cyclictest8035-21ntp_states20:35:221
1044992214,3cyclictest0-21swapper/100:29:391
104499221,3cyclictest86750irq/128-enp1s0-23:45:211
104499221,3cyclictest0-21swapper/100:24:491
104499221,19cyclictest0-21swapper/121:40:061
104499221,19cyclictest0-21swapper/119:20:121
104499221,15cyclictest0-21swapper/100:31:501
104499221,11cyclictest17087-21seq20:55:241
104499220,16cyclictest0-21swapper/123:17:351
103999229,3cyclictest0-21swapper/019:30:120
103999226,3cyclictest0-21swapper/021:31:540
103999224,15cyclictest17434-21idleruntime-cro00:40:010
103999224,15cyclictest17434-21idleruntime-cro00:40:010
103999221,10cyclictest1381-21perf22:50:010
211782211,3sleep10-21swapper/119:51:011
211782211,3sleep10-21swapper/119:51:011
105599216,6cyclictest0-21swapper/323:10:073
105599216,5cyclictest5526-21ntp_states00:10:223
105599215,6cyclictest0-21swapper/322:19:443
105599213,12cyclictest9927-21chrt23:06:503
105599213,11cyclictest812-21snmpd20:04:593
105599211,4cyclictest0-21swapper/300:19:103
105599211,14cyclictest0-21swapper/322:40:233
105599211,12cyclictest16518-21ntp_states00:35:223
105599211,12cyclictest16518-21ntp_states00:35:223
105199216,4cyclictest12050irq/123-ahci[0020:00:012
1051992116,3cyclictest6965-21head21:45:262
105199211,4cyclictest21173-21munin-run23:35:012
1051992114,4cyclictest23930-21df23:40:122
104499216,3cyclictest26983-21latency20:05:191
104499215,8cyclictest86750irq/128-enp1s0-19:45:231
104499214,4cyclictest851-21sssd_nss20:25:211
1044992117,3cyclictest0-21swapper/119:26:301
104499211,3cyclictest0-21swapper/121:20:241
104499211,18cyclictest23021-21switchtime19:55:231
104499211,18cyclictest0-21swapper/121:05:131
104499211,13cyclictest0-21swapper/123:14:161
104499211,12cyclictest0-21swapper/120:01:091
104499211,11cyclictest32212-21ntp_states21:30:191
104499211,11cyclictest17952-21/usr/sbin/munin21:00:191
104499211,11cyclictest17952-21/usr/sbin/munin21:00:191
104499211,10cyclictest15176-21strings22:05:191
104499210,3cyclictest21-21rcuc/123:00:201
104499210,19cyclictest0-21swapper/121:25:171
104499210,19cyclictest0-21swapper/119:10:151
104499210,15cyclictest0-21swapper/123:33:371
104499210,14cyclictest0-21swapper/119:44:561
103999216,3cyclictest0-21swapper/020:45:260
105599205,5cyclictest0-21swapper/323:24:113
105599205,3cyclictest268632cpuspeed_turbos21:20:133
105599205,3cyclictest12088-21kworker/u8:223:18:333
105599202,9cyclictest0-21swapper/300:30:233
105599202,4cyclictest1083-21ntp_states00:00:223
105599202,10cyclictest20160-21/usr/sbin/munin21:05:213
105599201,12cyclictest0-21swapper/320:20:233
105599201,11cyclictest0-21swapper/322:21:423
105599201,10cyclictest28632-21smart_sda22:35:253
105599201,10cyclictest17062-21hddtemp_smartct22:10:163
105599201,10cyclictest17062-21hddtemp_smartct22:10:163
105599200,9cyclictest0-21swapper/321:29:093
105599200,4cyclictest0-21swapper/323:58:343
105199208,2cyclictest86750irq/128-enp1s0-00:11:002
105199207,3cyclictest86750irq/128-enp1s0-22:23:242
105199206,3cyclictest86550irq/126-enp1s0-23:55:122
105199205,4cyclictest1655-21cpu22:50:132
105199202,5cyclictest8893-21diskstats19:25:152
105199201,6cyclictest6351-21/usr/sbin/munin19:20:192
105199201,6cyclictest4770-21/usr/sbin/munin20:30:242
105199201,6cyclictest14012-21ntp_states19:35:222
105199201,4cyclictest0-21swapper/222:49:572
104499204,8cyclictest2754-21grep22:50:241
104499202,15cyclictest22913-21grep23:35:261
1044992017,2cyclictest0-21swapper/120:15:161
104499201,18cyclictest0-21swapper/123:21:131
104499201,18cyclictest0-21swapper/120:11:021
104499201,17cyclictest2351-21awk20:25:021
104499201,17cyclictest0-21swapper/123:25:041
104499201,17cyclictest0-21swapper/123:25:041
104499201,17cyclictest0-21swapper/119:35:031
104499201,10cyclictest6736-21latency_hist20:35:011
103999207,3cyclictest0-21swapper/021:02:160
103999207,3cyclictest0-21swapper/021:02:160
103999201,2cyclictest0-21swapper/020:09:550
103999201,15cyclictest0-21swapper/021:25:240
103999200,10cyclictest0-21swapper/021:46:560
25942191,2sleep20-21swapper/221:35:282
105599199,2cyclictest7339-21sshd21:49:583
105599198,5cyclictest22674-21switchtime23:35:263
105599192,10cyclictest14186-21/usr/sbin/munin22:05:233
105599191,8cyclictest4920-21kworker/u8:222:00:193
105599191,6cyclictest0-21swapper/322:58:243
105599191,5cyclictest0-21swapper/300:20:233
105599191,12cyclictest0-21swapper/320:45:133
105599191,11cyclictest0-21swapper/322:27:333
105599191,10cyclictest960-21/usr/sbin/munin21:35:213
105599191,10cyclictest9226-21sshd21:50:583
105599191,10cyclictest22132-21diskmemload23:49:013
105599191,10cyclictest22132-21diskmemload21:40:033
105599190,8cyclictest0-21swapper/323:33:483
105599190,11cyclictest0-21swapper/322:30:233
105199196,3cyclictest86850irq/129-enp1s0-21:50:162
105199195,3cyclictest22132-21diskmemload23:38:022
105199195,3cyclictest12050irq/123-ahci[0020:20:012
105199194,3cyclictest86850irq/129-enp1s0-21:40:122
105199194,3cyclictest3667-21open_inodes20:25:222
105199192,5cyclictest20512-21sed19:50:182
105199192,5cyclictest20512-21sed19:50:182
105199192,5cyclictest16070-21smart_sda23:20:252
105199191,6cyclictest0-21swapper/221:00:232
105199191,6cyclictest0-21swapper/221:00:232
105199191,5cyclictest4718-21memory22:55:192
105199191,10cyclictest833-21runrttasks21:28:162
105199190,2cyclictest0-21swapper/200:09:232
104499193,8cyclictest0-21swapper/120:45:201
104499191,3cyclictest0-21swapper/119:30:231
104499191,17cyclictest0-21swapper/121:55:421
104499191,12cyclictest0-21swapper/123:55:061
104499191,12cyclictest0-21swapper/122:35:421
103999193,3cyclictest0-21swapper/021:10:500
105599186,4cyclictest12050irq/123-ahci[0021:15:023
105599186,3cyclictest12050irq/123-ahci[0019:50:013
105599184,8cyclictest41-21ksoftirqd/323:04:093
105599182,7cyclictest6351-21/usr/sbin/munin19:20:223
105599181,9cyclictest0-21swapper/300:06:233
1055991811,4cyclictest28568-21perl23:50:173
105599181,12cyclictest0-21swapper/319:15:113
105599181,10cyclictest0-21swapper/319:10:213
105199183,3cyclictest14815-21unixbench_singl00:30:262
105199183,3cyclictest11380-21sed23:10:182
105199181,9cyclictest0-21swapper/223:03:032
105199181,7cyclictest9247-21cpuspeed_turbos20:40:102
105199181,7cyclictest18409-21/usr/sbin/munin19:45:212
105199181,5cyclictest8473-21sshd00:18:132
105199181,5cyclictest16709-21smart_sda00:35:242
105199181,5cyclictest16709-21smart_sda00:35:242
105199181,4cyclictest20160-21/usr/sbin/munin21:05:262
105199181,3cyclictest851-21sssd_nss20:55:232
1051991813,3cyclictest26245-21ntp_states22:30:212
1051991812,3cyclictest32211-21ntpq21:30:192
105199181,15cyclictest0-21swapper/220:45:462
1051991811,4cyclictest16279-21pmu-power19:40:212
105199181,10cyclictest0-21swapper/222:00:312
105199181,10cyclictest0-21swapper/221:20:022
103999182,6cyclictest86850irq/129-enp1s0-21:24:460
103999181,13cyclictest0-21swapper/022:00:270
103999180,8cyclictest0-21swapper/021:37:420
310092170,2sleep30-21swapper/320:15:143
105599172,2cyclictest32432-21hddtemp_smartct22:45:133
105599171,3cyclictest0-21swapper/319:40:153
105599171,15cyclictest0-21swapper/320:42:303
1055991711,4cyclictest21849-21cut19:55:113
105599171,10cyclictest0-21swapper/321:56:593
105199172,3cyclictest28240-21cat20:10:012
105199171,7cyclictest0-21swapper/223:50:192
105199171,6cyclictest0-21swapper/222:08:182
105199171,4cyclictest10748-21/usr/sbin/munin19:30:122
105199171,3cyclictest0-21swapper/200:00:212
1051991713,3cyclictest0-21swapper/200:29:572
105199171,14cyclictest0-21swapper/222:10:032
105199171,14cyclictest0-21swapper/222:10:032
105199171,13cyclictest14524-21missed_timers20:50:182
105199171,10cyclictest0-21swapper/222:35:492
105199170,8cyclictest0-21swapper/221:23:522
1039991711,4cyclictest30675-21grep22:40:210
105599166,3cyclictest86550irq/126-enp1s0-20:35:153
105599161,6cyclictest29248-21memory20:10:183
105599161,6cyclictest0-21swapper/320:51:513
105599161,3cyclictest0-21swapper/321:00:113
105599161,3cyclictest0-21swapper/321:00:113
105599161,14cyclictest0-21swapper/319:26:213
1055991611,3cyclictest16089-21df20:55:133
1055991611,3cyclictest14081-21pmu-power19:35:243
1055991610,4cyclictest0-21swapper/319:50:263
1055991610,4cyclictest0-21swapper/319:50:253
105199164,3cyclictest86550irq/126-enp1s0-23:45:152
105199161,6cyclictest28334-21/usr/sbin/munin20:10:172
103999169,4cyclictest19001-21grep22:15:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional