You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-12 - 12:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sun Apr 12, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
133072238164,24sleep20-21swapper/219:07:562
131362230162,55sleep00-21swapper/019:05:460
130992217187,20sleep30-21swapper/319:05:273
133512203167,24sleep10-21swapper/119:08:321
2766521330,3sleep027664-21strings22:05:210
1355999106103,2cyclictest0-21swapper/220:47:252
13559999794,2cyclictest0-21swapper/221:03:412
111402630,3sleep339-21rcuc/323:55:203
37252510,3sleep23728-21sendmail_mailqu21:10:252
268592500,5sleep21355999cyclictest20:50:182
1356499361,6cyclictest0-21swapper/322:04:513
1356499341,13cyclictest0-21swapper/323:13:373
1356499321,15cyclictest0-21swapper/323:25:573
1355299301,20cyclictest0-21swapper/100:16:221
1354599305,12cyclictest1633-21munin-run19:55:020
13564992915,3cyclictest7146-40dnf23:45:243
1356499291,13cyclictest0-21swapper/322:38:133
1356499281,12cyclictest0-21swapper/319:34:103
13559992825,2cyclictest0-21swapper/223:34:592
1355999281,19cyclictest0-21swapper/200:16:432
1355999281,11cyclictest0-21swapper/222:20:032
13564992712,3cyclictest23926-21seq21:56:103
13564992711,3cyclictest14539-21ntp_states21:35:223
13564992711,3cyclictest14539-21ntp_states21:35:223
1356499271,13cyclictest0-21swapper/322:22:413
1356499271,12cyclictest0-21swapper/320:26:133
1356499271,12cyclictest0-21swapper/320:14:133
13564992710,9cyclictest0-21swapper/321:30:543
1355299274,9cyclictest0-21swapper/122:15:501
1355299271,17cyclictest0-21swapper/123:10:391
1354599275,11cyclictest0-21swapper/019:50:020
1356499261,14cyclictest0-21swapper/322:59:453
1356499261,13cyclictest7460-21latency_hist22:35:013
1356499261,13cyclictest0-21swapper/321:21:583
13564992611,3cyclictest0-21swapper/320:22:133
13564992611,3cyclictest0-21swapper/320:22:133
1356499261,12cyclictest0-21swapper/323:01:453
1356499261,12cyclictest0-21swapper/322:19:173
1356499261,12cyclictest0-21swapper/321:28:143
1356499261,12cyclictest0-21swapper/320:01:453
1356499261,12cyclictest0-21swapper/319:53:093
1356499261,11cyclictest0-21swapper/321:00:493
1356499261,11cyclictest0-21swapper/320:40:413
1356499261,11cyclictest0-21swapper/319:42:533
1356499261,11cyclictest0-21swapper/319:25:103
1356499261,10cyclictest0-21swapper/319:24:143
13564992610,3cyclictest0-21swapper/323:09:093
1355999261,11cyclictest0-21swapper/223:18:242
13552992611,13cyclictest16870-21kworker/u8:020:39:571
1354599261,14cyclictest11982-21cpuspeed_turbos22:45:130
1354599261,14cyclictest11982-21cpuspeed_turbos22:45:130
1356499259,3cyclictest0-21swapper/320:46:213
1356499259,3cyclictest0-21swapper/320:36:093
1356499259,3cyclictest0-21swapper/300:28:013
1356499258,3cyclictest0-21swapper/320:34:133
1356499258,11cyclictest0-21swapper/323:30:533
1356499257,3cyclictest0-21swapper/321:08:413
1356499251,14cyclictest0-21swapper/323:42:263
1356499251,14cyclictest0-21swapper/322:48:173
1356499251,14cyclictest0-21swapper/322:48:173
1356499251,13cyclictest0-21swapper/323:52:503
1356499251,12cyclictest2262-21diskmemload00:09:183
1356499251,12cyclictest0-21swapper/321:52:383
1356499251,11cyclictest0-21swapper/323:23:093
1356499251,11cyclictest0-21swapper/322:06:103
1356499251,11cyclictest0-21swapper/300:02:023
1356499251,11cyclictest0-21swapper/300:02:023
13564992510,8cyclictest0-21swapper/322:50:133
13564992510,3cyclictest0-21swapper/320:57:093
13564992510,3cyclictest0-21swapper/319:36:173
1355299256,8cyclictest0-21swapper/100:25:471
1355299251,9cyclictest0-21swapper/121:27:211
1355299251,8cyclictest0-21swapper/121:36:581
1355299251,8cyclictest0-21swapper/121:36:581
1356499249,8cyclictest0-21swapper/300:32:053
1356499249,3cyclictest0-21swapper/322:13:133
1356499249,3cyclictest0-21swapper/320:53:133
1356499249,3cyclictest0-21swapper/319:10:453
1356499249,3cyclictest0-21swapper/300:11:383
1356499248,4cyclictest0-21swapper/323:17:453
1356499248,3cyclictest0-21swapper/322:40:333
1356499248,3cyclictest0-21swapper/319:48:173
1356499248,3cyclictest0-21swapper/319:18:503
1356499247,3cyclictest0-21swapper/323:39:343
1356499241,12cyclictest0-21swapper/321:16:423
1356499241,12cyclictest0-21swapper/321:14:013
1356499241,11cyclictest0-21swapper/300:39:413
1356499241,11cyclictest0-21swapper/300:15:343
1355299246,7cyclictest0-21swapper/122:50:591
1355299241,21cyclictest2272-21perf21:10:011
1354599246,4cyclictest13538-21cyclictest19:45:020
1354599242,12cyclictest2262-21diskmemload00:31:280
1354599240,14cyclictest17790-21latency00:10:180
1356499239,3cyclictest2262-21diskmemload00:22:383
1356499239,3cyclictest0-21swapper/321:47:063
1356499239,3cyclictest0-21swapper/320:18:533
1356499239,3cyclictest0-21swapper/319:55:053
1356499238,3cyclictest0-21swapper/321:44:223
1356499231,10cyclictest0-21swapper/322:26:133
1355999235,10cyclictest0-21swapper/222:40:132
1355999231,11cyclictest18891-21runrttasks20:30:512
1355299232,9cyclictest4785-21ntp_states23:40:211
1355299231,4cyclictest7339-21ntp_states20:05:221
1355299231,19cyclictest2262-21diskmemload23:05:291
1355299231,16cyclictest0-21swapper/122:32:231
1355299230,15cyclictest87650irq/129-enp1s0-23:19:061
140952221,4sleep214120-21fschecks_count19:10:142
1356499229,3cyclictest0-21swapper/320:05:573
1355999221,6cyclictest0-21swapper/223:59:582
1355299228,12cyclictest7059-21kworker/u8:222:57:251
1355299227,3cyclictest156492chrt19:14:141
1355299224,10cyclictest0-21swapper/122:20:201
1355299222,9cyclictest0-21swapper/122:36:561
1355299221,5cyclictest0-21swapper/121:50:241
1355299221,4cyclictest0-21swapper/123:20:191
1355299221,3cyclictest7146-40dnf23:45:221
1355299221,3cyclictest31077-21users23:25:261
1355299221,3cyclictest13586-21ntp_states00:00:221
1355299221,3cyclictest13586-21ntp_states00:00:221
1355299221,15cyclictest0-21swapper/122:28:231
1355299221,12cyclictest0-21swapper/122:10:231
1354599227,6cyclictest0-21swapper/021:47:210
13545992215,4cyclictest12283-21turbostat00:00:000
1354599221,12cyclictest2262-21diskmemload22:37:050
1354599220,16cyclictest32284-21gltestperf23:30:150
1355999211,4cyclictest0-21swapper/200:00:282
1355999211,4cyclictest0-21swapper/200:00:282
1355999211,13cyclictest0-21swapper/222:33:292
1355299218,3cyclictest0-21swapper/122:05:131
1355299213,9cyclictest23-21ksoftirqd/123:50:191
1355299212,8cyclictest0-21swapper/122:41:231
1355299212,17cyclictest0-21swapper/120:27:151
1355299211,6cyclictest579-21systemd-journal19:45:011
1355299211,2cyclictest12637-21perf20:20:011
1355299211,18cyclictest0-21swapper/119:37:281
1355299211,16cyclictest21960-21cat19:25:191
1355299211,14cyclictest0-21swapper/121:11:291
1355299211,14cyclictest0-21swapper/100:13:201
1355299211,11cyclictest24009-21cut20:45:141
1355299211,11cyclictest18857-21/usr/sbin/munin23:00:151
1355299211,11cyclictest18610-21latency_hist19:20:011
1355299211,11cyclictest0-21swapper/100:38:441
1354599216,8cyclictest20870-21cstates23:05:130
1354599213,15cyclictest30228-21idleruntime-cro21:00:010
1354599213,11cyclictest11225-21kworker/u8:121:30:510
1354599212,12cyclictest0-21swapper/023:21:540
1354599212,12cyclictest0-21swapper/022:15:280
1354599212,12cyclictest0-21swapper/021:17:310
1354599211,3cyclictest0-21swapper/019:38:560
1354599211,13cyclictest0-21swapper/023:40:030
1354599211,12cyclictest27922-21/usr/sbin/munin00:35:230
1354599210,11cyclictest0-21swapper/022:54:580
10422210,5sleep121-21rcuc/119:50:251
53052201,15sleep05330-21cpu22:30:140
226202200,1sleep20-21swapper/221:55:152
1355999205,3cyclictest12050irq/123-ahci[0019:20:112
1355999201,5cyclictest0-21swapper/222:49:462
1355999201,5cyclictest0-21swapper/222:49:462
1355999201,2cyclictest0-21swapper/200:35:032
1355299209,2cyclictest1475-21latency_hist23:35:021
1355299207,3cyclictest26494-21fschecks_count20:50:151
1355299202,16cyclictest0-21swapper/120:41:161
13552992016,3cyclictest0-21swapper/123:37:391
13552992016,3cyclictest0-21swapper/121:04:021
1355299201,4cyclictest5943-21sshd20:03:541
13552992014,3cyclictest12321-21/usr/sbin/munin21:30:201
1355299201,3cyclictest15511-21cat00:05:171
1355299201,3cyclictest0-21swapper/122:46:271
1355299201,3cyclictest0-21swapper/122:46:271
1355299201,18cyclictest0-21swapper/121:55:351
1355299201,13cyclictest0-21swapper/121:41:391
1355299201,10cyclictest17609-21diskstats20:30:171
1355299201,10cyclictest0-21swapper/121:20:251
1355299200,2cyclictest87650irq/129-enp1s0-22:04:371
1354599205,7cyclictest16296-21/usr/sbin/munin22:55:160
1354599203,9cyclictest19455-21kworker/u8:221:41:030
1354599203,3cyclictest0-21swapper/000:01:380
1354599203,3cyclictest0-21swapper/000:01:380
1354599203,11cyclictest0-21swapper/021:58:430
1354599201,13cyclictest28392-21kworker/u8:023:10:080
1354599201,12cyclictest28392-21kworker/u8:023:02:100
1354599201,12cyclictest0-21swapper/021:13:090
1354599200,13cyclictest19455-21kworker/u8:222:01:120
1354599200,12cyclictest7059-21kworker/u8:223:16:520
138272191,1sleep00-21swapper/021:35:150
138272191,1sleep00-21swapper/021:35:150
1355999192,6cyclictest30847-21munin-run22:15:012
13559991912,4cyclictest6660-21idleruntime-cro21:20:012
1355999191,12cyclictest0-21swapper/223:02:132
1355999191,11cyclictest0-21swapper/222:09:482
1355299199,3cyclictest0-21swapper/120:20:201
1355299199,3cyclictest0-21swapper/120:20:201
1355299199,2cyclictest0-21swapper/121:45:191
1355299192,5cyclictest10664-21dnf23:55:151
1355299192,16cyclictest0-21swapper/119:56:551
1355299192,16cyclictest0-21swapper/100:20:461
13552991916,2cyclictest0-21swapper/121:15:571
1355299191,17cyclictest0-21swapper/119:32:541
1355299191,16cyclictest0-21swapper/120:55:591
1355299191,12cyclictest0-21swapper/100:31:251
1355299191,10cyclictest31017-21processes19:45:231
1354599197,2cyclictest8118-21/usr/sbin/munin23:50:180
1354599196,3cyclictest7990-21pmu-power21:20:220
1354599192,11cyclictest0-21swapper/023:46:010
1354599191,6cyclictest0-21swapper/023:27:230
13545991912,4cyclictest15322-21head00:05:160
1354599191,11cyclictest7059-21kworker/u8:222:43:570
1354599191,10cyclictest19137-21/usr/sbin/munin00:15:280
1354599190,12cyclictest11225-21kworker/u8:122:10:330
1354599190,12cyclictest0-21swapper/000:27:470
292362180,2sleep20-21swapper/223:24:542
1355999185,8cyclictest17582-40dnf21:44:252
1355999181,8cyclictest0-21swapper/200:25:402
1355999181,7cyclictest23823-21cat20:45:022
1355999181,7cyclictest0-21swapper/219:30:112
1355999181,7cyclictest0-21swapper/200:10:122
1355999181,6cyclictest0-21swapper/223:07:372
1355999181,6cyclictest0-21swapper/222:37:012
1355999181,5cyclictest22191-21runrttasks19:25:222
1355999181,15cyclictest0-21swapper/223:25:522
1355999181,11cyclictest0-21swapper/223:48:422
1355999181,11cyclictest0-21swapper/221:53:202
1355999181,10cyclictest0-21swapper/221:36:132
1355999181,10cyclictest0-21swapper/221:36:132
13552991813,2cyclictest20018-21processes19:20:211
1354599184,3cyclictest0-21swapper/022:28:450
1354599184,12cyclictest7-21ksoftirqd/019:10:240
1354599184,12cyclictest16571-21sed20:25:260
1354599183,8cyclictest4059-21/usr/sbin/munin20:00:170
1354599183,8cyclictest0-21swapper/021:50:220
1354599181,9cyclictest10023-21sed20:10:270
1354599181,8cyclictest23004-21latency_hist19:30:020
13545991813,3cyclictest19754-21memory19:20:170
1354599181,12cyclictest0-21swapper/023:36:560
1354599180,4cyclictest16870-21kworker/u8:021:25:280
1354599180,4cyclictest11966-21kworker/u8:200:21:000
1355999179,5cyclictest0-21swapper/222:50:272
1355999176,3cyclictest87350irq/126-enp1s0-20:55:152
1355999175,3cyclictest12050irq/123-ahci[0000:35:012
1355999171,7cyclictest0-21swapper/223:41:502
1355999171,6cyclictest579-21systemd-journal21:10:012
1355999171,6cyclictest21421-21latency_hist20:40:022
1355999171,6cyclictest15871-21irqstats20:25:192
1355999171,6cyclictest0-21swapper/222:59:112
1355999171,6cyclictest0-21swapper/221:26:052
1355999171,5cyclictest0-21swapper/221:45:172
1355999171,15cyclictest0-21swapper/200:22:202
13559991711,3cyclictest32025-21/usr/sbin/munin19:50:232
13559991711,3cyclictest2286-21fschecks_count19:55:152
1355999171,10cyclictest0-21swapper/221:30:172
1354599176,3cyclictest87350irq/126-enp1s0-20:50:140
1354599174,3cyclictest2921-21ntpq19:55:220
1354599171,8cyclictest2162-21mailstats22:20:230
1354599171,5cyclictest18618-21munin-run19:20:000
13545991712,3cyclictest1833-21grep21:05:260
1354599171,15cyclictest0-21swapper/020:46:570
1354599171,14cyclictest0-21swapper/020:24:070
1354599171,14cyclictest0-21swapper/020:24:070
1354599171,13cyclictest6870-21grep20:05:160
1355999165,3cyclictest808-21snmpd00:05:222
1355999162,12cyclictest3072-21grep23:35:262
1355999161,9cyclictest0-21swapper/222:25:452
1355999161,9cyclictest0-21swapper/222:01:162
1355999161,9cyclictest0-21swapper/220:02:282
1355999161,7cyclictest28660-21/usr/sbin/munin19:40:192
1355999161,6cyclictest14447-21users20:20:242
1355999161,6cyclictest14447-21users20:20:232
1355999161,6cyclictest11264-21grep20:15:162
1355999161,6cyclictest0-21swapper/223:50:102
1355999161,6cyclictest0-21swapper/220:10:132
13559991612,3cyclictest0-21swapper/220:05:182
1355999161,14cyclictest0-21swapper/219:45:292
13559991611,3cyclictest6882-21/usr/sbin/munin21:20:152
13559991611,3cyclictest32128-21ntp_offset22:15:212
13559991610,4cyclictest18607-21crond19:20:012
13559991610,3cyclictest23244-21diskstats23:10:142
13552991610,3cyclictest8608-21grep20:10:131
1354599169,4cyclictest23330-21tr19:30:110
1354599161,4cyclictest17492-21expr20:30:140
13545991611,3cyclictest22751-21open_inodes20:40:210
13545991611,3cyclictest19456-21/usr/sbin/munin20:35:120
13545991611,3cyclictest11248-21grep20:15:170
13545991610,3cyclictest30440-21/usr/sbin/munin21:00:120
1355999154,3cyclictest25492-21grep19:35:102
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional