You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-22 - 13:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jul 22, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
137532227154,23sleep20-21swapper/219:09:452
137362222192,20sleep00-21swapper/019:09:320
137572203166,24sleep10-21swapper/119:09:481
135742203167,24sleep30-21swapper/319:07:253
1442421720,3sleep01384799cyclictest19:10:140
241242560,2sleep00-21swapper/000:25:150
294262510,3sleep10-21swapper/100:35:221
312762430,4sleep232-21ksoftirqd/219:45:242
13863993826,3cyclictest0-21swapper/321:11:093
1386399380,28cyclictest0-21swapper/322:48:093
1386399380,28cyclictest0-21swapper/322:48:093
1386399361,21cyclictest0-21swapper/321:55:013
1386399351,27cyclictest0-21swapper/323:09:533
1386399351,23cyclictest0-21swapper/320:58:293
1386399341,26cyclictest0-21swapper/322:51:453
1386399341,20cyclictest0-21swapper/320:42:373
1386399341,16cyclictest0-21swapper/322:15:013
1384799347,12cyclictest28962-21idleruntime-cro22:10:020
1384799344,14cyclictest6968-21munin-run21:20:000
1386399331,25cyclictest0-21swapper/321:32:503
1386399331,21cyclictest0-21swapper/320:49:213
1386399331,20cyclictest0-21swapper/300:23:013
13863993219,3cyclictest0-21swapper/322:36:093
1386399321,24cyclictest0-21swapper/323:32:303
1386399320,22cyclictest0-21swapper/323:48:343
1386399320,16cyclictest0-21swapper/300:34:533
1384899321,18cyclictest31354-21expr21:00:161
13863993116,8cyclictest0-21swapper/323:35:303
1386399311,21cyclictest0-21swapper/319:25:253
1386399311,21cyclictest0-21swapper/300:09:373
1386399311,20cyclictest26799-21ntp_states19:35:213
1386399311,20cyclictest0-21swapper/322:33:053
1386399311,19cyclictest0-21swapper/321:28:303
1386399311,18cyclictest0-21swapper/300:18:413
1386399311,12cyclictest0-21swapper/322:16:493
1386399311,12cyclictest0-21swapper/321:01:423
1386399310,29cyclictest0-21swapper/322:23:413
1386399310,25cyclictest0-21swapper/322:03:293
1386399310,25cyclictest0-21swapper/321:19:333
1386399310,21cyclictest0-21swapper/321:09:373
1384799316,12cyclictest22359-21awk21:55:020
207932300,3sleep01384799cyclictest20:35:180
1386399301,21cyclictest0-21swapper/323:03:293
1386399301,13cyclictest0-21swapper/322:41:213
1386399301,12cyclictest0-21swapper/323:21:013
1386399301,12cyclictest0-21swapper/321:58:533
1386399301,12cyclictest0-21swapper/321:39:093
1386399301,12cyclictest0-21swapper/321:39:093
1386399300,20cyclictest0-21swapper/320:08:013
1386399300,20cyclictest0-21swapper/300:36:013
1386399300,18cyclictest0-21swapper/300:03:013
1386399300,15cyclictest0-21swapper/322:05:453
1384799305,12cyclictest19262-21munin-run00:15:020
1386399291,19cyclictest0-21swapper/323:14:413
1386399291,19cyclictest0-21swapper/320:51:413
1386399291,19cyclictest0-21swapper/320:36:173
1386399291,12cyclictest0-21swapper/319:11:373
1386399291,10cyclictest0-21swapper/319:19:333
1386399290,17cyclictest0-21swapper/322:27:213
1386399290,12cyclictest0-21swapper/319:33:533
1384899292,4cyclictest8550-21munin-run20:10:001
1386399281,14cyclictest0-21swapper/321:40:293
1386399281,11cyclictest0-21swapper/323:51:373
1386399280,26cyclictest0-21swapper/322:55:233
1386399280,20cyclictest0-21swapper/319:22:053
13863992723,3cyclictest0-21swapper/320:21:573
1386399271,19cyclictest0-21swapper/323:28:333
1386399271,11cyclictest0-21swapper/321:21:103
1386399271,11cyclictest0-21swapper/320:01:413
1386399271,11cyclictest0-21swapper/300:28:213
1386399271,10cyclictest0-21swapper/319:48:493
1386399270,19cyclictest0-21swapper/321:49:453
1386399270,19cyclictest0-21swapper/319:40:533
1386399270,11cyclictest0-21swapper/323:16:533
13863992616,3cyclictest0-21swapper/320:31:253
1386399260,11cyclictest0-21swapper/319:52:413
1386399260,10cyclictest0-21swapper/320:14:593
1384899265,12cyclictest87550irq/128-enp1s0-20:22:201
13848992617,7cyclictest2573-21diskmemload21:26:491
1384799267,16cyclictest30523-21munin-run21:00:010
1384799261,13cyclictest29918-21basename19:45:020
1384799261,13cyclictest23327-21awk19:30:010
1386399251,21cyclictest0-21swapper/320:25:153
1386399251,21cyclictest0-21swapper/320:25:153
1386399251,12cyclictest0-21swapper/323:42:063
1386399251,12cyclictest0-21swapper/300:13:453
1386399251,11cyclictest0-21swapper/323:55:373
1386399251,11cyclictest0-21swapper/323:55:373
13848992515,4cyclictest0-21swapper/120:57:411
1384799251,12cyclictest21468-21basename00:20:020
1384899240,15cyclictest0-21swapper/121:53:171
1384799245,16cyclictest3841-21munin-run23:40:000
1384799241,12cyclictest0-21swapper/022:45:010
1386399231,14cyclictest0-21swapper/319:59:013
1386399231,13cyclictest0-21swapper/320:16:493
13848992317,3cyclictest18365-21/usr/sbin/munin20:30:181
13848992315,2cyclictest0-21swapper/120:16:541
1384899231,16cyclictest0-21swapper/123:26:271
1384899231,16cyclictest0-21swapper/121:06:481
1384899231,16cyclictest0-21swapper/120:50:471
1384899231,16cyclictest0-21swapper/120:36:511
1384899231,16cyclictest0-21swapper/119:33:541
1384899231,15cyclictest0-21swapper/119:45:371
1384899230,15cyclictest0-21swapper/120:10:191
13847992316,4cyclictest16925-40dnf00:08:240
1384799231,11cyclictest23942-21basename20:45:010
131172230,2sleep20-21swapper/222:45:202
131172230,2sleep20-21swapper/222:45:192
1384899222,3cyclictest0-21swapper/120:28:061
1384899222,3cyclictest0-21swapper/120:28:061
1384899222,15cyclictest0-21swapper/100:05:371
1384899221,4cyclictest14710-21sshd00:03:191
13848992213,3cyclictest0-21swapper/122:26:471
1384899221,16cyclictest0-21swapper/123:55:131
1384899221,16cyclictest0-21swapper/123:55:131
1384899221,16cyclictest0-21swapper/119:29:491
1384899221,15cyclictest0-21swapper/121:39:101
1384899221,15cyclictest0-21swapper/121:39:101
1384899221,15cyclictest0-21swapper/121:34:431
1384899221,10cyclictest27696-21sh19:40:011
1384899220,16cyclictest0-21swapper/122:32:131
1384899220,16cyclictest0-21swapper/121:41:501
1384799223,10cyclictest501-21/usr/sbin/munin21:05:160
13847992210,3cyclictest14238-21ntp_states20:20:210
87722211,16sleep28932-21expr20:10:132
1385699216,8cyclictest0-21swapper/220:15:392
1385699216,3cyclictest23322-21awk19:30:012
1385699213,9cyclictest30689-21latency23:25:182
1385699211,6cyclictest25237-21timerandwakeup00:25:262
1385699211,11cyclictest20144-21latency_hist21:50:012
1384899217,4cyclictest15216-21grep22:50:171
1384899211,6cyclictest0-21swapper/122:06:091
13848992113,3cyclictest0-21swapper/122:17:001
13848992113,3cyclictest0-21swapper/121:45:481
1384899211,2cyclictest24153-21/usr/sbin/munin20:45:161
1384899211,18cyclictest0-21swapper/123:05:211
1384899211,15cyclictest0-21swapper/123:50:381
1384899211,15cyclictest0-21swapper/123:38:361
1384899211,15cyclictest0-21swapper/119:22:121
1384899211,14cyclictest2573-21diskmemload22:46:031
1384899211,14cyclictest2573-21diskmemload22:46:031
1384899211,14cyclictest0-21swapper/122:57:021
1384899211,14cyclictest0-21swapper/122:40:491
1384899211,14cyclictest0-21swapper/122:10:591
1384899211,14cyclictest0-21swapper/121:24:201
1384899211,14cyclictest0-21swapper/121:12:191
1384899211,14cyclictest0-21swapper/119:15:511
1384899211,14cyclictest0-21swapper/100:11:481
1384899211,11cyclictest8722-21memory22:35:211
1384899211,10cyclictest0-21swapper/122:22:031
13848992110,3cyclictest27463-21sed00:30:241
1384799211,11cyclictest10454-21ntp_states21:25:230
1384799211,10cyclictest22677-21kworker/0:223:06:490
262532201,3sleep126355-21memory23:15:211
204612201,2sleep20-21swapper/223:01:022
1385699206,5cyclictest14560-21kworker/u8:222:40:172
1385699203,9cyclictest0-21swapper/222:13:442
1385699202,5cyclictest27713-21awk19:40:012
13856992013,4cyclictest23895-21tail23:10:172
1385699201,11cyclictest1203-21gltestperf21:05:182
1385699201,11cyclictest0-21swapper/221:35:142
1385699201,11cyclictest0-21swapper/221:35:142
1385699201,10cyclictest31814-21ntp_states21:00:212
1385699200,10cyclictest0-21swapper/222:19:032
1384899202,16cyclictest0-21swapper/119:58:551
1384899202,10cyclictest26090-21switchtime22:00:261
13848992013,2cyclictest0-21swapper/100:20:211
13848992012,3cyclictest0-21swapper/100:17:001
1384899201,14cyclictest0-21swapper/123:46:151
1384899201,14cyclictest0-21swapper/123:10:331
1384899201,14cyclictest0-21swapper/120:02:271
1384899201,14cyclictest0-21swapper/119:42:331
1384899201,14cyclictest0-21swapper/119:10:541
1384899201,11cyclictest22962-21fschecks_count21:55:151
1384899201,11cyclictest0-21swapper/123:30:201
1384899201,10cyclictest27646-21bc23:20:121
1384899200,14cyclictest0-21swapper/119:51:161
1384799209,3cyclictest17566-21/usr/sbin/munin20:30:260
1384799208,2cyclictest13369-21kworker/0:221:35:090
1384799208,2cyclictest13369-21kworker/0:221:35:090
1384799207,4cyclictest0-21swapper/023:30:080
1384799207,4cyclictest0-21swapper/023:14:540
1384799207,3cyclictest0-21swapper/023:03:250
1384799203,15cyclictest29737-21grep00:35:250
1384799202,8cyclictest9571-21kworker/0:023:19:530
1384799202,5cyclictest18245-1kworker/u9:222:17:310
1384799201,8cyclictest27145-21kworker/0:223:20:330
1384799201,7cyclictest0-21swapper/023:50:560
1384799201,6cyclictest0-21swapper/022:21:150
1384799201,17cyclictest5381-21munin-run22:30:010
1384799201,11cyclictest0-21swapper/022:57:100
1384799200,9cyclictest29511-21kworker/0:123:29:090
47782190,4sleep121-21rcuc/123:40:161
1385699199,3cyclictest827-21runrttasks22:02:152
1385699195,4cyclictest26131-21perf20:50:022
1385699192,5cyclictest17178-21perl22:55:172
1385699191,9cyclictest27005-21sed00:30:182
1385699191,5cyclictest18245-1kworker/u9:200:02:482
1385699191,12cyclictest0-21swapper/221:31:202
1385699191,11cyclictest0-21swapper/200:09:142
1385699191,10cyclictest19255-21latency_hist00:15:012
1384899192,8cyclictest6475-21unixbench-2d21:15:251
13848991912,3cyclictest0-21swapper/123:00:361
1384899191,17cyclictest0-21swapper/120:43:541
1384799196,5cyclictest0-21swapper/022:49:400
1384799196,5cyclictest0-21swapper/022:49:400
1384799195,5cyclictest0-21swapper/022:51:470
1384799192,14cyclictest17612-21grep19:15:180
1384799191,6cyclictest0-21swapper/000:21:210
1384799191,11cyclictest0-21swapper/000:31:220
1384799191,10cyclictest6265-21/usr/sbin/munin23:45:150
1384799191,10cyclictest18180-21/usr/sbin/munin21:45:140
1384799191,10cyclictest0-21swapper/021:59:200
1384799190,6cyclictest0-21swapper/023:56:380
1384799190,6cyclictest0-21swapper/023:56:380
1384799190,6cyclictest0-21swapper/021:14:470
1384799190,5cyclictest0-21swapper/021:43:240
83242181,2sleep20-21swapper/220:07:202
82182180,2sleep00-21swapper/020:06:020
253282181,2sleep00-21swapper/019:32:510
220872181,2sleep20-21swapper/221:51:502
1385699185,4cyclictest7434-21sshd22:33:182
1385699184,9cyclictest0-21swapper/221:27:482
1385699184,3cyclictest87350irq/126-enp1s0-20:40:202
1385699183,4cyclictest3824-21netstat21:10:212
1385699182,9cyclictest0-21swapper/222:07:152
1385699182,7cyclictest13842-21hddtemp_smartct20:20:162
1385699182,4cyclictest15002-21gltestperf22:50:162
1385699181,9cyclictest0-21swapper/223:40:402
1385699181,7cyclictest0-21swapper/219:17:352
1385699181,7cyclictest0-21swapper/200:18:022
1385699181,6cyclictest4010-21grep22:25:172
13856991813,3cyclictest8266-21ntp_states21:20:232
1385699181,14cyclictest2573-21diskmemload00:21:012
1385699181,10cyclictest0-21swapper/221:58:052
13848991812,3cyclictest24552-21hddtemp_smartct00:25:171
1384799187,3cyclictest4402-21df23:40:130
1384799181,8cyclictest4629-21kworker/u8:121:32:180
1384799181,5cyclictest18245-1kworker/u9:221:21:220
1384799181,15cyclictest31161-21perf22:15:020
1384799181,11cyclictest0-21swapper/022:30:530
123582181,3sleep20-21swapper/223:56:432
123582181,3sleep20-21swapper/223:56:432
273372170,2sleep00-21swapper/019:35:290
1385699179,5cyclictest27473-21pmu-power20:50:242
1385699176,3cyclictest87650irq/129-enp1s0-20:25:222
1385699176,3cyclictest87650irq/129-enp1s0-20:25:212
1385699176,3cyclictest87350irq/126-enp1s0-23:55:012
1385699175,3cyclictest87650irq/129-enp1s0-23:30:182
1385699172,6cyclictest0-21swapper/221:42:322
1385699171,9cyclictest0-21swapper/219:31:292
1385699171,7cyclictest0-21swapper/223:15:042
1385699171,7cyclictest0-21swapper/223:06:472
1385699171,6cyclictest1885-21/usr/sbin/munin23:35:122
1385699171,5cyclictest8127-21expr22:35:142
13856991712,3cyclictest7761-21grep23:45:262
1385699171,15cyclictest0-21swapper/219:23:012
1385699171,14cyclictest0-21swapper/219:11:112
1385699171,14cyclictest0-21swapper/200:35:042
13856991710,4cyclictest28398-21grep19:40:152
13856991710,4cyclictest0-21swapper/223:20:142
1384799176,3cyclictest773-21NetworkManager20:53:070
1384799175,3cyclictest0-21swapper/022:36:320
1384799171,7cyclictest24769-21/usr/sbin/munin22:00:280
1384799171,6cyclictest13484-21gltestperf00:00:170
13847991713,3cyclictest0-21swapper/020:02:540
1384799171,13cyclictest9841-21ntp_states20:10:220
18832161,2sleep00-21swapper/019:54:270
1385699169,4cyclictest29471-21missed_timers20:55:212
1385699163,3cyclictest808-21snmpd21:16:322
1385699162,3cyclictest1211-21cpu22:20:122
1385699162,11cyclictest19042-21unixbench-2d20:30:262
1385699161,4cyclictest0-21swapper/219:55:242
13856991612,3cyclictest0-21swapper/220:36:502
1385699161,14cyclictest0-21swapper/219:51:072
13856991610,3cyclictest4363-21/usr/sbin/munin20:00:182
1384799162,5cyclictest4138-21munin-run20:00:020
1384799161,3cyclictest19146-21/usr/sbin/munin19:20:190
1384799161,2cyclictest0-21swapper/020:16:220
13847991612,3cyclictest0-21swapper/021:02:210
13847991611,3cyclictest16445-21ntpq20:25:230
13847991611,3cyclictest16445-21ntpq20:25:230
1384799158,4cyclictest839-21lldpd19:45:180
1384799151,9cyclictest0-21swapper/020:45:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional